Rapid Development and Testing of Behavioral Text Message Reminders for Antidepressant Adherence via Online Panels: Survey Study

Introduction

SMS text message reminders have been used to promote many health behaviors, including improving diet and physical activity, managing chronic health conditions, reminding patients of medical appointments, and medication adherence [1-4]. The scalability, low cost, and ability to reach people in real time make SMS text messages a feasible and effective strategy for health behavior change. However, crafting and validating helpful reminder messages that address the needs of specific patient populations is challenging and commonly involves recruiting and testing with target users [5]. Using traditional research methods that require patient recruitment can be time-consuming, costly, and resource-intensive.

Online panels offer a potentially rapid means of efficiently evaluating message content with target populations before committing to a full-scale evaluation of SMS text message reminder effectiveness. These large, diverse standing panels of research volunteers allow investigators to quickly recruit broadly representative samples and assess message content [6]. If successful, this approach could significantly reduce the cost and time required to gather participant feedback compared to traditional pretesting methods [7]. Although widely used in market research, online panels have been less often used in the development of digital health behavior interventions.

Evidence from systematic reviews and randomized controlled trials demonstrates the effectiveness of SMS text message reminders in promoting medication adherence across a range of health conditions [8,9]. Given this demonstrated utility, SMS text messaging may play a role in the support of antidepressant adherence. Antidepressant nonadherence is prevalent, with many patients discontinuing their medication early or taking it inconsistently, limiting its therapeutic effectiveness [10-12]. Early discontinuation is common, partly because antidepressants often require several weeks before subjective improvement occurs [13]. Tailored SMS text message reminders during this period offer an opportunity to deliver private, real-time, short-term support that might help patients maintain treatment adherence until symptom relief reinforces continued medication use.

Antidepressant nonadherence has many causes, such as forgetfulness, that may be modifiable [14,15]. Self-monitoring, social support, highlighting consequences, and fostering habit formation offer structured approaches that could address these causes [16]. SMS text messages that target these strategies of behavior change may influence message effectiveness in supporting antidepressant adherence. In addition to studying the feasibility of online panels, we were interested in comparing the perceived helpfulness of AI-generated and human-authored content. While there is some evidence suggesting that large language models such as the generative pretrained transformer series perform comparably to or better than humans on health message generation, other studies report mixed results regarding readability and empathy [17-20]. Given these inconsistent findings, it is important to directly compare AI- and human-authored message content to specifically support antidepressant adherence, which might be able to further accelerate the development and evaluation of SMS text messages.

In this study, we used an online panel to evaluate the perceived helpfulness and associated underlying behavioral mechanisms of messages designed to improve medication adherence among patients with depression receiving antidepressant treatment.

Methods

Overview

We conducted a cross-sectional survey in January 2025 to evaluate the perceived helpfulness of SMS text message reminders designed to promote antidepressant adherence. Messages were developed by the study team and AI and were informed by known barriers to medication adherence and evidence-based behavior change techniques (BCTs) [16]. Participants who self-reported a depression diagnosis and a current antidepressant prescription were asked to rate a subset of these messages on their perceived helpfulness. Analyses were conducted using survey-weighted regression models to examine associations between message characteristics and ratings.

Message Development

To guide message development, we first identified known modifiable barriers to antidepressant adherence, such as forgetting doses [21]. A content expert then drew on an established framework of evidence-based BCTs to pair each barrier with an appropriate intervention strategy [16,22]; for example, forgetfulness was paired with the strategy of prompting or cueing medication-taking behavior. Each message was designed using a single barrier and BCT pairing based on the capability, opportunity, and motivation–behavior framework [23].

To facilitate comparison of AI- and human-authored content, 2 sets of SMS text message reminders were developed. First, we drafted 37 messages based on barrier and BCT pairings. Barriers were later grouped into 4 categories: social stigma, unawareness of need for continued medication adherence, delayed benefit or no immediate relief (frustration with slow effects), and forgetfulness and lack of routine. BCTs were also grouped into 4 categories: action planning and habit formation (building routines and plans), self-monitoring and feedback (tracking progress), natural consequences (highlighting outcomes of behavior), and external support and influence (engaging physicians, mental health professionals, peers, or social norms). We then provided GPT-4 (OpenAI) with these examples along with the associated barriers and BCTs and prompted it to generate 46 additional messages. Multimedia Appendix 1 provides the prompts used to generate the messages. In total, 83 messages were created, and all were reviewed for accuracy, clarity, and adherence to the specified barrier and BCT pairing.

We classified messages as informational if the content conveyed a factual statement (eg, “Did you know that one in ten adult Americans take antidepressant medications?”) and motivational if the message prioritized encouragement to support adherence (eg, “It takes a lot of strength to seek care for depression.”). Each message was then coded according to 4 characteristics: AI status (human vs AI generated), message type (motivational vs informational), barriers to adherence addressed, and BCT used. The complete set of messages, along with their barrier-BCT pairings and additional coded characteristics, can be found in Multimedia Appendix 2.

Panel Participants

Once the full set of messages was developed, we evaluated their perceived helpfulness on antidepressant adherence using the CloudResearch Connect online platform, which provides access to a vetted panel of adult US residents. The platform enables rapid recruitment by allowing researchers to define eligibility criteria; stratify by demographic characteristics; and manage participant screening, enrollment, and compensation while maintaining participant anonymity [24]. Participants were eligible if they resided in the United States, spoke English, were aged 21 to 64 years, provided informed consent, and self-reported a diagnosis of depression and use of antidepressant medication. Within 131 minutes, the platform recruited a sample of 181 adults. Recruitment quotas were used to ensure sufficient sample sizes for subgroup analyses by sex and educational level, both of which are associated with differences in antidepressant use and adherence [25].

Eligible individuals were directed to a Qualtrics-hosted survey (Qualtrics International Inc) accessible via computer or mobile device where they completed a 10-minute questionnaire. Each participant was randomly assigned 41 or 42 of the 83 total messages and asked to rate how much each message would have helped them remember to take their antidepressant medication on a 5-point Likert scale ranging from 1 (“not at all”) to 5 (“very much”).

Ethical Considerations

The University of Pennsylvania Institutional Review Board determined this study to be exempt from review (857419). All participants provided informed consent and received US $3 for completing the questionnaire. Participant information was anonymized on the online platform prior to data download by the research team.

Statistical Analysis

We conducted survey-weighted analyses using Stata (StataCorp). We addressed the minimal missing data for key covariates (sex, race, ethnicity, marital status, income, and educational level) using hot-deck imputation, whereas missing age values were replaced with the sample mean. Iterative proportional fitting was applied using the svycal command to align the sample with national population benchmarks of adults with depression in the United States [26]. The resulting raked weights were applied in all subsequent analyses. We report weighted demographic characteristics of the survey sample, as well as comparisons to the national profile of individuals diagnosed with depression who were prescribed antidepressant medication.

To identify key factors that predicted how messages performed, we estimated unadjusted associations between message characteristics and item ratings using survey-weighted generalized linear models that account for clustering of responses within individuals. Adjusted models included AI status (human or machine generated), message type (informational or motivational), barriers (social stigma, unawareness of need for continued medication adherence, delayed benefit, and forgetfulness and lack of routine), and BCTs (self-monitoring and feedback, external influences and support, natural consequences, action planning, and habit formation) while controlling for demographic variables (age, sex, race, ethnicity, marital status, and income). Marginal means and contrasts were computed to examine differences across groups. All analyses were conducted at the item response level accounting for clustering of responses within individuals. We calculated survey-adjusted mean ratings for each message and identified the 5 highest-rated and 5 lowest-rated items.

Results

A total of 181 participants completed the survey, yielding 7520 message ratings. After weighting, 53% of participants were aged 46 years or older, 73% were female, 92% were White individuals, 94% were non-Hispanic individuals, and nearly 59% had lower than a bachelor’s degree (Table 1).

Table 2 shows that messages generated by AI were rated as significantly more helpful than those written by humans (adjusted mean difference 0.24, 95% CI 0.12‐0.36; P<.001). Among barriers to adherence, messages addressing delayed benefit or no immediate effect (adjusted mean difference 0.13, 95% CI 0.004‐0.25; P=.04) were rated as more helpful than messages addressing unawareness of need for continued medication adherence. Our results also revealed that BCTs such as self-monitoring and feedback (adjusted mean difference 0.48, 95% CI 0.33‐0.62; P<.001), action planning and habit formation (adjusted mean difference 0.43, 95% CI 0.29‐0.57; P<.001), and natural consequences (adjusted mean difference 0.47, 95% CI 0.33‐0.61; P<.001) were rated as significantly more helpful than those using external support and influence.

Table 3 highlights the 5 highest-rated and 5 lowest-rated messages. The most highly rated items were predominantly motivational messages focused on overcoming forgetfulness. In contrast, the lowest-rated items were all human authored and inspirational in tone and emphasized external support or influence from health care providers.

Discussion

This study demonstrates the feasibility of using online research panels to rapidly evaluate SMS text message reminders for antidepressant adherence. In just a few hours of fielding the survey, participants provided item-level helpfulness ratings across a large library of candidate reminder messages. Helpfulness ratings were generally high, although they varied by message source, barrier focus, and BCT. These results suggest that using online panels and AI provides a fast and low-cost [29] method for pretesting intervention messaging content.

A notable finding was that, when guided by examples developed by content experts within a structured framework of barriers and BCTs, AI was able to produce highly rated and sometimes preferred reminder messages targeting antidepressant adherence. This pattern aligns with emerging evidence that AI can support or improve health message design when prompts incorporate theoretical and domain-specific guidance [18]. Although further investigation is needed, one possible explanation for the observed preference for AI-generated messages may be large language models’ tendencies to produce responses that are agreeable or aligned with expectations [30]. As a result, AI models may be perceived as more relevant or appealing to users. To determine whether this preference translates into actual adherence, a randomized controlled trial is currently underway [31]. While future research is needed to confirm these findings, the results of this study suggest a scalable path to expanding message libraries while maintaining acceptability, which could support the design of personalized health messaging interventions.

Our analysis revealed important message preferences with implications for digital interventions and clinical practice. Messages that emphasized overcoming forgetfulness, developing a medication-taking routine, and highlighting downstream consequences were generally rated as more helpful than those emphasizing reliance on external support and influence. These results support prior work indicating that forgetfulness and disrupted routines are leading modifiable causes of unintentional nonadherence, with behavioral models underscoring habit formation and self-regulation as drivers of consistent medication use [12,32,33].

Messages encouraging patients to contact their health care providers were consistently among the lowest rated. While further study is needed, this finding is consistent with prior research showing that patients often view physicians and mental health professionals' communication as burdensome, time-consuming, or unhelpful in the context of medication adherence support [34]. Our findings may also reflect the tendency for individuals to attribute their health behavior to self-management efforts while underestimating environmental or interpersonal factors (eg, physician and mental health professional support) in shaping health behavior [35]. Importantly, this does not suggest that clinician communication is ineffective as previous work has demonstrated the positive effect that physician and mental health professional advice can have on health behavior change [36]. Rather, this tendency may help explain the observed preference for self-directed over physician- and mental health professional–mediated strategies and inform clinical guidelines on effective strategies for adherence counseling. For example, adherence interventions might be more successful if they focus on empowering patients with self-monitoring tools and highlight the benefits of consistent medication use. In general, autonomy-supportive and habit-based strategies have been demonstrated to improve adherence by strengthening patients’ self-efficacy and integrating medication taking into daily routines [33]. Applying these strategies alongside clinician support may provide the strongest benefit for health behavior change and represents an important area for further research.

Several limitations should be acknowledged. First, participants recruited through online panels may not be fully representative of all patients initiating antidepressant treatment. Additionally, the study team was unable to validate participants’ treatment histories and relied on self-report to determine study eligibility; therefore, it is possible that some participants may have misreported their diagnosis or medication use. Second, our evaluation focused on perceived helpfulness rather than actual behavioral outcomes, which may limit interpretability given that individuals’ intentions are imperfect predictors for their actual health behavior [37]. Although the response patterns appear credible, we have not yet tested whether any of these messages improve medication adherence. A randomized controlled trial is currently underway to address this limitation [31,38]. Third, although message development was grounded in prior theoretical considerations and empirical research, we may have missed some factors that meaningfully influence antidepressant adherence. Furthermore, the distinction between motivational and informational messages was based on criteria established by the study team; future research should explore more empirically grounded approaches to improve the reliability of this classification. Finally, the generalizability of these findings to other populations, health behaviors, and conditions remains unknown. While the methods appear broadly applicable for rapid message testing across various health interventions, the specific content preferences observed for antidepressant adherence may not generalize to other conditions or health behaviors. Furthermore, given that the sample was limited to adults aged 21 to 64 years as part of the eligibility criteria for a subsequent randomized controlled trial, the findings may not reflect the reminder needs and preferences of older adults.

Despite these limitations, the results suggest that online panels can offer a scalable, efficient approach for rapidly evaluating health behavior change messages. In addition, guided AI can play a productive role in message development. Together, these methods offer a practical approach for accelerating the design and pilot-testing of digital health interventions and reducing associated costs [39]. As SMS text messaging and related digital strategies continue to expand across health domains, the ability to efficiently identify and refine effective messages through these methods could accelerate advances in the field.