SMS text message reminders have been used to promote many health behaviors, including improving diet and physical activity, managing chronic health conditions, reminding patients of medical appointments, and medication adherence [1-4]. The scalability, low cost, and ability to reach people in real time make SMS text messages a feasible and effective strategy for health behavior change. However, crafting and validating helpful reminder messages that address the needs of specific patient populations is challenging and commonly involves recruiting and testing with target users [5]. Using traditional research methods that require patient recruitment can be time-consuming, costly, and resource-intensive.

Online panels offer a potentially rapid means of efficiently evaluating message content with target populations before committing to a full-scale evaluation of SMS text message reminder effectiveness. These large, diverse standing panels of research volunteers allow investigators to quickly recruit broadly representative samples and assess message content [6]. If successful, this approach could significantly reduce the cost and time required to gather participant feedback compared to traditional pretesting methods [7]. Although widely used in market research, online panels have been less often used in the development of digital health behavior interventions.

Evidence from systematic reviews and randomized controlled trials demonstrates the effectiveness of SMS text message reminders in promoting medication adherence across a range of health conditions [8,9]. Given this demonstrated utility, SMS text messaging may play a role in the support of antidepressant adherence. Antidepressant nonadherence is prevalent, with many patients discontinuing their medication early or taking it inconsistently, limiting its therapeutic effectiveness [10-12]. Early discontinuation is common, partly because antidepressants often require several weeks before subjective improvement occurs [13]. Tailored SMS text message reminders during this period offer an opportunity to deliver private, real-time, short-term support that might help patients maintain treatment adherence until symptom relief reinforces continued medication use.

Antidepressant nonadherence has many causes, such as forgetfulness, that may be modifiable [14,15]. Self-monitoring, social support, highlighting consequences, and fostering habit formation offer structured approaches that could address these causes [16]. SMS text messages that target these strategies of behavior change may influence message effectiveness in supporting antidepressant adherence. In addition to studying the feasibility of online panels, we were interested in comparing the perceived helpfulness of AI-generated and human-authored content. While there is some evidence suggesting that large language models such as the generative pretrained transformer series perform comparably to or better than humans on health message generation, other studies report mixed results regarding readability and empathy [17-20]. Given these inconsistent findings, it is important to directly compare AI- and human-authored message content to specifically support antidepressant adherence, which might be able to further accelerate the development and evaluation of SMS text messages.

In this study, we used an online panel to evaluate the perceived helpfulness and associated underlying behavioral mechanisms of messages designed to improve medication adherence among patients with depression receiving antidepressant treatment.

A total of 181 participants completed the survey, yielding 7520 message ratings. After weighting, 53% of participants were aged 46 years or older, 73% were female, 92% were White individuals, 94% were non-Hispanic individuals, and nearly 59% had lower than a bachelor’s degree (Table 1).

Table 2 shows that messages generated by AI were rated as significantly more helpful than those written by humans (adjusted mean difference 0.24, 95% CI 0.12‐0.36; P<.001). Among barriers to adherence, messages addressing delayed benefit or no immediate effect (adjusted mean difference 0.13, 95% CI 0.004‐0.25; P=.04) were rated as more helpful than messages addressing unawareness of need for continued medication adherence. Our results also revealed that BCTs such as self-monitoring and feedback (adjusted mean difference 0.48, 95% CI 0.33‐0.62; P<.001), action planning and habit formation (adjusted mean difference 0.43, 95% CI 0.29‐0.57; P<.001), and natural consequences (adjusted mean difference 0.47, 95% CI 0.33‐0.61; P<.001) were rated as significantly more helpful than those using external support and influence.

Table 3 highlights the 5 highest-rated and 5 lowest-rated messages. The most highly rated items were predominantly motivational messages focused on overcoming forgetfulness. In contrast, the lowest-rated items were all human authored and inspirational in tone and emphasized external support or influence from health care providers.

This study demonstrates the feasibility of using online research panels to rapidly evaluate SMS text message reminders for antidepressant adherence. In just a few hours of fielding the survey, participants provided item-level helpfulness ratings across a large library of candidate reminder messages. Helpfulness ratings were generally high, although they varied by message source, barrier focus, and BCT. These results suggest that using online panels and AI provides a fast and low-cost [29] method for pretesting intervention messaging content.

A notable finding was that, when guided by examples developed by content experts within a structured framework of barriers and BCTs, AI was able to produce highly rated and sometimes preferred reminder messages targeting antidepressant adherence. This pattern aligns with emerging evidence that AI can support or improve health message design when prompts incorporate theoretical and domain-specific guidance [18]. Although further investigation is needed, one possible explanation for the observed preference for AI-generated messages may be large language models’ tendencies to produce responses that are agreeable or aligned with expectations [30]. As a result, AI models may be perceived as more relevant or appealing to users. To determine whether this preference translates into actual adherence, a randomized controlled trial is currently underway [31]. While future research is needed to confirm these findings, the results of this study suggest a scalable path to expanding message libraries while maintaining acceptability, which could support the design of personalized health messaging interventions.

Our analysis revealed important message preferences with implications for digital interventions and clinical practice. Messages that emphasized overcoming forgetfulness, developing a medication-taking routine, and highlighting downstream consequences were generally rated as more helpful than those emphasizing reliance on external support and influence. These results support prior work indicating that forgetfulness and disrupted routines are leading modifiable causes of unintentional nonadherence, with behavioral models underscoring habit formation and self-regulation as drivers of consistent medication use [12,32,33].

Messages encouraging patients to contact their health care providers were consistently among the lowest rated. While further study is needed, this finding is consistent with prior research showing that patients often view physicians and mental health professionals' communication as burdensome, time-consuming, or unhelpful in the context of medication adherence support [34]. Our findings may also reflect the tendency for individuals to attribute their health behavior to self-management efforts while underestimating environmental or interpersonal factors (eg, physician and mental health professional support) in shaping health behavior [35]. Importantly, this does not suggest that clinician communication is ineffective as previous work has demonstrated the positive effect that physician and mental health professional advice can have on health behavior change [36]. Rather, this tendency may help explain the observed preference for self-directed over physician- and mental health professional–mediated strategies and inform clinical guidelines on effective strategies for adherence counseling. For example, adherence interventions might be more successful if they focus on empowering patients with self-monitoring tools and highlight the benefits of consistent medication use. In general, autonomy-supportive and habit-based strategies have been demonstrated to improve adherence by strengthening patients’ self-efficacy and integrating medication taking into daily routines [33]. Applying these strategies alongside clinician support may provide the strongest benefit for health behavior change and represents an important area for further research.

Several limitations should be acknowledged. First, participants recruited through online panels may not be fully representative of all patients initiating antidepressant treatment. Additionally, the study team was unable to validate participants’ treatment histories and relied on self-report to determine study eligibility; therefore, it is possible that some participants may have misreported their diagnosis or medication use. Second, our evaluation focused on perceived helpfulness rather than actual behavioral outcomes, which may limit interpretability given that individuals’ intentions are imperfect predictors for their actual health behavior [37]. Although the response patterns appear credible, we have not yet tested whether any of these messages improve medication adherence. A randomized controlled trial is currently underway to address this limitation [31,38]. Third, although message development was grounded in prior theoretical considerations and empirical research, we may have missed some factors that meaningfully influence antidepressant adherence. Furthermore, the distinction between motivational and informational messages was based on criteria established by the study team; future research should explore more empirically grounded approaches to improve the reliability of this classification. Finally, the generalizability of these findings to other populations, health behaviors, and conditions remains unknown. While the methods appear broadly applicable for rapid message testing across various health interventions, the specific content preferences observed for antidepressant adherence may not generalize to other conditions or health behaviors. Furthermore, given that the sample was limited to adults aged 21 to 64 years as part of the eligibility criteria for a subsequent randomized controlled trial, the findings may not reflect the reminder needs and preferences of older adults.

Despite these limitations, the results suggest that online panels can offer a scalable, efficient approach for rapidly evaluating health behavior change messages. In addition, guided AI can play a productive role in message development. Together, these methods offer a practical approach for accelerating the design and pilot-testing of digital health interventions and reducing associated costs [39]. As SMS text messaging and related digital strategies continue to expand across health domains, the ability to efficiently identify and refine effective messages through these methods could accelerate advances in the field.