Published on 19.12.13 in Vol 15, No 12 (2013): December
Use of a Text Message Program to Raise Type 2 Diabetes Risk Awareness and Promote Health Behavior Change (Part II): Assessment of Participants' Perceptions on Efficacy
Background: Although there is great enthusiasm in both the public and private sector for the further development and use of large-scale consumer-facing public health applications for mobile platforms, little is known about user experience and satisfaction with this type of approach. As a part of the Beacon Community Cooperative Agreement Program, txt4health, a public-facing, mobile phone-based health information service targeting type 2 diabetes, was launched in 3 Beacon Communities: the Southeast Michigan Beacon Community in Detroit, MI, the Greater Cincinnati Beacon Community in Cincinnati, OH, and the Crescent City Beacon Community in New Orleans, LA. This program was marketed via large public health campaigns and drew many users within the respective communities.
Objective: The purpose of this investigation was to use the RE-AIM framework to document txt4health efficacy by focusing on perceptions of satisfaction, usage, and behavior change among individuals who used txt4health in pilot studies in Southeast Michigan and Greater Cincinnati.
Methods: We conducted a multimodal user survey with txt4health users recruited via text message through the program to understand participant perceptions of program use and satisfaction, as well as self-reported perceptions of behavior change as a result of using txt4health.
Results: Txt4health users reported very high levels of program satisfaction, with 67.1% (108/161) reporting satisfaction scores of ≥8 on a 10-point scale, with 10 equivalent to most satisfied (mean 8.2, SD 1.6). All survey participants agreed/strongly agreed that the messages included in txt4health were clear and easy to understand (100.0%, 160/160), and most found txt4health made them knowledgeable about their risk for type 2 diabetes (88.1%, 140/159) and made them conscious of their diet and physical activity (88.8%, 142/160). Most participants reported that txt4health helped them to make behavior changes related to diet; after having completed txt4health, most agreed/strongly agreed that they are more likely to replace sugary drinks, such as juice or soda, with water (78.0%, 124/159), have a piece of fresh fruit instead of dessert (74.2%, 118/159), substitute a small salad for chips or fries when dining out (76.1%, 121/159), buy healthier foods when grocery shopping (79.7%, 126/158), and eat more grilled, baked, or broiled foods instead of fried (75.5%, 120/159).
Conclusions: Results from this study suggest that participants in txt4health, a large-scale, public health–focused text message program targeting type 2 diabetes, have positive perceptions of the program and that participation has led to positive behavior change.
J Med Internet Res 2013;15(12):e282
- diabetes mellitus, type 2;
- mobile health;
- cellular phone;
- text messaging;
- risk reduction behavior;
- program evaluation
In an era of health care reform occurring at a time when cell phones and mobile devices are near ubiquitous within the United States population , it is not surprising to see the rapid expansion of mobile health (mHealth) for the delivery of health information and services. Although there is great enthusiasm in both the public and private sector for the further development and use of consumer-facing health-related applications for mobile platforms, the momentum of mHealth is quickly outpacing the evidence base documenting the efficacy of these endeavors, as well as the traditional research designs used in the creation of the evidence base [ - ]. The efficacy of text message-based interventions to promote behavior change within the context of chronic illness has been previously documented within small-scale contexts [ - ], but most of the available evidence base is focused on underpowered small-scale pilot studies with few participants [ - ]. This fact leads many to believe that, similar to other innovations within the health care industry, moving beyond the pilot phase to operating at scale is an issue plaguing the mHealth field today.
Because of the affordances of a large-scale text message campaign, such as wide reach, low participant costs, and a small time investment on the part of participants, these types of programs appear to be perfectly suited to broad-based public health campaigns. Txt4baby is an example of this type of public health approach , which has garnered significant attention in the popular press and many people are attempting to replicate its successes. Txt4health, a text message program that offers participants a type 2 diabetes risk assessment followed by 14 weeks of tailored messaging focused on behavior changes for physical activity and diet, is one such program that was piloted in the Detroit, MI, Cincinnati, OH, and New Orleans, LA metropolitan areas through funding from the Beacon Community Cooperative Agreement Program. Although text message programs are well documented at the pilot stage, large-scale applications in the United States, such as txt4baby, are largely missing from the evidence base and little is known about participant experiences with these types of interventions. To better understand participant experiences within a large-scale, public health-focused, text message program targeting chronic illness, we conducted a multimodal survey of participants who had engaged in txt4health.
The purpose of this 2-part investigation is to evaluate the txt4health pilot studies in Southeast Michigan and Greater Cincinnati through the lens of the RE-AIM framework. In the present paper (Part II), we seek to document program efficacy in terms of participant perceptions of program satisfaction and usage, as well as its effects on behavior change. In our companion paper (Part I), we sought to document program reach and adoption. In comparison to the majority of previous work that has focused on small-scale implementations of mHealth programs [- ], this txt4health evaluation represents an effort to understand user perceptions of a program that is operating at scale.
This evaluation of txt4health was conducted in 2 parts. In Part I, we conducted a retrospective records analysis of individual-level txt4health system usage data from participants in Southeast Michigan and Greater Cincinnati to determine intervention reach and participant adoption of the program. Findings from Part I of the evaluation are presented in the companion paper. In Part II, we conducted a multimodal user survey with Southeast Michigan Beacon Community (SEMBC) and Greater Cincinnati Beacon Collaborative (GCBC) txt4health users recruited through the program. This survey sought to understand participant perceptions of program satisfaction and use, and self-reported perceptions of behavior change as a result of using txt4health. Txt4health is an automated, personalized, interactive, 14-week text message service targeting vulnerable at-risk populations, designed to help people understand their risk for type 2 diabetes and become more informed about the steps they can take to reduce that risk. Txt4health has been more extensively described in the Part I companion paper, as well as in recent work by Abebe et al .
Program Evaluation Framework: RE-AIM
We used the RE-AIM framework to guide this evaluation of the txt4health pilots in Southeast Michigan and Greater Cincinnati. As previously described in the companion paper, RE-AIM incorporates the dimensions of reach, efficacy, adoption, implementation, and maintenance, and is a framework that has been extensively used to guide the planning, evaluation, reporting, and review of health promotion interventions . Within the present paper, we focus strictly on the dimension of efficacy of the RE-AIM framework. Efficacy is an individual-level measure that refers to the degree to which use of an intervention creates the desired outcomes [ ]. Within the context of the present txt4health evaluation, efficacy refers to participant perceptions of txt4health, as well as perceptions of the effect of txt4health on behavior.
To be eligible to participate in this study, individuals had to be at least 18 years and enrolled in the txt4health program for at least 10 weeks. Beacon employees enrolled in txt4health were excluded from participation in this survey. For recruitment purposes, we sent a text message to all txt4health users in the SEMBC and GCBC pilots who were subscribed to the program at the time of solicitation. In total, 814 participants were active in the txt4health system when survey recruitment text messages were sent. Six batches of recruitment texts were sent to all active participants between September and December, 2012 (approximately every 2 weeks), and asked txt4health users to indicate whether or not they would be interested in participating in the survey. Those individuals who responded yes were contacted via phone to coordinate survey completion. Phone calls were not made to potential participants until at least 10 weeks had passed after a participants’ initial txt4health enrollment date. This allowed for participants to complete most of the program before taking the survey. Those individuals who responded no were removed from our solicitation list and they never received another recruitment text message. Participants who did not respond to the recruitment text messages remained on our solicitation list and continued to receive recruitment text messages. Up to 5 attempts were made by phone to contact all participants who expressed interest in survey participation.
Once contact was established, potential participants were again asked if they were interested in participating. Any individual no longer interested was thanked for their time. All eligible participants willing to participate were given the option of completing the survey via phone, Internet (survey hosted on SurveyMonkey ), or US Mail. Individuals who wished to take the survey via Internet or US Mail received periodic reminders via email or phone for up to 6 weeks, or until survey completion. All survey respondents were eligible to receive a US $10 gift card to a local retailer for completing the survey. The Wayne State University Institutional Review Board approved the user survey portion of this study with a waiver of documentation of written consent. In lieu of written consent, all participants who completed the survey were given information pertaining to the study, including study procedures, risks/benefits, and incentives. This information was distributed in electronic form for online survey participants, in hard copy for US Mail participants, and over the phone with a statement of oral consent for phone survey participants.
The investigator-developed user survey took approximately 20 minutes to complete and included questions assessing txt4health use and perceptions, health and health behaviors, mobile phone use, demographics, and activation as measured by the Patient Activation Measure (PAM). The PAM is a 13-item instrument that has been validated to assess individuals’ knowledge, skill, and confidence with managing their health. The PAM characterizes people into 1 of 4 levels of activation, with descriptions of each level as follows in[ , ].
We conducted descriptive statistics to describe participant characteristics, perceptions of txt4health, self-reported text4health usage, and self-reported effects of txt4health. Continuous variables were expressed as mean (SD) and means were compared using 2-tailed unpaired independent samples t tests. Categorical data were displayed as frequencies and percentages, and chi-square tests were used for comparison. It should be noted that because participants were not required to answer all items within the survey, the denominator of many of our analyses fluctuates slightly. Multiple regression analyses were used to predict satisfaction with txt4health controlling for the continuous variables age and body mass index (BMI), as well as the categorical variables gender, White or non-White race, income, education, level of activation as measured by the PAM, and Beacon Community affiliation (Southeast Michigan or Greater Cincinnati). Moreover, chi-square analyses were used to explore potential differences between participants affiliated with the Southeast Michigan or Greater Cincinnati Beacon Communities. Significance levels were set at a P value <.05. All statistical analyses were carried out using STATA version 11.0 (StataCorp LP, College Station, TX, USA).
We conducted a survey of Southeast Michigan and Greater Cincinnati txt4health participants recruited via text message. Of the 814 active txt4health participants who were sent recruitment text messages, 47.7% (388/814) expressed initial interest in participating in the survey via text message. We successfully contacted 70.4% (273/388) of those individuals by phone and asked again if they would be willing to participate. Overall, 14.7% (40/273) of txt4health users who previously expressed interest in our survey declined participation, and an additional 6 (2.2%) individuals were excluded from participation because they were Beacon employees. Of the remaining 227 eligible individuals still interested in survey participation, 70.9% (161/227) of participants completed the approximately 20-minute survey; 64.0% (103/161) completed it via the Internet, 31.1% (50/161) completed it via phone, and 5.0% (8/161) completed it via US Mail. Refer tofor a complete chart of participant flow.
Of the 161 participants who completed the user survey, the sample was predominantly female (73.7%, 115/156), African American (51.6%, 81/157) or White (44.6%, 70/157), smartphone users (78.5%, 124/158), with a mean age of 42.4 years (SD 11.6) and a mean BMI of 33.1 kg/m2 (SD 8.6). Most survey respondents were obese with a BMI ≥30 kg/m2 (53.9%, 82/152), or overweight with a BMI between 25.0 and 29.9 kg/m2 (31.6%, 48/152). In addition, 30.6% (48/157) had a history of diabetes. Most survey participants (75.6%, 121/160) self-reported good, very good, or excellent health. Overall, this sample was well educated with 58.6% (92/157) reporting that they were college graduates or had completed postgraduate work, and an additional 28.0% (44/157) reporting that they had completed some college classes. Only 8.9% (14/157) reported that they were high school dropouts, or had completed high school. Annual income of survey participants was well distributed, with 22.2% (34/153) reporting annual household incomes of less than US $25,000, and 20.9% (32/153) with more than US $100,000. Regarding levels of activation, 63.5% (101/159) of respondents were categorized as level 4 on the PAM, which is characterized as individuals who “have made most of the necessary behavior changes, but may have difficulty maintaining behaviors over time or during times of stress” . An additional 22.6% (36/159) were categorized as level 3 on the PAM, which is characterized as individuals who “have the basic facts of their conditions and treatments, some experience and success in making behavioral changes, and some confidence in handling limited aspects of their health” [ ]. For 53.8% (85/158) of respondents, txt4health was the first text message program that they had used. See for a complete breakdown of survey respondent characteristics.
Survey Participants’ txt4health Use
Survey participants first learned of txt4health through a variety of channels, including health fairs (20.0%, 32/160), radio advertisements (14.4%, 23/160), and directly from health care providers (13.8%, 22/160). Despite the number of respondents who first learned of txt4health through health fairs where individuals were able to initiate enrollment by providing a phone number and ZIP code to Beacon staff members, only 4.4% (7/160) of respondents initiated enrollment through that mechanism. Rather, 92.5% (148/160) self-reported initiating enrollment through texting the word “health” to the txt4health short code. Regarding use of the program, 83.2% (134/161) of survey participants reported using txt4health to set physical activity goals, and 67.5% (108/160) reported setting weight loss goals. The vast majority of survey participants (82.5%, 132/160) reported that they always read the text messages they received from txt4health, and 26.3% (42/160) of survey participants reported discussing txt4health messages with their health care provider. One component of txt4health was the inclusion of messages regarding resources of potential interest, such as contact information for community resources and organizations, links to additional health-related information, etc. Survey respondents reported using these resources never (25.0%, 40/160), rarely (30.0%, 48/160), occasionally (26.3%, 42/160), or frequently/very frequently (15.0%, 24/160).
Survey Participants’ Perceptions of txt4health
All survey participants agreed/strongly agreed that the messages included in txt4health were clear and easy to understand (100.0%, 160/160). Most reported txt4health made them knowledgeable about their risk for type 2 diabetes (88.1%, 140/159) and the program made them conscious of their diet and physical activity habits (88.8%, 142/160). Most survey participants reported that they agreed/strongly agreed that they enjoyed participating in the program (88.1%, 140/159). Although 6.9% (11/159) indicated that there were too many messages sent in the program, most participants indicated that the number of messages sent each week was just the right amount (79.9%, 127/159). Seefor a complete breakdown of participant responses to the txt4health perception items. Perceptions of satisfaction among survey participants were very positive with 67.1% (108/161) reporting satisfaction scores of 8 or higher on a 10-point scale, with 10 equivalent to most satisfied (mean 8.2, SD 1.6). Using multiple regression analyses, race was a significant predictor of satisfaction scores (beta=.81, P=.008) with non-White participants reporting higher mean satisfaction scores than White participants (mean 8.6, SD 1.5 and mean 7.7, SD 1.6, respectively; t155=–3.79, P<.001). In addition, PAM level was also predictive of satisfaction scores (beta=.33, P=.03) with individuals at PAM level 4 rating their satisfaction higher than those with PAM levels less than 4 (mean 8.5, SD 1.6 and mean 7.7, SD 1.6, respectively; t157=–2.96, P=.004).
Self-Reported Behavior Change Attributed to txt4health
Because the goal of txt4health is to promote behavior change among participants, survey respondents were asked to reflect on how txt4health may have changed their behavior. The majority of the participants agreed/strongly agreed that after having completed txt4health, they were more likely to replace sugary drinks, such as juice or soda, with water (78.0%, 124/159), have a piece of fresh fruit instead of dessert (74.2%, 118/159), substitute a small salad for chips or fries when dining out (76.1%, 121/159), buy healthier foods when grocery shopping (79.7%, 126/158), and eat more grilled, baked, or broiled foods instead of fried (75.5%, 120/159). Of those individuals who set goals within txt4health, most reported meeting their physical activity goals most of the time or always (60.6%, 80/132), whereas 25.7% (28/109) reported meeting weight goals most of the time or always. In addition, most (66.3%, 106/160) agreed/strongly agreed that the program helped them become more physically active, and almost half reported that txt4health helped them to lose weight (45.6%, 73/160). Seefor a complete breakdown of participant response to behavior change items.
Among txt4health users who participated in the user satisfaction and perceptions survey, participants reported very positive perceptions of txt4health. Overall, the majority of survey participants reported enjoying the program and rated their satisfaction as 8 or higher on a 10-point scale, with 10 equivalent to most satisfied. In addition to positive perceptions of the program, survey respondents self-reported making behavior choices related to diet and physical activity as a result of their participation, and approximately half of respondents self-reported that txt4health helped them to lose weight. Many of these positive perceptions were echoed in the open-ended response section of the survey, in which participants were given an opportunity to provide additional comments. Overall, the tone of these comments was positive, and survey respondents offered helpful and constructive suggestions on how to improve the program rather than any overtly negative feedback. Given our recruitment strategy solicited individuals who were active within the program and excluded any individuals who dropped out before recruitment, it is not surprising that participant response regarding the program was overwhelmingly positive. Undoubtedly, this recruitment strategy introduced a fair amount of bias into the survey results because only individuals who had not dropped out of the program received the invitation to participate. This, coupled with the self-selection bias that occurs from participants who were likely skewed toward favoring txt4health, is potentially problematic. Future research should seek to conduct an evaluation with a stronger research design that includes random sampling of both active and dropout participants to reduce the amount of bias within the sample.
Despite these limitations related to sampling, it is clear that for a subset of individuals who initially enrolled in txt4health, this approach to a behavior change program was met with great satisfaction from many participants, and suggests that this is a feasible and acceptable method to promote behavior change to large numbers of individuals who may derive some perceived benefit from participation. There appears to be a relatively discrete subset of individuals who responded very well to this type of program in terms of both program usage (see our companion paper, Part I) and overall perceptions of txt4health. In particular, it appears that minority participants respond more favorably to the txt4health intervention. Results from Part I of this investigation showed that non-White participants stayed in txt4health longer than White participants (see Part I companion paper), and results from the present user survey show that non-White participants had higher ratings of satisfaction than White participants.
Because it is unlikely that there exists a one-size-fits-all approach to behavior change programs that will be met with universal acceptance, future work should seek to understand which types of people are best suited to different types of programs, as well as to develop instruments for identifying these individuals. This may allow for better targeting of programs to those individuals who are likely to derive the most benefit. Throughout our analysis, demographic characteristics seldom predicted participant perceptions or usage of txt4health. This suggests that with the exception of non-White race predicting length of participation and satisfaction, demographics are not the drivers of use/acceptability in text message–based programs, and perhaps more psychological factors may be key.
In addition to positive perceptions of txt4health among survey respondents, many participants also self-reported that participation in the program led to positive behavior changes, which we hope will ultimately lead to positive health outcomes. Recent work by Ramachandran et al  supports the use of text messaging to gain positive health outcomes related to diabetes. In that study, 537 working men in India were randomized to a text message program (similar to txt4health) or usual care. The authors found after 2 years of program use, the text message program was effective at preventing type 2 diabetes; 27% of control group participants developed type 2 diabetes by the end of the trial compared to 18% of the text message group participants [ ]. This recent finding in India has increased the desire to more concretely establish the efficacy of this type of program among Americans, and to see if these findings could be replicated here. Unfortunately, in the present study, because of the limitations of our research design, we lack objective measures that would reveal whether txt4health actually caused people to change their behavior; thus, we are not able to make a judgment regarding the efficacy of text4health on health outcomes. Rather, our evaluation relies on self-reported survey data that only provides limited insight into participant perceptions of the program. Reasons for the limitations regarding research design have been previously described [ ], but briefly include issues related to program dissemination strategies, as well as limited time and financial resources. Although small-scale pilot studies exist that indicate text message programs can be effective at promoting behavior change, well-powered randomized controlled trials establishing efficacy of these types of programs among American adults are largely missing from the literature [ - ]. Given the participant response regarding self-reported behavior change from this study, future work should seek to test the efficacy of txt4health and similar broad-based, large-scale, public health text message interventions using well-thought-out experimental designs with appropriate controls.
Despite the proliferation of small-scale pilot studies demonstrating the efficacy of text message mHealth programs for chronic disease, little is known about user perceptions of these types of programs among American adults while operating at scale. Results from this study suggest that participants in txt4health, a large-scale public health–focused text message program targeting type 2 diabetes, have positive perceptions of the program and that participation has led to positive behavior change. Although these results come from a seemingly biased sample of txt4health users who self-selected to participate in the user survey, these findings do suggest that a large subset of individuals who participate in a broad-based, public health–focused intervention may respond quite favorably.
Txt4health and this evaluation were funded by the Office of the National Coordinator for Health Information Technology (Cooperative Agreement #90BC0017 and #90BC0016). The authors of this evaluation wish to thank Martin Salvi and Pamela Johnson, PhD, and the rest of the staff at Voxiva for assisting with evaluation recruitment and data extraction efforts, Rachelle Dawood for assistance with evaluation data collection and data entry, and Penelope Kopka from the Wayne State University College of Nursing Office for Health Research.
Conflicts of Interest
- Smith A. Smartphone Ownership: 2013 Update. Washington, DC: Pew Internet & American Life Project; 2013 Jun 05. URL: http://www.pewinternet.org/~/media//Files/Reports/2013/PIP_Smartphone_adoption_2013.pdf [accessed 2013-08-27] [WebCite Cache]
- K4Health. 2013. mHealth evidence: What is mHealth evidence? URL: http://www.mhealthevidence.org/ [accessed 2013-08-27] [WebCite Cache]
- Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236. [CrossRef] [Medline]
- Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med 2013;10(2):e1001382 [FREE Full text] [CrossRef] [Medline]
- Cole-Lewis H, Kershaw T. Text messaging as a tool for behavior change in disease prevention and management. Epidemiol Rev 2010 Apr;32(1):56-69 [FREE Full text] [CrossRef] [Medline]
- de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012;12:CD007459. [CrossRef] [Medline]
- Fjeldsoe BS, Marshall AL, Miller YD. Behavior change interventions delivered by mobile telephone short-message service. Am J Prev Med 2009 Feb;36(2):165-173. [CrossRef] [Medline]
- Whittaker R, Matoff-Stepp S, Meehan J, Kendrick J, Jordan E, Stange P, et al. Text4baby: development and implementation of a national text messaging health information service. Am J Public Health 2012 Dec;102(12):2207-2213. [CrossRef] [Medline]
- Abebe NA, Capozza KL, Des Jardins TR, Kulick DA, Rein AL, Schachter AA, et al. Considerations for community-based mHealth initiatives: insights from three Beacon Communities. J Med Internet Res 2013;15(10):e221 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [Medline]
- SurveyMonkey. URL: https://www.surveymonkey.com/ [accessed 2013-12-09] [WebCite Cache]
- Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and testing of a short form of the patient activation measure. Health Serv Res 2005 Dec;40(6 Pt 1):1918-1930 [FREE Full text] [CrossRef] [Medline]
- Insignia Health. Patient Activation Measure (PAM) 13 License Materials. Portland, OR: Insignia Health, LLC; 2011.
- Ramachandran A, Snehalatha C, Ram J, Selvam S, Simon M, Nanditha A, et al. Effectiveness of mobile phone messaging in prevention of type 2 diabetes by lifestyle modification in men in India: a prospective, parallel-group, randomised controlled trial. The Lancet Diabetes & Endocrinology 2013 Nov;1(3):191-198. [CrossRef]
|BMI: body mass index|
|GCBC: Greater Cincinnati Beacon Collaborative|
|mHealth: mobile health|
|PAM: Patient Activation Measure|
|SEMBC: Southeast Michigan Beacon Community|
Edited by G Eysenbach; submitted 30.08.13; peer-reviewed by R Whittaker, R Furberg, M McKinley; comments to author 19.09.13; revised version received 30.10.13; accepted 12.11.13; published 19.12.13
©Lorraine R Buis, Lindsey Hirzel, Scott A Turske, Terrisca R Des Jardins, Hossein Yarandi, Patricia Bondurant. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 19.12.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.