This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Mobile health (mHealth) interventions are effective in promoting physical activity (PA); however, the degree to which external validity indicators are reported is unclear.
The purpose of this systematic review was to use the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework to determine the extent to which mHealth intervention research for promoting PA reports on factors that inform generalizability across settings and populations and to provide recommendations for investigators planning to conduct this type of research.
Twenty articles reflecting 15 trials published between 2000 and 2012 were identified through a systematic review process (ie, queries of three online databases and reference lists of eligible articles) and met inclusion criteria (ie, implementation of mobile technologies, target physical activity, and provide original data). Two researchers coded each article using a validated RE-AIM data extraction tool (reach, efficacy/effectiveness, adoption, implementation, maintenance). Two members of the study team independently abstracted information from each article (inter-rater reliability >90%) and group meetings were used to gain consensus on discrepancies.
The majority of studies were randomized controlled trials (n=14). The average reporting across RE-AIM indicators varied by dimension (reach=53.3%, 2.67/5; effectiveness/efficacy=60.0%, 2.4/4; adoption=11.1%, 0.7/6; implementation=24.4%, 0.7/3; maintenance=0%, 0/3). While most studies described changes in the primary outcome (effectiveness), few addressed the representativeness of participants (reach) or settings (adoption) and few reported on issues related to maintenance and degree of implementation fidelity.
This review suggests that more focus is needed on research designs that highlight and report on both internal and external validity indicators. Specific recommendations are provided to encourage future mHealth interventionists and investigators to report on representativeness, settings, delivery agents for planned interventions, the extent to which protocol is delivered as intended, and maintenance of effects at the individual or organizational level.
The numerous health benefits of physical activity (PA) are well known, but still it is estimated that roughly 31% of the world’s adult population (28% men, 34% women) is classified as insufficiently active [
One such approach is the use of mobile technology, since ownership is on the rise in adults and children [
This growth in mobile technology ownership has led to the development of a number of mobile health (mHealth) intervention reviews [
Despite the popularity of commercially available health-related applications, there is little evidence that mobile phone-based interventions with demonstrated efficacy have been translated beyond the research setting and been broadly adopted [
To improve the reporting across behavioral interventions, Glasgow and colleagues developed the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework to evaluate the degree to which behavioral interventions, including those targeting PA, report on internal and external validity factors [
We replicated the search strategy used in a recently published meta-analysis publication that focused solely on effectiveness of mHealth interventions for PA promotion at the individual level [
Inclusion criteria.
Data type | Inclusion criteria |
Participants | Any age |
Language | English |
Study design | Experimental and quasi-experimental |
Control condition | Any comparator including active control, inactive control, or participants as their own control (ie, pre- and post-measures) |
Intervention | Implementation of mobile technologies |
Measurement | Assesses physical activity directly among participants |
Primary outcome | Physical activity |
Type of data | Original, quantitative outcome data |
Flow diagram of study selection.
Comprehensiveness of reporting was determined using a previously developed 21-item validated data extraction tool that included both internal and external validity indicators based on the RE-AIM framework [
All studies were coded independently by two members of the research team with the exception of the first three studies which were coded by five members of the research team to promote familiarity with the data extraction tool. For each of the 21 items, coders indicated whether or not the indicator was reported (ie, yes or no), and subsequently extracted specific data. After independently coding, the Kappa statistic [
To calculate the proportion reporting for each item, the number of “yes” codes was summed across the 15 studies and then divided by 15. Then the resulting number became the proportion reporting for that particular item. An overall comprehensiveness of reporting score for each article was calculated based on the number of reported indicators (possible score 0-21). Comprehensiveness of reporting score categories have been published in a past RE-AIM review [
RE-AIM internal and external validity indicators.
RE-AIM |
Indicator | Description | Importance |
|
|||
|
Individual level | The number, proportion, and representativeness of participants. |
|
|
Method to identify target population | Describe the process by which the target population was identified for participation in the intervention. | Helps investigators develop an approach to determining who may be suitable for the intervention. Examples include using an electronic medical record query or mass media approaches [ |
|
Inclusion criteria | Explicit statement of characteristics of the target population that were used to determine if a potential participant was eligible to participate. | Inclusion criteria should be as inclusive as possible to improve the external validity of findings [ |
|
Exclusion criteria | Explicit statement of characteristics that would prevent a potential participant from being eligible to participate. | Exclusion criteria should be considered carefully to prevent potential harm to prospective participants, but should also avoid excluding individuals based on criteria that could be related to SES (eg, ability to travel to intervention site), comorbidities, or other factors that could influence an externally valid depiction of intervention effects [ |
|
Participation rate | Sample size divided by the target population denominator. | Provides information on the acceptability of the study and interventions from the perspective of the target population [ |
|
Representativeness | Explicit statement of characteristics of the study participants in comparison to the target population. | Identifies disparities in participation and informs the degree to which the study results are generalizable to the target population [ |
|
|||
|
Individual level | The measure of the primary outcome, quality of life, and on avoiding unintended negative consequences. |
|
|
Measures/results for at least 1 follow-up | The study variable(s) are measured at a time point after baseline. | To evaluate whether the intervention outcomes were statistically significant or changed (positively/negatively) [ |
|
Intent-to-treat analysis utilized | Analyzing participants in trials in the groups to which they were randomized, regardless of whether they received or adhered to the allocated intervention. | Reduces bias from omitting individuals who were lost to follow-up and improves generalizability [ |
|
Quality-of-life (QOL) or potential negative outcomes | QOL: Includes a measure of quality of life with some latitude for coding articles that refer to well-being or satisfaction with life. |
Provide a metric to compare across interventions with different behavioral targets and provides a better sense of the impact that the intervention on the participants’ perceptions of health [ |
|
Percent attrition | The proportion that was lost to follow-up or dropped out of the intervention. | High attrition lowers statistical power and treatment-correlated attrition of participants from conditions threatens internal validity [ |
|
|||
|
Organizational level (setting and staff) | The number, proportion, and characteristics of adopting organizations and staff. |
|
|
Description of intervention location | The explicit statement of characteristics of the location of the intervention. | Provides an understanding of resources needed for future researchers [ |
|
Description of staff who delivered intervention | The explicit statement of characteristics of the staff who delivered the intervention. | Provides information on the characteristics may be needed to deliver an intervention and assist with retention of participants [ |
|
Method to identify staff who delivered intervention (target delivery agent) | Describe the process by which the staff was identified for participation in the study. | Helps investigators develop an approach to identify and engage staff that may be suitable for intervention delivery [ |
|
Level of expertise of delivery agent | Training or educational background in of those delivering the intervention. | Allows for the assessment of generalizability of those delivering an intervention to typical practice settings delivery [ |
|
Inclusion/exclusion criteria of delivery agent or setting | The explicit statement of characteristics of the setting/agent that were used to determine if a potential setting/agent is eligible to participate. | Inclusion criteria should be as inclusive as possible to improve the external validity of findings. Exclusion criteria should not systematically remove potential settings or staff that typical in the practice domain [ |
|
Adoption rate of delivery agent or setting | The number of participating delivery settings or agents divided by the number of eligible and approached delivery settings or agents. | Provides information on the acceptability of the study and interventions from the perspective of the setting and staff that will ultimately be responsible for intervention delivery [ |
|
|||
|
Organizational level | The degree to which the intervention is delivered as intended. |
|
|
Intervention duration and frequency | Duration: length the intervention over days, weeks, and months as well as the length of each intervention contact. |
Useful for replication and comparison of resources needed to resources available in a practice setting [ |
|
Extent protocol delivered as intended (%) | Description of fidelity to the intervention protocol. | This provides insight into the feasibility of delivering all components of an intervention at the pre-determined date and time [ |
|
Measures of cost of implementation | The ongoing cost (eg, money, time) of delivery across all levels of the intervention. | This is helpful for future researchers to be able to determine if conducting a specific intervention has economically feasible delivery [ |
|
|||
|
Individual and organization level | The measure of behavior at the individual level and sustainability of the intervention at an organizational level. |
|
|
Assessed outcomes ≥ 6 months post intervention | Description of follow-up outcome measures of individuals available at some duration after intervention termination. | Provides information on the maintenance of intervention outcomes over time [ |
|
Indicators of program level maintenance | Description of program continuation after completion of the research study. | Provides information on whether the intervention can be integrated into an existing system/organization [ |
|
Measures of cost of maintenance | The ongoing cost of maintaining delivery across all levels of the intervention. | Sustainability costs provides information for practice settings to determine the resources needed for long-term intervention delivery [ |
All trials were published after 2006 and 13 were conducted in Western countries. Six studies were conducted in the United States [
Five studies measured PA only through self-report [
In addition to PA, the majority of studies (n=11) reported on other outcomes. Eight studies reported on body mass index (BMI) [
The types of mobile devices used were similar across studies. Nearly all studies (n=13) used mobile phones while two used personal digital assistants [
Reach was the second most reported dimension at 53.3% (2.67/5). Approximately half of all studies reported on four of the five items (method used to identify target population, inclusion and exclusion criteria, and participation rate). The least reported component was representativeness, with only four studies reporting [
Efficacy/effectiveness was the most reported dimension at 60.0% (2.4/4). All studies reported on measures or results for at least one follow-up. Approximately three quarters of the studies reported on percent attrition, which ranged from 0-53%. Four studies reported on intent-to-treat analysis [
The majority of studies (n=12) reported whether the trial was an efficacy or effectiveness trial. Of these studies, eight were efficacy trials [
The average proportion reporting on Adoption items was 11% (0.7/6). Level of expertise of delivery agent was the most reported adoption component (n=5). The descriptions of staff level of expertise included a nutritionist [
Setting-level reporting was similar to staff-level reporting. Only five studies specified the intervention location: a school [
The average proportion reporting on Implementation indicators was 24% (0.7/3). Intervention duration and frequency were the most frequently reported items (n=6) [
Maintenance was the dimension that was reported least among the RE-AIM dimensions, with no items (0%, 0/3) reported. The reporting on indicators of individual-level or program-level maintenance were not reported in any trial.
The average comprehensiveness of reporting score was 6.9 out of a possible 21-item reporting coding sheet and scores ranged from 3-13. None of the studies were categorized as high reporting quality, six studies were moderate (range 8-11) [
Proportion of mobile health interventions reporting RE-AIM dimensions and components (n=15).
RE-AIM Dimensions | RE-AIM Components | Proportion Reportinga, % |
|
||
|
Method to identify target population | 60.0 |
|
Inclusion criteria | 80.0 |
|
Exclusion criteria | 60.0 |
|
Participation rate | 46.7 |
|
Representativeness | 26.7 |
|
Average across Reach Components | 53.3 |
|
||
|
Measures/results for at least one follow-up | 100.0 |
|
Intent to treat analysis utilized | 33.3 |
|
Quality-of-life or potential negative outcomes | 33.3 |
|
Percent attrition | 73.3 |
|
Average across Efficacy/Effectiveness Components | 60.0 |
|
||
|
Description of intervention location | 13.0 |
|
Description of staff who delivered intervention | 0.0 |
|
Method to identify staff who delivered intervention (target delivery agent) | 0.0 |
|
Level of expertise of delivery agent | 33.3 |
|
Inclusion/exclusion criteria of delivery agent or setting | 13.3 |
|
Adoption rate of delivery agent or setting | 6.7 |
|
Average across Adoption Components | 11.1 |
|
||
|
Intervention duration and frequency | 40.0 |
|
Extent protocol delivered as intended (%) | 13.3 |
|
Measures of cost of implementation | 20.0 |
|
Average across Implementation Components | 24.4 |
|
||
|
Assessed outcomes ≥ 6 months post intervention | 0.0 |
|
Indicators of program level maintenance | 0.0 |
|
Measures of cost of maintenance | 0.0 |
|
Average across Maintenance Components | 0.0 |
aBased on denominator of 15 intervention trials, reported across 20 articles.
Our review highlighted a recent increase in studies conducted to determine the efficacy or effectiveness of mHealth interventions for the promotion of PA. We identified gaps across and within each of the RE-AIM dimensions, potentially as a result of the relative early stages of this area of research. We also understand that there is a need to advance research by utilizing innovative, flexible, and rapid research designs and “rapid-learning research systems” where researchers, funders, health systems, practitioners, and community partners collaborate [
Still, the comprehensiveness of reporting on RE-AIM criteria across these mHealth articles was relatively low with a number of gaps in reporting on both internal (eg, extent that the protocol was delivered as intended) and external validity factors (eg, description of intervention location and staff). At the individual level (ie, reach, efficacy/effectiveness, and maintenance), the reporting on issues related to reach and maintenance are particularly problematic. At the organizational or delivery level (ie, adoption, implementation, maintenance), there are large gaps in reporting across each of the dimensions. These gaps extend to the reporting across the four CONSORT-EHEALTH standards of access as well as the degree to which intervention features and functionality were addressed. Based on our findings, the results reported on mHealth PA interventions, from both an internal and external validity perspective, should be considered with caution.
Consistent with past research, this body of literature does not typically describe the target population or give indications as to the degree to which the study samples are representative of a larger population [
Similar to other areas of research, efficacy or effectiveness based upon changes to the PA and percent attrition were reported consistently across the majority of studies while the maintenance of those changes were not [
Organizational or delivery level facets of RE-AIM have consistently been underreported across behavior change intervention studies; yet, studies on mHealth PA interventions appear to be even less likely to report on organizational adoption, implementation, and maintenance [
Understanding costs across RE-AIM dimensions is also key for dissemination [
Based upon the growth of research in the area of mHealth PA interventions and the review of this literature to date, there are a number of ways to improve the assessment and reporting on individual and organizational level factors that will improve our understanding of both the internal and external validity of this work. In
Recommendations.
RE-AIM component | Recommendations for reporting on future mHealth PA studies |
|
|
|
Report on characteristics (eg, demographics, behavioral outcomes) of nonparticipants and compare them to participants to understand the representativeness of the study sample. If not possible for Institutional Review Board reasons to compare nonparticipants directly, participants can be compared to the general local population. |
|
Indicate exclusion criteria so that it is clear as to why certain individuals were not eligible for participation. |
|
Report on inclusion criteria (eg, computer/Internet literacy [ |
|
Describe recruitment methods and adaptations to recruitment methods so that future researchers will know the best ways to recruit for mHealth PA interventions. |
|
Recruit participants from a known denominator that are representative of the target population. |
|
Calculate the participation rate based upon a known denominator: # eligible approached and agreed to participate/total # eligible and approached. |
|
Describe how participants accessed the application, and cost to access application [ |
|
|
|
Use intention-to-treat methods. |
|
Assess potential negative outcomes of the intervention and quality of life before and after the intervention. |
|
Indicate subgroup effects, especially those related to health equity issues. |
|
|
|
Report on characteristics of the location where the intervention is delivered and the staff who deliver the intervention and describe reasons for selection of this location and staff. |
|
If applicable, explicitly state inclusion/exclusion criteria of participating staff. |
|
If delivery locations or staff volunteer or are recruited for the study, calculate participation rate of settings/staff based on the number who volunteer divided by the number who were invited. |
|
Describe the level of human involvement required for the trial compared to the level of human involvement for a routine application [ |
|
Describe the level of prompts/reminders required for the trial compared to the level of prompts/reminders for a routine application [ |
|
Describe any interventions (including training sessions/support) that are implemented in addition to the targeted mHealth intervention [ |
|
|
|
Report on intervention content, duration, and frequency of in-person and virtual sessions (eg, SMS, applications). |
|
Provide information intervention costs (eg, price of mobile technology, mobile phone data plan, time it takes to implement each session). |
|
Indicate percent delivered as intended (eg, text messages sent/unsent/received/not received; any application functioning problems or other technology problems). |
|
Reports of engagement should use standard or harmonized reporting methods (eg, number of sessions, number of bug fixes). |
|
Describe adaptations made to the intervention during implementation. |
|
|
|
Include an assessment of maintenance of PA change 6 months after the completion of the intervention. |
|
Provide a description of how the intervention could be sustained or, if applicable, provide data on the degree to which the intervention is sustained over time. |
|
Report on strategies included during intervention design related to technical staff and potential participants to produce interventions that are functional and persuasive for a long period of time. |
Our review includes some limitations. First, our conclusions and recommendations are based on the degree to which these studies reported on specific RE-AIM dimensions. It is possible that some of these data have been collected, but not reported. To address this, we included all available articles on any given trial. Still, investigator plans and data for maintenance/sustainability or designing for dissemination may exist but go unreported; however, a transparent reporting of any existing plans would provide additional important context for any intervention study. In addition, a lack of reporting on an outcome cannot be equated to a lack of an intervention’s ability to achieve that outcome (eg, lack of reporting on maintenance cannot be equated to a lack of maintenance). Second, because mHealth PA interventions are relatively novel and this is an emergent research area, the goal of the studies included within this review may have been to establish internal validity (eg, effectiveness of study outcomes), and therefore we must be cautious of being overly critical of these studies relative to their reporting of organizational adoption or maintenance factors.
There is an emergent body of literature reporting on mHealth PA interventions. On average, the studies provide initial evidence that these interventions may have promise in helping participants initiate PA. However, few studies report on key internal (eg, delivery as intended) or external (eg, descriptions of participants, settings, and delivery staff) factors. As a result, the degree to which these findings are robust and generalizable cannot be determined. Improved reporting across RE-AIM dimensions and the use of intention-to-treat, tracking of costs, and mixed methods approaches are recommended to ensure mHealth PA interventions are developed that can be broadly applicable across target populations, intervention delivery locations, and staff of differing levels of expertise.
Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and online TeleHealth
mobile health
physical activity
Reach, Effectiveness, Adoption, Implementation, Maintenance framework
None declared.