Mobile health (mHealth) refers to using mobile communication devices, such as smartphones, tablets, or smartwatches, to support health, health care, and public health [1-3]. An mHealth app is software incorporated into a mobile device to improve health, health care services, and research [4]. Due to the pressure on health systems, there is an urgent need to invest in health and health systems via effective, feasible, and sustainable interventions [5,6]. mHealth has the potential to improve health and health care through different means. One example is apps that monitor users and collect information, such as exercise, heart rate, or medication adherence [7]. Other apps support health care providers in diagnostic procedures or clinical decision-making for treatments [8]. Finally, mHealth apps can also deliver health interventions, which is this study’s focus. Examples are apps to increase medication adherence or provide psychotherapeutic interventions (eg, cognitive behavioral therapies) to manage mental and physical health problems [3,9-14].

The mHealth market size is rapidly expanding. There are >350,000 health apps, and the global market will probably grow by 40% between 2022 and 2030 [15,16]. Developers often advertise mHealth apps as improving health and well-being, but sound research should support their claims on the effects of mHealth interventions on clinical outcomes. However, the evidence that mHealth interventions can improve health care processes, change patient behavior, and improve health outcomes is still uncertain [14,15,17]. Randomized controlled trials (RCTs) are considered the most rigorous method for evaluating the efficacy and effectiveness of health interventions because they minimize bias [18]. Therefore, RCTs are crucial for generating mHealth evidence and drawing firm conclusions about mHealth interventions’ effects [19-21].

mHealth interventions have peculiarities that distinguish them from other health care interventions, such as drug interventions. For example, mHealth interventions are usually behavioral interventions that are more complex to define and standardize than pharmacological interventions [22]. In addition, evaluating and reporting mHealth intervention integrity (the degree to which the study intervention was implemented as intended) in RCTs can be challenging [23]. A recent systematic review by Tønning et al [24] outlines key methodological challenges in conducting RCTs for smartphone-based treatments in psychiatry. The challenges highlighted include the rapid pace of technological advancements, which may make interventions obsolete by the time articles are published. In addition, privacy and security issues often do not receive the attention they deserve. Moreover, mHealth RCTs lack consistency in how adherence to interventions is measured and how primary outcome measures are selected, making it difficult to compare results across different RCTs. Challenges related to missing data and the methods used for statistical analysis further complicate the assessment of mHealth interventions’ efficacy. To address these issues, Tønning et al [24] recommend adopting more flexible trial designs, giving priority to ethical considerations concerning privacy and security, using technology for improved adherence tracking, and following detailed protocols that align with the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) guidelines [25], among other recommendations. Previous findings indicate that evaluating mHealth intervention effects via RCTs has specific methodological challenges. Our research is the first to solicit direct feedback from mHealth researchers on methodological challenges unique to mHealth RCTs.

A survey and a consensus exercise were chosen for this project to address the diverse methodological challenges in mHealth RCTs. Consensus methods allow for the integration of expert perspectives to produce recommendations. This approach is essential given the lack of consistent solutions to these challenges. The goal of the consensus exercise was to develop recommendations for researchers working in mHealth. The intended audience is global, with participation from experts across regions, making the recommendations relevant for both high- and low-resource settings.

Identifying and overcoming the methodological challenges specific to RCTs of mHealth interventions (from now on referred to as mHealth RCTs) can clarify whether mHealth improves health outcomes. This study had two aims: (1) to identify specific methodological challenges in RCTs evaluating the effects of mHealth interventions and (2) to develop consensus-based recommendations to address selected methodological challenges in mHealth RCTs.

We undertook a 2-phase participatory research study with mixed methods (quantitative and qualitative; Figure 1). This manuscript follows the Checklist for Reporting Results of Internet E-Surveys and the Accurate Consensus Reporting Document guidelines [26,27]. The study protocol was not registered.

The steering group, which included 4 researchers (CMW, LSW, JB, and JL-A), defined the project aims, collected potential methodological challenges of mHealth RCTs, identified the potential survey respondents, developed the survey, and selected the workshop participants. JB, an experienced researcher, facilitated the consensus meeting. The steering group also analyzed the survey results, summarized participants’ comments during the workshop, and integrated their feedback to generate this manuscript. No members of the public, patients, or carers were invited to participate in this study.

The study design was a cross-sectional web-based survey.

The consensus method involved a consensus meeting, wherein participants engaged in discussions to reach an agreement. Recommendations from the workshop were endorsed through consensus without structured voting [26] (Multimedia Appendix 4).

We submitted the study synopsis to the Kantonale Ethikkommission (Kanton Zürich), Switzerland. After review, they stated that the study did not fall under the regulation of the Human Research Act of Switzerland (Business Administration System for Ethics Committees Req-2022-00839). The survey and meeting participants gave their consent to participate via the SoSci Survey tool. We conducted an anonymous web-based survey using the SoSci Survey platform hosted by University Hospital Zurich. Personal details such as names, email addresses, or IP addresses were not collected. In the final section of the survey, respondents were invited to participate in an optional online workshop. The responses to this invitation were stored in a separate SoSci Survey project. Participants interested in attending the workshop provided their name, email, and place of residence, among other details. To ensure privacy and confidentiality, all data were securely collected and stored at University Hospital Zurich. In addition, participants’ data were managed in strict accordance with institutional data protection policies, ensuring that their privacy was rigorously protected throughout the research process. Survey participants did not receive compensation. While workshop participants were not offered financial compensation, they were given the opportunity to be coauthors of the manuscript if they met the authorship criteria.

A total of 80 authors of mHealth RCTs completed the survey. The completion rate was 40.2% (106/264, corresponding to the number of people who agreed to participate divided by the number of visitors). During testing, it consistently took <10 minutes to complete the survey. Furthermore, a similar survey conducted by our team, which had a comparable number of items but focused on challenges in systematic reviews of mHealth interventions [68], took an average of 5 (SD 2) minutes to complete, with a range of 2 to 11 minutes.

Figure 1 details the reasons for not completing the survey. The respondents’ academic backgrounds were diverse, and the most frequent were medicine (24/80, 30%), psychology (20/80, 25%), and epidemiology (12/80, 15%). Most participants (52/80, 65%) were experienced mHealth researchers who had conducted ≥2 mHealth RCTs. The survey respondents had been involved in at least 185 mHealth RCTs. Most trials (56/185, 30.3%) were conducted in North America and Europe. Few RCTs were conducted in Africa (7/185, 3.8%) and South or Central America (4/185, 2.2%; Table 1).

Most respondents (74/80, 92%) identified at least one methodological aspect as more or much more challenging in mHealth RCTs compared to non-mHealth RCTs. Survey respondents rated a median of 6 (IQR 4-9) out of 21 aspects as more or much more challenging (Multimedia Appendix 5). The methodological aspects most frequently perceived as more or much more challenging in mHealth RCTs were the following: managing low adherence to the mHealth intervention (43/77, 56%), defining adherence (39/79, 49%), measuring adherence (33/78, 42%), analyzing passive data (eg, data collected from smartphone sensors; 24/58, 41%), determining which active components of the mHealth intervention are used or received by the participant (31/75, 41%), and verifying the participants’ identity during recruitment (participants can register more than once using different identities; 28/68, 41%). In contrast, the aspects described by the lowest number of respondents as challenging were analyzing patient-reported outcomes (6/75, 8%) and analyzing clustered data (7/62, 11%; Table 2). Multimedia Appendix 6 lists additional challenges reported by the respondents.

Experienced and less experienced researchers’ perceptions did not differ markedly. However, a higher number of less experienced researchers considered the following aspects as more or much more challenging: achieving the required sample size (12/27, 44% vs 10/51, 20%) and analyzing data using repeated measurements across time (8/24, 33% vs 4/51, 8%). On the other hand, more experienced researchers considered the availability of suitable outcome measurement instruments for mHealth trials as more or much more challenging (17/52, 33% vs 3/26, 12%).

In total, 11 mHealth researchers attended the online workshop (2 hours; February 1, 2023; Table 3). Figure 2 outlines the topics that guided the discussion during the workshop.

The following recommendations were agreed upon during the workshop and refined via email afterward following a qualitative process.

We contacted 1535 authors of mHealth intervention RCTs, of whom 80 (5.21%) completed the survey. Our research identified specific methodological challenges in RCTs of mHealth interventions. The aspects most frequently reported as challenging were those related to mHealth intervention integrity, in particular managing low adherence to the mHealth intervention, defining adherence, measuring adherence, and determining which mHealth intervention components are used or received by the participants. Other challenges were also frequent, such as analyzing passive data and verifying the participants’ identity during recruitment to the RCT. The workshop concentrated on the integrity of mHealth interventions. Following the workshop and subsequent email discussions, we established 17 consensus-based recommendations to address selected methodological challenges related to mHealth intervention integrity.

Our study has several strengths. First, we likely identified the main methodological hurdles specific to mHealth RCTs. The process began with nonsystematic literature searches that informed the potential challenges listed in the survey. Interestingly, survey respondents identified only 1 methodological challenge not included in our survey: finding a suitable control condition, such as creating a placebo app. Second, the survey and workshop participants had diverse experiences, having been involved in 1 to >5 mHealth RCTs each. This range of expertise allowed our study to encapsulate the challenges encountered by mHealth researchers with varying experience levels. Furthermore, the workshop participants were involved in mHealth RCTs conducted in high- and low-resource settings, such as the United States, Argentina, Uganda, and Iran. This diversity probably helped capture the challenges that researchers face in different contexts. Third, a key strength of our project was adherence to reporting guidelines for online surveys and consensus documents. This alignment with best practices enhanced the credibility and reproducibility of our work (Multimedia Appendices 7 and 8).

This study has several limitations. First, the low response rate may have affected the representativeness of the results as participants may have had a specific interest or experience in mHealth RCTs. Contributing factors include inoperative emails, out-of-office replies, the specificity of the topic, and survey fatigue. Online surveys generally suffer from lower response rates compared to other methods [103,104]. In our survey, despite sending reminders and attempting to reach a diverse pool of researchers, this limitation persisted. Future studies could use additional strategies such as incentives or targeted recruitment to improve participation rates.

Second, only 15% (12/80) of the respondents had experience with both mHealth and non-mHealth interventions, limiting the survey’s ability to capture the challenges unique to mHealth RCTs. Including more researchers with experience in both areas could provide a more comprehensive understanding of these challenges in future studies.

Third, the study did not account for differences between various types of mHealth interventions. These interventions vary significantly, ranging from simple SMS text messaging to complex, multicomponent apps, each presenting distinct challenges. This variability may have influenced the findings. Future research should categorize mHealth interventions to provide more targeted recommendations for addressing specific challenges.

Fourth, the complexity of measuring adherence in mHealth RCTs was not fully explored. Adherence is difficult to measure due to factors such as user engagement, app use, and internet access. While adherence was identified as a key challenge, more research is needed to develop standardized methods for its measurement. Future studies should also examine how adherence differs across various types of mHealth interventions and how it impacts study outcomes.

To our knowledge, our survey is the first to ask mHealth researchers about methodological challenges specific to mHealth RCTs. Our findings align with those of previous research on the challenges in mHealth intervention research. A review of RCTs evaluating the effects of mHealth interventions in chronic disease management identified the following challenges as the most important ones: designing high-quality studies, developing robust interventions in combination with health professional input, and identifying tools and methods to improve patient adherence [105]. Another systematic review pinpointed methodological challenges in RCTs of smartphone-based interventions in psychiatry. These included the need for flexible trial designs that can quickly adapt to the fast-evolving nature of mHealth interventions, privacy and security concerns, the lack of standardized adherence metrics, and the difficulties in choosing primary outcomes and handling missing data [24].

Our research advances mHealth methodology by addressing specific challenges not comprehensively covered in other studies [24,25,105]. First, we examined intervention integrity, highlighting the intricacies of defining and measuring adherence and pointing to the need to define the essential mHealth intervention components. Second, we spotlighted the challenge of analyzing passive smartphone sensor data, a significant digital health issue overlooked in previous research focused on traditional data analysis challenges. Third, our work underscores the unmet need for effective digital identity verification in studies relying on online recruitment, a crucial aspect often overlooked in previous research in favor of recruitment strategies and sample size considerations.

Our study sets itself apart from previous research by calling for a clear definition of mHealth intervention adherence, emphasizing the need to compare between the actual and intended use of the intervention against predefined thresholds. Previous research has broadly addressed mHealth adherence challenges, noting that poor reporting standards in mHealth RCTs hinder adherence assessment [24,70,106]. In addition, our study distinguishes itself from previous research by advocating for a transparent framework that requires researchers to predefine the intended use of the mHealth intervention and choose clear thresholds for adherence evaluation. However, we recognize our limitation in not offering specific guidelines for measuring adherence, for example, by proposing adherence thresholds. Nevertheless, advocating for a transparent and operative framework to measure mHealth intervention adherence is a first important step.

Most of the methodological challenges that this study highlighted are associated with mHealth intervention integrity. There is no gold standard for measuring adherence in mHealth trials [70,107]. This construct is challenging to define and measure in mHealth trials for several reasons. First, mHealth trials have different factors influencing adherence compared to drug interventions, such as more diverse participant characteristics, preferences, and context, such as internet access [19]. The key difference likely stems from drug interventions occurring in controlled settings versus mHealth interventions occurring in real-world settings, which are less controlled and more variable [8,9]. Second, adherence can be measured at different trial phases, such as at intervention initiation (when the participants begin the intervention), during the implementation period, or at discontinuation (the permanent cessation of the randomized intervention) [108]. Third, the data collection sources and methods to measure adherence are diverse, such as self-reporting, interviews, or electronic tracking via mobile devices [109]. In this line, adherence definitions and thresholds can vary, such as the number of uses of the app, the time spent using the app, or the proportion of participants who achieved a certain level of adherence. Fourth, mHealth trials have different factors influencing adherence compared to drug interventions, such as more diverse participant characteristics, preferences, and context (eg, access to the internet network connectivity) [19].

Consequently, mHealth intervention integrity is seldom discussed in RCTs, likely due to the absence of universally accepted definitions and measurements [69]. Furthermore, existing reporting guidelines do not advise on how to report the integrity of mHealth interventions [25,110-114], and it remains uncertain how mHealth intervention integrity information should be managed by researchers and decision makers. Nevertheless, mHealth RCTs should strive to assess the integrity of interventions for each study arm and incorporate these data into their analyses, possibly using standardized data extraction forms. In this regard, a quantitative framework for evaluating the integrity of mHealth interventions could promote uniform assessment in RCTs. However, this is a complex task as there is no consensus on the model variables and their respective weights [69].

RCTs of mHealth interventions have specific methodological challenges compared to those of non-mHealth interventions, particularly those related to intervention integrity. Following our recommendations for addressing these challenges can lead to more reliable assessments of the effects of mHealth interventions on health outcomes.