Chronic health conditions have replaced acute diseases as the leading causes of both illness and death in the United States [1]. While overall mortality rates have declined, chronic health conditions are becoming more common [1]. There is a great need for empirically based treatment and support for these conditions [2]. While the accessibility and availability of health information is limited when it is delivered solely in medical settings, the Internet and mobile health (mHealth) apps hold promise for expanding the reach of evidence-based health interventions and information [3]. Both tools have opened up a new audience for health information [4]. According to a recent survey, approximately 284.5 million people in the United States (90% of the population) have at least one mobile phone [5]. An estimated 58% of mobile phone users have a smartphone, about 31% of mobile phone owners use their phones to look for health or medical information, and approximately 35% of mobile phone users download apps to track or manage their health [6,7]. A similar report found that 72% of adults look on the Internet for health information [8]. Sixty-five percent of Internet users say they are better informed about health because of Internet and mobile phone use, and 44% report that these technologies have greatly helped their ability to get this information [9]. mHealth apps hold power in their accessibility and reach [10], as well as their ability to lower the costs associated with relaying information, support, and assistance to those who need it [11]. While in-person interventions and consultations produce the highest efficacy, mobile technologies may address the disparity between supply and demand for health-related services and information, thus creating a potentially lower-cost solution to bridging that gap [4,12]. In addition, mHealth apps and Web-based interventions provide flexibility that in-person interventions are unable to provide, such as around-the-clock access to information and personalized feedback [4,13,14]. The benefits of this technology are promising but are in the development stage. However, mHealth is predicted to continue its growth as the technology becomes more pervasive in the US population [3].

Although research has been conducted for many years on Web-based health interventions, research in the area of mHealth is nascent. There are few published studies of mHealth apps designed using theoretical frameworks [4,14-16]. There are a large number of smoking cessation apps available for Apple, Android, and Windows Phone platforms, but the vast majority do not include basic evidence-based practices and are found to have low levels of adherence, meaning that participants quit using the app over time [17,18]. In addition, a majority of Web-based smoking cessation interventions are not evidence-based [19]. There is a clear need for evaluation of Web-based and mHealth interventions to ensure that public health problems, such as smoking, can be addressed with evidence-based tools [17,19].

Web-based interventions and mHealth apps are fundamentally different, but research in these areas can experience similar challenges. There are special issues involved with conducting virtual interventions. Research on mHealth apps and Web-based interventions is difficult due to unique challenges associated with recruiting, enrolling, and retaining participants [4,13]. Recruitment and retention of participants in these studies directly affects sample size, which determines the power and significance of the study [14,20]. If recruitment goals are not met in the initial period, it can negatively influence the chance of finding an effect or, if an extension on the recruitment period is required, it can expand study costs considerably [20]. Participant retention is another main concern for studies using mHealth interventions [4]. Some claim attrition may be due to early interest in the novelty of mHealth apps, which then fades as the innovation wears off [4], while others suggest the lack of personal contact can lead to higher dropout rates [21]. Attrition is inevitable, especially in lifestyle interventions such as diet or smoking cessation where dropout rates less than 20% are rarely achieved, but excessive attrition can reduce study power, increase bias, and lower generalizability [22].

Challenges in enrolling participants have led researchers to look towards alternative methods for identifying potential participants [23,24]. Web-based strategies for attracting participants, such as the use of Internet advertising, email invites, craigslist, online message boards, and more recently, social media, have been explored by researchers in order to find general and specific populations for studies [24]. Some studies indicate that online strategies can be effective at reaching a larger number of potential participants, as well as improving affiliated costs [4,13,24]. Online recruitment can cast a broader net than traditional recruitment, extending to previously hard-to-reach populations, such as young adults, racial/ethnic minorities, and those with low education attainment [25,26]. In addition, Internet-based recruitment can be affordable and reach a wider, more diverse population, thus increasing generalizability [27,28]. However, there is lack of consistency in reporting these population variables in studies employing these methods [29].

In a recently completed study (National Cancer Institute Grant #R41CA162502; J Gordon, PI), our research team developed and evaluated a theory-based mHealth app to improve medication adherence and provide behavioral support for tobacco cessation. Our team used online and in-app methods of recruitment with varying degrees of success. The research team, currently conducting another study (National Cancer Institute Grant #R21CA174639; J Gordon, PI) to test a multi-behavioral smoking, diet, and physical activity mHealth intervention, sought to identify the most effective methods for recruiting and retaining research participants. Our first-hand experience, as well as the desire to formulate recommendations for other researchers, prompted an analysis of the literature on methods of attracting and retaining participants in mHealth studies.

The cost of the recruitment method used and participant retention rates were our primary variables of interest. Recruitment cost can be important with a limited budget in order to find the largest, most representative recruitment sample with the allotted funds. Participant retention rates were a primary focus due to our suspicion that the lesser amount of investment required from participants in virtual research negatively affects retention [4,21]. In addition, the broad reach achieved by Web-based recruitment might bring in people who are unfamiliar with the research environment and are unaware of the extent of the commitment they are making. This might later affect the number of participants who enroll and complete all interventions. This predicted higher attrition rate was evident in our own experience, as well as in the literature on Web-based and mHealth studies [4,21]. Participant demographics were included in the review to allow for comparison to “traditional” methods of recruitment. The “traditional” recruitment methods are defined as telephone, newspapers, radio, TV, flyers, or word-of-mouth. Other variables included were duration of recruitment and intervention because of their influence on participant retention and engagement, which are important factors in the generalizability and success of mHealth research [4,22].

A review of the literature published between 2004 and 2014 was conducted from August 2014 to October 2014, using the search terms “mHealth”, AND “social media”, AND “health behavior change”, AND “online recruitment”, OR “recruitment”, in the databases PubMed, MEDLINE, Cumulative Index of Nursing and Allied Health Literature (CINAHL), PsycINFO, and EbscoHost. Articles’ reference lists were also screened for relevant literature. Two searches of the literature were conducted, with the first using more stringent selection criteria. The goal of the initial search was to focus solely on mHealth research. The second search was conducted with more expansive selection criteria to include Web-based interventions due to minimal results from the first search.

The initial review was limited to articles that were published in English, had an abstract available, and reported on research about behavioral interventions to improve health. While certain demographic components such as age can affect recruitment results [23], there were no specifications for age or race of participants included in the review. The first author (TL) screened the abstracts and titles of relevant articles for eligibility. Only peer-reviewed papers were included. Mobile health articles that detailed their recruitment process in their methods section were selected (TL). These articles reported online recruitment methods and outcomes from current or past experimental studies, retention rates, and costs related to online recruitment. This initial search produced only one publication [29].

The first search highlighted the novelty of the mHealth research field. Therefore, an additional, broader search was conducted to see if any Web-based studies, in addition to mHealth studies, published articles detailing their recruitment processes and outcomes. This search utilized the same keywords and similar study selection criteria. The only change was expanding the criteria to include Web-based studies, as well. This second phase search gave the researchers a better understanding of what types of articles were being published about online recruitment and what types of information were being reported. Upon conclusion of the broader search, 11 peer-reviewed publications were included for the literature review. This created a total of 12 peer-reviewed articles [24-27,29-36].

The following data were collected from each article and recorded in an Excel spreadsheet: (1) authors of the publication, (2) year of publication, (3) article digital object identifier (DOI) number, (4) health domain, (5) study design, (6) number of participants enrolled in the study, (7) general demographics of participants, (8) method of recruitment, (9) total time duration of recruitment, (10) total cost of advertising, as well as cost per click if applicable, (11) percentage retention of participants, (12) study or intervention duration, and (13) most effective methods of recruiting, including cost and number of participants, found by the study.

The use of mHealth has tremendous potential for improving public health through its convenience, wide reach, and flexibility [14]. The Internet and mHealth apps are increasingly used by individuals who seek health information, thus increasing the potential to reach underserved populations [32]. However, there is a need to develop and evaluate mHealth apps in order to establish consistent, effective methods for producing health behavior change. Research on mHealth may experience challenges in participant recruitment and retention due to its very nature, which is characterized by the virtual aspect of the intervention [27]. Virtual interventions may lead to less investment and fraudulent enrollment on behalf of participants due to lack of relationship with the study team and potential incentives to participate [27]. Personal relationships may help participants to understand their contribution to the research [27]. Currently the literature on Web-based interventions is greater than that for mHealth apps, but there is a lack of detail in both about participant recruitment and even less information on retention. More attention must be paid to these two factors as they affect the validity and generalizability of the research findings [15].

For researchers conducting online and/or mHealth studies, online methods of recruitment have the potential to achieve better affordability than traditional methods of recruitment [27]. However, the results of this review found inconsistent findings related to cost-per-participant. This review also found conflicting outcomes regarding whether Web-based or traditional methods of recruitment were more effective at enrolling participants. It appears that the type of intervention and target population influenced which type of recruitment method was most successful.

Of the 12 studies examined, there was no clear consensus on which method (Web or traditional) is best to use when conducting mHealth research. The success of recruitment method varied widely based on population, budget, intervention, cost, and study design. “Best method”, as defined in these articles, was the one that achieved the highest participant yield for the cost incurred. Assessing largest participant yield can be achieved through analyzing overall impressions and click-through rates. Four articles (33%) found Facebook ads to be effective for recruitment when the intention was to achieve a sample within a specific age range. The use of Facebook ads for recruiting those under 25 may be due to the fact that Facebook offers the ability to target a specific demographic or age group by displaying ads only on profiles within that listed age range. Google ads appeared to be the most effective Web-based recruitment method when there was no specific age range targeted. Four studies (33%) compared Web-based methods against traditional methods of recruitment and reached different conclusions about highest participant yield and affordability.

There are a variety of factors that could lead to these results. First, paying for advertisements is different for each online method. For example, the bidding method that both Facebook and Google use does not allow for a stable cost prediction that could be replicated by the same researcher or even future researchers. This is due to the intermittent presence of other bidders. Based on project budgets and desired participant demographics, very different results could be achieved. In addition, the business models of Internet advertising (eg, Facebook and Google ads) are constantly changing, presenting ongoing challenges for determining affordable strategies relating to online recruitment. Also, virtual interventions and methods of recruitment allow for a certain degree of ambiguity and anonymity. This might allow participants to feel more comfortable enrolling in a study or could lead to less investment in the process.

Overall, studies are not doing a good job of reporting retention rates. Only one of the five studies that utilized a long-duration intervention reported participant retention rates. Retention issues can create bias and complications with generalizability [22], and if the rates are not reported, it is unclear if the results are valid. Consort diagrams are needed to describe the recruitment process and sources of attrition. Retention is a factor that not only should be addressed, but also reported with its potential effects in all articles related to online (eHealth) and mHealth interventions.

We recommend that researchers report recruitment methods, costs, participant demographics, and participant engagement metrics in their publications. Studies evaluating the effectiveness of interventions, especially mHealth interventions, should include this information in order to allow for future replication. A set of guidelines should be established for informing eHealth and mHealth researchers about recruiting generalizable samples. These guidelines might include, for example, a list of successful methods for targeting a specific participant demographic or approaches to maximizing a limited recruitment budget.

This review has two possible limitations. First, the limited number of studies required us to modify the selection process to more broadly examine the literature in order to meet the specific needs of the research team. Therefore, our inclusionary and exclusionary criteria changed over the course of the study. Second, the studies identified during the search used very different designs and methods. These discrepancies made it difficult to form recommendations as to which methods of recruitment are most beneficial and cost-effective in conducting mHealth research.

Further evaluation of online methods of recruitment is necessary to better understand their effectiveness for use in Web-based and mHealth research. Due to the unique challenges that Web-based and mHealth interventions face, there is a critical need for published articles focusing solely on recruitment methods, including participant recruitment and retention rates. Although it was outside the scope of this review, researchers in this area may also experience the possibility of fraudulent participants. Little is known about how the mHealth research process may encourage/discourage “fake” participants from enrolling in this type of study. There is also a need for guidelines and recommendations for affordable ways of recruiting and retaining representative samples of participants in Web-based and mHealth research. It is difficult to assess which methods of recruitment will work best, and there is not a “one-size-fits-all” approach, which should be included as a part of a set of guidelines. Currently, recommendations for guidelines cannot be made due to the inconsistent nature of reporting of online recruitment strategies. As the body of literature grows around mHealth, and as researchers begin to report specific recruitment methods, associated costs, and retention methods and rates, guidelines may be formed.

Research on mHealth apps and Web-based interventions may experience challenges with participant recruitment and retention. The research available regarding ideal participant yield and the associated costs of online recruitment methods for Web-based and mHealth studies is minimal and inconsistent. Researchers in this area should routinely report metrics regarding recruitment methods used and participant attrition. This includes detailing specifically what type of ads were used (eg, banner or search ads), as well as specific cost information (eg, cost per click, cost per participant, and overall costs). In addition, retention methods and retention rates should be included, as it is an important factor for mHealth and Internet-based studies. Information needed would include participant engagement in the intervention (eg, metrics associated with use of the intervention), and how many participants successfully completed the intervention and associated research activities. A set of guidelines for successful and affordable methods of recruitment, and metrics for evaluating mHealth engagement and participant retention are needed.