This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Clinical trials are the gold standard of evidence-based practice. Still many papers inadequately report methodology in randomized controlled trials (RCTs), particularly for mHealth interventions for people with serious mental health problems. To ensure robust enough evidence, it is important to understand which study phases are the most vulnerable in the field of mental health care.
We mapped the recruitment and the trial follow-up periods of participants to provide a picture of the dropout predictors from a mHealth-based trial. As an example, we used a mHealth-based multicenter RCT, titled “Mobile.Net,” targeted at people with serious mental health problems.
Recruitment and follow-up processes of the Mobile.Net trial were monitored and analyzed. Recruitment outcomes were recorded as screened, eligible, consent not asked, refused, and enrolled. Patient engagement was recorded as follow-up outcomes: (1) attrition during short message service (SMS) text message intervention and (2) attrition during the 12-month follow-up period. Multiple regression analysis was used to identify which demographic factors were related to recruitment and retention.
We recruited 1139 patients during a 15-month period. Of 11,530 people screened, 36.31% (n=4186) were eligible. This eligible group tended to be significantly younger (mean 39.2, SD 13.2 years,
Patient recruitment and engagement in the 12-month follow-up conducted with a postal survey were the most vulnerable phases in the SMS text message-based trial. People with serious mental health problems may need extra support during the recruitment process and in engaging them in SMS text message-based trials to ensure robust enough evidence for mental health care.
International Standard Randomized Controlled Trial Number (ISRCTN): 27704027; http://www.isrctn.com/ISRCTN27704027 (Archived by WebCite at http://www.webcitation.org/6oHcU2SFp)
Serious mental health problems are a major problem around the world [
Recently, interventions with mobile phone technology (mHealth) have been applied to randomized controlled trials (RCTs) [
On the other hand, conducting mHealth-based research among people with mental health problems includes challenges [
To better understand how mobile apps could be developed, evaluated, and implemented into routine care, it is important to truly understand which study phases make the RCT the most vulnerable in the field of mental health care. Still, many important parts of the study methodology are inadequately reported in RCTs, particularly regarding interventions targeting people with serious mental health problems [
Mobile.Net is a nationwide multicenter randomized controlled two-armed trial. The Mobile.Net trial evaluated the effects of tailored short message service (SMS) text messages constructed to encourage patient medication adherence and outpatient care for adult patients with psychosis [
There were a total of 1139 participants, men and women, ranging in age from 18 to 65 years. Each participant had a continuing prescription for antipsychotic medication, access to a mobile phone, and the ability to use the Finnish language. After participants were recruited, they were then randomized. Forensic patients and those having a planned nonacute treatment period were excluded from the study [
Recruitment, including activities conducted before and during participant enrollment [
Attrition, including actions after enrollment in the study [
Data for this paper were divided into two categories: data concerning patient recruitment and data relating to attrition (
1) Screened: (n)
2) Eligible: n (%)
3) Consent not asked: n (%)
4) Refused: n (%)
5) Enrolled: n (%)
Recruitment speed: n/day
1) During SMS intervention: dropout rate
2) During follow-up
Telephone interview: dropout rate
Participant’s notification: dropout rate
Postal survey: dropout rate
Register data retrieval: dropout rate
Recruitment data were categorized into five groups: (1) patients screened for eligibility, (2) eligible participants, (3) eligible participants whose consent was not requested, (4) participants who refused to participate at the point of contact in the psychiatric ward, and (5) those who consented to participate. Outcomes were recorded as screened, eligible, consent not asked, refused, and enrolled. In addition, to track the pace of recruitment, a record of all identified, screened, eligible, unwilling, and successfully recruited patients was kept using a specific monitoring sheet developed for the trial. Patient flow was monitored and recorded daily on the study wards. Daily progress of patient recruitment was reported as “recruitment pace” (ie, how many new patients were recruited e each day) [
Attrition data were categorized into two groups: (1) attrition during SMS text message intervention and (2) attrition during the 12-month follow-up period [
Descriptive statistics (frequency, percentage, mean, standard deviation) were used to describe participants’ demographic characteristics, recruitment, and attrition metrics (study participants lost in the follow-up). The demographic variables examined included age, gender, marital status (lives alone, ie, single, divorced, or widowed; lives with someone, ie, married), vocational education (none, vocational education), employment status (employed/self-employed, retired, student, job seeker), diagnosis (
Multiple logistic regression analysis was used to determine predictors of dropping out of the 12-month postal survey follow-up. Participants’ demographic characteristics, including age, gender, marital status, vocational education, and employment status, were chosen as predictors and added to the analysis [
A total of 11,530 patients admitted within psychiatric inpatient hospital wards were screened during the 15-month (453 days) recruitment period. There were 6565 who did not meet the eligibility criteria. A total of 779 patients dropped out before the eligibility assessment because they were transferred to another ward or rapidly discharged from hospital. Of the candidates who were screened, 36.31% (4186/11,530) appeared eligible.
Of the 4186 eligible patients, informed consent was asked from 3417 (81.63%). Informed consent was not asked in 18.37% (769/4186) of the cases because the person was quickly discharged from the ward, absconded from hospital, or the research nurses simply forgot to ask.
When age and gender of the screened noneligible and eligible patients were compared, it was found that the eligible patients were generally younger than the noneligible patients (
Demographic characteristics comparable across all stages.
Stage of study | N | Age (years) | Gender (male) | |||||||
Mean (SD) | Range | n/N (%) | χ21 | |||||||
11,530 | 41.1 (14.6) | 16-90 | –9.86 (3492) | <.001 | 6164/11,461 (53.78) | |||||
Noneligible | 6565 | 43.7 (16.1) | 16-90 | 3633/6514 (55.77) | 30.7 | <.001 | ||||
Eligible | 4186 | 39.2 (13.2) | 18-65 | 2103/4181 (50.30) | ||||||
3.23 (1842) | .001 | 0.3 | .59 | |||||||
Refused | 2278 | 40.2 (13.9) | 18-65 | 1142/2274 (50.22) | ||||||
Randomized | 1139 | 38.3 (12.5) | 18-65 | 560/1139 (49.17) | ||||||
–0.73 (561) | .47 | 7.2 | .009 | |||||||
Completers | 536 | 38.5 (12.7) | 18-65 | 261/536 (48.7) | ||||||
Dropouts | 27 | 40.3 (13.0) | 21-63 | 6/27 (22.2) | ||||||
–1.28 (36) | .21 | 10.1 | .002 | |||||||
Completers | 1088 | 38.3 (12.5) | 18-65 | 545/1088 (50.09) | ||||||
Withdrawals | 35 | 41.1 (12.6) | 18-63 | 8/35 (22.9) | ||||||
–8.14 (1120) | <.001 | 18.5 | <.001 | |||||||
Completers | 534 | 41.5 (12.6) | 18-65 | 227/534 (41.0) | ||||||
Dropouts | 589 | 35.5 (11.8) | 18-65 | 326/589 (59.0) |
Out of the 3417 eligible participants whose consent was asked, 2278 patients (66.67%) refused to participate in the study. Although reasons for refusal were not asked due to ethical guideline requirements [
The pace of recruitment was analyzed based on the number of new patients recruited each day. At the beginning of the study, recruitment was slow. The recruitment rate reached its peak 15 months after enrollment started. For every 10 screens completed, one person was successfully enrolled, at an average recruitment speed of 76 participants each month (2.5 participants per day).
Of the 1139 patients who were enrolled in the study, the data of 16 participants were excluded due to either the withdrawal of informed consent (n=10), the patient did not meet the inclusion criteria (n=5), or a recruitment error (n=1). This left us with a total of 1123 participants (intervention group: n=563; control group: n=560).
A total of 569 eligible participants were allocated to a group to receive tailored SMS text messages for 12 months. The data of six participants were excluded from the analyses due to either a lack of written informed consent (n=2), the patient did not meet the inclusion criteria (n=3), or an erroneous randomization to study group (n=1). This left us 563 participants.
Of the 563 participants who received text messages, 27 dropped out during the 12-month intervention period (4.8%). In cases where a patient did not want to continue with the text message intervention, the researchers were notified by the participant, a relative, or a research nurse. Three participants dropped out before the intervention even began, and 24 within the 12-month intervention period [
Information about participants who dropped out after the 12-month follow-up was divided into four categories based on the data collection method: (1) telephone interview, (2) participants’ notification (ie, withdrew from the follow-up survey), (3) postal survey, or (4) register data retrieval.
First, telephone interviews (for the intervention group only) were conducted after the 12-month text message intervention to explore participants’ feedback on the text message service (n=569). We attempted to reach 558 participants by telephone for an interview; after the telephone calls were made, we had 403 completed questionnaires (response rate 72.2%, 403/558) [
Second, 35 participants expressed that they wanted to withdraw from the follow-up surveys (intervention group: 5.5%, 31/563; control group: 0.7%, 4/560; χ21=21.4,
Third, a postal survey (n=1123) was conducted after the 12-month study period to measure participants’ quality of life (Q-LES-Q [
Fourth, register data retrieval was conducted after the 12-month follow-up period. Out of 1123 participants, the register data of four participants were not available from the Finnish National Care Register for Health Care [
Through a logistic regression analysis,
Associations between participants’ demographic characteristics and risk of dropping out of the postal survey (N=1123).
Demographic characteristics | OR (95% CI) | ||
0.96 (0.95-0.97) | <.001 | ||
Female | 1 | <.001 | |
Male | 1.63 (1.27-2.11) | ||
Lives with someone | 1 | .46 | |
Lives alone | 1.12 (0.83-1.50) | ||
Vocational education | 1 | .04 | |
None | 1.37 (1.01-1.84) | ||
Student | 1 | .001 | |
Employed/self-employed | 1.65 (1.01-2.70) | .045 | |
Retired | 2.29 (1.45-3.61) | <.001 | |
Job seeker | 2.44 (1.50-3.97) | <.001 |
The results of this study demonstrate that it was challenging to recruit, and engage, participants in an SMS text message-based trial follow-up. One-third of patients (36.31%) appeared eligible, and two-thirds of eligible patients (66.67%) refused to participate. Participants were well engaged with the SMS text message intervention provided, but their engagement with the trial follow-up varied: the highest being with the register data retrieval (99.64%) and lowest with the postal survey (47.55%). Participants’ demographic characteristics (age, gender, vocational education, and employment status) were seen as dropout predictors.
In our study, within the context of psychiatric inpatient care, we were able to recruit 1139 individuals (33.33%) out of 3417 eligible participants, whose consent was requested. Age and gender tended to be factors influencing recruitment and refusal. Our refusal rate was 66.67% (2278/3417), which is in line with previous studies suggesting that high refusal rates are a major problem faced during the recruitment process [
Lack of interest in the trial [
The attrition rate during the SMS text message intervention was low (4.8%). This finding does not support previous findings stating that low engagement and discontinuation are major problems in intervention studies [
Patient engagement in the trial during the follow-up varied depending on the source of data collected, the highest being in the register data retrieval (99.64%) and the lowest in the postal survey (47.55%). Low engagement and discontinuation have been found as fundamental problems in technology-based intervention studies [
Our study has some limitations. The recruitment data concerning information about screened patients lacked some information, especially about patients’ ages. Therefore, results related to patient demographics concerning recruitment have to be handled with caution. More importantly, we did not gather knowledge about the participants’ actual SMS text message use. Therefore, we lack knowledge about participants’ true engagement with the SMS text message intervention. However, according to our findings before the study actually started, participants were very satisfied with the intervention [
A key strength of this study was in its large nationwide sample of people treated with antipsychotic medication. Another was that, to the best of our knowledge, this was the largest trial evaluating a text message system. Our findings regarding attrition are important for those conducting similar RCTs among people with severe mental health problems, although this group may well have different issues with the technology when compared with others [
Initial patient recruitment and then engagement in the 12-month postal survey follow-up were the most vulnerable phases in the SMS text message-based trial. This may indicate that people with serious mental health problems may need extra support during the recruitment process, and necessitate further support to engage in completion of these follow-up questionnaires—at least within SMS text message trials.
Researchers should acknowledge the possible digital divide for people with serious mental health problems, and choose convenient and efficient data collection methods for study follow-ups. At follow-up, for Mobile.Net, high-grade routine data were almost complete. Methods of trials should take much more consideration of the nature of the target group of participants; otherwise, evidence is dogged with high attrition with the accompanying speculation of researchers. No statistical technique or learned speculation can make up for loss to follow-up. The solutions are likely to vary for different client groups. We think more research is needed both to investigate the support of the recruitment process and methods of follow-up in technology-based RCTs. Asking people to complete forms that are likely to result in grossly incomplete datasets could be considered an unethical—and potentially dangerous—waste of time and resources.
Client Satisfaction Questionnaire
Quality of Life Enjoyment and Satisfaction Questionnaire
randomized controlled trial
short message service
The authors would like to warmly thank all the institutions who awarded grants for Professor Välimäki: the Academy of Finland (132581), Turku University Hospital (EVO 13893), Satakunta Hospital District (EVO 12/2010, 81096), Foundations’ Professor Pool, the Finnish Cultural Foundation, and the University of Turku. Many thanks also go to the patients participating in the Mobile.Net study, research nurses, and the staff of the research organizations and wards, without whom the realization of this study would not have been possible. The authors especially wish to thank Kaisa Kauppi, PhD, Minna Anttila, PhD, and Sanna Suni, MA, for their valuable help with monitoring, data collection, and data entry. They also thank the Mobile.Net Safety Committee Group for their efforts and support throughout this study.
None declared.
Additional demographic characteristics comparable across latter stages of study.