CONSORT-EHEALTH: Improving and Standardizing Evaluation Reports of Web-based and Mobile Health Interventions

Background Web-based and mobile health interventions (also called “Internet interventions” or "eHealth/mHealth interventions") are tools or treatments, typically behaviorally based, that are operationalized and transformed for delivery via the Internet or mobile platforms. These include electronic tools for patients, informal caregivers, healthy consumers, and health care providers. The Consolidated Standards of Reporting Trials (CONSORT) statement was developed to improve the suboptimal reporting of randomized controlled trials (RCTs). While the CONSORT statement can be applied to provide broad guidance on how eHealth and mHealth trials should be reported, RCTs of web-based interventions pose very specific issues and challenges, in particular related to reporting sufficient details of the intervention to allow replication and theory-building. Objective To develop a checklist, dubbed CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth), as an extension of the CONSORT statement that provides guidance for authors of eHealth and mHealth interventions. Methods A literature review was conducted, followed by a survey among eHealth experts and a workshop. Results A checklist instrument was constructed as an extension of the CONSORT statement. The instrument has been adopted by the Journal of Medical Internet Research (JMIR) and authors of eHealth RCTs are required to submit an electronic checklist explaining how they addressed each subitem. Conclusions CONSORT-EHEALTH has the potential to improve reporting and provides a basis for evaluating the validity and applicability of eHealth trials. Subitems describing how the intervention should be reported can also be used for non-RCT evaluation reports. As part of the development process, an evaluation component is essential; therefore, feedback from authors will be solicited, and a before-after study will evaluate whether reporting has been improved.


Comment on subitem 2a-i)
In this sense it is very important that is becomes clear that developing a webbased interventions was goal-driven instead of tool-driven. I find often that studies focus too much on the technology, while the goal behind it should be the drive for developing the intervention. mention the traditional interventions for the particular patient population and the advantages of the new system A significant number of people with mental diseases remain unnoticed due to the incorrect identification of the symptomatology, the resistance to seek either help or information regarding these services, among ...

Comment on subitem 2a-ii)
Accessing to EHRs through mobile devices provides a number of advantages both for health centers and clinical staff, and for patients. Among these advantages are: accessing to patients' information in real time (from wherever and whenever), resource savings, improving the information management, and reducing the delay in health care. In the field of mental health, there are important epidemiological studies releasing relevant information about types and rates of more frequent disorders. However, a significant number of people with mental diseases remain unnoticed due to the incorrect identif ...

Add a subitem under CONSORT item 2a
Possibility of wider implementation of the

3a) Description of trial design (such as parallel, factorial) including allocation ratio (no EHEALTH-specific subitems under CONSORT item 3a)
I agree, this is often too complex to perform in ehealth trials That will be mentioned except for the allocation ratio because the allocation was coded randomized during the programming process (and normally it is not a rule to provide the algorithm) mere mention of the design and allocation ratio would be ideal.
No. need to mention whether there's follow-up, and for how long How will this guideline relate to extensions for non-inferiority, cluster, etc.?

-essential 24 44%
Comment on subitem 3b-i) Especially the reasons are important to elaborate on. Conventional trials may have 'down time' etc which I would expect to be reported if they were significant, so major breaks should be reported. However, major changes to functionality or content or design would be essetial to report (score 5) Authors should mention if new content was added to system during the trial. In my opinion this is often the case. Essential to https://docs.google.com/spreadsheet/gform?key=0Aq_l0aMh4cwrdGZscXJBMGVRZDZWVjRSaWl0QzFVSFE&hl=en_US&gridId=0#chart allow for future comparisons across studies and allow the possibility of being part of meta-analyses Such change events can be represented in a timeline fashion. I think that "u ...

Add a subitem under CONSORT item 3b
Brief description of website layout and complexity ("After a brief welcome screen, participants enter their weight and length and proceed to the food diary in the taskbar etc.") More specifics regarding the software used during the interventions and whatever was used for analysis. After completion of trials, there may be a "lag" time for researchers to prepare and publish the results of their study. It may happen that the version(s) of the solutions used at the time of study design and conduct, change during this "lag" time. The discussion on these kinds of "version" changes seems necessary. T ...

Comment on subitem 4a-ii)
An important point is that it may be extremely difficult or even impossible for researchers to have a complete overview of the entire recruitment process due to the fact that, at least, the Internet is a medium which is hard to control. One such https://docs.google.com/spreadsheet/gform?key=0Aq_l0aMh4cwrdGZscXJBMGVRZDZWVjRSaWl0QzFVSFE&hl=en_US&gridId=0#chart example which serves to identify many of the problems I mention is viral recruitment where a researcher uses his or others' social networks to recruit participants. The researcher can control his 1st degree contacts, but this is hardly possible for 2nd, 3rd... degree contacts. That is simply the nature of online viral recruitment. This is similar to ...

4a-iii) Information giving durnig recruitment
subitem not at all importantessential Comment on subitem 4b-i) Data collection instruments and methods should be described I would ask for clarification on 1) the assessment medium (online, mail, telephone interview, etc.) and 2) perspective (self-report, interviewer, etc.) If possible, comment on how successful the adaptation of an existing pen-and-paper to an online questionnaire was (e.g., visual analogue scale required with pen-and-paper test but was not available in online questionnaire software, etc.) Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential Details of pilot-testing of ...

4b-ii) Report how institutional affiliations are displayed
subitem not at all importantessential

Comment on subitem 4b-ii)
This is common to face-to-face interventions where universities / hospitals may be regarded as more trust worthy than drug companies. This is the issue of the credibility of website or mobile intervention. It is not unique to ehealth (a pharmacotherapy trial by Harvard may be perceived differently than one run out of a private practice). This seems to be adding a new criterion category. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses A very nice idea This is an interesting aspect, however, I don't think it's an essential item. W ...

Add a subitem under CONSORT item 4b
If not self-assessed, it is needed to describe (a) how participants were assessed and (b) whether this was problematic given the setting of a web-based intervention. Are online questionnaire metrics reported, and how were "unlikely" questionnaire metrics handled (e.g., were questionnaires that were answered unusually quickly included (indicative of participant inattention or laziness)) I am not sure if this is the right heading for this item.
However, to my opinion it is important to state how participants got to know about the intervention. How was it spread/promoted to potential users? Often ...

5)
The interventions for each group with sufficient details to allow replication, including how and when they were actually administered

5-i) Mention names, credential, affiliations of the developers, sponsors, and owners
subitem not at all importantessential

Comment on subitem 5-ii)
Essential unless these were published previously elsewhere. Including a logical/process model of the intervention, for example in a flow chart, would really enhance the publications. It may be necessary to add this in an appendix. There should preferrably be some guidelines on how to develop a logical/process model. This is very important to for designers, practitioners, and academics to learn more about the success/pitfalls of intervention design and development. Also describe how formative evaluations shaped the intervention. If these are not previously published, I'm not sure that saying " ...

Comment on subitem 5-iii)
This may be difficult, especially for "social" networking-based interventions with a lot of user generated content.
Such interventions will change continuously as a direct consequence of the use of these interventions by the participants -something uncontrolled by the researcher / project leader, and probably even difficult to track / disclose in a

5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screencapture video, and/or providing flowcharts of the algorithms used
subitem not at all importantessential Comment on subitem 5-v) I am a psychologist and Research Scientist for a small business. Publishing our source code is not something the owners would allow. Replicability should hinge on the theory guiding the development of the intervention and the hypothesized active ingredients. It should be up to the replicators to develop something comparable, sufficient detail should be provided to enable this, but not the original source code. Source code should be proprietary. In the name of science and research, I completely understand why we would want researchers to publish source code and algorithms. If it does not p ...

5-vi) Digital preservation
subitem not at all importantessential

Comment on subitem 5-vi)
As with publishing the source code, it is not always possible to digitally archive an intervention. Again, sufficient detail should be provided to enable readers to understand all the components of the intervention to replicate or build upon prior work. If this is a funded research project, the URL will be inactive by the time the manuscript is published. This digital preservation is not likely or even possible in many cases. It would be a "demo" only -which is not very helpful. see my comment above. Essential to allow for future comparisons across studies and allow the possibility of being ...

5-vii) Access
subitem not at all importantessential Comment on subitem 5-vii) "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes..)" This would be absolutely wonderful. Participant access needs to be described -essentail (score 5) This item seems to confound participant access and reviewer access. I think separating out the requirements of reporting about the intervention from the methods of disclosure would clarify. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential Any specific constraint in access should be discussed; fo ...

5-viii) Mode of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework
subitem not at all importantessential

5-x) Clarify the level of human involvement
subitem not at all importantessential Comment on subitem 5-x) Also, identify any manuals, procedures or principals used in guiding the human support. this might especially provide important information for implementation strategies of ehealth interventions. It might be that in the beginning more human involvement is needed. Essential to allow for future comparisons across studies and allow the possibility of being part of metaanalyses Essential High-risk alerts of participants that may harm themselves etc. and how these were dealt with (automated, human-intervention etc).
In the intervention section. idem Clear explanation is required, as this item can be s ... The content of the prompts is also essential. Prompts may be extremely reinforcing the use of the intervention. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Very important These ways of reminding have different effects in different contexts and for different interventions. How one can assure the re ...

Add a subitem under CONSORT item 6a
It might be that also some unexpected side effects could be detected.
researchers should acknowledge the value of this for the young ehealth research field. These should also be stated when encountered. -No.

6b) Any changes to trial outcomes after the trial commenced, with reasons (no EHEALTH-specific subitems under CONSORT item 6b)
It might be that also some unexpected side effects could be detected.
researchers should acknowledge the value of this for the young ehealth research field. These should also be stated when encountered. It could provide important clues for future trials what we could expect to be outcomes of an intervention.

10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions (no EHEALTH-specific subitems under CONSORT item 10)
Yes, this should be included It must be specified in the methodology section as to the participants were recruited to different group randomly through a programming protocol or any other protocol. see above 8 a No.

11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how
NPT: Whether or not administering co-interventions were blinded to group assignment

11a-i) Specify who was blinded, and who wasn't
subitem not at all importantessential Comment on subitem 11a-i) Yes. FYI -I've seen growing numbers of journals ask for the work "masking" since blinding is purportedly pejorative. I would also ask that if an assessment of the effectiveness of blinding was performed, that it be provided.
(patients often mention their tx assigment to evaluators -it is important to know if that was monitored and if so, how it was managed when it occured). Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential Any effects of "non-blinding" on the possible outcomes of the study, should be discussed or at le ...

11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator"
subitem not at all importantessential

Comment on subitem 11a-ii)
This is the same as a face-to-face trial. This is an important issue. But it is setting a higher level for ehealth than other areas. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential !!! Challenges on developing "sham" e-interventions, "placebo" e-intervention or the "comparator" e-intervention (if appropriate) should be discussed. Now it involves skills as to generate a program which wont let the participants guess. mention in the methodology section. This could be associated to a form of "blinding" Unlike medication, when t ...

Add a subitem under CONSORT item 11a
-No.

11b) If relevant, description of the similarity of interventions
(this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention)

(no EHEALTH-specific subitems under CONSORT item 11b)
Surely it is relevant -eg. were the intervention and comparitor of similar intensity? did they require the same

Add a subitem under CONSORT item 12a
In the case of clustered randomized trials and repeated measurements, reserachers should be required to conduct multilevel analysis. It is a too common statistical mistake to conduct single-level analysis in cases where there are two or more levels of data. For example, time-level data which are repeatedly measured and person-level data which are measured on one occasion (typically baseline) in a repeated measurements study design. Otherwise, one runs the risk of getting entirely wrong results and drawing wrong conclusions. Include statistical procedures used and why they were chosen -No.

12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses (no EHEALTH-specific subitems under CONSORT item 12b)
Yes, definitely If

Applicable.
No. Subgroup analysis should be carefully evaluated, specially when originated from post-hoc analysis. In general they should be avoided.

x26-ii) Outline informed consent procedures
subitem not at all importantessential

X26-iii) Safety and security procedures
subitem not at all importantessential

Add a subitem under item X26
Considering the fact that most ehealth trial collect participant data online, and different database systems / programming languages are available -all with their unique merits and possible security problems -it may be desirable to inform on which database systems / programming languages have been used (both for webbased interventions and webbased datacollection), and how the security measures for the data(bases) were implemented. -No.

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center (no EHEALTH-specific subitems under CONSORT item 13a) Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Any specific method for analysis should be mentioned. For example "intention to treat" etc. Via a Flow Chart or a pictorial representation. No.
Number of people *in each analysis*. This is frequently unclear and the number of people in each analysis is essential for calculating an effect for meta-analysis. However, this isn't unique to e-health.

13b) For each group, losses and exclusions after randomisation, together with reasons 13b-i) Attrition diagram
subitem not at all importantessential

Comment on subitem 13b-i)
An interesting contrast to drug trials -where self-report or pill count is all that is often done to evaluate 'adherence' with treatment. I would distinguish attrition from intervention from lost-to-follow-up from assessment.
Specify both drop-out and non-use attrition separately We did not have any way to collect reasons for discontinuing participation -as we had no contact with our participants other than email/online intervention. We had no IRB permission to request details of discontinuing participation. There are many cases in which reasons for attrition would be available. Essential to ...

Add a subitem under CONSORT item 13b
Describe whether there was differential attrition (i.e., a differences in attrition between the two groups) and how was dealt with this. It is covered above under methods, but I would have use data here. It is really important to report a clear metric for use, separate from attrition. Someone who logs into a site every day over 12 weeks has a different dose from someone who logs in once every week. And I suspect this will be important to systematic reviewers in the future. Details of why people dropped out would be helpful to determine what didn't work and how this may have impacted on the ...

-essential 20 36%
Comment on subitem 14a-i) Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Japan is a recent example. Experiment wise, in the methodology section. only month and year would be enough. it's a co-intervention related to 3bi and 5iii Good point. However, if these events have been described in the "Method" section, it is probably a redundant to mention it again in the "Results" section here. I can see how this is important, but I'm not sure it's essential.

Add a subitem under CONSORT item 14a
Would this include events like Hurricane Katrina or 9/11, in which cases people would rather watch television that use a web-based intervention. -No.
14b) Why the trial ended or was stopped (early) (no EHEALTH-specific subitems under CONSORT item 14b) this could also be the case in ehealth trials. So I would not omit this item from this checklist. Yes, this is would be important to include in case anyone else may want to try and replicate the study Reason should be quoted in the discussion. No.
EHEALTH-specific additions here: Secular events, political regime change in the instance of countries choosing to withdraw from the Internet.

15) A table showing baseline demographic and clinical characteristics for each group
NPT: When applicable, a description of care providers (case volume, qualification, expertise, etc.) and centers (volume) in each group

15-i) Report demographics associated with digital divide issues
subitem not at all importantessential

-essential 31 56%
Comment on subitem 16-i) While these multiple outcomes are of interest surely the primary measure for an RCT should be ITT? Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential ! Already outlined in the Flow chart. Separately mentioning these things would increase the length of the paper. Somethings should be indirectly inferred and interpreted from the tables and figures which should be briefly described in the discussion section. Good point. N eligible to be exposed to intervention should also be indicated. This is a good principle, but too vaguely ...

16-ii) Primary analysis should be intent-to-treat
subitem not at all importantessential Comment on subitem 16-ii) Important to distinguish between effectiveness analyses (i.e. analysis of the whole sample) and efficacy analyses (i.e. sub-group analyses) which in the latter case is no longer a randomized sample. This should be made explicit in all papers. However, the intent-to-treat principle has its advantages, but also disadvantages. It seems to me that the statistical and methodological scientific community is becoming more and more pro imputation techniques because of the disadvantages associated with the ITT principle. Thus, I don't think that the primary analysis should or must be intentto-treat ...

Add a subitem under CONSORT item 16
17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval) 17a-i) Presentation of process outcomes such as metrics of use and intensity of use https://docs.google.com/spreadsheet/gform?key=0Aq_l0aMh4cwrdGZscXJBMGVRZDZWVjRSaWl0QzFVSFE&hl=en_US&gridId=0#chart subitem not at all importantessential

19) All important harms or unintended effects in each group
(for specific guidance see CONSORT for harms)

19-i) Include privacy breaches, technical problems
subitem not at all importantessential

19-ii) Include qualitative feedback from participants or observations from staff/researchers
subitem not at all importantessential

Comment on subitem 19-ii)
It is time to slice the salami! A good quality qualitative paper is justified but a paragraph in the primary paper seems unlikely to adequately address the issues. With space limitations, this report would be limited in scope to 1 or 2 sentences. I do not think it should be required. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Hmm . . . This really depends on the extent to which the qual data affected analysis and interpretation. In order to improve the website and make it more user friendly. It is very important as descriptiv ...

Add a subitem under CONSORT item 19
Maybe some guidelines about how to quantify qualitative feedback to make it more useful to others who may want to replicate the study -No.

22) Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence
NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group

22-i) Restate study questions and summarize the answers suggested by the data [2], starting with primary outcomes and process outcomes (use)
subitem not at all importantessential Comment on subitem 22-ii) Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses Essential I think mentioning the reproducibility of the intervention in other platforms is an appropriate suggestion for future research. This will provide direction and the the researcher's wisdom and perspective to the potential new researcher. Good point.
Is this not redundant? This doesn't add much to the current version of CONSORT.

Add a subitem under CONSORT item 22
-It would be very useful if researchers could emphasize and elaborate on their lessons learned in the "design" and "implementation" of the intervention based on both primary and process outcome findings as well as unintended outcome findings. This information would be not only important for readers to see the actual picture of the intervention effects but also very beneficial to future researchers and developers so that the same lessons won't need to be learned again.

20-i) Typical limitations in ehealth trials
subitem not at all importantessential Comment on subitem X27-i) As per the ISRII discussion -this is generally not explicity stated in face-to-face trials where it is a similar problem. Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses As applicable. Vary from journal to journal.
Good point. Absolutely right.

Add a subitem under item X27
Maybe some more detail about what is meant by "system being evaluated." A list of key words would be good for purposes of future online database searching. -No.

Last question
Do you want to become involved in the writing committee working on the elaboration document? If yes, please provide the subitems you wish to elaborate on 5-ii -developing a framework for a logical/process model 6a-i, 7a-i, 12a-i, 16-ii, X26 use: 5-ix, 5-xi, 6a-ii; differential attrition: 12a, 13b Possibly 4a-ii, 5viii, 5x, insertion of addition item in ...