Published on 06.08.19 in Vol 21, No 8 (2019): August
A Multicomponent eHealth Intervention for Family Carers for People Affected by Psychosis: A Coproduced Design and Build Study
Background: Psychosis, including schizophrenia, is the most common severe mental illness affecting 1% of the population worldwide. A large number of people provide long-term support and care for a relative with psychosis. Although psychoeducational interventions, especially those delivered through a face-to-face group format, have an established evidence base for improving the caregiving experience, well-being, and health outcomes, large-scale implementation and access remain limited. There is a demand for such provision to be made through the internet for greater flexibility and wider access.
Objective: This study aimed to integrate participatory research methodologies by the public, patients, and carers into the eHealth (electronic health) intervention design and build process to improve the product’s usability and acceptability.
Methods: We adapted a structured eHealth intervention build method to include participatory research activities involving key stakeholders and end users to co-design and coproduce our intervention. An expert advisory group (EAG) comprising public involvement members led the formative design and build work using an agile build process. Carers independent from the study were consulted on the evolving drafts of the intervention prototype through focus group meetings. These results were fed back into the intervention build work continuously to ensure end users’ input inform every stage of the process.
Results: An EAG comprising individuals with lived experience of psychosis, carers, health care professionals, researchers, voluntary organization workers, and eLearning experts (n=14) was established. A total of 4 coproduction workshops were held over 1 year during which the alpha and beta prototypes were designed and built through the participatory research work. Alongside this, 2 rounds of focus group study with carers (n=24, in 4 groups) were conducted to seek consultation on end users’ views and ideas to optimize the intervention design and usability. Finally, the EAG carried out a Web-based walk-through exercise on the intervention prototype and further refined it to make it ready for an online usability test. The final product contains multiple sections providing information on psychosis and related caregiving topics and interactive discussion forums with experts and peers for psychosocial support. It provides psychoeducation and psychosocial support for carers through the internet, promoting flexible access and individualized choices of information and support.
Conclusions: The participatory research work led to the coproduction of a eHealth intervention called COPe-support (Carers fOr People with Psychosis e-support). We believe the study methodology, results, and output have optimized the intervention design and usability, fitting the end users’ needs and usage pattern. COPe-support is currently being tested for its effectiveness in promoting carers’ health outcome through an online randomized controlled trial.
Trial Registration: ISRCTN Registry ISRCTN89563420; http://www.isrctn.com/ISRCTN89563420
J Med Internet Res 2019;21(8):e14374
With ever-advancing health care technologies and growing longevity worldwide, a significant proportion of people provide substantial and sustained help and support to friends or family members suffering from a long-term illness . In the United Kingdom and the United States, nearly one-fourth of the adult population identifies itself as a carer for a loved one who is ill, disabled, or elderly [ , ]. Many of these carers support a loved one affected by a severe and long-term mental illness such as psychosis [ , ]. Family caregiving often covers a huge amount of care and support ranging from emotional and psychosocial support (eg, engaging their loved one in social activities and sharing ups and downs) to financial and practical support (eg, provision of financial and practical help and monitoring of health and treatment compliance). Compared with paid or professional care workers, family carers also have unique advantages in knowing the individual’s strengths and interests in addition to their needs. Most have a well-established emotional bond and are committed to use all these preexisting knowledge and relationships to support their loved one in their recovery [ - ]. Caregiving imparts paramount emotional and psychosocial benefits to the cared-for individuals such that people in receipt of support from their family or social network have better prognosis, fewer relapses, and higher quality of life compared with those without such support [ - ]. Collectively, caregiving by family members amounts to significant economic savings to the wider society [ , ].
Conversely, it is well established that caring demands can jeopardize carers’ well-being [, , ]. For instance, population-level research data repeatedly show that carers experience higher levels of distress and poorer well-being when compared with age-matched counterparts in the general population [ , , ]. Indeed, distress in carers frequently reaches clinical thresholds, and their psychiatric symptom scores (eg, depression and anxiety) are found to be consistently inversely associated with the amount of care they provide, that is, carers’ mental health worsens with increasing demands on caregiving [ , , ]. Research evidence also suggests that poor well-being may hamper carers’ caregiving capacity. Carers who feel they are not supported and lack resources (eg, information about the illness condition and related management issues) to cope are less likely to engage in caring for their loved ones or more likely to exhibit critical or hostile behavior toward the cared-for individuals, albeit unintentionally [ , , ]. This, in turn, can impact negatively on both the carers and the cared-for individuals, leading to a vicious cycle of poor health and quality of life for all concerned.
Psychosocial Interventions Targeting Carers
Consequently, a body of research has been undertaken to explore interventions, which can best support carers [, , ]. Among a range of psychosocial interventions targeting whole families and/or carers alongside the individual treatment regime, psychoeducation (ie, information giving on the illness condition and related caregiving and problem-solving strategies) has the strongest evidence base for its effectiveness in enhancing carers’ knowledge and coping with their caring roles [ , ]. Commonly based on the stress-appraisal-coping theory as applied in family caregiving [ , - ], it has been hypothesized that psychoeducation, with education as its core feature and prime aim, works directly to improve carers’ knowledge about psychosis and related caregiving issues. Improved knowledge about coping strategies and available resources can lead to a more positive appraisal of their caregiving experiences and carers’ perceived self-efficacy in coping with the demands [ , ]. These, in turn, can translate into a more supportive home environment for all and better prognosis and reduced relapses in the cared-for individuals. In addition to information and advice on psychosis and related caregiving strategies, carers also identify that sharing mutual support and learning with other carers (ie, peers) as particularly useful in reducing their sense of isolation [ , ]. Consequently, psychoeducational interventions, especially those delivered in a group format, as a discrete treatment on their own or augmenting other treatments (eg, family intervention or mutual support programs) are widely recommended and practiced around the world [ , , ].
Carer-Specific eHealth Interventions
Carers have expressed their desire for interventions to support them to be delivered to them through a digital medium to fit with their caregiving and other commitments [- ]. The internet offers the potential to deliver interventions, which are highly flexible, accessible, and yet adaptable to individualized needs and schedule. With the popularity of eHealth (electronic health; ie, health care practice delivered through the internet) and mobile Health (ie, through the mobile network) interventions targeting a wide range of common public mental health issues (eg, insomnia and stress management) growing fast, internet-based interventions targeting carers have gathered momentum and popularity in the recent decade [ , , ]. eHealth interventions using an enriched online environment can integrate multiple components, especially educational and therapeutic information and network support with health care professionals and peers, and deliver such provisions to a critical mass of carers [ , ]. Carers, as the end users, particularly appreciate the autonomy that eHealth interventions offer, as they can decide which components or strategies resonate with them, how much time to spend accessing the intervention, and when to do so [ , ]. Systematic reviews on interventions targeting carers identify that eHealth interventions are particularly widespread in the field of dementia, stroke, or cardiovascular diseases. More importantly, research evidence to date shows promising effectiveness results in improving carers’ health outcomes [ , ]. Furthermore, these reviews suggest that effective Web-based interventions commonly have 3 essential ingredients. These include (1) multiple components such as information and well-being promotion strategies (including mindfulness and cognitive behavioral–oriented exercises), (2) psychoeducational content to enhance understanding of the illness condition and related caregiving issues, and (3) flexibility to self-pace and self-tailor the intervention to individual needs [ , , ].
Carers for People With Psychosis
Notwithstanding the evidence base for eHealth interventions targeting carers and their expressed desire for such provision, development in the field of psychosis appears to be lagging with few empirical studies documented to date. Of these studies, 2 were conducted in the United States nearly a decade ago. Rotondi et al developed and evaluated an online psychoeducational intervention with a peer forum for individuals with schizophrenia and their carers , and Glynn et al developed an online multifamily group program for carers of people with schizophrenia providing synchronous and asynchronous group sessions [ ]. More recently, 1 study was conducted in the United Kingdom, trialing a fully Web-based psychoeducation and peer-support intervention for siblings of individuals who developed first episode psychosis (The E Sibling Project) [ ]. A further psychoeducational intervention using an enriched online environment was developed and tested for acceptability in Hong Kong, China, targeting carers of people with psychosis [ ]. All these interventions were highly valued by the carers, albeit in either a pilot trial or usability study with a relatively small sample size (n=21, 26, 20, and 81, respectively) [ - ]. No further definitive studies on any of the aforementioned or other eHealth interventions targeting carers for those with psychosis are available to date (of note, a peer-facilitated online educational intervention study conducted in the United Kingdom is yet to be published [ ]).
Coproduction of eHealth Interventions
As eHealth interventions are for autonomous use by the end users, development methodologies commonly integrate users into the process. In the agile methodology commonly used in constructing software, iterative work sequences of technological expert–led sprints of developing and delivering working software are based on a brief commissioned by the clients. Through the cycles of set sprints, the developing software is shown to the clients or consumers for feedback, which is used to tune and adjust the software until its completion [, ]. Although the agile method has been commonplace in the software development field since the 1970s, more recently, the use of participative research methods, which involve key stakeholders and end users at the core of the intervention development process, rather than brokering out the build work with a commission brief, have gained popularity [ - ].
Within mental health research, there is an established tradition of service user–led and survivor-led research, with survivor researchers consistently arguing that, for example, the closeness of the researcher to the enquiry increases the validity of study findings . Although such an approach might help ensure that the build process responds closely to the experiences of end users in this study, the literature on, specifically, carer-led research in mental health is extremely limited and seemingly absent in the field of eHealth [ ]. In contrast, there is a growing literature documenting participatory design methodologies, including, but not limited to, involving patients, carers, and the public as end users in the core of the research team and research activities, to inform the design and development of interventions [ , , ]. Contrary to conventional interventions delivered through a face-to-face medium, where the intervention development and delivery are often driven solely by the professionals, consumers of eHealth interventions take on a much more active role [ , ]. In most eHealth interventions, including those that provide guided support from a professional or health care provider, the end users take on responsibility in initiating contact and engaging with the intervention, working through the content, and undertaking self-reported outcome measures online [ , ]. These make it paramount that the end users are involved actively in designing the intervention, not only to make sure the content meets their needs but also to ensure that the way the intervention is delivered keeps them engaged.
Overall, 2 recent systematic reviews on the topic have repeatedly identified the use of participatory research methods in eHealth intervention development as the 1 key factor in determining the acceptability and usability of most eHealth (especially electronic mental health [e-mental health]) interventions [, ]. Among publicly funded health studies conducted in the United Kingdom, there is evidence to suggest that involving patients and the public in participatory research is positively linked to study success, in terms of recruitment and retention rates [ , ]. It can be argued that the significance of participatory research could only be amplified when it is applied in the eHealth arena where the end users assume much more direct control in using and adhering to the interventions.
The E-Support for Families and Friends of Individuals Affected by Psychosis Project
The E-support for Families and Friends of Individuals affected by Psychosis (EFFIP) Project was set up to develop and evaluate an eHealth intervention for carers supporting a relative with psychosis . The overall EFFIP project lasts for 5 years spanning across the theoretical development work to the effectiveness evaluation of the end product on improving carers’ health outcomes based on the Medical Research Council Complex Interventions Framework [ , ]. The design of the overall project is illustrated in , focusing on this study.
This study reports the intervention building/modeling phase of the overall EFFIP project (see) [ ]. Before this phase, we conducted 3 studies in the theoretical development phase. These included 2 systematic reviews and a focus group study exploring research evidence and individuals with psychosis and carers’ ideas and views for the optimal intervention design including essential ingredients, contact hours, and facilitation considerations ([ , ]; also JS et al, unpublished data, 2019). In the build and modeling phase of the study, reported here, we aimed to integrate participatory research with end users alongside core research team activities to coproduce an eHealth intervention prototype to improve flexible access to high-quality psychoeducation and interactive support resources for carers [ , ]. We as a team decided to take this approach, rather than having carers lead the study, in part because of resource constraint and also because we considered that it was important that a range of consumer voices were included across all design decisions. Specific objectives of this study were to apply participatory methods with end users involvement to design and build the intervention, integrate iterative consultations with end users along the rapid prototyping work, report and consider the appropriateness of a participatory approach to eHealth intervention design, and test and refine the final draft of the intervention to get it ready for the usability evaluation, which is reported separately (JS et al, unpublished data, 2019).
The co-design and build process of the eHealth intervention followed the UK National Institute for Health Research online resource development cycle . This build method was chosen as we were to develop and deliver an eHealth intervention through an existing platform, rather than developing software or the platform ourselves [ , ]. There are 5 build steps illustrated in . These are as follows: (1) draft architecture and content, (2) mockup of shell, (3) fill shell with material, (4) cross-inference and integrate materials, and (5) prepare for piloting.
Coproduction Workshops With an Expert Advisory Group
The co-design and build work were directed by an expert advisory group (EAG) comprising individuals and carers with lived experience of psychosis and professionals working in health, social, or voluntary sectors with the target population. The EAG membership was devised to address the principles of participatory research that value contributions from expertise through experience, and that should never be underrepresented within the bigger whole-team context [, ]. We recruited EAG members with diverse demographic characteristics and varying degrees of ease and familiarity with digital communications from various clinical and voluntary service provider organizations across South East England. EAG members were paid a goodwill payment for meeting attendance and contributions and were also reimbursed for their travel expenses according to the INVOLVE payment scheme, a UK powerhouse that promotes patient, carers, and public involvement (PPI) in research [ ].
The EAG comprised 3 individuals with lived experience of psychosis, 3 family members having different relationships with a loved one affected by psychosis in the family (including a cousin, a sister, and a daughter), 1 clinician who also has personal experience of family caregiving, and 1 voluntary service lead. The EAG worked with the core research team (comprising 2 clinical academics, 1 health services researcher, 2 eLearning experts, and 1 administrator) to lead the build work.
Over the build phase lasting 12 months, 4 EAG workshops (1-4) were held, equivalent to the sprint cycles used in the agile approach. We used the coproduction in mental health improvement work method to organize the workshops [, , ]. This approach emphasizes and values the hidden capacity and capability of end users, families, and significant others. At each workshop, all members assumed equal decision-making role, whereas we contributed with our respective strengths and expertise [ , ]. These participatory design workshops (and follow-up work) were mapped to fit the 5-step development process of eHealth products [ ], and hence, each had specific aims and expected output, progressing from generating ideas and design to reviewing and refining product from rapid prototyping cycles.
After each participatory design workshop, the knowledge and ideas generated were translated to produce draft hand-sketched plans and wireframes, mockups of Web pages, and source materials for the intervention (eg, videos or textual information). The mockups and output produced between workshops and end users’ feedback obtained from consultations (see below) were then presented at the next workshop, enabling content and broad design ideas to be critically discussed and reviewed and then further developed. Toward the end of the co-design and build work (build steps 4 and 5), the EAG undertook final development work before conducting a walk-through exercise of the online beta-build of the intervention [- ]. Final revision and refinement work were undertaken in build step 5 to produce the intervention prototype (ie, the delta-build in ) ready for the real-world usability test.
Iterative Consultations With Carers
At the interfaces of build steps 3 and 4, we incorporated 2 rounds of iterative consultations with target end users . For the consultations, we recruited carers who had no prior involvement with the EFFIP project to a focus group meeting lasting up to 2 hours. During the first consultation, we showed the carers the offline first alpha-build of the intervention and asked for their feedback on its design and likeability, the flow and readability of the content, and invited them to preempt potential usability issues and identify ways to promote end users’ engagement. The second consultation followed a similar format with another set of carers, with an online second alpha-build of the intervention, which had been further developed by the EAG. The carers’ views were fed back to the rapid prototyping cycles as overseen by the EAG. illustrates the inputs made by the focus groups.
The consultation was approved by the UK National Health Service Research Ethics Committee (REC) process (REC approval reference number: 16/LO/1300) and Health Research Authority (HRA IRAS project ID: 210571). Carers aged 18 years or older and who provided unpaid support and care for a loved one affected by psychosis were recruited from 3 mental health trusts in South London and Berkshire, South East England. Recruitment strategies for carers included the posting of study flyers at clinical areas and informing the carers support workers and mental health professionals at each trust to disseminate the study information to carers.
All the workshops and consultation meetings were digitally recorded and transcribed verbatim. We took photos of the sketches produced at the workshops, on which the wireframes were developed later. The qualitative data were analyzed using the thematic analysis method [, ], suiting the social constructionist and realist paradigms. We used an inductive approach to identify themes from the data without trying to fit it into a preexisting coding frame but grounded our understanding on the EAG members’ and carers’ perception and experiences of the evolving drafts of the intervention. As the study focused on designing and reviewing the build of the intervention, the data were analyzed descriptively to generate themes concerned broadly with the design and content of the intervention, general look and feel, usability factors, privacy and security, and ways to enhance engagement and usefulness.
Throughout the study, to ensure the rapid prototyping and build of the intervention was grounded in the data, the participatory design workshops, focus group consultation, and data analysis were performed in parallel with one another. This ensured timely and robust feedback to inform the evolving drafts of the intervention (see).
Expert Advisory Group Members and Consultation Group Participants
The EAG comprised 8 members, whereas the core team had 6 members. Over 1 year during which 4 participatory workshops were held, all EAG members and core team members attended all the workshops and selected follow-up work to create the materials as indicated.
During each round of consultations, 2 meetings involving 5 to 8 carers were held (seeand ). In total, 24 carers participated in the iterative consultations along the evolving build process: 10 on the first alpha-build, and 14 on the second alpha version. The participants comprised 10 men and 14 women. The range of the carers’ ages was 22 to 83 years (mean 59 [standard deviation, SD 12.7]) with the median age being 61 years. Half of the carers had retired (n=9) or stayed at home being a full-time carer (n=3). The other half were in gainful employment: 5 working full time, 6 working part time, and 1 actively seeking employment. Most of the carers were a parent (19/24, 79%), and there were 2 spouses (8%, 2/24), 2 close friends (8%, 2/24), and 1 adult child (4%, 1/24). Just more than one-third of the carers (38%, 9/24) lived with their cared-for person at the time of the study. The gender mix of the cared-for persons was similar to that of the carers; 11 were male (46%, 11/24), and 13 were female (54%, 13/24). The ages of the cared-for persons ranged from 17 to 61 years (mean 35.3 [SD 15.8], median=32). In terms of diagnosis, half of the participants reported that their cared-for persons (50%, 12/24) had a diagnosis of psychosis, 9 were diagnosed with a schizophreniform disorder (45%, 9/24), and 3 had type 1 bipolar disorder (13%, 3/24). As reported by the participants, the cared-for persons had been unwell for less than 1 year to the longest for 36 years (mean 10.6 [SD=10]; median=6). The demographic characteristics and caring situation of the participants and their cared-for persons are summarized in .
|Characteristics||Carer participants (n=24)||Their cared-for person (n=24)|
|Mean (SDa)||59 (12.7)||35.3 (15.8)|
|Median (range)||61 (22-83)||32 (17-61)|
|Male, n (%)||10 (41)||11 (46)|
|Ethnicity, n (%)|
|Work, n (%)|
|Full-time work||5 (21)||—|
|Part-time work||6 (25)||—|
|Not working||1 (4)||—|
|Looking after home/family||3 (13)||—|
|Marital status, n (%)|
|Relationship with the cared-for person, n (%)|
|Close friends||2 (8)||—|
|Accommodation arrangement of carer, n (%)|
|Living with cared-for person||9 (37)||—|
|Not living with cared-for person||15 (63)||—|
|Diagnosis of cared-for person, n (%)|
|Schizophreniform disorder||—||9 (37)|
|Type 1 bipolar disorder||—||3 (13)|
aSD: standard deviation.
Expert Advisory Group Coproduction Workshops and Iterative Consultations
Expert Advisory Group Workshop 1
The inaugural workshop was set to design the study website and the online intervention, including their (domain) name, content, and look. The EAG was provided with results obtained from prior studies conducted in the theoretical development phase of the EFFIP project ([, ]; also JS et al, unpublished data, 2019) to aid their design decision.
Following an extended brainstorming exercise, we decided to call the eHealth intervention COPe-support, an acronym of Carers fOr People with Psychosis e-support. We subsequently secured the domain name of cope-support.org and hosted our project website on the World Wide Web. With carers as end users in mind, we set out to design a clean and user-friendly website giving information about the project and its research program. The EAG believed that the testimonials from other carers who helped develop the intervention would add credibility to the intervention; hence, these testimonials and photos of the EAG working together were added to the website (see).
The EAG reviewed the essential ingredients as identified by our earlier development work (JS et al, unpublished data, 2019) to produce the draft architecture and mockup of shell (build step 1). We used nondigital design tools such as magic sheets, post-it notes, index-cards with content items written on them, papers, and pens to encourage a creative design atmosphere involving all members, regardless of their competency level of information and communication technology. EAG members did hand-drawn sketches of Web pages and organized the content and ingredients in a structure that they saw fit.
The EAG decided the master plan for the content and key functions of COPe-support. The intervention comprised 10 sections of psychoeducational materials, communication and problem-solving knowledge and skills, reflective exercises, and discussion points. There were 2 interactive discussion forums: one with a panel of expert members comprising professionals and experts through experience and another one with carers as peers.provides samples of early designs and sketches regarding the project website and the intervention.
Expert Advisory Group Workshop 2
The second workshop was held 6 weeks later. For hosting and running COPe-support, our eLearning experts (LW and AS) identified a virtual learning environment (VLE) called Canvas . The COPe-support intervention, accessible via the Canvas VLE, was designed to work on desktop (or laptop) Web browsers, as well as smartphones or tablets through a Canvas app.
Building on the framework and functions of Canvas, the EAG members worked together to build shell with materials at build step 2 . We structured all the content items onto 1 platform hosting 12 sections. These include the following:
- Two modules on psychosis, common symptoms, and comorbid problems and evidence-based treatment for psychosis;
- Two modules on caring strategies for common symptoms and problems (eg, supporting your loved one with paranoid beliefs) and on ways to promote recovery;
- Two modules on wider social and service issues related to psychosis including ways to deal with stigma and discrimination and navigating the health and social care systems;
- Two modules focusing on well-being–promotion strategies for carers themselves;
- A virtual discussion forum and blog space for carers to share experiences and discuss commonly encountered issues;
- An Ask the Experts forum where participants can post questions to an expert panel comprising health and social care professionals and campaigners;
- A Further Resources section with supplementary Web links to relevant external resources; and
- A support page where carers can get in contact with the online facilitator for technical or emotional support directly.
During this workshop, the EAG members also discussed the best ways to present the different elements and topics. We devised a detailed work plan to source or produce the materials in various formats, ranging from textual documents to making videos with experts speaking on the specific topics and Weblinks to external sites (with explicit agreement sought).
The following 2 months after these workshops saw the EAG producing the materials and the sketch design of the intervention being turned into wireframes with materials developed filling the shell—the build step 3 . provides screenshots of the first alpha build of COPe-support.
Consultation With End Users on the First Alpha-Build
Once the first alpha-build of COPe-support (including the home page where the intervention menu and content list sat and mockups of 3 modules and the 2 forums) was prepared, we ran the first consultation showing the carers its offline version to seek their feedback.
Feedback from carers indicated that the general presentation and the content of COPe-support were well received. However, carers found the linear program of sections and elements too prescriptive (see), whereas they envisaged that carers as end users would prefer flexibility in choosing relevant content suiting their own caring situation. Although carers rated positively the differing elements including psychoeducational information, reflective exercises, and practice guides, which encourage the integration of skills learnt into their day-to-day life, they found the default terminology used by the Canvas VLE alienating. Some examples included the terms quizzes and grades. The findings obtained from the focus groups were fed back to the rapid prototyping cycle overseen by the EAG.
Expert Advisory Group Workshop 3
Although the rapid prototyping build work progressed onto build step 4—cross-inferencing and integrating materials, the consultation findings on the initial alpha-build of COPe-support were reviewed by the EAG at this workshop. The original linear program menu design was changed to a grid-based visualization, which implied no order for its content yet still provided the functionality to link to the relevant areas. The 3 key elements, that is, psychoeducational information, interactive forums, and further resources, were color-coded in the content menu. Explicit guidance notes were added to encourage participants to pick and choose the content relevant to their needs. We changed the terminology used by the Canvas VLE, which was originally designed for eLearning students, as much as possible to suit the carer population. These included labeling all the exercises and prompts for reflection and integration as reflection and leaving out submission or grading and sharing of reflections through peer-to-peer forum as some participants might perceive that as a source of unintended pressure. In both forums, we added ground rules and notes to explain the privacy and security measures and provided examples in writing posts or questions without giving personally identifiable data away. We also developed further sections using these design principles in producing the second alpha-build of COPe-support.provides screenshots of the second alpha-build of the intervention.
Consultation With End Users on the Second Alpha-Build
A second round of focus group consultation was subsequently organized when we showed the carers an online version of the second alpha-build of COPe-support (see). Carers’ feedback for this was positive in general. They found the home page welcoming and particularly appreciated the content menu design, which they found inviting and intuitive to use. The explicit display of the ground rules and guidance notes to ascertain privacy and security in the home page and again in forums was identified by carers as enhancing their sense of trust and ease of use. The layout of pages focusing on psychoeducational information given was well evaluated with the use of pictures and graphics to intercept the text. We asked carers to review the 2 forums and the discussion topics during the consultation. Most found that the topics gave some structures for the participants to focus their posts and felt confident and comfortable to use the forums with the guidance notes and the anonymous participation arrangement. Carers identified that there was a need for clearer instruction in how the participants can contact the COPe-support facilitator directly for support if necessary. They liked the cognitive behavioral–oriented reflections and exercises that promote self-care but would like more instruction on downloading the materials to allow practicing in their own time.
Expert Advisory Group Workshop 4
In the 2 months between the last consultation and the fourth coproduction workshop, further rapid prototyping work was carried out to address the focus group findings.
Workshop 4 concluded the build step 4— cross-inference and integrating materials —of the coproduction work and preceded the final build step 5— ready for usability test. During this workshop, the EAG members conducted a walk-through exercise as a group on the beta-build of COPe-support [- ]. Through the walk-through exercise starting from login to log-off, the EAG members navigated through every section and tried out every function of COPe-support from an end user’s perspective. A number of minor usability issues related to navigation and use of various functions were identified (eg, how to raise a new post?). We took note of these findings and carried out further refinement accordingly. We also captured the EAG members’ experience and feedback expressed through the exercise to inform the production of a navigation video (and a written navigation guide in parallel).
We discussed and confirmed the privacy and data security measures based on the exercise and the data output from it. In addition to the anonymous participation mechanism through the use of pseudonyms by all enrolled carers, the EAG considered it useful to state explicitly the monitoring, moderation, and facilitation provided by a qualified mental health professional (JS) daily during the week. We believed that it helped create and maintain a safe online environment as well as enhance the credibility of the intervention. The facilitator also posts weekly updates on COPe-support online forum to all carers with an aim to keep them engaged.
These considerations and final refinement work were undertaken in the final build step 5— preparing for usability test —with the delta-build of COPe-support developed by the end of the build and modeling phase.
Our study used an innovative approach to develop a Web-based intervention for psychosis carers. COPe-support provides psychoeducation and emotional support using health care professional contribution and peer support . The participatory research design method ensured that carers as end users were fully involved in all phases of the design and build process [ , , , ]. We believe that carers, together with other key stakeholders such as individuals with lived experience of psychosis and health care professionals, provided some insightful foresight of target end users’ expectations and usage pattern for the intervention. These were addressed and taken on board in the build process to optimize the matching of end users’ expectations and needs and the final product design and delivery. Furthermore, these end users’ inputs helped the core team, especially the eLearning experts, to understand how best to manipulate the technological remits as afforded by our chosen software to enhance the relevance and suitability of the intervention for the end users ([ , ]; also JS et al, unpublished data, 2019).
Our study combined 2 distinct participatory research coproduction methodologies with end users—the participatory research workshops to design and build the intervention and consultation meetings to provide feedback on drafts along the rapid prototyping cycles—in parallel and interacting with one another [, , ]. Furthermore, 2 different groups of end users and stakeholders were involved in the different methods and activities (carers with no involvement with the project joined the consultation meetings on initial and second alpha-builds, alongside an established set of EAG members). As we embedded the participatory research elements within a structured eHealth build model originally developed for e-mental health interventions [ ], the repeated development cycle borrowed from the agile method enhanced the continuous generation of new ideas and feedback being inputted into the coproduction build process. Our approach integrating PPI into the agile build process, enabled end users to have hands-on involvement in producing and revising the developing intervention drafts; their experiential insights were not lost through interpretation by external technological experts which, we hope, the insights gained likely to increase end users’ engagement with the intervention [ , ]. We would argue that our participatory approach was better suited than a carer-led method, as we were able to integrate a full range of stakeholders, including eLearning experts, clinicians, and researchers into the build process alongside carers.
The participatory design and build work described in this paper have produced a final prototype of COPe-support. This study has produced a tangible output, that is, an eHealth intervention meeting the required and desired expectations of both EAG members and the carer participants who contributed to the iterative consultation alongside the prototype development. However, our results are descriptive in nature and limited in establishing the usability or acceptability of the intervention, and our sample of carers was limited by size and representativeness. The success of our approach will ultimately be tested in experimental evaluation of the intervention itself (see below).
Implications and Future Directions
This study illustrates the importance of coproduction of COPe-support, an eHealth intervention, which is designed to be used by carers autonomously. Another important contribution this study made to research in the field is the documentation of a rigorous and innovative build process, which combined intervention development method and participatory research methodologies throughout its life cycle . Across the spectrum of eHealth intervention build methods, we recognize our participatory research method being in the middle ground of the 2 polar approaches: agile or technologist-driven and carer-led. Our approach establishes itself as a third way adopting the agile process and principles while integrating PPI with the technological and research core team conducting the build work as directed by the EAG.
Following this study with the prototype ready for feasibility and usability testing, a usability study of the final delta-build of COPe-support has been completed with both a remote usability trial and a think-aloud study (JS et al, unpublished data, 2019). The results were promising and provided further feedback from end users, in terms of facilitation and delivery strategies (JS et al, unpublished data, 2019). Following further refinement work as informed by the usability study (JS et al, unpublished data, 2019), COPe-support is currently being tested for its effectiveness in supporting carers through an online randomized controlled trial (RCT) . The trial will also test levels of carers’ engagement both with the intervention and the study and for any correlation of engagement or adherence with the effects on carers’ health outcomes. These investigations will therefore enable us to consider, in the future, if our participatory research approach has been successful.
To the best of our knowledge, COPe-support is one of the few comprehensive Web-based interventions targeting carers for psychosis patients. With the benefits of access, facilitation, and delivery completely through the internet, our product has the potential to provide an evidence-based psychoeducational intervention with its key ingredients [, ]. These include providing health outcomes monitoring, psychoeducational information, and real-time interaction with health care professionals and other carers as peers. We expect the internet delivery will overcome some of the implementation and access barriers from both the service providers’ and carers’ perspective [ ]. Dependent on the trial outcomes, we have considered several areas for further development and innovation in the future. These range from investigating further implementation strategies to optimize large-scale rolling out of COPe-support upon positive trial results to examining and incorporating additional health behavioral change techniques or methods used in eHealth interventions to enhance engagement and effects upon unclear effectiveness results [ , ]. Participatory research methods and principles will, no doubt, be an important approach integrated within such work.
We integrated participatory research methodologies with a structured eHealth intervention development process to develop COPe-support through this study. COPe-support is one of the few eHealth interventions dedicated for family carers of individuals affected by psychosis. It provides information and psychosocial support for carers through the internet, promoting flexible access and individualized choice. Following usability evaluation of the intervention prototype, we are currently undertaking an online RCT to evaluate its effectiveness in promoting carers’ health outcomes.
The authors thank all the input and contributions from the expert advisory group members who helped design COPe-support. They are Ellen Harris, Jacqueline Marks, Angela Ryan, Storm Ryan, Lana Samuels, Dr Clive Travis, Leigh Wallbank, and Dr Elen Williams . The authors are grateful to the carers who participated in the consultations and supported the development of COPe-support. For recruiting participants into the study, the project team acknowledges the support of the National Institute for Health Research (NIHR) through the Clinical Research Network (Division 4).
JS is funded by a National Institute for Health Research (NIHR) Post-Doctoral Research Fellowship for this research project. This paper represents independent research. The views expressed are those of the author(s) and not necessarily those of the National Health Service, the NIHR, or the Department of Health and Social Care.
Conflicts of Interest
- Shahly V, Chatterji S, Gruber MJ, Al-Hamzawi A, Alonso J, Andrade LH, et al. Cross-national differences in the prevalence and correlates of burden among older family caregivers in the World Health Organization World Mental Health (WMH) Surveys. Psychol Med 2013 Apr;43(4):865-879 [FREE Full text] [CrossRef] [Medline]
- Smith L, Onwumere J, Craig T, McManus S, Bebbington P, Kuipers E. Mental and physical illness in caregivers: results from an English national survey sample. Br J Psychiatry 2014 Sep;205(3):197-203. [CrossRef] [Medline]
- National Alliance for Caregiving. 2009. Caregiving in the US URL: http://www.caregiving.org/data/Caregiving_in_the_US_2009_full_report.pdf [accessed 2019-02-12] [WebCite Cache]
- Rethink Mental Illness. 2012. The Abandoned Illness: A Report by the Schizophrenia Commission URL: https://www.rethink.org/media/2303/tsc_main_report_14_nov.pdf
- NICE. London, UK: The National Institute for Health and Care Excellence; 2014. Psychosis and schizophrenia in adults: prevention and management URL: https://www.nice.org.uk/guidance/cg178
- Smith J, Birchwood MJ. Relatives and patients as partners in the management of schizophrenia. The development of a service model. Br J Psychiatry 1990 May;156:654-660. [CrossRef] [Medline]
- Sin J, Moone N, Wellman N. Developing services for the carers of young adults with early-onset psychosis--listening to their experiences and needs. J Psychiatr Ment Health Nurs 2005 Oct;12(5):589-597. [CrossRef] [Medline]
- Sin J, Moone N, Harris P, Scully E, Wellman N. Understanding the experiences and service needs of siblings of individuals with first-episode psychosis: a phenomenological study. Early Interv Psychiatry 2012 Feb;6(1):53-59. [CrossRef] [Medline]
- Carers Trust. 2014. Key Facts About Carers and the People They Care For URL: http://www.carers.org/key-facts-about-carers [accessed 2019-02-12]
- Sin J, Murrells T, Spain D, Norman I, Henderson C. Wellbeing, mental health knowledge and caregiving experiences of siblings of people with psychosis, compared to their peers and parents: an exploratory study. Soc Psychiatry Psychiatr Epidemiol 2016 Dec;51(9):1247-1255 [FREE Full text] [CrossRef] [Medline]
- Sherifali D, Ali MU, Ploeg J, Markle-Reid M, Valaitis R, Bartholomew A, et al. Impact of internet-based interventions on caregiver mental health: systematic review and meta-analysis. J Med Internet Res 2018 Dec 3;20(7):e10668 [FREE Full text] [CrossRef] [Medline]
- Pharoah F, Mari J, Rathbone J, Wong W. Family intervention for schizophrenia. Cochrane Database Syst Rev 2010 Dec 8(12):CD000088 [FREE Full text] [CrossRef] [Medline]
- Cooper C, Blanchard M, Selwood A, Walker Z, Livingston G. Family carers' distress and abusive behaviour: longitudinal study. Br J Psychiatry 2010 Jun;196(6):480-485. [CrossRef] [Medline]
- Sin J, Gillard S, Spain D, Cornelius V, Chen T, Henderson C. Effectiveness of psychoeducational interventions for family carers of people with psychosis: a systematic review and meta-analysis. Clin Psychol Rev 2017 Aug;56:13-24 [FREE Full text] [CrossRef] [Medline]
- Beaumont J, Thomas JH. Measuring National Well-Being: Health, 2012. London, United Kingdom: Office for National Statistics; 2012. URL: https://www.bl.uk/collection-items/measuring-national-wellbeing-health-2012
- NHS Digital. 2011. Health Survey for England - 2010, Respiratory Health URL: https://digital.nhs.uk/data-and-information/publications/statistical/health-survey-for-england/health-survey-for-england-2010-respiratory-health
- Kuipers E, Raune D. The early development of expressed emotion and burden in the families of first onset psychosis. In: Birchwood MJ, Fowler D, Jackson C, editors. Early Intervention in Psychosis: A Guide to Concepts, Evidence and Interventions. Chichester, England: John Wiley and Sons; 2000:128-140.
- Sin J, Norman I. Psychoeducational interventions for family members of people with schizophrenia: a mixed-method systematic review. J Clin Psychiatry 2013 Dec;74(12):e1145-e1162 [FREE Full text] [CrossRef] [Medline]
- Sin J, Henderson C, Spain D, Cornelius V, Chen T, Gillard S. eHealth interventions for family carers of people with long term illness: a promising approach? Clin Psychol Rev 2018 Dec;60:109-125 [FREE Full text] [CrossRef] [Medline]
- Yesufu-Udechuku A, Harrison B, Mayo-Wilson E, Young N, Woodhams P, Shiers D, et al. Interventions to improve the experience of caring for people with severe mental illness: systematic review and meta-analysis. Br J Psychiatry 2015 Apr;206(4):268-274. [CrossRef] [Medline]
- Lazarus RS, Folkman S. Stress, Appraisal and Coping. New York: Springer Publishing Company; 1984.
- Szmukler GI, Burgess P, Herrman H, Benson A, Colusa S, Bloch S. Caring for relatives with serious mental illness: the development of the experience of caregiving inventory. Soc Psychiatry Psychiatr Epidemiol 1996 Jun;31(3-4):137-148. [CrossRef] [Medline]
- Joyce J, Leese M, Kuipers E, Szmukler G, Harris T, Staples E. Evaluating a model of caregiving for people with psychosis. Soc Psychiatry Psychiatr Epidemiol 2003 Apr;38(4):189-195. [CrossRef] [Medline]
- Szmukler G. From family ‘burden’ to caregiving. Psychiatr Bull 1996;20(8):449-451. [CrossRef]
- Rotondi AJ, Haas GL, Anderson CM, Newhill CE, Spring MB, Ganguli R, et al. A clinical trial to test the feasibility of a telehealth psychoeducational intervention for persons with schizophrenia and their families: intervention and 3-month findings. Rehabil Psychol 2005 Nov;50(4):325-336 [FREE Full text] [CrossRef] [Medline]
- Rotondi AJ. Schizophrenia. In: Cucciare MA, Weingardt KR, editors. Using Technology to Support Evidence-Based Behavioural Health Practices: A Clinician's Guide. New York: Routledge; 2010:69-90.
- Schook RM, Linssen C, Schramel FM, Festen J, Lammers E, Smit EF, et al. Why do patients and caregivers seek answers from the internet and online lung specialists? A qualitative study. J Med Internet Res 2014 Feb 4;16(2):e37 [FREE Full text] [CrossRef] [Medline]
- Powell J, Chiu T, Eysenbach G. A systematic review of networked technologies supporting carers of people with dementia. J Telemed Telecare 2008;14(3):154-156. [CrossRef] [Medline]
- Vaughan C, Trail TE, Mahmud A, Dellva S, Tanielian T, Friedman E. Informal caregivers' experiences and perceptions of a web-based peer support network: mixed-methods study. J Med Internet Res 2018 Dec 28;20(8):e257 [FREE Full text] [CrossRef] [Medline]
- Chi NC, Demiris G. A systematic review of telehealth tools and interventions to support family caregivers. J Telemed Telecare 2015 Jan;21(1):37-44 [FREE Full text] [CrossRef] [Medline]
- Boots LM, de Vugt ME, van Knippenberg RJ, Kempen GI, Verhey FR. A systematic review of internet-based supportive interventions for caregivers of patients with dementia. Int J Geriatr Psychiatry 2014 Apr;29(4):331-344. [CrossRef] [Medline]
- Rotondi AJ, Anderson CM, Haas GL, Eack SM, Spring MB, Ganguli R, et al. Web-based psychoeducational intervention for persons with schizophrenia and their supporters: one-year outcomes. Psychiatr Serv 2010 Nov;61(11):1099-1105. [CrossRef] [Medline]
- Glynn SM, Randolph ET, Garrick T, Lui A. A proof of concept trial of an online psychoeducational program for relatives of both veterans and civilians living with schizophrenia. Psychiatr Rehabil J 2010;33(4):278-287. [CrossRef] [Medline]
- Sin J, Henderson C, Norman I. Usability of online psychoeducation for siblings of people with psychosis. Int J Technol Assess Health Care 2014 Oct;30(4):374-380. [CrossRef] [Medline]
- Chan SK, Tse S, Sit HL, Hui CL, Lee EH, Chang WC, et al. Web-based psychoeducation program for caregivers of first-episode of psychosis: an experience of Chinese population in Hong Kong. Front Psychol 2016;7:2006 [FREE Full text] [CrossRef] [Medline]
- Lobban F, Robinson H, Appelbe D, Barraclough J, Bedson E, Collinge L, et al. Protocol for an online randomised controlled trial to evaluate the clinical and cost-effectiveness of a peer-supported self-management intervention for relatives of people with psychosis or bipolar disorder: relatives education and coping toolkit (REACT). BMJ Open 2017 Dec 18;7(7):e016965 [FREE Full text] [CrossRef] [Medline]
- Robertson EG, Wakefield CE, Cohn RJ, O'Brien T, Ziegler DS, Fardell JE. The development of Delta: using Agile to develop a decision aid for pediatric oncology clinical trial enrollment. JMIR Res Protoc 2018 May 4;7(5):e119 [FREE Full text] [CrossRef] [Medline]
- Sadasivam RS, Delaughter K, Crenshaw K, Sobko HJ, Williams JH, Coley HL, et al. Development of an interactive, web-delivered system to increase provider-patient engagement in smoking cessation. J Med Internet Res 2011 Oct 18;13(4):e87 [FREE Full text] [CrossRef] [Medline]
- Adenuga OA, Kekwaletswe RM, Coleman A. eHealth integration and interoperability issues: towards a solution through enterprise architecture. Health Inf Sci Syst 2015;3:1 [FREE Full text] [CrossRef] [Medline]
- Barry MJ, Edgman-Levitan S. Shared decision making—The pinnacle of patient-centered care. N Engl J Med 2012 Mar 1;366(9):780-781. [CrossRef] [Medline]
- Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res 2010 Dec 19;12(5):e74 [FREE Full text] [CrossRef] [Medline]
- Muller MJ, Kuhn S. Participatory design. Commun ACM 1993;36(6):24-28. [CrossRef]
- Sjöberg C, Timpka T. Participatory design of information systems in health care. J Am Med Inform Assoc 1998;5(2):177-183 [FREE Full text] [CrossRef] [Medline]
- Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Hum Factors 2015 Jul 9;2(2):e12 [FREE Full text] [CrossRef] [Medline]
- Rose D. Service user/survivor-led research in mental health: epistemological possibilities. Disabil Soc 2017 May 16;32(6):773-789. [CrossRef]
- Richardson E, Laird SE. Involving carers from minority ethnic backgrounds in carer-led research. Disabil Soc 2012 Jul 12;28(1):67-80. [CrossRef]
- Staley K. National Institute for Health Research. 2013. A Series of Case Studies Illustrating the Impact of Service User and Carer Involvement on Research URL: https://www.nihr.ac.uk/nihr-in-your-area/mental-health/documents/Impact-of-service-user-and-carer-involvement%20PDF.pdf
- Schmidt U, Wykes T. e-Mental health - a land of unlimited possibilities. J Ment Health 2012 Aug;21(4):327-331. [CrossRef] [Medline]
- Trivedi P, Wykes T. From passive subjects to equal partners: qualitative review of user involvement in research. Br J Psychiatry 2002 Dec;181:468-472. [CrossRef] [Medline]
- Simblett S, Greer B, Matcham F, Curtis H, Polhemus A, Ferrão J, et al. Barriers to and facilitators of engagement with remote measurement technology for managing health: systematic review and content analysis of findings. J Med Internet Res 2018 Dec 12;20(7):e10480 [FREE Full text] [CrossRef] [Medline]
- Ennis L, Wykes T. Impact of patient involvement in mental health research: longitudinal study. Br J Psychiatry 2013 Nov;203(5):381-386. [CrossRef] [Medline]
- Sin J, Henderson C, Sese A, Woodham L, Krishnamoothy K, Gillard S. European Academy of Nursing Science Summer Conference 2018: Leadership in Nursing: challenges for the future—EFFIP (E-support for Families & Friends of Individuals affected by Psychosis): Developing & evaluating an online complex intervention for carers. In: BMC Nursing. 2018 Presented at: EANS'18; July 11-12, 2018; Ghent, Belgium p. 34. [CrossRef]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. ResearchGate. 2008. Developing and Evaluating Complex Interventions: New Guidance URL: https://tinyurl.com/y6c98kke
- ISRCTN Registry. 2018. Randomised Controlled Trial of COPe-Support Online Resource for Carers URL: http://www.isrctn.com/ISRCTN89563420 [accessed 2019-02-12]
- Cahn ES. No More Throw-Away People: The Co-Production Imperative. Washington DC: Essential Books; 2001.
- Needham C, Carr S. Social Care Institute for Excellence. 2009. SCIE Research Briefing 31: Co-Production - An Emerging Evidence Base for Adult Social Care Transformation URL: https://www.scie.org.uk/publications/briefings/briefing31/
- Pinfold V, Duggan A, Huxley P, Tomlin A, Rapaport J. NETSCC. 2010. The Development of an Online Training Resource for Mental Health Professionals to Involve Carers in Information Sharing: Report for the National Institute for Health Research Service Delivery and Organisation Programme URL: http://www.netscc.ac.uk/hsdr/files/project/SDO_FR_08-1711-160_V01.pdf
- Green SA, Carnegie W, Barber S, Matthew D, Matthews R. Co-designing interventions within quality improvement initiatives: notes from the field. J Health Des 2018 Mar 22;3(1):49-56. [CrossRef]
- INVOLVE. 2018. Guidance on Co-Producing a Research Project URL: https://www.invo.org.uk/wp-content/uploads/2019/04/Copro_Guidance_Feb19.pdf
- Lewis A, King T, Herbert L, Repper J. ImROC - Implementing Recovery Through Organisational Change. 2018. Co-Production – Sharing Our Experiences, Reflecting On Our Learning URL: http://imroc.org/wp-content/uploads/2017/10/ImROC-co-pro-briefing-FINAL-4.pdf
- Baines RL, de Bere SR. Optimizing patient and public involvement (PPI): identifying its 'essential' and 'desirable' principles using a systematic review and modified Delphi methodology. Health Expect 2018 Dec;21(1):327-335 [FREE Full text] [CrossRef] [Medline]
- Boren T, Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Profess Commun 2000;43(3):261-278. [CrossRef]
- Nielsen J. Usability Engineering. Boston, MA: Academic Press; 1993.
- Nielsen J. Nielsen Norman Group. 1994. 10 Usability Heuristics for User Interface Design URL: https://www.nngroup.com/articles/ten-usability-heuristics/ [accessed 2019-02-12]
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [CrossRef]
- Ritchie J, Lewis J, editors. Qualitative Research Practice: A Guide for Social Science Students and Researchers. Second Edition. Los Angeles: Sage Publications; 2014.
- Canvas the Learning Management Platform. 2019. URL: http://www.canvasvle.co.uk [accessed 2019-02-12] [WebCite Cache]
- Blackburn H, Hanley B, Staley K. Turning the Pyramid Upside Down: Examples of Public Involvement in Social Care Research. Eastleigh, England: INVOLVE; 2010.
- Hickey G, Richards T, Sheehy J. Co-production from proposal to paper. Nature 2018 Dec;562(7725):29-31. [CrossRef] [Medline]
- Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G, Powell J, Englesakis M, Rizo C, Stern A. Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. Br Med J 2004 May 15;328(7449):1166 [FREE Full text] [CrossRef] [Medline]
- COPe-Support. URL: http://cope-support.org/team/
|COPe-support: Carers fOr People with Psychosis e-support|
|EAG: expert advisory group|
|EFFIP: E-support for Families and Friends of Individuals affected by Psychosis|
|eHealth: electronic health|
|e-mental health: electronic mental health|
|NIHR: National Institute for Health Research|
|PPI: patient, carers, and public involvement|
|RCT: randomized controlled trial|
|REC: Research Ethics Committee|
|VLE: virtual learning environment|
Edited by G Eysenbach; submitted 12.04.19; peer-reviewed by J Amann, N Miyoshi, O( Danilina; comments to author 14.06.19; revised version received 28.06.19; accepted 28.06.19; published 06.08.19
©Jacqueline Sin, Claire Henderson, Luke A Woodham, Aurora Sesé Hernández, Steve Gillard. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 06.08.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.