This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The use of electronic health (eHealth) technologies in practice often is lower than expected, mostly because there is no optimal fit among a technology, the characteristics of prospective users, and their context. To improve this fit, a thorough systematic development process is recommended. However, more knowledge about suitable development methods is necessary to create a tool kit that guides researchers in choosing development methods that are appropriate for their context and users. In addition, there is a need for reflection on the existing frameworks for eHealth development to be able to constantly improve them.
The two main objectives of this case study were to present and reflect on the (1) methods used in the development process of a virtual reality application for forensic mental health care and (2) development model that was used: the CeHRes Roadmap (the Centre for eHealth Research Roadmap).
In the development process, multiple methods were used to operationalize the first 2 phases of the CeHRes Roadmap: the contextual inquiry and value specification. To summarize the most relevant information for the goals of this study, the following information was extracted per method: (1) research goal, (2) explanation of the method used, (3) main results, (4) main conclusions, and (5) lessons learned about the method.
Information on 10 methods used is presented in a structured manner. These 10 methods were stakeholder identification, project team composition, focus groups, literature study, semistructured interviews, idea generation with scenarios, Web-based questionnaire, value specification, idea generation with prototyping, and a second round of interviews. The lessons learned showed that although each method added new insights to the development process, not every method appeared to be the most appropriate for each research goal.
Reflection on the methods used pointed out that brief methods with concrete examples or scenarios fit the forensic psychiatric patients the best, among other things, because of difficulties with abstract reasoning and low motivation to invest much time in participating in research. Formulating clear research questions based on a model’s underlying principles and composing a multidisciplinary project team with prospective end users appeared to be important in this study. The research questions supported the project team in keeping the complex development processes structured and prevented tunnel vision. With regard to the CeHRes Roadmap, continuous stakeholder involvement and formative evaluations were evaluated as strong points. A suggestion to further improve the Roadmap is to explicitly integrate the use of domain-specific theories and models. To create a tool kit with a broad range of methods for eHealth development and further improve development models, studies that report and reflect on development processes in a consistent and structured manner are needed.
Electronic health (eHealth)—a technology to support health, well-being, and health care—can offer many benefits, such as increased quality of care, easily accessible health care, and increased self-management [
To create a tool kit, more generalizable knowledge on eHealth development methods is necessary. To build this knowledge base, more case studies that explain and reflect on specific development methods used seem to be necessary [
In this study, the aforementioned CeHRes Roadmap [
eHealth development should be a participatory process—structurally and actively involving stakeholders during development is important [
eHealth should not be seen as a separate, stand-alone tool but has to be integrated in a health care context, which also implies changes in the way health care is delivered [
eHealth development and implementation should be intertwined; implementation is a very complex activity that should be accounted for from the start of the development process [
eHealth technologies should be based on theories from persuasive design, which can be used to support behavior and attitude change via technology [
Continuous, formative evaluation in eHealth development is important to enable creating by evaluation [
The CeHRes Roadmap (the Centre for eHealth Research Roadmap) [
These principles are translated into a model with 5 phases with accompanying goals, which are presented in
Due to the involvement of 2 of the researchers in the development process, this research can be labeled as an action study. In this study, the development process of a VR application for the treatment of forensic psychiatric patients is presented. This project was initiated and mostly took place at Transfore, a forensic hospital in the east of the Netherlands, which offers forensic mental health care to both in- and outpatients. Forensic mental health care is a complex branch of mental health care, which is situated at the intersect between mental health care and the law because it deals with the combination of mental illness and delinquent behavior. In forensic mental health care, inpatients who reside in a closed setting and outpatients who are living at home are treated for sexual or aggressive criminal behavior [
Multiple studies have pointed out the potential of VR for the assessment and treatment of forensic psychiatric patients [
In this study, multiple methods were used to operationalize the first 2 phases of the CeHRes Roadmap. The development process started with the contextual inquiry. In this phase, the stakeholders were identified, a literature review was conducted, and a multidisciplinary project team to coordinate the project was constituted. Also, focus groups and interviews with forensic patients and therapists were held. In the value specification phase, 6 scenarios with concepts for VR applications were generated by the multidisciplinary project team. These concepts were presented to the patients, therapists, and stakeholders in a Web-based questionnaire. Next, values were formulated and used to create a concept for a VR app. This concept was visualized in a low-fidelity prototype and presented to the patients and therapists in an interview to examine their opinions and preferences. These activities were not performed sequentially: several methods were conducted alongside each other or were updated throughout the process [
An overview of the used methods in the contextual inquiry and value specification phases of this study.
To reflect on the suitability of the methods and overall development process, we provided the most relevant information about each research method in a comprehensive table. The aim of this table is to present the goals, methods, results, and experiences with each method as clearly and concisely as possible. For each development method, the following information is reported:
In the contextual inquiry, we generated an overview of relevant stakeholders and their roles and tasks. Furthermore, the current situation and its points of improvement were analyzed to determine if and in what way VR could contribute to treatment in forensic mental health [
An overview of the methods and outcomes and reflection on these methods of the contextual inquiry.
Research goal | Method | Target group | Main results | Conclusions | Lessons learned |
Creating an overview of people and organizations who had a stake in the development process | Not applicable | Identification of a broad range of stakeholders, such as end users, financers of care, knowledge institutes, and other forensic care organizations—see |
Stakeholder identification was useful to identify potential financers, participants, or institutions for data collection and to look for potential development partners | This method served well as a starting point for the project, but as in-depth information about (key) stakeholders was lacking, additional research into stakeholder perspectives was necessary, for example, via interviews. The stakeholder identification was constantly revised over the course of the project to keep it up-to-date. The identification proved to be important in preventing the relevant stakeholders from being overlooked in the development process and also in supporting the researchers in identifying participants for studies. |
|
Constituting a multidisciplinary project team comprising patients, therapists, managers, and researchers to coordinate the project | Not applicable | The project team with 2 patients, 3 therapists, 2 researchers, and 1 policy advisor (n=8) was responsible for content-related and practical activities, such as structuring the development process, setting up studies, and accompanying research goals, interpreting results, decision making, and planning | The multidisciplinary project team was found to be essential for the coordination of the project, mostly because of the integration of different perspectives. |
Including potential end users in the project team was useful to ensure that decisions were aligned with their perspective. In hindsight, the team might have benefited from someone with more technical knowledge on VRa, for example, a developer. Practical issues can influence the project team composition, for example, sometimes therapists or patients did not have enough time. It was important to make agreements on what to do when this occurred. Structure was needed to keep members involved: setting regular meetings, clear communication in between meetings, and keeping minutes of meetings. Coordination by a project manager was important to achieve this. The project team members had individual, concrete, and specific tasks that helped in keeping everyone actively involved. Patients indicated that participating in the project team gave them a sense of purpose and helped them with their treatment. |
|
Determining how far there is support and enthusiasm for VRa in forensic mental health care and identifying the ideas of therapists and patients about potential ways of using VR in treatment | Patients (n=14) and therapists (n=23) | Most participants were very positive about VR. There was a broad range of ideas about using VR, for example, to improve skills, enhance insight by therapists or loved ones, or treat specific disorders, such as psychosis or posttraumatic stress disorder. See |
There appeared to be many possibilities, but further specification and insight into why and how VR should be used was required |
Focus groups were a good and efficient way to start this broad, complex project with many possible outcomes, mostly to get an idea of attitudes and potential end users. These focus groups aimed to generate idea, so provided little in-depth information about needs and goals. It was necessary to complement them with other methods, such as interviews. The way this focus group was set up was seen as a strong point: there was a clear structure without much steering on content, which enabled all participants to brainstorm freely and individually. This resulted in a very broad range of ideas, which was relevant for this phase of the development process. It was relatively easy to find participants for the focus groups. An important reason for this seemed to be the possibility to learn more about and try out VR. |
|
Gaining an overview of all studies and current initiatives concerning VR in treatment of (forensic) psychiatric patients | Not applicable | In July 2017, only 6 relevant studies were found, mostly focused on the assessment of sexual delinquents [ |
Not much is known about VR in forensic mental health care in both practice and research, so there appeared to be a need for a bottom-up development process to identify why and in what way VR could be used |
Especially, desk research proved to be relevant for the project because there were no publications (yet) about many recent, ongoing initiatives/projects. The strategy for desk research could have been more structured, for example, by creating an activity plan and planning recent updates of desk research. It was important to look outside of the focus of the project (eg, studies on VR in general), either by conducting a literature study (which is time consuming) or by searching for published reviews or meta-analyses. It might have been useful to systematically collect the literature on theories and models on delinquent behavior, as in this project, it was done in a more ad hoc manner. |
|
Identifying points of improvement in the existing forensic mental health treatment of in- and outpatients and possible applications of VR, which could improve the current situation, according to therapists and patients | Therapists (n=8) and patients (n=3), working or treated at multiple locations of Transfore, the forensic hospital. | Via inductive coding [ |
The interviews gave much information about why and how VR could be of added value. However, there were still too many possible directions to make a grounded decision about the goal and content of VR. Additional research into the needs and wishes of end users was required | The participants were asked to provide scenarios about their own experiences and ideas in an open, explorative manner to prevent too much steering by the researchers. To gain in-depth information, good interviewing skills and probing questions appeared to be important. Eliciting scenarios in participants proved to be unsuitable for (most) patients, mostly because of the broad questions that required much abstract reasoning. The part with examples from the focus groups worked better but was still experienced as difficult. Also, the interview took 1 hour, which proved to be a threshold for participating and resulted in difficulties with inclusion. The type of information collected via the interviews would have been hard to retrieve via questionnaires because of the need for probing questions. The research questions might have also been answered by means of (small) in-depth focus groups, which might have been less time-consuming. |
aVR: virtual reality.
In the value specification phase, the outcomes of the contextual inquiry were used to further specify what the added value of a technology should be according to the key stakeholders. Again, multiple methods were used to identify the stakeholders’ preferences and opinions on VR in forensic treatment and prototypes to specify these abstract values were created. These methods are provided in
An overview of the methods and outcomes and reflection on these methods of the value specification.
Research goal | Method | Target group | Main results | Conclusions | Lessons learned |
Generating multiple ideas on the use of VRa in forensic mental health care, based on the outcomes of the contextual inquiry | Not applicable | A short video was created for each of the 6 ideas. All videos had the same underlying structure: the goal of VR, its use during treatment, an example, and the desired outcomes. The videos (with English subtitles) can be watched on YouTube [ |
The videos made clear that there are a lot of promising possibilities for VR in forensic mental health, so it appeared to be necessary to make decisions about what to prioritize and why |
The structured approach in which multiple templates were used worked well in this project: it forced all different members of the project team to work and think in a similar way. Each member of the project team had a clear role with individual responsibilities. This was experienced as helpful in motivating the team members and ensuring that all of their perspectives were present in the 6 ideas. Creating scripts and videos was very time-consuming, so motivated members who are willing to invest time and effort and enough budget were necessary preconditions for making videos. |
|
Identifying (1) the preferences of stakeholders of the 6 ideas and (2) the stakeholders’ values regarding VR in forensic mental health care | Patients (n=19); therapists (n=89); other stakeholders (n=38), such as parole officers or researchers from different Dutch forensic institutions | There were no significant differences between the grades and PII scores for ideas. A broad range of positive and negative aspects and remarks were identified via inductive coding. These can be found in |
The results of the questionnaire were mostly in line with the interviews but provided more detailed and specific information, for example, how VR should be personalized and which skills should be trained |
The answers of the patients fitted the research questions of the questionnaire better than the answers that were given by patients in the interviews. This indicated that the concrete, scenario-based videos were a better way to include the patient perspective than the broad, abstract interviews. Although the goal was to make this method less time-consuming, filling in the questionnaire still took about 30 minutes, which might explain why a large share of the participants (55.4%) did not fully complete it. A shorter questionnaire might have led to more response but also would mean that less information would have been retrieved. The quantitative measures indicated no major differences between opinions about ideas. Although it was not clear if this was an issue regarding validity or if there actually were no differences, it was still useful to ask for a grade for each idea. The PII was not of added value in this questionnaire. Although this method proved to be useful to further specify previously found results, it would not have been suitable as an initial method to gather in-depth information, partly because no probing questions could be asked, and answers were relatively short. |
|
Formulating values that capture what the added value of the technology should be for people and context, according to the stakeholders | Not applicable | A total of 43 attributes and 13 values were formulated. An example of how a value was created can be found in |
Formulating values proved to be a very good way to |
Values might be difficult to understand for outsiders as they are abstract, concise summaries of the needs and wishes. Consequently, clear definitions of the values were provided to prevent misunderstandings. Besides their importance for development, the project team determined that values could also be useful to determine what to evaluate: to what extent was the added value actually achieved in practice? This way of thinking about values allowed the project team to think ahead in terms of implementation and evaluation and facilitated a broader view on the VR application. In hindsight, the process of formulating values was more complex than expected. The project team had to account for the results of all used research methods, combine them in an abstract way, and make decisions about conflicting values, such as the importance of visual realism. A clear guideline for formulating values would have been useful. |
|
Generating a concept for a VR application based on the values and previously gathered results | Not applicable | The main goal of the VR application was to support therapists and patients in identifying |
The developed concept was a combination of elements of all 6 videos that were created by the project team. Also, important concepts that already arose from the interviews were present in the idea, for example, personalization, skills training, and new insights |
To ensure the consistency of the development process, the idea generation process started with discussing the implications of all earlier conducted studies, even though it was more appealing for the project team to start creating the idea right away. Visualization of ideas via low fidelity (lo-fi) prototypes appeared to work well during the idea generation process to make abstract concepts more concrete. For example, the team drew multiple dashboards and visualized the structure of the dashboard with post-its. This was experienced as helpful by all members of the project team. |
|
Investigating (1) how far the stakeholders’ opinions of the concept match the previously formulated values and (2) if changes to the concept are required for it to optimally fit the stakeholders’ preferences | Patients (n=10) and therapists (n=8) from all different locations of Transfore, the forensic hospital | The first part was coded deductively using the constructs of the TAM (see |
Overall, the idea fits the values of the participant, mostly with regard to the unique added value to treatment. No major changes to the basic idea were necessary. In later stages, attention should be paid to the usability of the application, training, and protocols to successfully embed VR in treatment |
This second set of interviews was considerably shorter than the first one: they only took about 15 to 20 minutes. It proved to be easier to include patients, which might be because of the relatively little time that was required to participate. Using the values to code these interviews was useful to determine the positive and negative aspects of the idea in relation to the added value that it should have had. In this way, it became very clear what the points of improvements were, which might not have been the case with an inductive, bottom-up coding process. It also allowed the project team to check whether the idea was still in line with the values. The TAM was used in the interview scheme and coding process. Although it helped to structurally ask about and analyze the participants’ attitudes and intentions, it provided hardly any information about the treatment context and characteristics of (other) persons [ Merely using the TAM would not have sufficed in this interview. |
aVR: virtual reality.
bPII: personal involvement inventory.
cTAM: technology acceptance model.
The main goals of this study were to analyze the suitability of the development methods for participatory eHealth development in a complex context and reflect on the development model used: the CeHRes Roadmap. This study can contribute to the development of a broad tool kit from which researchers can choose appropriate methods for the stage of their development process, participants, and context. In hindsight, this study would have benefited from such a tool kit, as the results showed that all methods generated valuable information, but not each method proved to be very suitable for the target group and their context. Besides generating knowledge on suitable methods, this type of study can also facilitate reflection and accompanying improvements of the development model used. Although this study offers a contribution, more studies that pay attention to development methods and models are required to make generalizable statements about methods and models.
The first goal of this study was to reflect on the suitability of different development methods. The relevance of this goal became clear from the experiences of the project team, as a major challenge was to identify the suitable methods for the forensic psychiatric patient population. These types of vulnerable patient populations are often difficult to involve in research, and not much is known about the suitable methods for these types of population [
A first set of recommendations focuses on involving patients in research. First of all, working with concrete examples seemed to work better than merely asking patients for their opinion or ideas without much guidance or input [
The second set of recommendations centers on combining multiple methods and perspectives to paint a clear and complete picture of the context and stakeholder perspective. First, although involving patients proved to be very valuable, the development process also benefited from the perspectives of other types of stakeholders, such as therapists, managers, researchers, and technology developers, as they might have different needs or a more overarching view [
Besides reflections on development methods, this study also aimed to reflect on the application of the development model that was used: the CeHRes Roadmap. It is of course not possible to conclude whether the development process guided by the CeHRes Roadmap resulted in better outcomes than another development method, partly because that would require 2 parallel development processes in identical settings [
On the basis of the experiences of this study, several recommendations can be made on how to operationalize the CeHRes Roadmap and similar development models. First of all, an important principle of eHealth development is that it should comprise multiple formative evaluation cycles. The experiences of this study confirmed that the Roadmap should not be used as a linear, sequential approach with a fixed order of phases and accompanying activities [
A second important finding was that the formulation of clear, specific research goals was pivotal in structuring this development process. A pitfall of an elaborate development process in a complex setting is that it might become unstructured or vague [
Third, although constituting and managing an interdisciplinary project team was complex and time-consuming, the team was found to be an important part of the development process as it facilitated decision making from multiple perspectives [
The structure of the goal-driven development process with multiple formative evaluation cycles.
While using the CeHRes Roadmap to shape the development process, we identified several strong points but also some points of improvement. First of all, the participatory development principle was used to determine what the main goal of the VR application should be in a bottom-up manner. According to this principle, it is important to involve users from the start to ensure that a technology addresses actual problems or points of improvement and is of added value for them [
Second, the Roadmap emphasizes the importance of formative evaluation and use of multiple methods. This indeed proved to be essential in this development process, especially because at the start of the project, there was no knowledge about the use of VR in forensic treatment. Consequently, much information had to be generated to make substantiated choices for the goal and content of the VR app. Just using 1 or 2 research methods would not have sufficed. This can be illustrated by the following example on personalization of VR. The first interviews and literature study indeed pointed out that personalization was important [
Finally, when reflecting on the development process, a more systematic approach toward involving domain-specific theories and models could have been used. Owing to the involvement of researchers and professionals with much knowledge on existing treatment models and theories on offending, this information was included but in an ad hoc manner. As other studies and models such as intervention mapping point out, it is important to incorporate theories that explain and change behavior in eHealth interventions [
This study described and reflected on the methods and development model used in a development process of a VR application for a complex setting: forensic mental health care. To take the domain of eHealth development to the next level, more studies need to report and reflect on the development processes in a standardized way to generate more knowledge on suitable methods. This might result in a tool kit that researchers can use to choose and operationalize methods. Based on this study, we conclude that eHealth development is much more than programing a technology or just going with the flow; it requires thorough research via methods that fit the participants, stage in the development process and context, structured project coordination by a multidisciplinary project team, a flexible and open mind-set, and the inclusion of multiple perspectives in every decision.
Main results of the research methods.
Centre for eHealth Research
electronic health
personal involvement inventory
technology acceptance model
virtual reality
Funding for this study was provided by Stichting Vrienden van Oldenkotte. The authors would like to thank the members of the
None declared.