The CHARIOT program has led the field in clinical applications and innovations in VR for pediatric procedural anxiety and pain. Having used VR in more than 5000 patients since 2017, the scope of impact and dissemination of VR software tools across 25 institutions worldwide reflects a motivation to rapidly equip hospitals with VR tools for patient care, particularly in the realms of procedural discomfort and anxiety. In addition to the direct distribution of content to these hospitals, the software developed by Stanford CHARIOT has been included on hospital VR platforms used by more than 200 hospitals and clinics in the United States.

Since 2018, the Stanford Children’s Health Pediatric Rehabilitation Program (PReP) for youth with chronic pain has used VR interventions during physical therapy, occupational therapy, and individual psychology sessions. The program obtained a dedicated VR therapy room for use during the PReP treatment sessions. In collaboration with the Stanford CHARIOT program and through support from The Mayday Fund, a VR platform called Fruity Feet was developed with the PReP interdisciplinary team. This platform was created through an iterative process from PReP provider and patient feedback for improvements and modifications. The overall goal was to improve function and reduce the fear of pain. The initial implementation of Fruity Feet VR tested the acceptability and feasibility of facilitating increased upper and lower extremity engagement [29]. This platform was simultaneously tested with an inpatient population during physical therapy sessions with large effects on movement observed in VR compared with the standard of care physical therapy (Caruso et al, unpublished data, August 2019).

In response to the COVID-19 pandemic, this work involves pilot testing of at-home VR for pain rehabilitation. With PReP operating in a telehealth format for all treatment interventions, youth in PReP were provided with VR equipment for home use during telehealth sessions with providers and independently for their daily home exercise plan. Preliminary data for the at-home VR intervention are forthcoming, but the initial interview feedback is positive.

Since 2018, the use of VR in patients with pain at CCHMC has followed a -pronged approach: a clinical research arm for managing acute postoperative pain and a clinical application arm for patients with chronic pain undergoing inpatient rehabilitation. Our team has conducted 2 pilot studies in children with moderate to severe pain after surgery, one of which used distraction-based virtual reality (VR-D) and guided-relaxation virtual reality (VR-GR) using the Mindful Aurora app [30-32]. This pilot study demonstrated that a single, postoperative session of both VR-D and VR-GR was associated with small changes in pain and anxiety lasting up to 30 minutes. This pilot study supports the implementation of VR therapy in managing acute postoperative pain in children and the need for future studies. The CCHMC team worked with Stanford CHARIOT to use VR programs appropriate for pediatric patients. This partnership has afforded the clinical team access to VR content that is highly engaging and motivating for children and adolescents.

Going forward, the VR team continues to explore engaging and innovative ways of using VR for pain management. By adding VR to postoperative pain management, we hope to augment our primary strategy of multimodal analgesia by increasing patient motivation and engagement in alternative therapies (eg, distraction and relaxation). Plans are in place to continue the implementation of VR into chronic pain rehabilitation with an emphasis on increasing social interaction in group settings through cooperative VR activities and improving movement and body awareness by incorporating VR into physical and occupational therapy.

Clinicians and researchers have been implementing VR at SickKids over the past 8 years in diagnostic imaging and emergency departments for intravenous insertions [33], perioperative services to prepare children for anesthesia during the perioperative period, and for subcutaneous port-a-cath access in oncology (Aqua KindVR) [34,35]. In a single-site pilot randomized controlled trial comparing VR (peaceful underwater gamified environment) with iPad (Apple Inc; underwater video with headphones) for oncology procedures, nurses, parents, and children reported the interventions in both groups to be acceptable, with the VR participants reporting significantly higher immersion, an underassessed outcome metric in studies of VR for pediatric pain. In addition, there were trends toward reports of less pain and distress during procedures in the VR group compared with the iPad group [35].

More recently, the SickKids Chronic Pain Clinic rehabilitation team used the Fruity Feet program developed through collaboration with Stanford CHARIOT in a quality improvement (QI) project to implement and evaluate the hospital’s rehabilitation department. The QI project involved interviews with 10 adolescents and their physical therapists (PT) and found that participants reported a high acceptance of the program, high immersion and high satisfaction, no adverse events, and lower pain scores after using VR. PTs also reported that VR was easy to use and feasible to implement in the pediatric rehabilitation setting. Given the success and clinical need, SickKids has successfully advocated for a dedicated child life specialist to help with the clinical implementation of VR coupled with a programmatic approach to rolling out new interventions using evidence-based methods.

With support from The Mayday Fund, the BCH has played a central role in developing the INOVATE-Pain network. This partnership has afforded BCH with opportunities to gain VR expertise from initiatives undertaken by the leaders of the pediatric pain VR field. In addition, this collaboration has provided BCH with opportunities to creatively explore and expand the utilization of VR in the treatment of pediatric chronic pain and to plan research initiatives to evaluate these novel applications.

Since 2010, the BCH’s intensive pain rehabilitation day program has used a basic interactive gaming platform, Xbox Kinect (Microsoft Corporation), to engage youth with chronic pain in their daily treatment. Given this technology’s success and popularity, BCH is actively seeking the integration of newly released VR into their pain rehabilitation program. Initial efforts in this process focus on replicating the success established at Stanford, SickKids, and CCHMC in using VR-based physical therapy, occupational therapy, and psychological interventions in the context of our intensive pain rehabilitation day hospital program. In addition, the BCH team is leading the effort in the development of a novel, immersive VR-based intervention targeting successful school reentry in the context of intensive pain rehabilitation. These innovative efforts have successfully garnered the philanthropic support needed to bring them to fruition.

Clinician-scientists at CHLA have been exploring the use of VR for acute pain management for more than 20 years, pioneering its use with children for routine painful procedures [8,36-38]. Since its inception, the BioBehavioral Pain Lab at CHLA has partnered with academic institutions, technology companies, software developers, and other VR collaborators to maximally leverage content expertise in health care, technology advancements, and input from clinician-scientists to design and implement rigorous VR clinical trials to scientifically investigate the feasibility, usability, and efficacy of VR for managing pain with routine medical procedures. Early and recent efforts with technological advancements have fundamentally changed clinical service lines at CHLA in phlebotomy, where children coming for outpatient phlebotomy can now request VR as a standard of clinical care for blood draws. Recently, VR clinical research has focused on VR home-based systems. As virtual care has been catapulted by telehealth, home-based VR has become critical for ongoing VR investigations.

The importance of interdisciplinary, multi-institutional, and cross-contextual (industry, academic, and clinical) collaborations toward developing VR software and clinical investigations cannot be understated. However, there are challenges inherent in aligning approaches to the development and evaluation of products in each of these contexts. Navigating logistical, regulatory, and intellectual considerations is a challenge when collaborating with similar institutions (eg, academia), and they are amplified when hospitals and universities attempt to work with industry and commercial partners. Common hurdles that arise include the differing pace of development, the need to navigate multiple regulatory systems, and distinctions in the desired scope or target of the intervention. Start-ups and industries tend to move very quickly with innovation efforts and face fewer regulatory barriers than academic medicine. Large academic and clinical institutions have many layers of regulation, such as intellectual property (IP) considerations when developing products and licensing and billing issues when delivering these interventions in the clinical setting. This raises particular challenges in the realm of digital health and emerging technologies. After a rigorous randomized clinical trial is funded, institutional review is granted, and data are collected and analyzed (a course of action often measured in years), the technology under investigation may be rendered obsolete. However, opportunities exist within these challenges when these 2 approaches can work in a complementary fashion to pursue more nimble approaches to evaluation and dissemination. For example, clinical settings can provide real-world testing grounds and patient populations that are difficult for the industry to access alone. Meanwhile, performing some of the product development in the industry setting may allow the process to move more quickly than projects housed entirely within the academic environment.

In addition to working out issues of project pace and addressing regulatory requirements, partners may differ in their views of project scope. There are advantages and disadvantages to developing interventions tailored to a specific clinical population versus creating widely applicable tools, but potentially less targeted to a particular population (eg, disease condition and age group). This may be particularly true of VR, where there is a huge range of potential software applications and a variety of potential mechanisms of effectiveness that can be harnessed. There is tremendous opportunity to be leveraged when industry and academic partners come together early in the design process so that these advantages and disadvantages can be considered together and interventions can be designed to fit real-world clinical needs, rather than trying to retrofit existing products to use with specific patient populations. Fruity Feet is an excellent example of a collaboration that capitalized on the collaborative process to develop a project that held broad enough appeal to be useful in multiple contexts while also meeting specific needs of children with chronic extremity pain undergoing rehabilitative pain treatment.

Related to, but in some ways separate from, the challenges around evaluation design, resources are a major point of negotiation in developing productive partnerships for VR innovation. Resource sharing between collaborators can offer huge benefits in evaluating new interventions, but resource sharing can also multiply hurdles. Given the obligations one assumes when undertaking this type of endeavor and the time and effort required to succeed, it is critical for academic partners to enter carefully into these partnerships to avoid becoming an open testing ground for too many industry products.

For academic and clinical researchers, opportunities lie in the fact that industry partners can bring specific financial benefits to fuel the research endeavor, with tech-savvy personnel and computer science skills exceeding those typically found within academia. In turn, academic partners bring expertise and access to some funding opportunities (eg, National Institutes of Health [NIH] Small Business Innovation Research and similar mechanisms, seed funding to pilot new ideas), along with the ability to gather data for extra- or intramural grant applications, culminating in peer-review publications contributing to the evidence base in VR. Other benefits to academic partners include the possibility of IP or equity in a start-up that could also contribute to academic success or contribute to building a financially secure lab for ongoing research and future collaborations. The industry tends to focus on validating their product through institutional acceptance and the association with large prestigious medical and/or university settings. Conversely, some industry partners want to engage with content expertise to iterate and improve their products to increase efficacy and impact. Ideally, similar-minded academic-industry collaborations focus on creating an ecologically sound product that delivers on the idea that technology can assist in solving complicated health care problems.

The major challenge in the vast immersive technology marketplace is finding effective products that arise from sound research-based development. Over time, the cost of VR hardware has significantly reduced. Once an experimental scientific ivory tower effort with limited clinical application, it has evolved into affordable, off-the-shelf products that operate with minimum components to increase ease of use. The accessibility of VR products provides new opportunities for integrating these interventions into clinical care. VR hardware has become highly sophisticated and affordable such that most clinician-scientists can now afford to add it to their toolkit or even work with equipment owned by patients. In addition, as the industry expanded, the inventory has grown from very few available virtual experiences (VEs) to hundreds. Even so, very few VEs have been scientifically well established as gold standard interventions for clinical problems.

Although there are typical barriers to the adoption of new technologies in pediatric health care, VR research and adoption are particularly challenging. First, given the lack of VR safety research in pediatrics, most hardware manufacturers do not recommend VR for children before adolescence. This can further delay institutional review board approval and require additional safety monitoring that is not required with widely accepted hardware, such as tablets. Second, given the typical age of users, the headsets often require custom modifications to appropriately fit children’s head circumference to facilitate such safety and efficacy research. Third, because VR is applied directly to the patient, near the nose and mouth, infection control practices require close attention. Many consumers facing VR products include cloth straps and face pads, which present challenges to sanitation among patients. Finally, although reduced costs have made VR headsets more affordable, they are still a relatively costly adjunct given the lack of reimbursement offered by insurance for their utilization. In the future, appropriate reimbursements will mitigate the cost of implementation.

This is not an easy task, as industry start-ups are often quite eager to partner with academic VR leaders and to gain access to clinical populations to pilot and test their VR experiences. It is critical as a clinician-scientist to thoroughly vet potential partners. In these initial meetings, it is also critical to right-size the project and the timeline, that is, determine goals and benchmarks that are agreeable to all partners and appropriate to the desired outcomes. Aligning project scope, timeline, and financial concerns; assembling a team that includes all necessary domains of expertise; and clearly defining each team member’s role are vital preliminary steps to determining the match between academia and industry (Textbox 1).

It is essential to carefully define the target population in the early phases of designing a VR and AR-based intervention. There are advantages and disadvantages to developing highly tailored products. On the one hand, clinical utility is paramount, and having an intervention that clearly targets a specific clinical problem increases the effectiveness of the intervention. On the other hand, the time and effort needed to create and evaluate such interventions steer away from creating products that appeal to a broader population in terms of clinical condition, age, and gender. A product that is not sufficiently specific in its intended application may be more challenging to validate through research trials. Ideally, the goal should be to create a suite of tools with options for customizing the experience to achieve a balance between specificity and broad application to a range of patients who can benefit from the intervention.

After establishing a partnership and aligning the direction and goals of the collaboration, a well-designed study to evaluate the project is vital. As with any clinical research, the goals, available sample, resources, and timeline should drive the study design. Studies to evaluate VR-based interventions can take a variety of scientifically rigorous forms before or in lieu of traditional clinical trials. Depending on the scope and goals, feasibility or usability studies, QI studies to evaluate clinical implementation, pilot studies to inform more definitive trials, and adaptive trial designs to reduce time to results may be appropriate. Ultimately, the use of rigorous design, methods, and standardized patient-reported outcomes will be critical to both the academic and the start-up and industry. The use of flexible and agnostic technology is critical for testing new software programs and moving toward disseminating new information and evidence-based results. It is necessary to set realistic shared visions, missions and objectives and to develop task-oriented timelines with deliverables. There is a need to move away from the traditional, cumbersome randomized controlled trial approach to evaluating VR interventions and toward more adaptive and efficient evaluation methods [40].

A critical element of research design is the thoughtful selection of measures to assess processes and outcomes. To date, there is no clear consensus on how to evaluate a VR experience appropriately or how to tailor the evaluation to the specific context of the intervention. As described earlier, immersion may be central in the context of acute pain distraction, whereas interactivity is paramount for pain rehabilitation and exposure-based VR protocols. Unfortunately, even in the VR field more broadly, there are no gold standard measures of embodiment, presence, and immersion, and more research is needed (see Multimedia Appendix 3 for an example measure of some of these constructs used in pediatric research). In clinical settings, it may also be important to assess provider satisfaction and feasibility in terms of how the intervention fits into the overall workload and workflow. Failure to understand and address these aspects can lead to lack of buy-in from clinical partners (see Multimedia Appendix 4 for an example measure of these constructs).

Although the factors that influence outcomes may vary, the outcomes of interest should remain uniform and consistent across studies and clinical contexts. On the basis of the discussion and consensus at our meeting, we present recommended domains of measurement to evaluate VR interventions for pediatric chronic pain (Textbox 2). Some of these domains have been deemed critical for pediatric chronic pain within the NIH common data elements (CDEs [41]). These CDEs include pain intensity, pain interference, functioning, pain catastrophizing, and treatment satisfaction [41]. Additional general domains considered particularly important for VR include affect and fear of pain or movement. Moreover, we highlight the VR-specific metrics to consider. These include physical movement, energy expenditure, physiology, and immersion. In the context of pain rehabilitation, increasing movement is a highly relevant outcome that can be measured in real time [29]. An added consideration is the frequency of assessment, as it is likely that data collection may range from continual, session-contingent, daily, or milestone-based (eg, start of treatment and discharge). Ideally, a thorough measurement of an intervention will consist of a combination of these measurement timelines. For example, a study could include objective, continuously collected measures of physical activity and movement metrics cataloged during VR sessions, brief questionnaires deployed daily to capture gradual changes and enable single case experimental design analyses [42], and a longer battery of questionnaires completed at specified time points (eg, baseline and 3-month follow-up) to assess changes over time from repeated VR exposure.

Not all collaborative efforts were the same. From the onset, it is critical to set mutually agreed-upon deliverables. As previously noted, academic goals are often quite different from industry objectives. Some of these discrepancies need to be addressed in the first phase of the academic-industry collaborative, establishing clear goals or deliverables. The framework (for-profit or nonprofit) can greatly influence the end goal. Ultimately, both the collaborative process and the findings derived from the academic-industry collaborative need to be published in peer-reviewed journals and disseminated widely to promote the innovation and proliferation of evidence-based technology in health care. To that end, healthy, transparent, well-communicated projects can lead to fruitful and rewarding collaborations for the academic and medical partners, industry, patient and their family, and ultimately, society. INOVATE-Pain is working to create a repository for protocols, products, resources, and recommendations to guide study design and clinical implementation that can be openly accessed in an effort to advance dissemination of the work in this field and increase opportunities for collaboration.