This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
As mental illness continues to affect 1 in 5 individuals, and the need for support has increased during the COVID-19 pandemic, the promise of digital mental health tools remains largely unrealized due to a lack of uptake by patients and providers. Currently, most efforts on supporting the uptake of digital mental health tools remain fragmented across organizations and geography. There is a critical need to synthesize these efforts in order to provide a coordinated strategy of supporting the adoption of digital mental health tools.
The specific aim of this project is to develop a web-based resource document to support the engagement of mental health providers and patients in the use of digital mental health tools.
The web-based resource was developed using a multimethod approach. A grey literature review was conducted in 2019 to identify relevant toolkits that are available in the public domain. This was supplemented with an environmental scan where individuals with expertise in the development, acquisition, implementation, and evaluation of digital mental health tools were invited to contribute additional tools or documents not identified in the grey literature search. An engagement workshop was held with stakeholders to explore how the resource document should be developed and delivered. These findings were collectively used to develop the final iteration of the resource document.
Based on a gray literature review and environmental scan with 27 experts, 25 resources were identified and included in the resource guide. These resources were developed for patients and providers by organizations from 5 countries. An engagement workshop was held with 14 stakeholders, and barriers related to cultural sensitivity, sustainability, and accessibility of the toolkit were identified. The final iteration of the resource document was developed by the research team using findings from the gray literature review, environmental scan, and engagement workshop. The contents of the 45-page resource guide are directed at mental health care providers, administrators, and patients (inclusive of families and caregivers).
The use of a multimethod approach led to the development of a resource guide that builds on existing evidence on digital mental health tools and was co-designed with stakeholders and end-users. The resource guide is now publicly available online for free and is being promoted through digital health and mental health websites. Future work should explore how this document can be integrated into clinical care delivery and pathways.
Mental illness continues to be a global challenge, particularly during the COVID-19 pandemic [
Digital mental health interventions, such as mobile apps, have been advocated by organizations, including the American Psychiatric Association (APA) [
The objective of this project is to develop a comprehensive web-based resource guide to support mental health providers and patients in the selection and adoption of digital health tools through consideration of relevant factors (eg, demographics, clinical needs). The intended audiences of the guide are mental health care providers (eg, psychologists) and administrators (eg, implementation specialists) who are interested in integrating digital health tools into clinical practice, as well as people with lived experience and families or caregivers looking to use the resource to select helpful tools for their own needs. In this article, we share our approach and methodology for developing the resource document and outline how the main findings from each phase of the method informed the final development of the resource document. In addition, the implications of the resource document and challenges identified throughout the development process are discussed.
Following guidelines of the Agency for Healthcare Research and Quality [
Overview of the multimethod approach for the development of the resource document.
The objective of the gray literature review [
Preference was given to Canadian sources and bilingual (French and English) resources. Relevant websites from mental health organizations (eg, Canadian Mental Health Association), medical organizations or hospitals (eg, British Medical Association), patient organizations (eg, The Mental Elf), and governmental organizations (eg, US Department of Health and Human Services) were also included. Tools of documents were excluded for one or more of the following reasons: they were more than 3 years old; they were digital mental health tools (eg, mental health apps, telemedicine portals); information was intended for policymakers, industry, or other audiences outside of providers, clients, or caregivers; they were academic or research articles; the tools had significant contextual information (eg, legal context or policy context) that rendered the information irrelevant for the Canadian context.
Blogposts or other lists (usually of apps) were also excluded due to a concern for the information being outdated. Included tools or documents were then catalogued using a Microsoft Excel spreadsheet for analysis. Relevant information related to the scope and utility was extracted from each tool or document. A content analysis [
• (electronic OR digital OR mobile) AND “mental health” AND (tool OR resource or e-tool OR e-resource OR toolkit OR app OR web)
• (electronic OR digital OR mobile) AND patient AND (tool OR resource OR e-tool OR e-resource OR toolkit OR app OR web)
• digital mental health tool
• digital tools to help my mental health
The purpose of the environmental scan was to identify relevant documents and tools that currently exist and are used in the field. To maximize the impact of the environmental scan and the number of documents and tools found, experts in Canada and the United States were identified using a snowball sampling approach through the professional networks of the project team and those who published, conducted research, or worked in the field. Individuals who were knowledgeable across various digital health activities (eg, implementation, evaluation, design) were eligible to participate. Experts were contacted via email and telephone by the project team. Each expert was asked if they were aware of any tools or documents relevant to guide the uptake of mental health tools in the delivery of mental health care. These tools or documents were added to the list from the gray literature review (phase 1) and screened using the same inclusion and exclusion criteria. Content analysis [
We conducted an engagement workshop to increase the relevance and use of our research findings in practice [
In order to develop the resource guide, the aforementioned efforts were consolidated by the research team. Foremost, after screening of the identified toolkits from the gray literature scan and environmental scan, a member of the research team (DM) consolidated and organized the list of resources based on the purpose and description provided by each toolkit. Each toolkit was also characterized by the intended audience, format, scope, language, and country of origin. The findings from the engagement workshop were then used by the research team to refine the draft of the resource document to a format and delivery that aligned with the need of stakeholders.
The gray literature review and environmental scan led to the identification of 25 resources that were deemed relevant for the resource document. The engagement workshop, which was conducted with 14 participants, was used to inform the development of the resource document (
The gray literature review was conducted in September 2019, and a total of 19 resources were identified. Most of the identified resources from this phase of the project were websites or blog posts that contained a collection of apps (n=9). Other types of resources included app rating frameworks (n=3), implementation guides for clinicians (n=2), evaluation tools (n=1), electronic health record–related comic strips (n=2), and social media and info guides (n=2).
Of the 19 resources that were identified from the gray literature review, only 11 (60%) of the tools or documents met the inclusion criteria and were summarized for the final iteration of the toolkit. These included the HITEQ (Health Information Technology Evaluation, and Quality Center) Health App Decision Tree [
A total of 27 experts from Canada and the United States participated in the environmental scan. The demographics of the experts are outlined in
Demographic characteristics of participants in the environmental scan.
Characteristic | Number of participants (N=27), n (%) | |
|
||
|
British Columbia | 1 (4%) |
|
New Brunswick | 2 (8%) |
|
Nova Scotia | 8 (30%) |
|
Ontario | 10 (37%) |
|
Prince Edward Island | 2 (8%) |
|
Quebec | 1 (4%) |
|
Saskatchewan | 1 (4%) |
|
Outside of Canada | 2 (8%) |
|
||
|
Academic institution | 10 (37%) |
|
Government | 9 (33%) |
|
Hospital | 3 (11%) |
|
Nonprofit organization | 5 (19%) |
The gray literature review (phase 1) and environmental scan (phase 2) led to the identification of 25 resources in our web-based resource guide. Among the 25 resources, 9 resources (36%) provided ratings or reviews of digital mental health tools, and 3 resources (12%) provided guidance on the implementation of these technologies. Additionally, there were 4 resources (16%) that were tools for patients, 6 resources (24%) for clinicians, and 3 (12%) resources designed for both patients and clinicians.
In terms of the resources identified, most were developed in Canada (n= 10) and the United States (n=10), with other resources being from the United Kingdom (n=2), Australia (n=2), and New Zealand (n=1). Only resources developed in Canada were found to be available in French. In addition, only 60% of the resources (15/25) were developed or updated in 2018 and later. The latter resources do not have an updated date or were last updated before 2018. Most resources were developed collaboratively with private or not-for-profit organizations (n=15), academic groups (n=6), provider associations like Canadian Medical Association (n=5), health care organizations (n=2), and governments (n = 2). Likewise, funding for the development of the resource originated from not-for-profit organizations (eg, One Mind), health service organizations (eg, Ministry of Health of New Zealand), government-funded organizations (eg, Ontario Telemedicine Network), provider organizations (eg, British Medical Association), and academic institutions (eg, University of Chicago).
In terms of the audience, identified resources included content relevant to patients (n=10) and clinicians (n=20). In particular, 15 resources had clinician-specific resources, 5 resources contained patient-specific resources, and 5 resources had content for both populations. Some resources indicated a specific audience, such as primary care or general providers (n=2), frontline workers (n=2), physicians (n=1), medical school students (n=2), and researchers or app developers (n=1). Some resources targeted a specific mental health condition (eg, depression), while 13 resources focused broadly on mental health and e-mental health technologies.
As per the typology outlined by the Mental Health Commission of Canada (MHCC) [
The report from the MHCC [
A total of 14 participants from mental health organizations across Canada took part in the engagement workshop that was facilitated by 6 members (GS, DM, LC, HDS, AM, and JC) of the research team. The demographics of participants and facilitators in the engagement workshop are outlined in
Demographic characteristics of participants in the engagement workshop.
Category | Number of participants including facilitators (N=20), n (%) | ||
|
|||
|
Female | 16 (80%) | |
|
Male | 4 (20%) | |
|
|||
|
British Columbia | 1 (5%) | |
|
New Brunswick | 2 (10%) | |
|
Newfoundland | 1 (5%) | |
|
Nova Scotia | 2 (10%) | |
|
Ontario | 13 (65%) | |
|
Quebec | 1 (5%) | |
|
|||
|
Clinician (eg, nurse, psychologist) | 13 (65%) | |
|
Graduate trainee | 4 (20%) | |
|
Indigenous perspective | 1 (5%) | |
|
Person with lived experience | 2 (10%) | |
|
Research personnel/expert | 4 (20%) |
In the first part of the exercise, the research team asked the participants, “What would make the worst toolkit ever?” Respondents suggested that accessibility barriers were an important consideration. Examples of accessibility barriers included a lack of “searchability” and poor user-friendliness of the resource guide itself. A document with too much text and use of jargon would make it difficult for the end-user to effectively integrate it into practice. In addition, respondents highlighted the need for making the resource a “living document” that does not contain outdated information and broken links. It was further noted that resources that are not culturally sensitive and not trauma-informed may also be dangerous for the end-user and can impede the value of the resource. Other factors discussed included a lack of a dissemination plan, discussion on privacy issues, and the absence of patients, families, or the community in the development of the resource guide.
The subsequent discussion aimed to address the challenges identified by exploring the questions “How could we prevent this from happening?” and “What would the best possible toolkit look like?” Participants made several suggestions including a focus on evidence-based development of the document, co-design with the audience, and development of a postdevelopment sustainability plan. It was indicated that the evidence-based development should be inclusive of the views and perspectives of intended end-users and include open-source links for readers to explore if they are interested. It was further suggested that the methodology of the resource guide be transparent in the resource document. Participants explained that the content of the resource document should also be inclusive of the different learning styles of individuals and manage the expectations of the reader (ie, relatively new field). With regard to the postdevelopment sustainability plan, participants suggested a “review cycle” where the materials would be revisited after a certain period of time to ensure up-to-date content. In addition, a follow-up/feedback loop with participants was encouraged to allow for continuous improvement of the resource document.
The afternoon session of the workshop focused on dissemination of the resource document. When participants were asked where they seek information on digital mental health tools, a variety of academic (eg, school) and professional (eg, regulatory college) organizations were listed. Other approaches included conferences, word of mouth, and the intranet of their employer. With regard to the challenges of implementing the resource document for uptake of digital mental health tools, there were concerns on the definition of the “toolkit” and who the target audience is. There was also discussion on how the scope of the toolkit may not be compatible with current structure of care systems. For example, some clinicians may not have a choice in deciding which tools would be made available to the patient, and the process would require engaging stakeholders across project management, clinical services, and privacy domains. It was also unclear if this resource document would be based on current principles of mental health care, such as the stepped care model [
The findings from phases 1-3 of this project were used to develop the final iteration of the resource document (
Moreover, many suggestions and concerns from the engagement workshop (phase 3) were incorporated in the development of the final version of the resource guide. For example, the language used throughout the document was reflective of the suggestions of the stakeholders (eg, neutral, welcoming, and free of jargon) and brief instructions were provided at the beginning of the resource document to orient the end-user on usage of the document. The guide was also optimized for the search functionalities of the application.
Although digital mental health tools have gained significant traction and interest from patients, caregivers, family members, providers, and mental health organizations [
In our experience, the use of a multimethod approach [
This paper introduces a web-based resource guide [
During the development of the document, a number of evidence gaps were identified. Chiefly, the identified resources included in the resource guide fail to cover many of the technologies outlined in the MHCC typology [
Although this resource document was developed prior to the COVID-19 pandemic, we expect that it will continue to be of value for supporting the ongoing mental health needs and demands during and beyond the pandemic. However, it is important to note that the pandemic has greatly accelerated the uptake of some digital mental health tools [
Although this resource guide has been developed with extensive input from gray literature and experts in the field, it has yet to be integrated into clinical workflows and refined by providers with experience using this resource with patients [
This document is the product of a careful and meaningful synthesis of resources that encourage uptake of digital mental health tools, and future work should explore how these tools can be or have been integrated into clinical care pathways for mental health conditions (eg, depression). This may involve promoting and sharing the resource document across organizations that may be interested in the uptake of digital mental health tools. This may include the identified recommendations from the engagement workshop on expanding the delivery of the resource document to web-based approaches (eg, website, mobile app). It would also be useful to validate the findings from the resource document (eg, with other similar toolkits or documents) and to examine the efficacy of this resource document in addressing barriers and opportunities of digital mental health tools [
This paper describes the development of a web-based resource guide that we designed to guide the uptake of digital mental health tools into the clinical environment through a multimethod approach. The document, which is available online for public use, includes a number of resources to guide the selection, implementation, and evaluation of digital mental health tools. Although these resources cover many objectives and audiences, there are disproportionately fewer resources available for emerging technologies like virtual reality. Moreover, the lack of resources designed for caregivers warrants further research. There is also a critical need to ensure that resources are inclusive of the needs of diverse cultures, including the First Nations, Inuit, and Métis people of Canada. Finally, future work should explore how this resource guide can be adopted and integrated into clinical environments.
Final version of the resource document.
American Psychiatric Association
Health Information Technology Evaluation, and Quality Center
Mental Health Commission of Canada
Theory of Inventive Problem Solving
Authors would like to sincerely thank all the stakeholders, participants, and topic experts who participated in the workshop and environmental scan during this project. This project was funded by Canada Health Infoway.
GS contributed to the conceptualization of the study, methodology, formal analysis, investigation, supervision, project administration, and reviewing and editing of the manuscript. DM contributed to the conceptualization of the study, methodology, formal analysis, investigation, project administration, and the reviewing and editing of the manuscript. NT contributed to the conceptualization of the study and methodology. EM contributed to the conceptualization of the study and methodology. HDS contributed to the investigation, visualization, literature review, and the reviewing and editing of the manuscript. NS contributed to the conceptualization of the study and methodology. VS contributed to the literature review. JC contributed to the stakeholder workshop. BL contributed to methodology and the writing of the manuscript. LC contributed to the conceptualization of the study, methodology, and stakeholder workshop. AC contributed to the gap analysis, methodology, and stakeholder workshop. All authors gave final approval of the manuscript.
None declared.