This section begins with acknowledging the global burden of mental illness and barriers to care, as well as the increasing need of patients, clinicians, and scientists for digital, mobile, remote mental health care and research. We have then highlighted the advantages of mobile apps, passive monitoring, and experience sampling. To provide context for the rest of the paper, we have described 2 studies: the Healthy Brain Network study and the National Institute of Mental Health (NIMH) Family Study of Affective Spectrum Disorders.

The global burden of mental illnesses is staggering. Epidemiologic studies indicate that 75% of all diagnosable psychiatric disorders begin before the age of 24 years [1] and the lifetime prevalence of a severe disorder among children and adolescents is 21.4%. The most common diagnoses during childhood are anxiety disorders, attention-deficit/hyperactivity disorder, and mood disorders [2]. Despite their high prevalence, only about 50% of children with a mental health disorder receive treatment (Centers for Disease Control and Prevention, NIMH). Adults and children remain untreated, even though effective treatments exist. The World Health Organization has reported large median treatment gaps for alcohol abuse and dependence (78%), generalized anxiety disorder (58%), obsessive-compulsive disorder (57%), depression (56%), dysthymia (56%), panic disorder (56%), bipolar disorder (50%), and schizophrenia (32%) [3,4]. There is a general dearth of mental health resources, especially in low- to middle-income settings [5] and for disadvantaged youth [6], and there are many barriers to care that do not involve finances, insurance, and availability of treatments [6-16]. Simply scheduling an appointment in the United States can be difficult [8], and wait times may take months [8,11]. Additional barriers include concerns about stigma and privacy, as well as exceptional circumstances when telehealth is the only option, such as with the current COVID-19 crisis.

People seeking mental health care can use web-based resources to overcome barriers to treatment, such as distance, cost, long waits, inconvenience, stigma, privacy, and problems associated with in-person visits. In recent years, people have been increasingly turning to the internet as a preferred source of knowledge about mental health [7,17,18]. However, the abundance of inaccurate and misleading information on the internet makes it difficult to evaluate the credibility of information sources, the veracity of claims, and the effectiveness of recommendations and therapies. In addition, there are many scenarios in which obtaining data or care without access to the internet or a computer is important. Furthermore, receiving scheduled notifications at any time and place can be critical for a realistic assessment and effective intervention. In the context of these requirements, mobile devices with dedicated apps offer a better way of receiving vetted, curated content and of communicating relevant information without having to sit in front of a computer or find the right person to talk to at the right time.

Clinicians and health organizations, particularly those in the mental health sector, are overwhelmed. General health practitioners, in particular, struggle to meet the needs of their local communities. Better assistance is required to identify mental health issues during checkups to ensure that these concerns are caught early on and that patients benefit from referral to a specialist. Health organizations struggle to provide broader assistance to the vast majority of people around the world who have limited or no access to mental health resources. They need better tools to scale up mental health efforts that reduce or remove face-to-face interactions to inform, assess, and provide therapy. Digital and mobile tools are especially important to reach people who need care and who are in remote areas, in socioeconomically disadvantaged conditions, or in situations where there is profound stigma surrounding mental illness. Telehealth options that involve video teleconferencing with a health care provider are becoming more prominent and have some evidence of scalability [19]. However, these services require the time and attention of expensive and limited human resources, and therefore do not scale as well as digital, mobile, and remote resources that do not require human interaction. The current COVID-19 crisis has made it starkly apparent that telehealth, with or without human interaction, may at times be the only option for seeking and receiving care. This pandemic is causing a widespread impact on mental health, from anxiety and stress related to helplessness, fear, and uncertainties of the crisis itself to the isolation and loneliness of home confinement and social distancing [20,21]. The need for remotely administered information, assessment, and therapy will only grow as nations have been forced to discover the extensive possibilities of these novel technologies.

Scientists often struggle to acquire relevant mental health and behavioral data when using traditional research paradigms. Study participant recruitment efforts are usually restricted to individuals who can be evaluated in person and provide written consent, which drastically reduces the size and scope of study samples. Smaller sample sizes are less likely to detect effects that exist at the population level, so for data to be relevant, robust, and replicable, it is important to frequently sample from large and diverse populations. Off-the-shelf, web-based tools, such as mobile apps for collecting data, do not meet the needs of most scientists who must configure their data collection tools to reach the right population and address specific questions of scientific and clinical relevance. Even tools that are configurable often require considerable mobile app software engineering expertise that is outside of most laboratories’ capabilities. For laboratories attempting to create their own mobile app from scratch, they may turn to outsourcing for its development only to find that it is far more costly than anticipated and requires consistent maintenance and upgrading to keep up with changing software dependencies, operating system versions, and hardware, let alone the changing requirements of the research itself. In addition to the engineering, content, design, and financial challenges of creating mobile apps, there can be considerable logistical and governance challenges involved in dissemination and the incorporation of an electronic consent process.

Appropriately constructed, configured, and vetted mobile apps are currently the most promising avenue for satisfying all of the abovementioned requirements, as mobile devices with internet access are rapidly becoming widely available to diverse populations around the world. It is currently estimated that 60% of the world’s population uses the internet and that 93% of users gain internet access through mobile devices [22]. They provide a convenient way to passively or actively collect and present information that can be relevant to the natural variations and social and environmental contexts of people’s lives. Passive methods usually involve sensors carried or worn on some part of the body and can collect objective, real-world data about participants to monitor motor activity, sleep, heart rate, cognition, behaviors, mood, and physiological states, as well as detect important outcomes such as medication response [23]. Some recent examples of sensor-based technologies that track mental health include phone apps, such as Northwestern University’s Center for Behavioral Intervention Technologies’ Purple [24,25] and Intellicare [26,27] platforms, Harvard University’s Beiwe platform [28,29], and apps such as mPower [30,31] built on top of Apple’s ResearchKit for iOS [32] and ResearchStack for Android [33]. The Beth Israel Deaconess Medical Center maintains an extensive database of apps that claim to respond to mental health needs [34] and provides a protocol for their evaluation [35,36]. There is an unmet need for a free, open, configurable platform to translate pencil-and-paper assessment instruments, cognitive tasks, and therapies into attractive, engaging, and effective digital, mobile, and remote tools that exceed current standards of privacy, security, and accessibility.

One important limitation in the interpretation of data acquired through passive monitoring is the lack of contextual information on variables that may influence sleep, activity, or mood changes inferred from speech, texting, GPS location, or other interactions with mobile devices. Although studies have combined subjective symptom ratings with passive monitoring, retrospective reporting over the past week or month is common among many traditional clinical questionnaires and may miss important real-time associations between subjective and objective data. Gaining insight into the directional associations between events and psychological states and their association with sleep and physical activity can be enhanced through the administration of tools that simultaneously capture descriptions of symptoms of mood, cognition, and other subjective experiences, which remain the core components of psychiatric disorder criteria. These active methods of recording individuals’ internal states and experiences include explicit self-reports that may range from occasional and detailed survey instruments to more frequent, brief, and in-the-moment questionnaires. Repeated assessment of people in real time is best known as experience sampling [37] and in the context of medicine and physiological or event-related data capture has been referred to as ecological momentary assessment (EMA) [38]. As pointed out by Marije aan het Rot et al [39] in their excellent review of experience sampling methods and EMA [39], the 2 terms are increasingly used together, so we will simply refer to them as experience sampling.

As described by Trull and Ebner-Priemer [40], experience sampling offers major benefits over traditional clinical assessments, including the reduction of retrospective bias, real-time tracking of dynamic processes, simultaneous integration of multi-level data (eg, biological and psychological), characterization of context-specific relationships of behaviors and symptoms, inclusion of feedback, and enhanced generalizability of results. Experience sampling has been shown to be highly feasible and valid in the assessment of diverse categories of mental illness, including patients with mood disorders, anxiety disorders, substance use disorders, and psychosis [41-45] as well as in the assessment of transdiagnostic mental health issues such as suicidal ideation [46]. More recently, experience sampling has been used to assess cognitive functions [47]. Batteries of cognitive tasks, such as the NIH Toolbox [48,49], ACE [50], and Cambridge Cognition [51], are primarily used for research and are not currently intended for clinical practice; they are for the most part commercial, proprietary, and permit limited (if any) configuration options for presenting and collecting data. Although mental health professionals may not adopt new technologies for clinical assessment more readily than traditional tools [52], their use in clinical practice may be encouraged by increasing their personalization and integration into standard care [53].

The Child Mind Institute’s Healthy Brain Network study [54] is an ongoing initiative focused on creating and sharing a biobank of data from 10,000 New York area participants (age group 5-21 years). The biobank houses data on psychiatric, behavioral, cognitive, and lifestyle phenotypes, as well as multimodal brain imaging (resting and naturalistic viewing functional magnetic resonance imaging (MRI), diffusion MRI, and morphometric MRI), electroencephalography, eye tracking, voice and video recording, genetics, and actigraphy.

The NIMH Family Study of Affective Spectrum Disorders was a large, community-based, controlled family study [55] that collected assessment data 4 times per day for 2 weeks from phones provided to participants of the study. The assessments included questions about daily life experiences and behaviors at the moment of acquisition (current location, social company, performance of specific behaviors, and mood states) and since the previous assessment or since waking up (experience of daily events and event negativity, food intake, substance use, experience of headache, and specific symptoms). Assessments at the beginning of the day also included questions about duration, quality, and problems with sleep, and at the end of the day included ratings of the stressfulness of the day, food craving for the day, and specific physical symptoms (gastrointestinal symptoms and muscle pain). Response options included Likert scales for dimensional constructs (such as mood or event negativity) and checklists for multiple responses (such as for food consumed) or single responses (such as current physical location).

The data from the NIMH Family Study that evaluated the association between daily events and emotional experience yielded important differences in patterns of reactivity among the major subtypes of mood disorders, including bipolar I disorder, bipolar II disorder, major depression, anxiety disorders without a mood disorder, and controls [56]. These findings demonstrate how experience sampling is a particularly well-adapted tool for assessing affective dynamics as well as emotional reactivity following daily life events. The value of combined passive and active monitoring in this study further showed bidirectional associations between energy, motor activity, and sleep, and unidirectional associations between activity and mood, suggesting that increased activity could be used as an intervention for depression [57]. Using the novel analytic approach of fragmentation to test the stability and instability of emotional states in this study showed greater instability of energy and attention in people with a history of bipolar I disorder, whereas those with bipolar II disorder or major depression exhibited greater fragmentation of mood and anxiety [58].

In this paper, we (1) review customizable, mobile, experience sampling products for configurable data collection and content delivery, (2) summarize the motivation for and development of a new mobile platform called MindLogger, and (3) describe MindLogger applets, including an initial use case that applies a MindLogger version of the NIMH Family Study’s app as part of the Healthy Brain Network study.

In this section, we have discussed our criteria for selecting and evaluating customizable, mobile, experience sampling products; provided an overview of the development of the MindLogger platform for experience sampling and interventions; and presented an example app of MindLogger in a large-scale mental health study. The Results section follows up on each of these, presenting the results of the product review, the current state of the MindLogger platform, and example apps of MindLogger.

In this review, we did not consider mobile apps limited to specific assessments, cognitive tasks, or therapies, but rather examined platforms that enable the creation and distribution of such apps. We wanted to find products that have an administrator interface for creating and scheduling recurring, customized questionnaires, where users receive and respond to scheduled notifications on a mobile device.

The detailed outline of the protocol we followed has been provided in Multimedia Appendix 1, which is briefly summarized in the paper. Over the last 3 years, clinical and research collaborators and colleagues have helped us gather information about products with desired characteristics. To extend this search, we conducted 3 queries in Google’s search engine (without quotation marks or Boolean operators): (1) digital electronic data capture systems, to broadly identify any electronic tools for capturing data; (2) mobile phone software sensor data collection, to identify mobile data collection software that may involve sensors; and (3) alternative to Qualtrics, to identify alternatives to one of the most prominent products that enables web-based customization of surveys (although Qualtrics itself does not currently have a mobile app). We visited the websites of the first 20 search results from each query (not including advertisements) and identified the candidate products. For example, if a website listed the Top Ten Apps for..., we would include these 10 apps in our initial set of candidates. We then filtered this set using the following inclusion criteria: the candidate product had to (1) be in current use, (2) have (Android and iOS) mobile apps, and (3) have an administrative user interface for creating and scheduling times and days for recurring, customized questionnaires, where end users receive and respond to scheduled notifications via an iOS or Android app on a mobile device. Where there was any ambiguity, we contacted the company or organization to clarify, scheduled a web-based demonstration, and requested a free trial to explore the product. We excluded products that do not currently fulfill the above requirements or for which we could not receive a demonstration or trial without a legal agreement. We also excluded products that require SMS text messaging, email, or other modes of communication outside of their mobile app to send and receive notifications.

Because the type of notification is important in experience sampling applications, we identified which products can deliver local operating system notifications or push notifications, where an end user receives notifications in their mobile device’s notification bar at scheduled times, and a tap on a notification takes them to their scheduled activity within the mobile app. Local operating system notifications do not require an internet connection at the time that the notification is to be received, whereas push notifications do, and both of these are distinct from simple in-app notifications, which require the end user to use the app to see their notifications.

We collected additional information from product websites, and via teleconferences and web-based demonstrations with the product creators, and from free trials to determine the degree to which each is (1) customizable in its information presentation and data capture capabilities, (2) private and secure, and (3) accessible (easy to use, economical, and open source). Customization is important to ensure that content, language, and presentation can be adapted and updated to meet the specific needs of a given population of intended end users, and to expand the scope of possible dimensions to assess, analyses to perform, and inferences to make. Privacy and security is a central concern, especially as we conduct research involving a triply vulnerable population of end users: (1) child and adolescent (2) patients with (3) mental health and learning disorders. Data access, encryption, and deletion capabilities are the primary considerations in this domain. Accessible can mean many things; here we refer to whether and how the administrator can access the product, its source code, or its data. The degree to which a product is affordable will often determine the degree to which it is adopted. Possibly, the most stringent accessibility criterion one could have is for the software to be open source. Open-source software is important because the product does not live or die with a given company, provides an opportunity for anyone to build on and improve the software, and it is open to greater scrutiny to ensure the quality of the software, accuracy of any claims made about the software, and transparency of clinical and scientific practices that use the software. Each of these characteristics offers a competitive advantage; however, they should be considered together. For example, a company offering a free product can store or transmit user data insecurely.

MindLogger [59] is intended as an open ecosystem to create, edit, share, and administer mobile or web applets for data collection and content delivery. We use the term applet to refer to a customized collection of activities within the MindLogger app administered to target end users. Our focus with MindLogger is to easily create and edit digital mental health assessments and interventions and administer them to users via mobile or web applets. The 3 key innovations we sought to accomplish with MindLogger were customizability of content, response options, and appearance; an extensive library of applets built using open standards (open, reusable parts defined by an open protocol), and distribution as a single app that appears differently to different user groups.

MindLogger’s development began in 2017, with a focus on mental health assessments. After the significant development of an early prototype for mental health research and clinical colleagues at the Child Mind Institute and the Child Mind Medical Practice in New York City, we revisited development with a greater emphasis on human-centered design [60,61]. We included a variety of key stakeholders to ensure that their needs would be met by the platform: clinicians (psychiatrists, psychologists, social workers, etc), scientists (neuroscientists, cognitive psychologists, etc), and directors of schools specializing in learning and developmental disorders, as well as technology consultants. We integrated their feedback at different stages throughout the development of the platform. Regarding user experience, both structured and informal user feedback collection has been iterative and ongoing throughout the development process. The MindLogger apps described in this paper use an assessment frequency, duration, and question content that has been substantially validated over recent years by members of our team for a wide range of age groups (children, adults, and older adults) [62-66], for healthy individuals [62,63,67], and for persons with diverse forms of mental [62,63,67-71] or physical [72-74] disorder. The NIMH’s predecessor to the MindLogger applet described below (using a highly similar protocol for daily assessments and with identical question content) documented an average completion rate of 77.9% (SE 0.81%) for repeated daily assessments, with no significant fatigue effect, defined as an increase in missing data as a function of time in the study (ordinary least squares linear model β coefficient=–.041; P<.001). In addition to user experience considerations and other stakeholder feedback, the information we gathered over the last 3 years that preceded the above review of customizable, mobile, experience sampling products also helped to guide MindLogger development.

To obtain more in-depth information on real-time tracking of emotions, behavior, daily activities, and their contextual influences in the Healthy Brain Network study [54], we adapted the combined actigraphy and experience sampling mobile assessment tools and content from the NIMH Family Study of Affective Spectrum Disorders [55]. Although the findings of that study (summarized in the Introduction section) were primarily based on adult samples, the inclusion of a substantial subset of offspring aged 10-18 years of parents with mood disorders and controls provided compelling evidence for the feasibility, acceptability, and clinical significance of experience sampling in youth. Therefore, the goal of the present initiative was to create a version in MindLogger, with updated content (particularly with regard to sleep, positive and negative thoughts, food and drink, internet, and social media), enhanced with clarification of the content, inclusion of colorful images, and formats adapted for children and young adults to encourage engagement [75].

In this section, we present the results of our review of customizable, mobile, experience sampling products, the current state of the MindLogger platform (including roles and permissions, software architecture, and current set of features), and the applet we have deployed in the Healthy Brain Network study.

Our search resulted in 392 products, of which 315 appear to be in current use. Multimedia Appendix 1 contains a list of 101 products that have Android and/or iOS mobile apps. Upon closer inspection of their websites, 59 appeared relevant to scheduling questionnaires and notifications for a group of respondents, so we contacted the 59 products’ companies/organizations through their web-based contact forms or via email to clarify their products’ capabilities and exchanged emails with the 47 companies that responded. On the basis of these exchanges, we were able to identify 21 products that appeared to satisfy our primary criteria (administrator interface for creating and scheduling recurring, customized questionnaires, where users receive and respond to scheduled notifications on a mobile device). Five potential candidates were omitted as they are intended for use by internal business employees of a company, not by patients or by participants of a study, and require individual licenses, log-ins, or fees per device. Two more candidates were omitted as they required a legal agreement to demonstrate their products. Of the 14 remaining products for which we engaged in demonstrations and free trials, 10 met the primary criteria and are included in Figures 1-3. In these figures, yes or no (filled vs empty) correspond to the questions in the bulleted lists below. The questions represent specific instances of broader criteria that can be complex and nuanced.

Figure 1 contains information about access to the product, software, and data (for both Android and iOS).

Figure 2 contains information about presentation and data capture features:

Figure 3 contains information about privacy and security:

MindLogger’s NIMH-EMA applet has been launched as part of the Healthy Brain Network study [54] to a vulnerable transdiagnostic New York City community sample (current n=4315; enrollment rate: 90 per month; >90% have mental health or learning disorders). Participants of the study receive multiple notifications per day on their Android or iOS devices to respond to morning, afternoon, and evening assessments. Figure 8 shows screenshots of the NIMH-EMA applet. We are currently enrolling children and adolescents who are at least 11 years old to use the NIMH-EMA applet as part of the Healthy Brain Network study. This applet is about to be deployed in the NIMH research program as well.

Having developed and deployed the NIMH-EMA applet as part of one research study, we were able to rapidly develop and deploy a second assessment applet as part of a different research study on reading disability. Reading disability is the most common learning disability, affecting 10%-15% of school-age children [95]. It causes major functional impairments at all stages of life. A wealth of data documents lifelong disadvantages in educational and occupational attainment. Current evidence-based reading interventions largely rely on services by trained specialists, either in well-resourced classrooms or clinical settings. As such, under-resourced schools (or regions) are often unable to provide reading interventions for their students. The significance of this dilemma is compounded when considering that children of lower socioeconomic status and children with other serious comorbid behavioral health conditions may have more severe or complex reading disability profiles [96-98]. Thus, the children most in need are the least likely to have access to evidence-based treatment.

The EDUCATE study, funded by the National Institute of Child Health and Human Development, is a collaborative effort between researchers at the University of Connecticut and the Child Mind Institute. This clinical trial will examine the effectiveness of an at-home, game-based intervention for reading disorders. Parents of participants in this study are completing daily and weekly assessments in MindLogger, which will allow researchers to assess the home environment and compliance with the intervention protocol throughout the study period. These data are critical for evaluating the impact of this clinical trial.

In this paper, we reviewed customizable, mobile experience sampling products for configurable data collection and content delivery, summarized the motivation for and development of a new mobile platform called MindLogger, and described an initial use case that applies a MindLogger version of the NIMH Family Study app as part of the Healthy Brain Network study.

Our review returned an initial set of 392 products, of which 59 appeared to be in current use, had Android and iOS mobile apps, and were capable of scheduling questionnaires and notifications for a group of respondents. Of the 47 companies that responded to our inquiries and did not require a legal agreement, we identified 10 products that satisfied our primary criteria (administrator interface for creating and scheduling recurring, customized questionnaires, where an arbitrary number of users can receive and respond to scheduled notifications on their mobile devices). Four of these products supported end-to-end encryption, 2 enabled restricted access to individual users’ data, 1 provided open-source software, and none provided all 3 of these capabilities. Our review is not exhaustive regarding either the products that could possibly be used for experience sampling or the range of features that these products support. Existing products include an impressive assortment of nonoverlapping features and complement each other in ways that reflect the different niches or markets for which they were intended. The limitations of our search for existing products are as follows: (1) our search queries and Google’s search algorithm may not reflect the optimal search criteria for some relevant products; (2) product websites are sometimes unclear and incomplete and may even misrepresent the capabilities of their products; and (3) some companies did not respond to questions even through their website’s question or support page. Products are also adding features over time, and some companies or organizations offer paid services to build the desired features. Our comparison should be seen as a snapshot of the current state of a subset of features provided by software products intended for experience sampling.

We successfully developed MindLogger, a new platform that meets most of the stated needs of our collaborators around the world, who desire an open source, mobile mental health platform to inform, assess health, acquire data, and administer therapies. We prioritized clinicians’ and researchers’ needs and users’ experience during development, aligned technologies to meet these priorities, and are on track to achieve the full feature set we set out to include. MindLogger has end-to-end encryption, enables restricted access, is open source, and supports a variety of data collection features. One limitation of MindLogger is that it currently does not support passive data collection or interaction with peripheral devices. This was intended to reduce concerns about surveillance in an app whose first use case was for assessing children and adolescents. However, in the future, we intend to support passive monitoring of location and behaviors, and communication with other devices, so long as these are opt-in by the end user and there are clear reminders to the end user concerning what data are collected and how they will be used. Another potential limitation lies in its core strength: by making MindLogger flexible, modular, cross-platform, and configurable to help meet the unforeseen needs of future applet builders and users, the creation of variants of even well-vetted instruments is likely and will necessitate their careful validation. We have recently built a web-based applet library for viewing, copying, editing, and sharing applets. Applets in the library are labeled to indicate which have been created or vetted by the Child Mind Institute and which have been contributed by others.

We have demonstrated the flexibility and applicability of the MindLogger platform through the deployment of the NIMH-EMA applet in the Healthy Brain Network, a large-scale, longitudinal, mobile mental health study. The NIMH is about to launch the same applet in its own research program. For future directions in the near term, there are a variety of other mental health–related applets in preparation for deployment and in the planning stages as described below.

We are currently refining and testing MindLogger applets to assess and administer interventions targeting specific subgroups of youth with particular mental health and learning disorders. Although some of these applets support specific collaborators’ research, others are for broader use (listed below, with video screencast demonstrations on the MindLogger website [59]).

We intend to replace all of the Child Mind Institute’s and the Child Mind Medical Practice’s pencil-and-paper assessments with MindLogger applets and, where appropriate, create and test digital renditions of therapies as MindLogger applets, and make these universally accessible as part of a web-based public library of MindLogger applets for anyone to use, modify, and translate. In addition, there are a few applets in the planning stages.

In the same manner as we want to enable anyone to create, configure, and administer their own mobile questionnaires, we also want to enable anyone to do the same for different types of cognitive assessments. In this spirit, we created a Flanker task applet that is thoroughly configurable (presentation and timing of a fixation target, stimulus, and feedback), and we are collaborating with the NIMH to create a small battery of cognitive tasks for research in modeling behaviors.