The prospective, observational real-world study accessed data from a mobile health platform (Mobio Interactive Pte Ltd, Singapore) equipped with computer vision and artificial intelligence to objectively quantify psychological stress [14,17] and benchmarked EMAs to subjectively measure stress, valence, and arousal [21]. Asynchronous and on-demand psychotherapy was available as audio files and has been clinically validated across the mental illness severity spectrum [21-24,26,27,30-33]. Users consented to the use of their anonymous data for research purposes via the platform’s Terms of Use and were not involved in a formal process to inform the design of this research. Patient identity remained anonymous throughout the study. An exception by the institutional review board for consent was granted by Advarra (Ontario, Canada). This study has been reported in line with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines (Multimedia Appendix 1).

Data were collected between March 2015 and December 2022 on 36,160 unique users, compliant with the HIPAA (Health Insurance Portability and Accountability Act), Personal Health Information Protection Act (PHIPA), and General Data Protection Regulation (GDPR). Of these, 2786 unique individuals engaged in biometric and self-report data collection before and after engaging with asynchronous psychotherapy. No audio or video was collected, transferred, or stored.

To protect the real-world applicability of the results, users were not given any special instructions or information about the nature or possibility of the current analyses. As a consequence, user data varied greatly in terms of engagement and app-use characteristics. To create a single, unified, and consistent dataset that could be leveraged across all intended analyses, data were filtered to include only English-language psychotherapy sessions that contained the required session payloads for algorithm inclusion (refer to the “Recommendation Algorithm” section below) and only when the objective and 2 subjective measures were completed before and after each session. A power analysis indicated a minimum of 66 unique users were required to achieve 80% power for detecting a medium effect size via the intended analyses. This limit afforded over 28 sessions with all pre- and postsession measures from 67 unique users.

The recommendation algorithm was developed internally at Mobio Interactive in advance of conducting this study and was not altered during analysis. The algorithm uses a noncollaborative user-item interaction design, relying exclusively on pre- and postsession data from a consistent user. Prior knowledge of user characteristics (eg, gender, age, ethnicity, and medical diagnosis) is not necessary. The content selections and responses of other users were not considered and thus did not influence the algorithm’s recommendations for a given user. This algorithm design thus leverages neither collaborative filtering nor content-based filtering, instead treating each individual as an individual without prior assumptions about how therapy may influence them based on demographics, diagnoses, or the response patterns observed at large. A design of this nature was considered necessary due to the global footprint of the dataset and an understanding that how patients respond to treatment is highly idiosyncratic and unpredictable. Thus, the only assumption used by the algorithm is that the forms of therapy that are most effective for a given patient at one point in time will reliably predict the forms of therapy that will be most effective for the same patient at a later point in time.

A total of 3 types of pre- and postsession data were available to provide an evaluation of historical efficacy: ∆OSL, ∆SRS, and ∆SRM (as explained above within the “Materials and Variables” section). The algorithm was trained independently on each of these 3 measures, first by calculating the post-pre delta:

Where WM is the well-being measure (ie, ∆OSL, ∆SRS, or ∆SRM). For measures of stress, the more negative the ∆WM, the more efficacious the session is considered for that user (and vice versa for mood). For each user, all sessions were then ranked according to efficacy.

Sessions that are intended to be directly therapeutic, that is, are not purely (psycho)educational, are each paired with a payload of (1) 3 out of a total of 36 possible “mood words” and (2) 3 out of a total of 24 possible “intent words.” The pairing of sessions to mood and intent words was conducted according to a consensus among the content creation team at Mobio Interactive and was later verified by the session’s guide (the voice within each audio recording), including clinical psychologists, psychiatrists, mindfulness teachers, and other relevant professionals involved in content creation for the platform. Mood and intent words paired to a given session were next assigned a “raw word score,” R, equal to the ∆WM value from that session.

Next, the algorithm then allowed for potentially important factors of influence to scale R:

Where S is the scaled word score, and A, B, and C are scalers. At the time of analysis, one scaler was used, A, to control for the potential influence of the session guide, and according to the formula:

Where is the mean ∆WM of the session guide for the user, is the mean ΔWM of sessions delivered by the guide that is least efficacious for the user, is the mean ΔWM of sessions delivered by the guide that is the most efficacious for the user, and W is the “guide weight,” an arbitrary constant that controls the degree to which scaler A influences S. For simplicity, this analysis used W=1. Formula 3 increases the influence of mood and intent words delivered by session guides that are generally more efficacious for a given user.

Next, for each mood and intent word, S for each session was summed to produce the “total word score,” T:

Finally, the 3 mood words and 3 intent words for each user with the greatest T were input into a proprietary algorithm that ranks sessions within the entire possible library according to their association with that specific combination of 6 words. The top 3 sessions in this ranking were considered the algorithmically selected (AS) sessions for the patient.

All statistical analyses were conducted using R (version 4.0.2, R Core Team).

The study protocol was reviewed and approved by an independent review board (Pro00084303; Advarra; Ontario, Canada). Participant safety and privacy were protected through standard operating procedures defined in the DCB0129 and ISO27001 certifications for software as medical software and cybersecurity, respectively, awarded to the developer (Mobio Interactive, Singapore).

Of the 67 patients eligible for analysis, 43 (64%) identified as female, 19 (28%) identified as male, and 5 (7%) selected “other” under gender. Self-reported age ranged from 18 to 66 years, with the largest age group being 25-34 years (n=21, 31%), and the smallest being 18-24 years (n=7, 10%). Geolocation data indicated predominant use in Australia (n=27, 40%), New Zealand (n=12, 18%), and Canada (n=11, 16%). The mean number of sessions completed by each user (with or without pre- and postsession measures) was 154.9 (SD 228.1, median 76, range 40-1328). English-language sessions with all pre- and post-session measures completed by the participants had a mean of 42.7 (SD 16.8, median 37, range 28-108).

Inspection of the 3 measures intended to train the recommendation algorithm suggested a mean reduction in stress and an improvement in mood across all sessions. In all cases, the CIs did not include zero, as shown by ∆OSL –0.003 (95% CI –0.026 to –0.00091), ∆SRS –0.09 (95% CI –0.12 to –0.0075), and ∆SRM 0.13 (95% CI 0.01-0.24). These real-world data corroborate findings from controlled clinical investigations [21-24,26,27,30-32].

Pearson correlations between the 3 measures intended to train the recommendation algorithm suggested independence between the objective measure and the 2 subjective measure deltas. ∆SRS and ∆SRM correlated significantly with each other, but neither correlated with ∆OSL (Figure 2). The objective measure, at least within this mental health platform, may therefore be a valuable, differentiated source of information about how mental well-being is impacted by therapy in real time.

Algorithm performance was evaluated using linear regression analysis, within-subject ANOVA, and 10-fold bootstrapping to compare the efficacy of AS sessions against US sessions, using the same well-being measure (∆OSL, ∆SRS, or ∆SRM) for both training and testing.

When ∆OSL or ∆SRM data were used, AS sessions demonstrated increasing efficacy relative to US sessions as a function of training set size, as revealed by linear regression analysis (∆OSL: F_1,16=5.37, P=.03; ΔSRM: F_1,16=3.69, P=.50; Figures 3A and 3C). No effect of training set size was observed when ∆SRS data were used (F_1,16=0.046, P=.83, Figure 3B).

Within-subject ANOVA comparisons allowing the algorithm to “learn” from 15 training sessions revealed that ∆OSL training data informed an algorithm that recommended significantly more efficacious sessions when compared with US sessions (F_1,66=8.22, P=.005; Cohen d=0.17; Figure 3D). In contrast, the same comparisons when self-reported data were used did not reach significance (∆SRS: F_1,66=1.24, P=.09, Cohen d=0.095; ∆SRM: F_1,66=2.21, P=.12, Cohen d=0.041; Figures 3E and 3F).

Ten-fold bootstrapping, consistent with the ANOVA results, suggested an absence of oversampling (Figures 3G-I).

The purpose of the recommendation system explored in this study is to ensure that therapy sessions delivered to patients are as efficacious as possible. This being the case, we were curious to examine the overlap of AS and US sessions. Post hoc analysis via an ANOVA using the expected versus actual quantity of sessions with overlap (1, 2, and 3+) revealed a sharp rise in the number of patients with overlap between AS and US (Figure 4). This rise in overlap quickly exceeded chance (∆OSL: F_1,14=30.94, P<.001; ΔSRS: F_1,14=19.33, P<.001; ΔSRM: F_1,14=67.17, P<.001) and followed a quadratic relationship as revealed by regression analysis (∆OSL: F_2,22=204.1, P<.001; ΔSRS: F_2,22=23.98, P<.001; ΔSRM: F_2,22=51.55, P<.001). These results may hint that users intuitively know, perhaps subconsciously, which sessions are providing benefit.

In this study, we trained and evaluated a recommendation algorithm with an objective measure of psychological stress. We also performed the identical analyses using 2 EMAs. We compared the efficacy of algorithmically recommended sessions against sessions selected by patients themselves. Our results indicate that a precision psychiatry recommendation algorithm, when trained with an objective measure of psychological stress, can faithfully predict what forms of therapy will be more efficacious on average than the therapy patients will choose for themselves. The same clinical benefit of recommendations was not observed when the algorithm was trained with subjective EMA data.

To our knowledge, this is the first prospective study that leverages a rapid and scalable objective measure of well-being to train and evaluate a precision psychiatry therapy recommendation algorithm. The results demonstrate the predictive ability of the approach and provide a framework for statistical analysis, measurement methodology, and algorithm design in the pursuit of precision psychiatry at a scale that meets global demand. Several interesting and, at times, surprising observations emerged during our analysis.

First, patients garnered a mean benefit from engaging with content on the platform. This observation is an important real-world corroboration of previous research [21-24,26,27,30-33]. Without a strong foundation of efficacy from mobile delivery of asynchronous and on-demand therapy, recommendation algorithms embedded within these products are of no value to patients or health systems.

Second, the deltas in subjective versus objective data appear to measure separate therapy-response phenomena (Figure 2). Additional support for divergence between objective and subjective data deltas was also revealed in finding that users consistently self-reported larger changes to their well-being than what was revealed by objective data. Similar divergence between objective and subjective measures of mental well-being has been reported by others [39] and may arise in part from the susceptibility of self-reported data to biases of recall, confirmation, response, and expectation. While EMAs may circumvent recall bias, the biases of confirmation, expectation, and response likely persist. In addition, EMAs rely on a precise awareness of one’s immediate psychological states, which likely varies between (and within) individuals. Meanwhile, convergence of the EMAs suggests the potential for diminishing returns when collecting multiple sources of self-reported data, at least when compared with a divergent measure like the psychological stress DNN. Thus, irrespective of the mechanism underlying divergence between objective and subjective well-being data deltas, their dissociation may serve as an independent rationale for including objective data on mental well-being in the practice of psychiatry.

Third, objective data on psychological stress captured immediately before and after therapy sessions were sufficient to train an algorithm to prospectively predict what therapy sessions would be more efficacious for a patient compared with the content patients would choose for themselves (Figures 3A, 3D, and 3G). This finding is a realization of the general potential for objective measures of well-being in psychiatry. Beyond facial biomarkers, other methods, such as biomarker analysis of speech [40], will likely be critical. Being less susceptible to biases and independent from interoception accuracy, objective data could be more sensitive to factors important for predicting well-being outcomes [41,42]. The finding that an objective measure of stress can train an algorithm to select ideal content for patients is especially exciting considering its rapidity of collection (30 s) and potential for broad adoption within health care since the software runs on hardware most patients have with them all day (smartphones).

Fourth, subjective data in this study were not sufficient to train the recommendation algorithm to an extent that its predictions were statistically superior to self-selected sessions (Figures 3E and 3F). While other studies have reported success with recommendation algorithms trained with subjective data [34], these studies generally compared algorithm recommendations against randomly selected content (instead of the more meaningful patient-selected content, as was the case in our analysis). As discussed in the “Introduction” section, the limitations of subjective data for algorithm training are likely rooted in the multifarious, well-described biases. In particular, an expectation bias was easily observable in our dataset (Figure 3D-3F, green markers) and may have impaired the utility of self-report data for training the recommendation algorithm. It is also possible that subjective data have greater session-to-session variability (more noise), giving rise to a less consistent training dataset.

Fifth, in self-reports, when the recommendation algorithm was trained with ∆SRM data, algorithm predictiveness improved as a function of the number of sessions in the training dataset (Figure 3C). No such relationship was found for ∆SRS (Figure 3B). This result may be due to the direct versus implicit nature of the stress slider versus the mood board EMAs, respectively. The conclusion, if these results can be generalized, is that the less subjectivity inherent to a given measure, the better it may be for recommendation algorithm training.

Sixth, there was a surprising overlap of AS and US sessions beyond what would be expected by chance. This finding may suggest that patients intuitively gravitate toward more efficacious therapy regimens. While this is surely a comforting finding for both health care professionals and patients, we have little insight into why this overlap occurs. We did note that patients completed identical sessions multiple times, potentially signaling a preference to repeat sessions that led to greater feelings of well-being. Even if patients were not consciously aware of the subtle response differences to various forms of content, they nevertheless appear to act as their own recommendation algorithm. More simply put, the sessions patients “like” tend to also be more beneficial. The power of an accurate automated recommendation algorithm lies in improving upon this process regardless of an individual’s ability to self-select efficacious content or gain access to a clinical expert, thereby rendering the delivery of appropriate care more equitable across patient populations.

The work here demonstrates an opportunity for objective measures of well-being, such as psychological stress when rapidly and accurately obtained via a smartphone camera [14], to meaningfully inform precision psychiatry therapy recommendation algorithms. There are additional, broader benefits for psychiatric practice. In particular, removing bias will facilitate higher accuracy and standardization, ensuring a more uniform approach to diagnosis and treatment. Similarly, with the ability to observe measurable impact in real time, patients may become more engaged in beneficial treatment regimens. Objective measures may also lead to earlier detection, allowing intervention before issues become so severe they are undeniable or result in serious harm.

More broadly, using objective well-being data may reduce the stigma associated with an “invisible” self-reported condition and place mental health on the same level as physical health. When readily accessible for clinicians, objective data obtained remotely before a consultation may also reduce the time clinicians spend collecting patient data in person, freeing up time for clinicians to better understand and connect with their patients on a personal level. Objective data will also facilitate the proper inclusion of mental health into the broader health care economy, given that the payors of health care services are reluctant to cover costs that lack standardization and are susceptible to “gaming.” Finally, as we show here, objective data will facilitate better personalization of mental health care as recommendation methods continue to improve.

Our future work within precision psychiatry may explore the impact of additional weighted scalers for the current recommendation algorithm, as well as other algorithm designs, such as indication-specific content filtering. For example, patients with major depressive disorder may respond best to consistency with the therapy guide, while patients with generalized anxiety disorder may be particularly influenced by the degree of silence afforded to them during a given therapy session. Thus, there is room to introduce additional content classifiers for consideration by the algorithm. Separately, the efficacy of psychotherapy may be moderated by various demographic factors, including cultural background [43]. Mindfulness training and meditation courses are indeed often adapted for specific clinical populations [44-46], age groups [47,48], and cultural groups [49-51]. Incorporating diagnoses, medical history, and demographic factors into the algorithm design may lead to further improvement in its predictions. However, these types of data can also introduce undesired biases into the algorithm that counteract our intent to balance functionality across individuals. Caution will therefore be required, keeping in mind that content filters may not impart any net benefit.

Future analysis may also examine additional objective measures of well-being beyond psychological stress. In particular, biomarkers from microexpressions and facial blood distribution could be leveraged to produce accurate and objective measures of affect across the domains of valence and arousal. Once validated, multiple objective measures may prove synergistic in their ability to capture the psychological profile of patients as they engage with therapy and tailor treatment recommendations with ever-increasing accuracy.

There are a few limitations to this study. First, while our DNN quantifies stress from HRV with unprecedented precision [14], the use of facial biomarkers as the ultimate data source requires patients to actively engage with each measurement by way of a mobile selfie scan, ultimately limiting the length of the analysis period and complicating data capture during therapy (as opposed to before and after). Additional technology developments are underway to facilitate continuous passive facial biomarker data capture without direct and continuous engagement from the patient. Moreover, the increased convenience and accessibility incurred from avoiding wearables, along with the additional data that can be captured from the human face compared with the finger or wrist, make the facial biomarker method preferable for precision psychiatry applications at a global level. Second, the dataset used in the current study was restricted to English therapy sessions, and therefore, most of these patients likely use English as a first language. This constraint was necessary to ensure homogeneity among the therapeutic content. While there are no immediately obvious reasons why the results with English content would not generalize to other languages, the restriction does create a bias to a certain patient demographic. It would be prudent and worthwhile to replicate the analyses with non-English therapy sessions once the prerequisite volume of data is at hand.

While our analysis focused on the clinical utility of leveraging objective versus subjective data on patient well-being for applications within precision psychiatry, the study design and clinical implementation of the technology also demand that the recommendation algorithm training data be obtained rapidly and at scale. Rapidity is essential since the likelihood of patients completing assessments is reduced as a direct function of how long the assessment takes. Meanwhile, scalability is essential to maximize the breadth of patients that can meaningfully benefit from the technology. Wearables and other medical hardware are major barriers for underserved populations, especially in lower socioeconomic regions of developed nations and throughout low- and middle-income countries and lower-income countries. The general theme is removing barriers and increasing convenience.

Globally, more than 10% of people today need some form of mental health care, and more than 50% of the global population will require mental health care at some point in their lives [52]. This study demonstrates that effective and accessible precision psychiatry can be delivered at scale. We hope our work helps prepare the field to join the rest of medicine in consulting objective data as a regular course of action in the pursuit of optimal patient care. We also hope efficiencies generated via the technology will allow health care professionals to spend more time with their patients.

In summary, we present evidence that artificial intelligence–derived objective stress data, when captured through a mobile device’s front-facing camera before and after asynchronous therapy sessions, can identify the forms of therapy that are most efficacious for each patient and accurately predict which future therapy sessions will result in the most clinical benefit for each patient. While we feel this finding marks a major step toward precision psychiatry at scale, it is imperative to keep in mind that precision psychiatry recommendation algorithms are, in fact, recommendations. No in silico solution, no matter how accurate and powerful, will be able to adequately address all the needs of a patient.