The participants in this study were from an ongoing RCT of metacognitive training, which is being carried out at the Hospital Universitario Fundación Jiménez Díaz (Madrid, Spain) [33]. Briefly, these outpatients (age 18-64 years) with an SSD diagnosis (F20-29 of 10th revision of the International Statistical Classification of Diseases and Related Health Problems), according to the Mini International Neuropsychiatric Interview, 5th edition [34], from June 10, 2019 to March 11, 2020, were considered to be eligible. Recruitment had to be stopped on March 11, 2020 due to the COVID-19 outbreak in Spain. Exclusion criteria were (1) an intelligence quotient ≤70, which was assessed with the short form of the Wechsler Adults Intelligence Scale-IV [35]; (2) a history of head injury or a neurological condition; (3) having received a metacognitive intervention within the previous year; (4) low level of communication in Spanish; and (5) clinician judgment of the participant being unable to complete all aspects of the RCT (eg, clinician-perceived cognitive difficulty of the patient to complete all assessment procedures or weekly therapy group sessions). Of relevance, eligible candidates were reassured that refusing to participate in or dropping out of the study at any time would have no implications on care provision. While customary in research projects, financial compensation of participants may prevent us from fully understanding the acceptability of EMA apps in real-world conditions. In particular, it should be noted that the vast majority of patients with mental health problems receive publicly funded care in Spain, which is free at the point of delivery. Although participants may have been financially compensated for their time, we considered that this may have affected the external validity of our results in terms of EMA acceptability, particularly in our setting. This RCT obtained ethical approval from the local research ethics committee and is registered at clinicaltrials.gov (NCT04104347). Participants gave written informed consent to the research project principal investigator (JDLM) who led the first face-to-face interview with eligible candidates. Those who agreed to participate in the trial were asked if they owned a smartphone. If this was the case, consent to eB2 installation (which is explained below) was attempted to be obtained. Another researcher (VGRR) provided participants in the trial with all the relevant information on eB2, particularly regarding their passive role. More specifically, participants were explained that after installation of the app by one researcher (VGRR) at the clinic in front of them, they did not have to upload data. Equally, they were informed about how to uninstall the app at any time and were provided with the team telephone number. Those who consented to eB2 (users) and those who did not (nonusers) were compared on sociodemographic, clinical, neurocognitive, psychopathological, insight, and metacognitive variables as putative predictors of eB2 acceptability.

The eB2 [36] is a “passive” smartphone-based platform [37] available on Android and iPhone operating systems, designed for recording functioning-related data such as mobility (location, distance, speed), physical activity (number of steps), sleep data, and social activity (phone use, active apps, social network data) without the subject’s collaboration, that is, running in the background of users’ phones (it is passive) other than the initial configuration, which was assisted by one researcher (VGRR). All the above information is gathered from inertial sensors, physical activity, phone calls, message logs, app usage, nearby Bluetooth and Wi-Fi connections, and location. In addition, more detailed activity information and nearby location data can be accessed through Google Play services. Additional resources include a noncontinuous recording schedule and automatic sleep/wake function so that the battery can be safely saved. Further, if eB2 is stopped due to user’s actions or failures/reboots, the operating system can relaunch itself. Data are anonymized and sent to a secure server, thereby allowing continuous feedbacks from digital phenotyping [5].

In terms of sociodemographic variables, we collected data on age, gender, and education level. Regarding clinical variables, diagnosis (schizophrenia-F20 versus all other SSD), previous suicidal behavior (present vs absent), duration of illness (≤5 years vs >5 years), and previous admissions were included. Premorbid adjustment was measured with the premorbid adjustment scale (PAS) [38], which provides 3 scores on childhood, early adolescence, and late adolescence premorbid adjustment (higher scores indicating poorer premorbid adjustment).

Two neurocognitive measures were used, namely, intelligence quotient, which was estimated with the vocabulary subtest of the Wechsler Adults Intelligence Scale-IV [35], and executive function, which was evaluated using the Trail Making Test (TMT) [39]. TMT involves connecting numbers (Task A) or alternating numbers and letters (Task B); therefore, time to complete each task (in seconds) was taken. Subtracting the time to complete task A from time to complete task B provides an overall measure of the executive function (set shifting), having controlled for processing speed (TMT B-A). Psychopathological symptoms were rated using the Spanish version [40] of the Positive and Negative Syndrome Scale (PANSS) for schizophrenia [41].

Clinical insight, which was the primary outcome of the RCT [33], was assessed with the Spanish version [42] of the Schedule for Assessment of Insight [43]; it provides scores on global insight and 3 insight dimensions: illness recognition, symptom relabeling, and treatment compliance. The higher the score, greater the insight. Three metacognitive dimensions were considered, namely, jumping to conclusions (JTC), cognitive insight, and theory of mind (ToM). JTC was measured with the Beads Task [44]. On the basis of probability (in task 1, the probability is 85:15, while in task 2, the probability is 60:40), the individual must decide the jar to which the extracted bead belongs. JTC is considered if a decision is made after extracting 1 or 2 beads. Cognitive insight was measured with the Spanish version [45] of the Beck Cognitive Insight Scale (BCIS) [46], which is a 15-item self-rated scale and yields 2 factors, namely, self-reflectiveness (9 items) and self-certainty (6 items). An overall measure of cognitive insight, that is, composite index, can thus be calculated by subtracting self-certainty from self-reflectiveness. ToM was evaluated by means of the Hinting Task [47] Spanish version [48] by using 2 stories: scores ranged from 0 to 4, where higher scores indicated better ToM performance; and the Emotions Recognition Test Faces activity [49], which is composed of 20 different photographs showing people’s emotions and 2 answers at the bottom of each picture, one of each is right and the other is wrong. Scores ranged from 0 to 20; the higher the scores, the better was the ToM performance.

First, we calculated acceptability in terms of (1) recruitment, that is, the proportion of RCT participants who consented to eB2 installation (users), and (2) retention, that is, the proportion of users whose eB2 remained installed till the end of the study period. With regard to the predictors of eB2 acceptability, we first inspected the variable distributions by using histograms and the Kolmogorov–Smirnov test. Second, exploratory bivariate analyses investigated differences between users and nonusers in sociodemographic, premorbid adjustment, clinical, neurocognitive, psychopathological, insight, and metacognitive variables. Parametric and nonparametric tests were used as appropriate. Third, a multivariable binary logistic regression model tested the real influence of the above (independent) variables on “being user” (vs nonuser), that is, acceptability, which was the main outcome measure. A 95% significance level was set for all the above analyses, which were performed using SPSS version 25.0 (IBM Corp).

From June 10, 2019 to the March 11, 2020, 351 eligible patients were approached and invited to participate in the RCT by the treating consultant psychiatrist (LMI or SSA) or psychologist (LML). Of these, 77 individuals (22%) agreed to take part in the RCT, met the inclusion criteria, and were available at the baseline assessment; therefore, they were asked to install eB2. Twenty-four subjects (31%) agreed and gave consent (users) (Figure 1).

Although in 4 users, eB2 installation could not be completed owing to technical issues, in the analyses below, we compared all the individuals who accepted eB2 (users, 24/77, 31%) with those who did not (nonusers, 53/77, 69%). Only 6 subjects uninstalled the app prior to the end of the study, which occurred at a median of 14.50 weeks. Hence, there were 14 users who did not uninstall eB2 (ie, retention was 14/20, 70%) of whom 7 subjects continued to use the app (ie, data being sent to the server) till the end of the study. For the remaining 7 users, there was no evidence of data being sent to the server (eg, they had no wireless access or the battery may have run out without being replaced). Time of eB2 data uploading ranged from 0 to 44 weeks (median 12 weeks).

We asked nonusers (n=53) for the reasons underlying refusal. For descriptive purposes, 4 categories (see below) were created by 2 researchers (JDLM and VGRR) who independently classified participants’ free-text verbal responses according to such categories. Any inconsistency was resolved by consensus by 2 researchers (EBG and MLB). Results were (1) lack of interest (31/53, 58%), (2) privacy issues (5/53, 9%), (3) lack of a mobile device (4/53, 7%), and (4) eB2 incompatible with mobile device (prior to installation) according to the patient (12/53, 23%). One individual gave no reason. However, no nonuser raised issues about the configuration/installation of eB2, which was carried out in front of him/her by one researcher (VGRR), that is, at no point was the smartphone taken away from the owner. Time from eB2 installation to last eB2 data registration by the server (n=20) ranged from 0 to 553 days (median 187.5). One subject experienced a psychotic relapse and asked the research team to uninstall the app, which was honored. However, no causality association between eB2 installation and mental state relapse could be established. The sample population characteristics (N=77) and differences between users (n=24) and nonusers (n=53) are presented in Table 1.

At the time of the study inception, users were younger than nonusers (42.33 [SD 10.75] vs 47.69 [SD 9.76] years, respectively; t₇₅=–3.46; P=.001). The proportion of less educated people among users was lower than that among nonusers (1/24, 8% vs 8/53, 23%, respectively; χ²₁=4.0; P=.045). No between-group differences in gender, diagnosis, history of suicidal behavior, illness duration, or number of previous admissions emerged from the analyses. PAS scores among users were lower (ie, better premorbid adjustment) than those among nonusers, which reached significance in early adolescence (5.33 [SD 2.76] vs 8.71 [SD 4.95], respectively; t_71.21=–3.80; P<.001). No significant differences between users and nonusers in psychopathological symptom severity (PANSS factors) or clinical insight scores were found. Although intelligence quotient did not significantly differ between groups, users had better executive function performance, that is, it took them shorter time (in seconds) to complete TMT-A (45.38 [SD 12.51] vs 65.90 [SD 30.16], respectively; t₆₃=–4.16; P<.001) and TMT-B (102.13 [SD 34.27] vs 138.57 [SD 62.20], respectively; t₆₃=–3.18; P=.002) than nonusers. Differences in time to complete task B minus time to complete task A were nonsignificant (P=.09). In terms of metacognitive tasks, the BCIS self-reflectiveness (17.18 [SD 3.86] vs 14.69 [SD 5.41], respectively; t₇₂=1.95; P=.06) and the BCIS Composite Index (9.95 [SD 4.78] vs 6.73 [SD 7.17], respectively; t₆₈=1.92; P=.06) among users were higher than those among nonusers (indicating better metacognitive performance). Neither JTC (P=.83) nor ToM measures (Hinting Task: P=.47; Emotions Recognition Test Faces activity: P=.79) distinguished users from nonusers significantly.

Age, education level, early adolescence premorbid adjustment, TMT-A, TMT-B, and cognitive insight (only composite index was taken to avoid multicollinearity) were significantly associated with being a user, and they were therefore entered into the binary multivariable logistic regression model (Table 2).

Age (OR 0.93, 95% CI 0.86-0.99; P=.048) and early adolescence PAS score (OR 0.75, 95% CI 0.61-0.93; P=.01) remained significant. The final model (χ²₆=25.3; P<.001) explained 44.7% (Nagelkerke R²) of the variance on being a user (or acceptability, ie, the outcome variable) and correctly classified 77% (59/77) of the subjects, that is, 54% (13/24) of the users and 88% (47/53) of the nonusers (Table 2).

We used data from an ongoing RCT of metacognitive training with an unselected sample of patients with SSD [33] and we compared those participants who consented to installation (users) of a passive smartphone-based EMA app, eB2, with those who did not (nonusers) in order to investigate eB2 acceptability and the factors that predicted the acceptability. First, as postulated, acceptability was lower than 50% (approximately one-third of the participants), thereby indicating low recruitment. However, contrary to our second hypothesis, retention was higher than expected since most users had not uninstalled eB2 till the end of the follow-up period (at 14 weeks) and in half of the users, there was evidence of continued eB2 use. However, this may have been due to the chronicity of the participants (illness duration was longer than 5 years in almost 90% of them) or negative symptom severity (based on the PANSS ratings). In other words, not having uninstalled the app may well reflect a lack of interest in eB2 rather than the other way round. Our third hypothesis was in part supported by the study results. eB2 users were younger, had a higher education level, better premorbid adjustment and executive function, and higher levels of cognitive insight than nonusers, although only being young and good early adolescence premorbid adjustment survived the multivariable regression model.

In spite of previous concerns about EMA tool use by patients with SSD, almost one-third of the trial participants agreed to the installation of the eB2 passive mobile app, which opens new directions for clinical practice and research. Participants did not have to take any active role in uploading data or installing the app, which was completed by one researcher (VGRR), which may have increased acceptability levels. EMA-based methods may also pave the way toward remote monitoring of such a vulnerable group of patients. However, further studies should explore more successful strategies aimed at increasing EMA acceptability among patients with SSD. Information leaflets explaining evidence-based benefits from EMA in lay terms within a proper patient-researcher/clinician relationship may contribute to this. Unlike most previous studies [14], participants in this RCT did not receive a financial compensation. Although subject to further debate, by doing so, we may have avoided a potential selection bias. In other words, our findings do reflect the extent to which real-world patients with SSD consented to a mobile-based EMA app, and what appears to underlie this, regardless of the potential financial incentives. Specifically, the vast majority (31/53, 59%) of those who refused to install eB2 reported lack of interest. Although a financial incentive may have reduced the proportion of nonusers, this would have not reflected a real patient involvement in using EMA.

This study findings should be considered in light of some limitations. First, participants came from an RCT and therefore gave consent and completed a comprehensive set of assessments. Lack of cooperativeness was also an exclusion criterion. Thus, referrers (LLI, SSA, LML) only found 351 patients to be eligible over the study recruitment period, which had to be stopped owing to the COVID-19 outbreak in Spain. Regretfully, we did not systematically record the total number of patients that they saw in clinic, which was much higher. Only 77 agreed to take part in the RCT. Hence, those with poorer insight levels were therefore less likely to take part in this study, which may limit the generalizability of the results. However, this ethical requirement and the subsequent limitation in terms of generalizability applies to most studies on insight in psychosis. Nonetheless, this study is part of an RCT of metacognitive training (detailed above), which was not originally designed to test this study hypotheses. Not only this may have affected the representativeness of the study sample, but also much caution should be taken when applying the study results to patients with SSD in other settings. Second, all participants were mental health service users living in Madrid, which is an inner urban area, and results may not apply to people with psychosis receiving mental health input from primary care (ie, only from the general practitioner) or those residing in rural areas. Third, only a small proportion of the RCT participants agreed to eB2 installation; therefore, some between-group (users vs nonusers) comparisons may have lacked sufficient power. Fourth, other nontested variables may affect eB2 acceptability. Finally, future studies may involve families and carers in EMA app installation and compliance. Regretfully, we did not systematically collect data in the first face-to-face interview with participants in the RCT in terms of variables related to the researchers asking for consent (JDLM and VGRR) and whether their relationship with the participants may have affected eB2 acceptability. This said, we suggest registering information on these variables in future EMA acceptability studies. Further, it should be noted that as per the protocol of the RCT, these EMA users had to have consented to participating in the trial. Hence, we cannot rule out that a number of those who refused to enroll the RCT may have accepted eB2, although this seems unlikely since RCT participants tend to be those individuals with higher levels of cooperativeness.

Our work, therefore, adds to the growing field of e-mental health. Within the context of the COVID-19 pandemic, remote telemedicine-based mental health services need to be prioritized, which is in line with previous mental health policies focused on information and communication technology [62]. Not only the COVID-19 pandemic is likely to have a negative impact on mental health outcomes, including increased suicide rates [63] via unemployment rise [64], but also underfunded/underresourced services will have to continue delivering mental health care to vulnerable patients in need [65]. Although eHealth tools may mitigate this [66], long-term outcomes remain unknown, particularly regarding patients with SSD.

In keeping with this, our results highlight that EMA methods may need to be tailored to patients with SSD. Otherwise, there is a high risk that this group of vulnerable patients may be neglected by newly developed approaches to mental health care with the subsequent very negative impact on outcomes, including increased stigma. Specifically, much attention should be paid to patients with SSD when getting older and to those with low education level, poor premorbid adjustment, and deficits in executive function and cognitive insight. Worryingly, not only are these the most vulnerable individuals affected by such a serious mental illness, but also, based on our results, they appear to be the most reluctant ones to use remote resources such as EMA, which are likely to be prioritized by mental health services in the post-COVID-19 pandemic era. The question, therefore, arises: how can we practically tailor EMA methods to subjects with SSD? First, as noted above, we think that passive apps should be more widely recommended than active apps, given the overall poor compliance in patients with SSD. Second, patients need to be properly reassured that these resources do not control their thoughts or monitor them personally since data are anonymized, which requires a proper doctor-patient relationship. In other words, patients with SSD may be less likely to consent to researchers who are not involved in their clinical care. This said, this is definitely an area in which further research is needed.

More specifically, 2 main interlinked unmet challenges need to be addressed by the so-called e-mental health in the years to come. First, the satisfaction of the patients with SSD toward, and adherence to, new technologies and related devices should be improved. Second, financial incentives should not be considered to achieve this, which may stress further underfunded mental health services, particularly in low-income countries.