The field of digital health has become a multibillion dollar market, powering a paradigm shift by enabling the continuous capture of multimodal data including activity, sleep, vital signs, and contextual information. Novel machine learning applications are pioneering the conversion of these multimodal data into measures for health-related quality of life (QOL)–relevant symptoms like fatigue [1], stress [2], and depression [3,4]. These insights have the potential to enable improved care delivery [5] and a deeper understanding of patients’ lived experiences and better, more personalized medicines. However, important barriers remain to realize these benefits, both in technical and social aspects of real-world adoption.

On July 27, 2021, as part of the IEEE-EMBS International Conference on Biomedical and Health Informatics jointly organized with the 17th IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks [6], a workshop was held on “Measuring Quality of Life with Multimodal Data.” The workshop was divided into two sessions, the first focusing on disease detection and the second focusing on the measurement of well-being. Abstracts from the keynotes and talks are presented in Table 1; this meeting report summarizes key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.

To begin discussions, participants shared their perspectives on some of the terminology relevant to this emerging area of research. In Table 2, we restate some of the key points raised to orientate readers in the following report.

The first session focused on the use of multimodal data and machine learning for disease detection. Detecting deviations from normal behaviors and processes is a key step in triggering further actions, whether that be a follow-up with a health care provider or a direct digital intervention [28-30].

The session started with a keynote from author LF of Evidation Health, who spoke about how person-generated health data (PGHD; adapted from [31]) are transforming public health applications of disease detection. Author LF provided an overview of how PGHD are being used to detect and measure disease progression in a range of indications, the use of PGHD for detecting COVID-19, and the challenges of distinguishing COVID-19 from other influenza-like illnesses and infections [7]. LF underlined that machine learning model performance needs to be evaluated with a specific context of use in mind and that, without such context, model performance is ultimately of little relevance in terms of clinical utility and large-scale adoption. Finally, the keynote closed with a discussion of how what we classically think of as evidence can be a source of value to patients themselves, by helping them manage and understand their own health.

The following talks covered a wide range of indications, including author FL of Roche who discussed the use of smartphone-based apps to monitor neurological conditions including Parkinson disease [9] and multiple sclerosis [10,32]. Author BV shared work from the Byteflies platform showing how the system is being deployed for longitudinal monitoring of sleep disorders and cardiorespiratory and neurodegenerative conditions, and for detecting seizures in epilepsy [11,12]. Finally, author GG presented recent work examining how unsupervised measurements of autonomic nervous system signals, including photoplesmography and electrodermal activity (EDA), are showing value in the detection and staging of mental health conditions like anxiety and depression, and how these measures play a complementary role to traditional biomarkers, becoming a useful tool in enhancing clinical trials and precision psychiatry [13-15].

The session was closed with a short panel discussion featuring all the speakers that focused on questions raised by the attendees. One question addressed the pros and cons of data collection via bring your own device (BYOD; ie, allowing participants to connect their own devices) versus data collection via an app versus provisioned devices. The speakers agreed that there are different advantages to each approach. For example, BYOD enables comparison to a personal baseline and has advantages for device adherence, whereas provisioned devices can enable higher data uniformity and eliminate barriers to participation due to lack of access to appropriate hardware. Overall, the key is to select the right data collection approach for a given setting; where data consistency or a specific data type or density is priority, for example, in a smaller randomized controlled trial, provisioning may be preferred [33]; BYOD may in turn be preferred in settings where scale becomes limiting or where long-term “pervasive” monitoring places an emphasis on measuring ecologically valid natural behavior [34]. It was noted that while progress has been made around BYOD for patient-reported outcomes [35-37], similar progress for digital measures has not been seen and will be a key step in unlocking the value previously outlined. Another question focused on challenges to integrating objective (ie, from wearable devices) and subjective (eg, from surveys of patient-reported outcomes) inputs. The panel pointed out that many disease detection applications combine both objective and subjective inputs, for example, asking participants to confirm a signal or get a follow-up test. They also pointed out that subjective and objective inputs measure different aspects, so we should not expect them to correlate; however, this also means that they may have different relationships to a given concept of interest [38]. Thus applications that combine objective and subjective inputs can have an advantage in signal detection for disease detection. To help clarify this point, consider the following example on general well-being: a range of objective characteristics can be measured that are informative of overall well-being, including social media activity, patterns of sleep and activity, news consumption, patterns of independence, and many other objective data sources; these sources are informative of several aspects of subjective well-being (eg, perception of health), but none have a direct relationship to any specific aspect of subjective well-being, and what relationship there is differs between individuals [39].

There were also questions on the value of specific objective features (eg, EDA in stress), and GG discussed how this is a special case because this objective marker directly measures autonomic nervous system activation and thus gives a very good signal on psychological state. This was contrasted against other objective measures (eg, step counts) that have a more indirect relationship to symptoms like depression and anxiety.

The panel also discussed the impact of covariates within a cohort (eg, comorbidities) and how it influences model performance. Specifically, when trying to derive more “generalizable” models, which perform well across a broad range of unseen individuals, there is a need to incorporate a large number of covariates, and these covariates can have highly varying relevance across individuals. Progress on this topic has been made in other fields [40], but it was noted that such considerations are particularly relevant to multimodal measures.

The second session focused on measuring QOL and well-being. This ever-growing field has seen proof of concepts for measures across a range of health-related QOL-relevant symptoms, including fatigue [1,41], depression [3,42,43], stress [2], anxiety [44], and independence [8,45], and significant resource is being invested to understand what “wellness” means for diverse populations [46].

Author AS of Rice University started the session with a keynote on multimodal sensor data analysis and modeling for health and well-being, discussing her vision for how measures can underpin decision support and behavior change interventions. Her examples included schizophrenia [47], mood, and stress [16]. She also discussed challenges in in-the-wild multimodal data modeling such as model personalization and adaptability to new users/patients and missing data.

Author BS of the University of California, San Diego then discussed current limitations in generalizability [48]. He proposed that the growth of personal sensor devices should enable us to augment classification by demographics and genetics, by including time series of physiology and behavior in our understanding of human diversity [17,18,49]. BS suggested developing algorithms that account for these dynamical differences, especially in health and wellness settings. Author FC of Cambridge Cognition then presented her work on the measurement of fatigue, which is increasingly understood to be a highly patient-relevant symptom across a large range of conditions [50-53]. She discussed the heterogeneity of the manifestations of fatigue, and their approach to combining active tests, voice biomarkers, and passive data collection to capture this complex symptom. Author CvH of imec then presented his work on digestible sensors and sensorized toilets for examining gut physiology. Finally, author SF of the Massachusetts Institute of Technology presented work on digital signals from wearables and smartphones, with application for the assessment of depression symptoms [22] and suicidal thoughts [23].

Overall, the panel discussion focused on measuring well-being as a whole versus specific aspects of QOL. While developing measures for specific aspects remains highly relevant for clinical development and treatment—for example, a measure of anxiety severity enables drug development and management of diagnosed individuals—personalized measures of general well-being have substantial application in public health and in engagement with individuals’ prediagnosis. Advancements to date have focused more on the former, as the relevance to the pharmaceutical industry is higher and the validation pathway is simpler [27]. The former also limits the diversity of experience captured and so frames an opportunity for reconceptualizing wellness, health, and QOL derived from broader participation in mapping individuals’ perceived needs. The panel also discussed whether it is possible and valuable to stratify mood predictions (ie, creating semipersonalized models where individuals with similar manifestations, personas, or journey stages are grouped together). Stratification based on objective behavioral data and digital signals can also advance our understanding of a condition by delineating commonalities across patients.