Published on in Vol 21, No 4 (2019): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/12910, first published .
Using Machine Learning to Derive Just-In-Time and Personalized Predictors of Stress: Observational Study Bridging the Gap Between Nomothetic and Ideographic Approaches

Using Machine Learning to Derive Just-In-Time and Personalized Predictors of Stress: Observational Study Bridging the Gap Between Nomothetic and Ideographic Approaches

Using Machine Learning to Derive Just-In-Time and Personalized Predictors of Stress: Observational Study Bridging the Gap Between Nomothetic and Ideographic Approaches

Original Paper

1Center for Behavioral Cardiovascular Health, Columbia University Irving Medical Center, New York, NY, United States

2Feinstein Institute for Medical Research, Northwell Health, New York, NY, United States

Corresponding Author:

Alan Rozet, BA

Center for Behavioral Cardiovascular Health

Columbia University Irving Medical Center

Presbyterian Hospital Building, 9th Floor

622 W 168th Street

New York, NY, 10032

United States

Phone: 1 212 342 4493

Email: ar3793@cumc.columbia.edu


Background: Investigations into person-specific predictors of stress have typically taken either a population-level nomothetic approach or an individualized ideographic approach. Nomothetic approaches can quickly identify predictors but can be hindered by the heterogeneity of these predictors across individuals and time. Ideographic approaches may result in more predictive models at the individual level but require a longer period of data collection to identify robust predictors.

Objective: Our objectives were to compare predictors of stress identified through nomothetic and ideographic models and to assess whether sequentially combining nomothetic and ideographic models could yield more accurate and actionable predictions of stress than relying on either model. At the same time, we sought to maintain the interpretability necessary to retrieve individual predictors of stress despite using nomothetic models.

Methods: Data collected in a 1-year observational study of 79 participants performing low levels of exercise were used. Physical activity was continuously and objectively monitored by actigraphy. Perceived stress was recorded by participants via daily ecological momentary assessments on a mobile app. Environmental variables including daylight time, temperature, and precipitation were retrieved from the public archives. Using these environmental, actigraphy, and mobile assessment data, we built machine learning models to predict individual stress ratings using linear, decision tree, and neural network techniques employing nomothetic and ideographic approaches. The accuracy of the approaches for predicting individual stress ratings was compared based on classification errors.

Results: Across the group of patients, an individual’s recent history of stress ratings was most heavily weighted in predicting a future stress rating in the nomothetic recurrent neural network model, whereas environmental factors such as temperature and daylight, as well as duration and frequency of bouts of exercise, were more heavily weighted in the ideographic models. The nomothetic recurrent neural network model was the highest performing nomothetic model and yielded 72% accuracy for an 80%/20% train/test split. Using the same 80/20 split, the ideographic models yielded 75% accuracy. However, restricting ideographic models to participants with more than 50 valid days in the training set, with the same 80/20 split, yielded 85% accuracy.

Conclusions: We conclude that for some applications, nomothetic models may be useful for yielding higher initial performance while still surfacing personalized predictors of stress, before switching to ideographic models upon sufficient data collection.

J Med Internet Res 2019;21(4):e12910

doi:10.2196/12910

Keywords



Deeper knowledge of the day-to-day effects of both weather and physical activity on stress can be valuable for creating personalized stress-reduction interventions on a just-in-time basis. Previous investigations have often focused on a nomothetic approach, pooling data to identify influential features across individuals [1,2]. However, this approach typically has a drawback: Insight into any particular individual is limited due to heterogeneous effects of factors on individual-level stress and may not be generalized due to biological variability or overfitting [3]. For example, hot weather may reduce stress for some participants, but increase stress for others. To remedy this, some have taken an ideographic approach, developing personalized stress-prediction models for each person [4-6].

Although this ideographic strategy resolves the issue of achieving per-individual insights, it does so by discarding potentially useful data from other individuals. The ideographic approach also requires the acquisition of at least some data about a given individual before making predictions for that individual. Identifying predictors of potentially low-frequency events such as occurrences of high stress may require substantial data collection before the ideographic model becomes sufficiently robust to confidently identify predictors. Thus, prior to obtaining reliable predictors of stress, a given individual may have to complete a long period of self-tracking, which may not be acceptable to some individuals.

Here, we first compared individual-level predictors of stress identified through nomothetic models to those identified in ideographic models. We next explored the accuracy of a model-switching paradigm that begins with a nomothetic model and progressively changes to an ideographic model for data for the individual accumulate. We hypothesized that beginning with a nomothetic model would maximize accuracy during the early phase of data collection (referred to as a “warm start”) and subsequently, switching to ideographic modeling for higher personalization and performance would be the most effective approach to maximizing accuracy throughout data collection.


Overview

This paper used the data collected in an observational study of 79 participants who were followed for up to 1 year, starting between January and July 2014; the study continuously and objectively monitored the physical activity of the participants by actigraphy and the perceived stress rating through ecological momentary assessment (EMA) reports on a mobile app [6]. Participants were healthy individuals, aged 18 years or older, who responded to fliers posted throughout the buildings of Columbia University Medical Center (New York City, NY) and who, on phone screening, reported only intermittent engagement in exercise and having access to a personal computer and iOS or Android smartphone. Individuals with significant medical comorbidities, occupational work demands requiring rigorous physical activity, or inability to read and speak English were excluded. During a baseline interview, demographic characteristics including age, sex, race, ethnicity, education, partner status, and living situation were collected.

Measures

Stress was measured using an end-of-day text message survey on the participant’s own iPhone or Android phone, with the question “Overall, how stressful was your day?” Each evening, the participant was also asked, “Overall, how stressful do you think tomorrow will be?” Each morning, the participant responded to the questions “How stressful do you expect today to be?” and “How likely are you to exercise today?” All responses were rated on a scale from 0 (not at all) to 10 (extremely). All surveys were administered using Qualtrics software (Qualtrics, Seattle, WA). Two participants’ data were excluded for almost no variance in the self-reported stress ratings, leading to a total of 77 participants for the analysis.

Physical activity was measured using a wrist-worn Fitbit (Fitbit, Inc, San Francisco, CA) to track daily physical activity, including the steps taken, calories burned, and intensity of physical activity for each minute of the day. Participants were instructed to sync and charge the device every 5 to 7 days. In this analysis, a bout of “exercise” was defined as any consecutive 30-minute period within which 24 or more minutes of moderate- or vigorous-intensity activity was performed. We followed the recommendations of Ward et al [7] regarding best practices for the use of accelerometer data in research on physical activity. Specifically, physical activity guidelines recommend exercising for at least 30 minutes a day while accommodating interruptions. Further, when analyzing accelerometer data, the conventional approach is to quantify exercise in bouts of 10 minutes with allowances for 2 minutes of interruption (ie, total exercise for 8 of 10 min). Extrapolating the definition based on 10 to 30 minutes of activity, this yielded 24 of 30 minutes of activity. Software was written to determine, for each day, whether there was any 30-minute period within which at least 24 minutes of moderate or vigorous activity was performed; this was our objectively assessed measure of a 30-minute period of exercise. Days in which the Fitbit device was worn for fewer than 10 hours were excluded from all analyses.

External and environmental variables, including temperature (high, low, average, and range), hours of daylight, precipitation, and day of the week, were retrieved from the meteorological station in Central Park (New York City, NY); these data are made publicly available by the National Oceanic and Atmospheric Administration’s National Center for Environmental Information.

Statistical Analysis

We developed models for stress rating using neural network, decision tree, and linear approaches across all participants, as well as participant-specific decision trees. Each model predicted a person’s self-reported stress rating (range, 0-10), using the previous 3 days of weather, self-reported stress, and actigraphy data. Because the previous 3 days were used as inputs, periods missing a stress rating were excluded.

The continuous prediction for stress was then converted into a binary classification as either above or below a participant’s median self-reported stress value in the training set. We chose this approach, because if the stress rating was left as a continuous value, it was not clear at what rating an intervention should be initiated. Further, using the participant’s median as a reference slightly adjusts for participants who did not utilize the full range of the 11-point scale. In this way, the high-stress rating was individualized for each participant. The Keras Python library [8] was used to train neural network models, and the scikit-learn Python library [9] was used to train linear and decision tree-type models. Dropout, a regularization method for neural networks, was also used in the neural network models during training to decrease overfitting.

Model performance was compared based on regression (mean absolute error) and classification (area under the curve [AUC], F1 score, accuracy) error in predicting stress self-report. Feature importances were also compared between models. The AUC of a classifier is equivalent to the probability that the classifier will rank a randomly chosen positive instance

higher than a randomly chosen negative instance [10]. A model with an AUC closer to 1 is generally better. An F1 score is the harmonic mean of a model’s precision and recall, with values between 0 and 1, in which values closer to 1 are better. Model selection was performed through exhaustive grid search of the corresponding hyperparameters for each model using 3-fold cross-validation and optimizing on the mean absolute error.

For the ideographic decision trees, the Gini importance, as implemented in scikit-learn, was used to derive the significance of each input variable for each participant. Layer-wise relevance propagation (LRP) [11] was used to interpret our neural network models. LRP propagates the relevance of each input variable back through the network from a specific prediction. In other words, for this dataset, a participant with 30 valid days in the dataset would yield 30 sets of LRP relevance scores, each set having one score for each input variable. These scores can be positive or negative in a similar fashion as linear coefficients, and the greater the magnitude of the score, the more that particular variable increased or decreased that particular prediction for stress rating. LRP was used in favor of other methods such as Deep Learning Important Features (DeepLIFT) and Integrated Gradients, because it does not require a baseline or reference value. However, LRP can be viewed as an approximation of DeepLIFT when bias terms are 0 and the reference values are set to 0 [12]. This, in turn, approximates Shapley values [13], which is another method of interpreting neural network output. As such, to leverage the visualizations for Shapley values built in the Shapley Additive Explanations Python library and to add an additional method of visual interpretation, bias terms for the neural net were locked to 0; this did not impact model performance.

The following variables were used for the ideographic models: the number of hours with ≥10 activities, total exercise duration in minutes during bouts of exercise with ≥24 minutes of activity out of 30 minutes, the number of exercise bouts with ≥24 minutes of activity out of 30 minutes, the binary presence of any exercise bout of ≥24 minutes of activity out of 30 minutes, the total number of exercise bouts, the binary presence of any exercise bout, total exercise duration, stress rating, minimum outdoor temperature, maximum outdoor temperature, average outdoor temperature, total daylight minutes, and total precipitation. Several person-level variables were included in the nomothetic models that were not included in the per-participant models, as they would have been static for a given participant and would not have contributed to performance. These included height, weight, age, and sex. The inclusion of these variables in the nomothetic models slightly improved their performance.

Additionally, all models were trained with varying training set sizes to test the hypothesis that ideographic models may be more reliable at large-enough training set sizes for each participant. Training set sizes ranged from 30% to 80% of valid days for each participant. For example, the first 30% of days in which a participant documented his or her stress was used to train a model predicting the subsequent 70% of days. The results for an 80% train/20% test split are highlighted here, with a total of 4050 training set samples and 678 testing set samples available to all nomothetic models.

All models were compared to a baseline model that simply predicted the median of the stress self-report values from a participant’s training set, for all test samples.


Overview

The average age of the participants was 32 years (range, 20-58 years), with a height of 169 cm and weight of 75 kg. The study sample included 34 men and 43 women in the dataset. Table 1 depicts the basic descriptive statistics on the age, height, and weight of our set of participants.

The pattern of stress ratings differed significantly among participants. Figure 1 shows plots of stress by day for the 10 participants with the greatest number of valid responses in the dataset. Days without responses are shown without points plotted.

Table 1. Basic summary statistics for the participants (N=77).
StatisticMean (SD)Min25%a50%a75%aMax
Age (years)31.62 (9.42)2024273858
Height (cm)168.71 (8.49)149.4162168176187
Weight (kg)75.68 (17.46)43.362.574.785.5138.6

aQuartile ranges.

Figure 1. Plots of stress ratings for the participants with the greatest amount of responses.
View this figure
Table 2. Summary statistics and count of stress ratings, grouped by occurrence of missing stress ratings for the 3 days before the date of the predicted stress rating.
ResponseCountMean stress (SD)

Min25%a50%a75%aMax
00017283.60 (2.58)023610
00110773.40 (2.57)013510
0102923.36 (2.63)013510
0119733.31 (2.51)013510
1009733.16 (2.54)013410
1012393.35 (2.63)013510
1108893.17 (2.48)013410
11140602.93 (2.35)013410

aQuartile ranges.

Table 3. Percentage of stress ratings grouped by the occurrence of missing stress ratings for the 3 days before the date of the predicted stress rating. Each column corresponds to the value of the stress rating, and each cell represents what percentage of stress ratings had that value and response pattern.
Response012345678910
00011.9212.2114.4714.4715.745.738.918.394.401.502.26
00113.6512.2616.0616.3412.916.418.736.412.791.582.88
01017.479.9313.3617.1213.706.167.887.193.420.003.77
01114.5911.5114.5920.4511.517.097.715.963.190.412.98
10017.5713.0513.7714.8016.146.066.376.372.260.722.88
10117.5710.8813.3915.9013.815.026.699.215.020.422.09
11016.2011.2515.7518.6714.405.297.315.292.250.792.81
11115.6215.1217.0218.9213.425.914.983.892.090.642.39

Table 2 shows the mean stress rating grouped by the occurrence of missing stress ratings for the 3 days before the date of the predicted stress rating. The response column describes the occurrence of stress ratings: 000 represents no stress ratings in any of the 3 days, 001 represents presence of only one stress rating recorded just before the predicted day, and 111 represents presence of all three stress ratings. Periods missing a stress rating were more often followed by a higher stress rating than periods that were not missing any stress ratings.

Table 3 shows the percentage of stress ratings grouped by the occurrence of missing stress ratings for the 3 days before the date of the predicted stress rating. The frequency of high-stress ratings is slightly higher for response patterns with missing stress ratings.

Accuracy of Nomothetic Versus Ideographic Models

With an 80% training set/20% testing set split, the nomothetic recurrent neural network model AUC was 74.20% and the F1 score was 79.21%. In addition, the per-participant decision tree AUC was 0.67 and the F1 score was 0.83. The full results can be found in Multimedia Appendix 1. Nomothetic models, on an average, outperformed ideographic models for shorter training set sizes. As the training set size increased (ie, allowing for longer duration of assessments), group-level model performance generally increased across all models, and the nomothetic and ideographic model performances converged. However, when separately applying models to individual participants, ideographic models generally outperformed the nomothetic ones once the days with valid data exceeded 50 days.

Table 4 lists performance metrics for each model, grouped by training set proportion and model used. Precision, recall, F1 score, AUC, and accuracy are performance metrics. Training set size ranges from 30% to 80%. “Ideographic decision tree over 100 days” represents the performance metrics, only for participants who had at least 100 valid days, which is the same for the 50-day model.

Table 4. Performance metrics across all models, grouped by training set size.
Average number of days for training set sizePrecision (%)F1 score (%)Area under the curveAccuracy (%)Training set size (%)Model
16.721574.2271.9553.3960.6930.00Ideographic Decision Tree
16.721575.7672.6354.4061.5330.00Ideographic Decision Tree Over 50 Days
16.721576.6171.8652.7360.2130.00Ideographic Decision Tree Over 100 Days
16.721577.8078.7762.8869.7530.00Random Forest
16.721578.8378.7264.2570.1030.00Gradient Boosted Decision Tree
16.721579.9077.9365.3269.6930.00Recurrent Neural Network
16.721581.0972.6964.7165.1830.00Baseline
16.721581.2081.0368.2173.3630.00Elastic Net
16.721582.4177.3968.0070.0130.00Neural Network
21.759575.3572.4054.4461.3740.00Ideographic Decision Tree Over 100 Days
21.759576.4873.0657.1262.7940.00Ideographic Decision Tree
21.759576.9672.9656.3262.3740.00Ideographic Decision Tree Over 50 Days
21.759577.8777.0962.0867.9640.00Gradient Boosted Decision Tree
21.759578.1579.2363.1670.2540.00Random Forest
21.759580.9371.1163.7463.6140.00Baseline
21.759581.4776.2566.2368.4740.00Recurrent Neural Network
21.759581.7080.1568.1572.4840.00Elastic Net
21.759582.3379.4068.5571.9040.00Neural Network
27.329175.7675.3652.4163.4750.00Ideographic Decision Tree
27.329177.0975.9552.5264.0650.00Ideographic Decision Tree Over 50 Days
27.329177.2775.5650.1063.0750.00Ideographic Decision Tree Over 100 Days
27.329182.4182.6567.5474.6050.00Random Forest
27.329182.4581.9467.2973.8050.00Gradient Boosted Decision Tree
27.329184.0282.7169.7975.1550.00Recurrent Neural Network
27.329184.8176.3167.8968.5750.00Baseline
27.329185.8483.8972.7377.0050.00Elastic Net
27.329188.7683.0975.6976.7950.00Neural Network
32.721578.5476.9954.6365.5960.00Ideographic Decision Tree
32.721578.8577.4553.4065.7760.00Ideographic Decision Tree Over 100 Days
32.721579.4177.6954.8766.3460.00Ideographic Decision Tree Over 50 Days
32.721585.9083.7369.1375.7560.00Random Forest
32.721586.5376.0566.8367.3660.00Baseline
32.721588.7683.4973.2476.1960.00Gradient Boosted Decision Tree
32.721589.8085.1975.6278.4760.00Elastic Net
32.721590.7882.2075.2075.1560.00Recurrent Neural Network
32.721591.3580.7375.0873.6260.00Neural Network
38.164680.6980.9062.8771.6570.00Baseline
38.164684.5282.5067.1774.0970.00Ideographic Decision Tree
38.164684.6483.7869.0675.9670.00Ideographic Decision Tree Over 50 Days
38.164687.3585.7174.6979.2370.00Random Forest
38.164687.8383.8174.2377.0870.00Gradient Boosted Decision Tree
38.164688.5784.2175.3677.7270.00Elastic Net
38.164688.6088.9976.7683.2570.00Ideographic Decision Tree Over 100 Days
38.164689.1680.6974.0773.8870.00Recurrent Neural Network
38.164689.5782.7675.6976.2870.00Neural Network
43.202581.4481.7963.9472.8180.00Baseline
43.202584.1083.0067.7274.8280.00Ideographic Decision Tree
43.202588.3190.0776.8084.6280.00Ideographic Decision Tree Over 50 Days
43.202589.0084.2075.6377.7180.00Elastic Net
43.202589.0083.4375.1876.8080.00Gradient Boosted Decision Tree
43.202589.2584.7076.2278.3580.00Random Forest
43.202590.3878.9174.2072.1680.00Recurrent Neural Network
43.202590.5790.7879.8785.8280.00Ideographic Decision Tree Over 100 Days
43.202590.8973.8772.0667.2780.00Neural Network

Predictors of Stress

In our ideographic models, there was significant heterogeneity in the effect of features of weather and exercise. Figure 2 shows a series of horizontal bar plots, visualizing the Gini importance values, or the factors that significantly predicted day-to-day variability in stress rating for the 9 individuals with at least 100 valid days of data.

The most frequent important predictors for each participant included daylight minutes, temperature, and exercise behavior for the current or preceding 3 days. Often, variables that were assessed closer (temporally) to the stress rating being predicted exhibited greater importance.

Viewing feature importance in aggregate for all the participants in the dataset, the Gini importance values from their corresponding decision tree model were sorted and then ranked. Thereafter, the number of times that each input variable ranked in the top 5 spots across participants’ rankings was retrieved to create Figure 3. Decision tree models appear to have most frequently and highly ranked exercise-related features, followed by environmental variables.

Figure 4 depicts the relevance scores for the recurrent neural network model, which were derived using LRP. Like the decision tree models, the scores ranged widely, even among participants. To retrieve a metric more comparable with the strictly positive Gini importance values derived from the decision trees, the absolute value of the LRP scores was taken for each participant, followed by the median value for each input variable. The neural network model often ranked the highest, preceding stress self-reports, but there was also significant variety across features such as weather (eg, average or minimum temperature on the day of the stress rating), exercise (eg, number of minutes of exercise or number of exercise bouts the previous day), age, height, and weight.

In Figure 5, as with the decision tree models, the relevance of each input variable was ranked and then counted across participants.

Figure 6 depicts the F1 score, AUC, and accuracy for the nomothetic recurrent neural network model and the ideographic decision trees across multiple training set sizes. For individuals with fewer than 50 valid days, the nomothetic models generally outperformed or performed comparably as the ideographic ones. Nomothetic model performance and ideographic model performance converged as training set size per participant increased. However, for the 16 individuals with more than 50 valid days and the 9 individuals with over 100 valid days, ideographic performance overtook nomothetic performance. Each dot corresponds to a training set size proportion, ranging from 30% to 80% in increments of 10%. The number of days included in the training set for each proportion across all participants, on an average, was as follows: 17 days for 30%, 22 days for 40%, 27 days for 50%, 33 days for 60%, 38 days for 70%, and 43 days for 80%.

Using LRP for the nomothetic model, a per-sample feature importance method, allows us to gain insight on what influenced a participant’s predicted stress score for a given day. In Figure 7, a particular participant’s actual stress self-reports, predicted stress self-reports, outside temperature, and the neural network model’s LRP values are depicted across several consecutive days. On day 5, the neural network model assigned a high importance to the average temperature. As shown in the figure, as the average temperature increased (temperature here is standardized to fit on the same scale as the other values), predicted stress and the true stress value increased. Note that in the following graphs, the left-hand y-axis contains the scale for the input variable and the LRP value, and the right-hand y-axis contains the scale for both predicted and true stress rating. The x-axis simply denotes consecutive dates, marked as integers, rather than true dates.

Figure 2. Most important predictors for the ideographic decision tree models for the 9 individuals with at least 100 valid days of data.
View this figure
Figure 3. A horizontally stacked bar chart of occurrences of the most frequently appearing predictor variables, and how often they ranked in the top 5 spots across participants’ predictor variable importance rankings from the ideographic models.
View this figure
Figure 4. A horizontal bar chart of the 15 predictor variables with the highest median absolute layer-wise relevance propagation scores. LRP: layer-wise relevance propagation.
View this figure
Figure 5. A horizontally stacked bar chart of occurrences of the most frequently appearing predictor variables, and how often they ranked in the top 5 spots across participants’ predictor variable importance rankings from the nomothetic recurrent neural network model.
View this figure
Figure 6. A plot comparing the F1 score, area under the curve, and accuracy across the neural network model and the ideographic models. Training set size varies on the x-axis, and performance of participants with more than 50 valid days and more than 100 valid days is shown separately for the ideographic models. The y-axis is the same in all 3 figures.
View this figure
Figure 7. A plot of the actual stress rating, predicted stress rating, layer-wise relevance propagation value, and average temperature for an individual participant, with average temperature standardized to fit on the same graph. LRP: layer-wise relevance propagation.
View this figure
Figure 8. A plot of the actual stress rating, predicted stress rating, layer-wise relevance propagation value, and average temperature for an individual participant, with the average temperature standardized to fit on the same graph. LRP: layer-wise relevance propagation.
View this figure

For another participant represented in Figure 8, an increase in the maximum temperature on a particular day was associated with a decrease in predicted stress, demonstrating individual-specific response patterns to environmental variables.

Exercise, as measured through total minutes or number of separate bouts, was often inversely related with stress rating, where less exercise increased the predicted stress rating (Figure 9).

However, there is a risk when interpreting feature importance presented as a time series. Although a variable may carry a high positive influence for a given day, it may be outweighed by a high cumulative negative impact of other variables, causing an improper inference. As such, it is useful to visualize a prediction for a given day not as a time series, but as a force plot of contributions from different variables, allowing us to more easily disentangle their influences. Treating LRP values as approximations for Shapley values, we can retrieve the following force plot depicted in Figure 10 of feature importance for a specific prediction.

Here, the predicted stress self-report value is 1.07. The stress rating 2 days ago of 2 and the stress rating 3 days ago of 6 push the prediction downward. The minimum temperature of 35°F 3 days ago and the absence of any exercise bouts of greater than 24 minutes of moderate-to-vigorous physical activity out of a 30-minute period push the prediction upward.

We can also aggregate these force plots across a set of predictions to understand how our nomothetic neural network model behaves more generally. In Figure 11, each horizontal line displays the LRP values for an input variable, across the entire testing set. The variables are sorted by the sum of the magnitudes of their values, in descending order. The figure shows that higher stress ratings preceding a predicted stress rating typically increase predicted stress, whereas lower minimum temperatures typically increase predicted stress.

Figure 9. A plot of the actual stress rating, predicted stress rating, layer-wise relevance propagation value, and exercise minutes for a participant, with exercise minutes standardized to fit on the same graph. A lower value for exercise minutes on the previous day was associated with a high layer-wise relevance propagation value and a higher predicted stress rating. LRP: layer-wise relevance propagation.
View this figure
Figure 10. A force plot visualizing the contributions of different input variables to a predicted stress rating for a single participant.
View this figure
Figure 11. Layer-wise relevance propagation values for each input variable across the entire testing set. Each horizontal line corresponds to a single input variable. LRP: layer-wise relevance propagation.
View this figure

Overview

Many studies have examined relationships among exercise, weather, and stress using both linear and nonlinear approaches, and a mix of self-report questionnaire responses and automatically gathered sensing data. Some have taken an exclusively nomothetic approach; Wang et al [1] used both EMA data and automatically sensed activity and sociability data to explore correlates of stress, depression, and loneliness, but did not use a personalized machine learning approach or investigate individual predictors. Using a large set of meteorological data paired with responses to a self-report questionnaire and a mixed modeling approach, Beecher et al [2] found that increased sun exposure decreased reported distress.

In contrast, other studies have taken an ideographic approach. Tuarob et al [4] used a variety of machine learning techniques and questionnaire data to train ideographic models predicting participant mental states but relied on self-reported input data and did not investigate predictors for each individual. Sharmin et al [14] used sensor data and self-reported stress reports to create personalized visualizations that were then manually examined for temporal trends in stress. Plarre et al [5] trained ideographic decision trees using electrocardiographic and respiration-related data to predict self-reported stress after performing activities in the laboratory setting, such as public speaking or mental arithmetic. Burg et al [6] analyzed the same EMA and exercise data as those used here and estimated ideographic random coefficient mixed models; they found that the influence of exercise on self-reported stress was heterogeneous, as was the effect of self-reported stress on exercise.

Finally, taking an integrative approach and using the same dataset as that used by Burg et al [6] and us, Cheung et al [15] compared the performance of ideographic and nomothetic methods to predict whether an individual will exercise on a given day, again showing that for some, but not all participants, self-reported stress was a predictor of exercise.

In this paper, heterogeneity in the effects of predictors on stress was confirmed, highlighting the value of using an ideographic modeling approach. Further, it was demonstrated that the nomothetic model performs better (ie, is more accurate) than the ideographic model initially, but as data collected accumulates, the performance of the ideographic model equals and then surpasses that of the nomothetic model, providing a motivating example for a “warm start” strategy to leverage the advantages of each modeling approach. Put into practice, one might continuously monitor the performance of a nomothetic model and an ideographic model on a particular individual and adjust the weight of each model’s contribution to the predicted output accordingly.

Retrieving individual predictors yields hypotheses that we can test for a particular participant and, perhaps, act on. For example, if a clinician repeatedly sees that low average temperatures are driving higher predicted stress values, they may be able to recommend a particular intervention for the patient. Of course, not all situations are so easily interpretable, and the relationship of a predictor to the outcome may not be linear. This is both an advantage and a disadvantage, allowing a model to potentially be flexible to multiple climates but limiting the ease of generalizability to other participants, and requires either manual inspection as mentioned above or a more rigorous and automated method of consistent linear predictor detection.

Here, the LRP values from our nomothetic neural network model, and Gini importance values from our ideographic decision trees, suggested different predictors for each individual. The nomothetic neural network model gave preceding stress self-reports higher weight on an average (Figure 10). This may be a result of training the neural network using dropout, in which increasing the weight given to preceding stress reports was an optimal way to manage the heterogeneity of participant response patterns. Notably, Sarker et al [16] reported similar results that stress episodes increase the likelihood of subsequent episodes, although these were within-day data.

Although Figure 4 presents a small sample of individuals and a relatively small dataset overall, it depicts the LRP values for the nomothetic neural network model and motivates discussion of real-world applications of such a model for just-in-time predictions. From this, it seems likely that it is effective to start with a nomothetic model to maximize predictive performance and gain initial insights into the possible correlates of stress before switching to (or increasing the weight of, with an ensemble-type implementation) an ideographic model once enough responses have been collected. This assumes that immediate interventions are necessary, valuable, and worth the cost of potentially introducing bias in the dataset for that individual. Alternatively, predictions could be withheld until sufficient data are collected for ideographic models to be used, but this might come at the expense of disengaging participants during the process of data collection. More generally, adopting a framework, as proposed by Nahum-Shani et al [17], to specify specific proximal outcomes while managing participant engagement is prudent.

Future Directions and Limitations

There is no current industry or academic standard for retrieving feature importance values from neural networks. Layer-wise relevance propagation, the method used here, has drawbacks of not meeting certain axiomatic properties of ideal feature importance methods [18]. In this case, however, LRP was used because it does not require a baseline, whereas other methods require some reference input for comparison. This instance is not one in which it is immediately clear what that input would be. Other model-agnostic, sensitivity-based approaches exist, such as Locally Interpretable Model Explanations [19]. Regardless of the approach for retrieving feature importance values, we retrieved per-participant predictors from the nomothetic neural network model by taking the median of the absolute value of LRP scores across a subset of the highest-stress events in a participant’s test set. Other strategies may yield features that are more representative of an individual’s stress. Further research in the field may be required to elaborate on strengths and weaknesses of different approaches in terms of interpretability, generalizability, and suitability for inclusion in a clinical decision process.

Next, in this setting, individuals who deviate significantly from the rest of the population may create large gradient updates to a neural network, potentially reducing performance for other individuals. Although this can be mitigated through techniques such as batch normalization and dropout, these strategies, in turn, reduce the degree to which the model can closely fit the data and obscure important differences among individuals. Similarly, individuals contributing a larger number of samples to the training set can bias the model. If an exhaustive hyperparameter search is performed without regularization strategies like L1/L2 penalties, max-norm constraints, dropout, or early stopping, the model may be further overfit to these individuals. As a result, some settings might still be best served by an ideographic approach for safety, but even N-of-1 decision trees may need to be constrained in their complexity to prevent overfit, especially while still acquiring data.

With self-report values, similar to item ratings, variable distributions may be skewed for particular individuals based on their perception of the scale. For example, in this dataset, some individuals never reported their stress to be above a value of 7, despite the scale going up to 10. Examples of these differing response patterns can be found in Figure 1. It is also highly unlikely that responses are missing at random. In fact, these days may be more stressful than the ones reported. The same issue may exist with Fitbit nonwear data. Time series forecasting methods often rely on either excluding or interpolating time windows that have missing data. Here, we do not interpolate missing data and instead, choose to exclude them. As a result, in situations in which self-report data are used, individual response patterns should be taken into account, whether through techniques such as feature engineering or increased data collection.

In our analyses, samples with missing stress ratings prior to the predicted stress rating were excluded. Based on the observed difference in distributions between stress ratings preceded by missing stress ratings, and ones that were not preceded by such ratings, it is possible that the ratings may not be missing at random. Instead, unobserved stress ratings may be indicative of higher stress in some patients. As a result, we primarily explored imputation using a dummy value, but models trained using these values severely underperformed, likely due to the relatively small size of the dataset; therefore, those results were not reported here. Alternatively, missing stress ratings could be imputed using a rule-based approach such as incrementing the most recently observed stress rating. However, for a practical implementation, we believe that with sufficient dataset size, missing stress ratings could be replaced by dummy values to avoid manually biasing predictions and improve generalization. Further, current Fitbit devices and other accelerometers provide continuous heart rate monitoring data, which may provide additional useful predictors and mitigate the effect of missing stress ratings.

Finally, as confidence intervals were not retrieved, we lacked a measure of certainty per prediction. This could be alleviated by using Bayesian neural network or dropout-based methods.

Conclusions

Through the combination of a nomothetic neural network model, recent advances in retrieving per-sample feature importance, and ideographic decision trees, we show that high predictive performance can be achieved while recognizing individual differences and surfacing personalized predictors of stress. Key predictors in the nomothetic models were typically related to recent stress experience and weather activity. In addition, key predictors in the ideographic models displayed significant heterogeneity but were often weather or exercise related for individuals from whom more data were collected. Environmental variables were also shown to affect stress differently in different participants; for example, high temperatures predicted high stress in one individual but low stress in another. These predictors can be used to provide individuals with insights into what may contribute to their stress, as indicated by Yoon et al [20]. These models can also be operationalized to generate interventions or encouragements just before instances of high stress when the model predicts, with a sufficient degree of confidence, impending stress based on what is expected to be effective for that particular individual. Finally, ideographic models surpassed a nomothetic one after sufficient data collection, supporting the use of a “warm start” model-switching approach. Further work is needed to explore interpretable and repeatable ways to assess personalized predictors in nonlinear settings, as applied to disentangling correlates of stress.

Acknowledgments

This work was supported by funding (R01-HL115941) from the National Heart, Lung, and Blood Institute at the National Institutes of Health. IK was also supported by the National Center for Advancing Translational Sciences (UL1TR001873).

Conflicts of Interest

None declared.

Multimedia Appendix 1

Performance metrics for each model.

PDF File (Adobe PDF File), 90KB

References

  1. Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S. StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones. 2014 Presented at: UbiComp: ACM International Joint Conference on Pervasive and Ubiquitous Computing; Sep 13-17, 2014; Seattle, WA. [CrossRef]
  2. Beecher ME, Eggett D, Erekson D, Rees LB, Bingham J, Klundt J, et al. Sunshine on my shoulders: Weather, pollution, and emotional distress. J Affect Disord 2016 Dec 15;205:234-238. [CrossRef] [Medline]
  3. Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2009.
  4. Tuarob S, Tucker CS, Kumara S, Giles CL, Pincus AL, Conroy DE, et al. How are you feeling?: A personalized methodology for predicting mental states from temporally observable physical and behavioral information. J Biomed Inform 2017 Dec;68:1-19 [FREE Full text] [CrossRef] [Medline]
  5. Plarre K, Raij A, Hossain S, Ali A, Nakajima M, Al-absi M, et al. Continuous inference of psychological stress from sensory measurements collected in the natural environment. 2011 Presented at: Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks; April 12-14, 2011; Chicago, IL.
  6. Burg MM, Schwartz JE, Kronish IM, Diaz KM, Alcantara C, Duer-Hefele J, et al. Does stress result in you exercising less? Or does exercising result in you being less stressed? Or is it both? Testing the bi-directional stress-exercise association at the group and person (N of 1) level. Ann Behav Med 2017 Dec;51(6):799-809 [FREE Full text] [CrossRef] [Medline]
  7. Ward DS, Evenson KR, Vaughn A, Rodgers AB, Troiano RP. Accelerometer use in physical activity: best practices and research recommendations. Med Sci Sports Exerc 2005 Nov;37(11 Suppl):S582-S588. [Medline]
  8. GitHub. 2018. Keras team   URL: https://github.com/keras-team/keras [accessed 2018-11-27] [WebCite Cache]
  9. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O. Scikit-learn: Machine learning in Python. JMLR 2011;12:2825-2830.
  10. Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters 2006 Jun;27(8):861-874. [CrossRef]
  11. Bach S, Binder A, Montavon G, Klauschen F, Müller K, Samek W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 2015 Jul;10(7):e0130140 [FREE Full text] [CrossRef] [Medline]
  12. Shrikumar A, Greenside P, Kundaje A. Learning Important Features Through Propagating Activation Differences. 2017 Presented at: Proceedings of the 34th International Conference on Machine Learning; August 6-11, 2017; Sydney, Australia p. 3145-3153.
  13. Lundberg S, Lee S. A unified approach to interpreting model predictions. 2017 Presented at: Neural Information Processing Systems; Dec 4-9, 2017; Long Beach, CA.
  14. Sharmin M, Raij A, Epstien D, Nahum-Shani I, Beck J, Vhaduri S. Visualization of time-series sensor data to inform the design of just-in-time adaptive stress interventions. 2015 Presented at: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing; Sep 07-11, 2015; Osaka, Japan.
  15. Cheung Y, Hsueh P, Qian M, Yoon S, Meli L, Diaz KM, et al. Are Nomothetic or Ideographic Approaches Superior in Predicting Daily Exercise Behaviors? Methods Inf Med 2017;56(6):452-460 [FREE Full text] [CrossRef] [Medline]
  16. Sarker H, Hovsepian K, Chatterjee S. From markers to interventions: the case of just-in-time stress intervention. In: Mobile Health: Sensors, Analytic Methods, and Applications. New York: Springer; 2017:411-433.
  17. Nahum-Shani S, Smith S, Tewari A, Witkiewitz K, Collins L, Spring B. Just-in-Time Adaptive Interventions (JITAIs): An Organizing Framework for Ongoing Health Behavior Support (Technical Report No. 14-126). University Park, PA: The Methodology Center; 2014.
  18. Sundararajan M, Taly A, Yan Q. Axiomatic Attribution for Deep Networks. In: ICML. 2017 Presented at: ICML; June 10-15, 2017; Long Beach, CA.
  19. Ribeiro M, Singh S, Guestrin C. "Why Should I Trust You?": Explaining the predictions of any classifier. 2016 Presented at: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 13-17, 2016; San Francisco, CA. [CrossRef]
  20. Yoon S, Schwartz J, Burg M, Kronish IM, Alcantara C, Julian J, et al. Using Behavioral Analytics to Increase Exercise: A Randomized N-of-1 Study. Am J Prev Med 2018 Dec;54(4):559-567 [FREE Full text] [CrossRef] [Medline]


AUC: area under the curve
EMA: ecological momentary assessment
LRP: layer-wise relevance propagation


Edited by G Eysenbach; submitted 28.11.18; peer-reviewed by E Hekler, C Fernández; comments to author 03.01.19; revised version received 24.01.19; accepted 03.03.19; published 26.04.19

Copyright

©Alan Rozet, Ian M Kronish, Joseph E Schwartz, Karina W Davidson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 26.04.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.