Samples 2 and 3 used a mood monitoring control condition through the CARE app. Participants assigned to the mood monitoring control condition were prompted to report on state mood and rumination 3 times a day without the chance of receiving a mindfulness intervention. After receiving a notification to use the app, participants reported on their current mood (ie, sad, anxious, happy, and calm) and state rumination on a scale from 0 (not at all) to 100 (extremely). The work by Hilt et al [48] provides more information.

Samples 1, 2, and 3 used a mindfulness meditation condition through the CARE app. Participants assigned to the mindfulness condition answered the same mood and rumination questions as those assigned to the mood monitoring control condition, and they also received mindfulness exercises. To prevent participants from learning which responses would result in receiving a mindfulness exercise, there was a 67% probability of receiving an exercise each time the app was used. If participants indicated high levels of anxiety or sadness (ie, ≥90 on the 0-100 scale), their chances of receiving a mindfulness exercise increased to 85%. If prompted to receive a mindfulness exercise, participants could select how much time they had to complete the exercise (ie, approximately 1, 5, or 10 minutes), and an exercise was randomly assigned within those parameters. After completing the mindfulness exercise, participants reported again on their current mood and rumination. The work by Hilt et al [48] provides more details.

Sample 4 used the 4-week Foundations program from the HMP mobile app. HMP includes meditation practices linked with 4 dimensions of well-being: awareness, connection, insight, and purpose [55]. Briefly, awareness practices aim to cultivate mindfulness and attention regulation, connection practices aim to cultivate healthy relationships with oneself and others, insight practices aim to cultivate an understanding of how our internal experiences (eg, emotions and thoughts) shape our wellbeing, and purpose practices aim to cultivate a connection with one’s values and a sense of meaning in daily life. The works by Goldberg et al [56] and Hirshberg et al [57] provide more details. Within this study, participants were randomly assigned to use HMP for either 5 or 15 minutes per day (defined as low- and high-dose conditions, respectively). In addition to using HMP, participants were sent EMAs assessing various aspects of well-being (awareness, connection, insight, and purpose dimensions), psychological distress (depression and anxiety), stressor exposure, and rumination 4 times per day over the 4-week intervention period.

For adolescents (samples 1 and 2), depressive symptoms were assessed using the Children’s Depression Inventory (CDI) [58]. The CDI is a 27-item measure of depression adapted for children and adolescents from the Beck Depression Inventory [12]. The CDI assesses the frequency and severity of depressive symptoms over the previous 2 weeks. Participants report on depressive symptoms and their severity using a 4-point scale, with higher scores on the CDI indicating greater frequency and severity of symptoms. Previous research demonstrates that the CDI is reliable and valid in child and adolescent samples [59,60]. In sample 1, the CDI showed good reliability at baseline (Cronbach α=0.82), the postintervention time point (Cronbach α=0.86), the 6-week follow-up (Cronbach α=0.88), and the 12-week follow-up (Cronbach α=0.91). In sample 2, the CDI showed excellent reliability at baseline (Cronbach α=0.90), the postintervention time point (Cronbach α=0.92), the 6-week follow-up (Cronbach α=0.91), and the 12-week follow-up (Cronbach α=0.92).

The Beck Depression Inventory–II (BDI-II) [12] was used to assess depressive symptoms in college students (sample 3). The BDI-II is a 21-item measure that asks participants to self-report on depressive symptoms experienced in the previous 2 weeks using a 4-point scale. Higher scores on the BDI-II indicate greater symptom frequency and severity. The BDI-II has demonstrated validity and reliability in college samples [61]. The measure demonstrated excellent reliability at baseline (Cronbach α=0.94), the postintervention time point (Cronbach α=0.93), the 6-week follow-up (Cronbach α=0.88), and the 12-week follow-up (Cronbach α=0.91). For samples 1 to 3 (all of which used the same CARE app), missingness for depression measures was 0.3% (1/320) at baseline and 6.9% (22/320) at the postintervention time point.

The computer-adaptive version of the PROMIS depression scale [62] was used to assess depressive symptoms in adults with elevated depression or anxiety (sample 4). The computer-adaptive PROMIS depression scale draws on a bank of 28 items. Participants rate their experience of various depression symptoms in the previous 7 days on a 5-point scale ranging from 1 (never) to 5 (always). The measure has shown strong convergent validity with legacy measures of depression, including the BDI-II [50]. As not all participants received the same items, internal consistency could not be calculated. However, short-form versions of the PROMIS depression scale have shown excellent reliability (eg, Cronbach α=0.97 for the 8-item PROMIS depression scale [63]). Missingness was very low for sample 4—0% at baseline and 3% (3/92) at the postintervention time point.

Adolescent self-reported trait rumination (samples 1 and 2) was assessed using the 13-item rumination subscale of the CRSQ [17]. For each item, participants are instructed to report on how they usually respond to feelings of sadness or stress. Instructions were modified to include stress in line with current conceptualizations of rumination [64]. In response to each item, participants indicate whether they respond in the way described by the item on a scale from 0 (almost never) to 4 (almost always). Higher scores indicate greater rumination. Previous research suggests that the CRSQ is reliable and valid in an adolescent sample [17]. In this study, the rumination subscale of the CRSQ demonstrated good reliability in sample 1 (Cronbach α=0.89) and excellent reliability in sample 2 (Cronbach α=0.92) at baseline. At the postintervention time point, the CRSQ showed excellent reliability in sample 1 (Cronbach α=0.90) and sample 2 (Cronbach α=0.92).

College student rumination (sample 3) was assessed using the Ruminative Response Scale (RRS) from the RSQ [65]. The RRS is a 22-item measure in which participants are asked to rate whether they generally respond as described by the item on a scale from 0 (almost never) to 4 (almost always). Higher scores indicate greater frequency and severity of rumination. Previous research in college student samples suggests that the RRS has good internal consistency and moderate test-retest reliability [66]. Similar to the CRSQ, instructions were adapted to ask participants to report on their response to sadness or stress [64]. In this sample, the RRS showed excellent reliability at baseline (Cronbach α=0.93) and follow-up (Cronbach α=0.94). For samples 1 to 3, missingness for rumination measures was 0% at baseline and 6.6% (21/320) at the postintervention time point.

Repetitive negative thinking in the adult sample with elevated depression or anxiety (sample 4) was assessed using the Perseverative Thinking Questionnaire (PTQ) [67]. It should be noted that repetitive negative thinking is a broader construct than rumination (not solely focused on past-oriented repetitive negative thoughts but also includes, eg, future-oriented worry and present concerns). The PTQ is a 15-item measure in which participants are asked how often they respond in a particular way on a scale from 0 (never) to 4 (almost always). Items query various forms of repetitive negative thinking (eg, “I keep thinking about the same issue all the time”). Similar to the CRSQ and RSQ measures, participants are asked to rate how they typically respond. Higher scores indicate greater frequency of repetitive negative thought. Internal consistency and test-retest reliability, along with convergent and predictive validity, have been established for adults [67]. In this sample, the PTQ showed excellent reliability at baseline (Cronbach α=0.94) and follow-up (Cronbach α=0.95). PTQ missingness for sample 4 was 0% at baseline and 3% (3/92) at the postintervention time point.

In samples 1, 2, and 3, state rumination was assessed by asking participants the following questions—“How much were you focusing on your emotions?” and “How much were you focusing on your problems?”—just before seeing the prompt to use the app. Participants rated the degree to which they had been ruminating as described by these questions on a scale from 0 (not at all) to 100 (extremely). These questions were created in line with previous research [68,69]. In the analyses below, we focus on the latter question (problem-focused rumination) to assess rumination given (1) some concerns about the extent to which focusing on emotions in fact assesses problematic rumination (eg, participants may direct greater attention to their emotions by virtue of mindfulness training) [7] (see also the study by Nolen-Hoeksema et al [64], which focused on problem-focused rumination); (2) the relatively modest correlation between problem-focused and emotion-focused rumination (r=0.41) in this sample, which raises questions regarding internal consistency; and (3) the fact that these 2 items have yielded different patterns of findings in previous work [7]. Multimedia Appendix 1 [70] provides additional analyses based on the emotion-focused item. Participants in samples 1 to 3 had a mean survey compliance of 74% (46.71/63; SD 14.52), 79% (50.02/63; SD 14.24), and 72% (45.45/63; SD 17.78) during the 3-week intervention period, respectively. There was a nonsignificant trend for week-to-week decreases in EMA compliance (β=−0.043; P=.10). Multimedia Appendix 1 provides analyses showing no significant differences in EMA compliance during COVID-19 lockdowns (only 1 sample included participants enrolled during COVID-19 lockdowns).

In sample 4, participants first completed a stressor exposure item that asked them to “Think about the most stressful or negative thing that happened since you completed the last survey” and then indicate how stressed they felt “at the worst point” on a scale from 1 (not at all) to 7 (very much). Rumination was then assessed using a subsequent item—“After the stressful or negative thing happened, I was dwelling on my mistakes, failures, or losses”—which was also rated on a scale from 1 (not at all) to 7 (very much). These items were also created in line with previous research [71,72]. EMA compliance was 80.2% (89.85/112; SD 20.6%) and decreased over time by 3.4 percentage points per week (b=-0.034; P<.001).

Before estimating the reliability of rumination change scores, we first report the reliability (internal consistency) of conventional retrospective self-reported rumination (for brevity, we refer to the latter simply as “retrospective rumination” in the following sections) at baseline (T1) and the postintervention time point (T2). Next, to compute the reliability of residualized change in retrospective rumination, we used the following formula, which has the reliability of retrospective rumination scores at T1 (α_T1) and T2 (α_T1) and the squared correlation coefficient () between these 2 scores [73] as inputs:

Similar to what was mentioned previously, before testing the association between change scores, we first tested the correlation between baseline (T1) scores on the retrospective rumination measure and mean EMA rumination over the initial 3 days of EMA. Similarly, we correlated retrospective rumination at the postintervention time point (T2) and mean EMA rumination over the previous 3 days. We selected a 3-day window for averaging in an effort to obtain a relatively representative estimate of an individual’s typical tendency to ruminate (see Multimedia Appendix 1 for analyses averaging over different numbers of days, which yielded similar findings). We also tested the association between change in retrospective rumination and EMA rumination from T1 to T2. To compute change in retrospective rumination, we saved the residuals from a model in which T2 retrospective rumination scores served as the dependent variable and T1 retrospective rumination was the predictor variable, creating a residualized change score. We focused on residualized change scores because they control for effects of regression to the mean, providing estimates of change that are independent of initial status (in addition to typically having slightly better reliability). Biostatisticians have often argued against using raw change scores for these and other reasons [79]. To compute change in EMA rumination, the slope of rumination change over the course of the trial (ie, 3 weeks or 21 days for samples 1, 2, and 3 and 4 weeks or 28 days for sample 4) was computed from participant-specific regressions of rumination scores on intervention day [74,75]. It should be noted that, as our focus in this analysis was solely on obtaining and interpreting participant-level estimates of systematic linear change over time, alternative multilevel reliability approaches were perceived as less relevant. However, the section on multilevel reliability in Multimedia Appendix 1 illustrates the application of such methods. Given the differences between samples 1 to 3 (adolescents and college students who all used the identical CARE mindfulness app for 3 weeks) and sample 4 (adults who used the HMP meditation app for 4 weeks), all analyses were conducted separately for the CARE (with sample included as a covariate in the analyses) and HMP studies (henceforth referred to as the CARE and HMP samples, respectively).

Given that the CARE sample included a mood monitoring control condition, we tested whether group differences emerged in change in rumination from T1 to T2. For retrospective rumination, we ran a regression predicting T2 rumination (adjusting for T1 rumination scores) with group as the predictor of interest. For EMA rumination, a robust linear mixed-effects model (LMM) using the statistical package robustlmm (version 3.1) in R (version 4.2.2; R Foundation for Statistical Computing) with a group × time interaction was specified. We specified random intercepts and slopes for participants with an unstructured covariance matrix for the random effects, allowing for correlation between random intercepts and slopes within individuals. The residual errors were assumed to have an independent covariance structure (conditional independence given the random effects). These covariance specifications are the defaults in the robustlmm package and were used consistently across all robust LMM analyses. A standardized effect size for the group × time interaction was estimated using the d_{growth-modeling analysis − raw} (β₁₁[time]/SD_raw) formula recommended by Feingold [80] (for brevity, d_{growth-modeling analysis − raw} is referred to simply as d in the following sections).

For the HMP sample, in which both groups had access the HMP app, we examined whether EMA rumination changed over time. To do this, we fit a robust LMM with EMA rumination as the dependent variable and time in days (ranging from 0 to 28) as the independent variable, with the nesting of observations within participants modeled using a random intercept and slope. A standardized effect size for the effect of time was estimated following the aforementioned d_{growth-modeling analysis – raw} formula.

Finally, to test whether change in traditional self-reported rumination or EMA rumination predicted improvement in depression severity (criterion), we used robust LMMs. Consistent with prior work arguing that targeting repetitive negative thinking (rumination) improves depressive symptoms [6,7,81], these analyses were an attempt to test whether improvement in either retrospective rumination or EMA rumination were related to improvement in depression. For the CARE sample, the dependent variable incorporated repeated assessment of depression over 4 time points (ie, baseline, posttreatment time point, 6-week follow-up, and 12-week follow-up) that were nested within individuals. Time was centered to represent estimated symptom scores at the final follow-up (12 weeks), and baseline depression was included as a covariate. Random intercepts and slopes were specified. To test whether either change measure (baseline to posttreatment time point) predicted improvement in depression over time (baseline to the 12-week follow-up), we included each term in an interaction with time (ie, retrospective rumination change × time and EMA rumination change × time). Each term was simultaneously included in the same model.

For the HMP sample, traditional self-reported depression was assessed only at the pre- and posttest time points. Therefore, we used OLS regression models with posttest depression as the dependent variable and pretest depression, retrospective rumination change, and EMA rumination change entered as predictors.

To reduce the influence of outliers on both the CARE and HMP samples, we winsorized any extreme values (winsorize function in the DescTools R package). Correlation and regression analyses used listwise deletion for the specific variables included in each model. Missing EMA observations within participants were handled using maximum likelihood estimation in mixed-effects models, which uses all available data and provides unbiased estimates under the missing at random assumption. Data were analyzed using R (version 4.2.2).

For correlation analyses between conventional self-report and EMA measures, our combined CARE sample (n=320) provided >80% power to detect correlations of r≥0.16 at an α value of .05. For the HMP sample (n=92), we had >80% power to detect correlations of r≥0.29 (using the pwr package in R). For analyses testing group differences in conventional self-reported rumination improvement in the CARE sample, we had >80% power to detect a small effect size (Cohen f²≥0.025) for between-group comparisons (pwr package). For mixed-effects models testing group differences in EMA rumination trajectories over time (condition × day interaction) in the CARE sample, we had >80% power to detect a small effect size of Cohen d≥0.25 (using the mixedpower package). For the HMP sample, all participants had access to the HMP app (no control group), so we tested within-subject rumination change over time. We had >80% power to detect a small to medium effect size of Cohen d≥0.30 for paired t tests of pretest-posttest rumination change (pwr package) and Cohen d≥0.22 (small effect size) for mixed-effects models testing EMA rumination decline over the study period (mixedpower package).

As internal consistency is computed based on the raw item scores, the 2 adolescent samples completed the CRSQ, and the other (college student) sample used the RSQ, we computed reliability separately for both measures. Reliability (internal consistency) for retrospective rumination was high at T1 (Cronbach α=0.89 for adolescents and Cronbach α=0.93 for college students) and T2 (Cronbach α=0.90 for adolescents and Cronbach α=0.94 for college students) but decreased for the residualized change score (r=0.71 for adolescents and 0.84 for college students). Reliability (split-half) for EMA rumination was high (r=0.89; SB adjusted=0.94) but decreased substantially for the change score (r=0.50; SB adjusted=0.67; using the reliability estimate from the MLM: r=0.58). There were fewer EMA time points per participant in the CARE samples (mean 50.2, SD 17.6; median 55.0) relative to the HMP sample (mean 89.9, SD 23.1; median 98). Reliability estimates are often higher when there are more time points as the slopes are estimated with less error. We recomputed reliability for the change score excluding individuals with <40 time points, which increased the reliability coefficients from r=0.50 (SB adjusted=0.67) to r=0.61 (SB adjusted=0.76).

Reliability (internal consistency) for retrospective rumination was high at T1 (Cronbach α=0.94) and T2 (Cronbach α=0.95). The residualized change score in this sample also retained high reliability (r=0.90). Reliability (split-half) for EMA rumination was high (r=0.96; SB adjusted=0.98) but decreased for the change score (r=0.77; SB adjusted=0.87; it was also 0.87 using the MLM-based reliability estimate). It should be noted that the relatively high residualized change reliabilities for the HMP and CARE college student samples were due to the high reliabilities of the constituent T1 and T2 rumination scores (see the previous paragraph) and their relatively low intercorrelation (CARE college student r=0.66; HMP r=0.55 vs CARE adolescent r=0.69), implying more substantial variability in real change (see, eg, formula 7.10 in the work by Crocker and Algina [82]). Multimedia Appendix 1 provides additional analyses, including convergent and divergent validity tests for EMA and the reliability of raw rather than residualized change scores that were similar.

Retrospective rumination at T1 was significantly positively correlated with mean EMA rumination over the subsequent 3 days (r=0.35; P<.001; Figure 1A). Similarly, the correlation between retrospective rumination at T2 and EMA rumination over the previous 3 days was significant (r=0.28; P<.001; Figure 1C). The magnitude of these correlations is conventionally considered to be in the medium range [83].

Residualized T1 to T2 change in retrospective rumination was not significantly correlated with change (slope) in EMA rumination (r=0.03; P=.66; Figure 1E).

Retrospective rumination at T1 was significantly positively correlated with mean EMA rumination over the first 3 days of EMA (r=0.31; P=.003; Figure 1B). Similarly, the correlation between retrospective rumination at T2 and EMA rumination over the final 3 days of EMA was significant (r=0.47; P<.001; Figure 1D). These correlations were medium to large in magnitude [83].

Residualized T1 to T2 change in retrospective repetitive negative thinking was not significantly correlated with change (slope) in EMA rumination (r=0.20; P=.06; Figure 1F).

A significant group difference emerged for change in retrospective rumination such that the mindfulness group exhibited significantly greater improvement relative to the mood monitoring control group (d=0.37; b=−0.34; SE 0.096; t₂₉₄=−3.51; P<.001). In contrast, there were no group differences in EMA rumination change (d=0.14; b=−0.20; SE 0.138; P=.14). Overall change in the CARE trials (averaging across both conditions) was small for retrospective rumination (d=0.19) and EMA rumination (d=0.20).

There was a significant pretest-posttest reduction in retrospective repetitive negative thinking in the HMP sample (d=0.77; t₈₈=7.60; P<.001). EMA rumination decreased over time (b=−0.012; SE 0.0050; P=.02). However, the associated effect size was small (d=0.17).

Greater pre- to postintervention improvement in both retrospective rumination (b=0.04; SE 0.017; P=.03) and EMA rumination (b=0.03; SE 0.014; P=.04) were significantly positively associated with improvement in depressive symptoms from baseline to follow-up.

Similarly, greater pre- to postintervention improvement in both retrospective rumination (b=0.23; SE 0.060; P<.001) and EMA rumination (b=20.49; SE 9.94; P=.04) were significantly positively associated with improvement in depression symptoms from baseline to the posttest time point.

Researchers are increasingly incorporating EMA into intervention studies in an effort to obtain a more fine-grained and ecologically valid assessment of change. EMA can provide high-resolution information about treatment-relevant processes (eg, emotional, cognitive, and behavior change) while minimizing memory biases relative to conventional (retrospective) self-report measures. However, repeated EMA surveys can be burdensome to participants, which could lead to poor compliance; careless responding (eg, random or stereotyped responses to items); or a bias toward the enrollment of participants who are more motivated, thus limiting generalizability. In addition, the psychometric properties of EMA measures are often not well established before implementing them in a study (eg, researchers often create new EMA items to assess a construct of interest or adapt them from an existing self-report scale without evaluating or reporting their psychometric properties) [84]. In short, intervention researchers must carefully consider whether the benefits of implementing an EMA measure in a study outweigh potential costs and limitations. In this study, we attempted to help address the question of whether EMA is worth including by combining data from 4 trials (total N=412), all of which examined a meditation app and measured outcomes (change in rumination) via both conventional retrospective self-report questionnaires and EMA. We did so by evaluating 4 relevant criteria.

While the reliability of both retrospective rumination (at baseline and the postintervention time point) and mean EMA rumination scores was high, our findings revealed a decline in reliability when assessing change over time, especially for EMA. These results align with long-standing concerns about the unreliability of measures being compounded when measuring change [85,86]. Reliability is generally defined as the ratio of systematic (“true”) variance of interest to total observed variance. In addition to high measurement error, another reason for the low reliability in change scores is that true change variability may be low, either because of limited change or because all participants change to a nearly equivalent degree [87]. Previous EMA studies have reported relatively lower reliability for change scores [88-90]. Critically, low reliability can result in biased inferences and a loss of statistical power, leading to an inaccurate estimation of the true effect on the population [91]. Consequently, when interpreting findings, it becomes extremely challenging to determine whether the results are trustworthy or whether they are attributable to measurement error, thereby reducing the likelihood of successfully replicating findings [92]. Despite the importance of establishing reliability, many EMA studies do not adequately report the psychometrics of their measures [84].

In theory, 2 approaches to improving the reliability (and likely the validity) of EMA measures are to increase the number of time points or add more items that capture the construct of interest [93]. In this study, we relied on single-item measures of EMA rumination, which may provide less coverage of the construct of interest relative to multi-item measures [89]. However, it is important to note that increasing the number of items or assessment time points also increases the burden on participants and, consequently, may reduce compliance or increase careless responding [94]. Therefore, researchers should balance the need to improve measurement (via more items or assessment time points) with a careful consideration of participant burden. In this study, using single-item measures (a common practice in many EMA studies) also prevented us from calculating internal consistency, the method we used to compute reliability for the conventional self-report measures. Consequently, we were unable to directly compare the reliabilities of the conventional self-report and EMA measures.

Similar to increasing the number of relevant items, averaging scores across multiple measurement time points is known to improve reliability [84,93]. This approach aligns with our finding of high reliability for mean EMA rumination. For example, in the context of studying day-to-day changes in rumination, averaging data collected from multiple within-day occasions (as opposed to relying on a single measurement per day) to estimate rumination levels each day is expected to improve the precision of within-day rumination. In addition, the higher reliability of EMA change scores in the HMP sample relative to the CARE sample may be due to the larger number of time points collected per participant in the former sample, improving the precision of slope estimates.

In summary, the relatively low reliability of change scores (in particular for EMA measures of rumination) is concerning. It would be helpful to better understand the scope of this issue if researchers more consistently reported the reliability of change where it is evaluated given that this is not common practice yet can have a profound impact on study findings (eg, low reliability attenuating effect sizes). This can aid readers in evaluating EMA findings in a given study in light of potential psychometric limitations. The field may also benefit from additional innovative approaches to assessing the reliability of EMA measures [89].

The results revealed statistically significant medium to large (r=0.28 to 0.47) correlations [83] between rumination assessed via conventional, retrospective self-report and the mean of EMAs. However, there were no significant correlations between changes in retrospective rumination and change in EMA rumination (r=0.03 to 0.20). These results are in line with those of previous studies that have reported low to moderate correlations between trait and state measures of various constructs, including anhedonia [95], affective lability [96,97], and personality traits [98,99].

Several factors may account for the relatively modest association between conventional retrospective self-report and EMA measures of rumination (especially for the change scores). First, less than perfect reliability can attenuate associations between any 2 variables. Reliability estimates were particularly low for change scores, which may explain (at least in part) the especially low correlation between those measures. Second, the distinction by Kahneman [100] between the “experiencing self” and the “remembering self” provides a useful framework for interpreting these results. According to this distinction, when participants are asked to recall their past experiences on conventional, retrospective self-report measures (in this study, asking about past ruminative thoughts), they may be especially influenced by the most intense (peak) and recent (end) levels of rumination (ie, the so-called peak-end bias) rather than equally weighting and simply averaging all time points, as reflected by the mean of repeated EMAs [27-30]. Third, EMA requires introspection of current states, whereas retrospective self-report measures require participants to access autobiographical (episodic) memory and provide an average report of past experiences. The accuracy of these reports is limited by the extent to which past experiences have been accurately encoded and consolidated in memory [11,31,32]. Moreover, when recall time frames are longer, individuals may shift from episodic memory to semantic memory, which includes more abstract beliefs or generalization about the self [11,34]. Thus, when asked to report on their typical tendency to ruminate on self-report measures, individuals’ responses may be influenced by relevant self-concepts (“I tend to overthink” or “I’m a worrier”), how they believe others view them, or how they would like to be perceived [101]. In summary, in addition to reduced reliability, the relatively modest correlations between conventional retrospective self-report and EMA measures of rumination may be due to the fact that the 2 measurement approaches ultimately tap into different conscious or functional “selves” [31]. As described in the following sections, the decision to include EMA or retrospective measures of change in an intervention study should be informed by which of these “selves” (experiencing vs remembering self) the researchers want to assess and expect to change.

Compared to conventional retrospective self-report, EMA measures detected smaller group (intervention vs control) differences in rumination improvement (effect sizes: d=0.37 vs d=0.14 for self-report and EMA, respectively). EMA also showed less linear change in rumination over time in the intervention group (effect sizes: d=0.77 vs d=0.17 for self-report and EMA in the HMP sample, respectively). These findings have important implications for the selection of outcome measures in intervention studies. They imply that a trial may yield very different findings (eg, significant vs nonsignificant differences in outcomes between the intervention and control group) based on whether a conventional retrospective or EMA instrument was used as the outcome measure. This raises the critical question of which result is closer to the “ground truth.” The larger effect size observed for retrospective self-report could be due to this measure being a more sensitive method for capturing changes in rumination. Although speculative, EMA may be less sensitive to detecting change due the repeated nature of EMA surveys introducing unique forms of bias. For example, perhaps participants tend to align their responses with their (relatively recent) previous answers (a type of anchoring bias), potentially resulting in reduced detection of actual change. If a participant rated their rumination as a 4 on a scale from 1 to 5 on the last EMA survey, an anchoring bias may lead them to report a similar score on the next survey even if their “true” rumination score is now lower. Alternatively, EMA may provide a more accurate estimation of real change (which may, in fact, be modest), whereas the conventional retrospective self-report measure may overinflate estimates of change due to influences such as social desirability bias, regression to the mean, or initial elevation bias [39].

If, as discussed previously, EMA and retrospective self-reports indeed capture related but somewhat different constructs, it is possible that the pattern of results we observed is due to the intervention differentially affecting each of these facets. For example, it may be that the intervention had a larger impact on one’s episodic or semantic representations of being a ruminator (as measured via retrospective self-report) but had a more modest impact on actual moment-to-moment rumination (as measured via EMA). There may also be a causal relationship from the latter to the former. Namely, relatively modest improvements in day-to-day momentary rumination may have a proportionally large positive impact on one’s self-concept of being a ruminator.

Even if changes in retrospective self-report are biased due to the limitations of memory and there is little change in actual experience, these changes in retrospective recall may still be meaningful. Research has shown that future behaviors (eg, whether to return for a colonoscopy screening or end a romantic relationship) are better predicted by retrospective measures of experience than by actual momentary experiences [102]. In summary, retrospective self-reports (eg, related to one’s self-concept and tendency to engage in repetitive negative thought) may, of course, be very personally meaningful to the individual who underwent the intervention. Therefore, if the goal is to understand what individuals experienced in the moment, EMA arguably provides a more accurate measure. However, if the focus is on individuals’ global impressions of their experiences and themselves, then retrospective self-report measures may be more valuable.

Finally, the results of this study revealed that both retrospective and EMA measures of rumination change predicted improvement in depressive symptoms, with each measure contributing significantly beyond the effects of the other (ie, evidence of the incremental predictive validity of each measure). These findings suggest that these 2 methods assess meaningfully distinct aspects of rumination, both of which hold clinically important information. Therefore, if feasible, combining retrospective and EMA data could provide a more comprehensive picture of clinically meaningful improvement over time. It is also important to highlight that there are, of course, other reasons why a researcher may want to include EMA measures in an intervention study. For example, EMA enables the examination of symptom or emotion dynamics over time, such as changes in inertia or instability, which cannot be captured using traditional self-report methods [103]. In addition, EMA enables more sophisticated analyses of mechanisms of change, such as temporal mediation analyses that can examine how changes in putative mediators precede and predict subsequent changes in outcomes within individuals over time, addressing key limitations of traditional mediation approaches [46,104]. Nevertheless, these methodological advantages require careful attention to data quality as poor reliability in EMA can obscure the ability to detect meaningful temporal dynamics in these processes.