The data for this study were sourced from a larger study utilizing secondary data analyses of preexisting EMA datasets for characterizing momentary pain experiences in patients with chronic noncancer-related pain [16]. The inclusion criteria were study sample of at least 30 adult patients (excluding studies with pediatric patients) and administration of a minimum of 3 EMA pain intensity prompts per day for at least 4 days with a fixed- or random-assessment schedule that was assessed via electronic diaries, mobile phones, or interactive voice responses. Studies using paper diaries were excluded because of problems (eg, back-filling and forward-filling) that can undermine the validity of the estimated completion rates [17,18]. Observational studies and clinical trials were included, but clinical trials were limited to no-intervention or baseline assessment periods. EMA pain assessments needed to focus on monitoring momentary pain intensity. Studies that used EMA exclusively as an intervention trigger (eg, just-in-time adaptive interventions [19]) were excluded.

Eligible datasets were identified through a systematic literature search conducted in October 2016 using PubMed and Web of Science databases with the following search terms: ([“Ecological Momentary Assessment” or “Experience Sampling” or “Electronic Diary” or “Electronic Diaries” or “Electronic Interview” or “Electronic Interviews” or “Interactive Voice Response” or “Intensive Diaries” or “Ambulatory Monitoring” or “Ambulatory Assessment”] and “Pain”).

Completion rates were calculated as the percentage of EMA prompts completed (relative to the number of prompts received) for each person and day of the study. For EMA protocols with a fixed sampling scheme (6 studies), we considered the number of prompts received as the number of prompts specified by the protocol (and reported in respective articles). For EMA protocols with a random-sampling scheme (4 studies), we obtained the specific number of executed prompts from the datasets, because this number could vary across days (this was especially the case when studies allowed the number of momentary prompts to vary according to patients’ waking hours). Given the proportional nature of the completion rates, we tested our models with both the original and arcsine-transformed scores [20]. In this paper, we report the results based on the original completion rates, because the analyses yielded nearly identical results.

To examine changes in completion rates over time, the study day was coded as a within-person (day-level) predictor variable. Data for patient-level predictors of EMA-completion rates—age and sex—were taken directly from the databases. Patients’ chronic pain diagnosis was coded as osteoarthritis, rheumatoid arthritis, fibromyalgia, or other diagnoses. The following features of the EMA protocol were coded as study-level predictors of EMA completion rates: Study duration was coded as the total number of days of the EMA protocol, EMA sampling density was coded as the average number of EMA prompts received per day, and EMA survey length was coded as the number of EMA items presented at each prompt.

We used the momentary pain intensity ratings available in each dataset to examine whether EMA-completion rates were associated with pain intensity at the day, person, or study level. The number of scale points used to measure momentary pain differed across studies (range, 5-101 points), and the pain ratings were converted, so that the ratings were on a 101-point scale. For the conversion, we used the following equation: New rating=100*(original rating+0.5)/(number of scale points). Because momentary pain ratings were not assessed at the time of the missed EMA prompts, the analyses were based on averages of the nonmissing pain ratings, as per a previous analysis [10]. Specifically, study- and person-level averages of all available pain ratings were calculated to examine whether studies or patients who, on an average, reported higher pain levels showed lower (or higher) completion rates. Additionally, daily average pain levels were computed for each patient (and within-person centered) to examine whether day-to-day variations in pain in a given patient were related to daily completion rates.

Our multilevel models subsequently incorporated day-level (Level 1), person-level (Level 2), and study-level (Level 3) predictors of EMA-completion rates. Specifically, a model without predictors (Step 1) was followed by analyses of the day-level predictors, which examined changes in daily completion rates over time (ie, over the course of the EMA sampling protocols; Step 2). Step 3 added patient-level predictors including age, sex, and chronic pain diagnosis. Step 4 added the following study-level predictors: study duration, EMA sampling density, and EMA survey length. In the final step (Step 5), day-, patient-, and study-level averages of pain intensity were added as predictor variables of completion rates on each of the 3 levels. Analyses were conducted using maximum likelihood parameter estimation in Mplus, version 8 [21]. Values of P<.05 were considered statistically significant.

Statistical power in multilevel models depends on several factors such as the sample sizes at each level of analysis and the intraclass correlations due to the clustering effect or observation dependence of lower-level units nested in higher-level units. We conducted a power analysis using Monte Carlo simulation [22] to determine the minimum effect sizes that would be detectable with 80% power (alpha=.05), given the sample sizes and intraclass correlations of the data analyzed in the present study (see Sample Size and Design Characteristics section below). The minimum detectable effect sizes were 0.04 at Level 1 and 0.1 at Level 2, corresponding with small effects following the Cohen [23] conventions. Owing to the limited sample size at Level 3 (the study level), the minimum detectable effect size at this level was 0.7, which was a large effect.

Our literature search identified 20 eligible databases from 37 articles (Figure 1). Authors of these articles were contacted by the research team. Original patient data were received for 10 of the 20 databases: Nine datasets were not received because the authors did not respond to the request or declined to provide the data or because the data were no longer available, and one dataset was not included because it provided only partial data without information on demographic predictor variables. One database consisted of three substudies comprising independent patient samples with different EMA sampling designs. These were separated into three datasets; thus, a total of 12 independent datasets were included in the analyses.

Characteristics of the participants and studies are summarized in Table 1. Overall, our analyses included 7956 study days from a total of 701 patients. The study duration ranged from 4 to 28 days. The number of prompts per day ranged from 3 to 12 prompts, and the number of items per prompt ranged from 6 to 63 items.

The distribution of average daily completion rates by individuals is presented in Figure 2. The average completion rate was 85%, with daily completion rates <70% for 13% of the patients, <80% for 27% of the patients, and <90% for 60% of the patients. Initial multilevel models without predictor variables showed that 58% of the total variance in completion rates was attributable to within-person (day-to-day) variation: 26% to reliable differences between patients and 15% to differences among studies. Thus, investigation of predictors from different levels was warranted.

Findings for the multilevel model predicting completion rates are summarized in Table 2. On the within-person level, we found a significant linear decline in daily completion over time (b=–2.29, P<.001); on average, completion rates decreased by approximately 2.0% per week of EMA sampling. On the between-person level, age showed a curvilinear relationship with completion rates (b_{age_linear}=1.77, P=.002; b_{age_quadratic}=–0.62, P=.009). Completion rates were highest among older patients (age≥60 years), and younger patients showed less completion (Figure 3). In addition, we found that the linear term of age significantly moderated the magnitude of changes in completion rates over time (b=0.56, P=.02). Younger patients had steeper declines over time as compared to older patients (Figure 4). Other patient characteristics—gender and chronic pain diagnosis—were not significant predictors in the model. On the between-study level, study duration, sampling density, and survey length were not significantly related to completion rates. Similarly, pain intensity levels were not significantly related to day-to-day variation in within-patient, between-patient, or study-level differences in completion rates.

The goal of this study was to determine whether completion rates in EMA studies on chronic pain differ systematically according to characteristics of the situation, person, or EMA protocol. Analyzing data of 701 patients from 10 studies, we found an average EMA-completion rate of 85%. Although this rate may seem high, this finding is consistent with that of other studies on chronic pain [12,16]. The results revealed lower completion rates among younger patients than among older patients. In addition, we observed a decline in completion rates over the course of study protocols. The rate of this decline was moderated by respondent age: Younger patients showed a faster decline in completion rates than older patients.

The effect of age may be a result of fewer competing demands from elderly patients in their daily lives due to, for example, retirement, as compared to younger patients. In fact, evidence suggests that younger respondents are more prone to inattention and carelessness when completing online surveys than older respondents [34]. Similarly, the observed decline in completion rates over time may be due to survey fatigue or loss of motivation over the study days. Okifuji et al [11] recommend limiting the period of EMA assessments to 1 week, but this limit may not be desirable when, for example, evaluating responses to changes in treatment or adjustment to new medications using EMA. Research is needed to identify feasible ways to ensure sustained patient engagement in EMA assessments over time including the use of monetary or motivational incentives, close participant-researcher interactions over the course of the study, and the use of emerging data-collection strategies aimed at reducing participant burden, such as “microinteraction-based” EMA (a method developed to answer few EMA items very quickly) [35] or “measurement burst” designs (multiple brief EMA periods repeated over time) [36].

Our analyses did not support several a priori hypotheses. We did not detect any gender differences in the EMA-completion rates. In fact, previous findings of lower completion rates among men were primarily based on healthy samples [6-8], and studies on chronic pain did not identify such a pattern [10,11]. In addition, we found no evidence that either day-to-day variations in pain intensity or differences in pain intensity levels between participants or studies were systematically associated with EMA completion. This finding is especially important because an association between high pain levels and a low likelihood of responding to EMA surveys could severely undermine the validity of the EMA data collected to monitor patients’ pain in everyday life. An important caveat of our analyses is that they were necessarily based on averages of pain from EMA prompts that were not missed by patients; considering that pain levels for missed EMA prompts are not known, the possibility that EMA prompts are more likely missed when patients are in higher (or lower) pain at the time of the prompt cannot be excluded. Our findings may have also resulted from the salience of pain experiences in the study samples. Participation in research wherein pain is of immediate relevance and intrinsic importance might have contributed to patients’ motivation to complete the momentary assessments even in times of high pain intensity. This perspective aligns with our result of no differences in the EMA-completion rates according to the chronic pain diagnosis. The characteristics of the chronic pain experience can vary substantially between diagnoses; for example, high levels of fatigue and cognitive difficulties are more strongly associated with some diagnoses than with other pain diagnoses, and this might have contributed to differences in completion rates. However, the present results suggest that more complex symptomatology per se might not preclude patients’ engagement in the momentary assessments.

At the between-study level, we did not detect any associations between design features related to participant burden (overall length of a given study, sampling density, and number of EMA survey items) and completion rates, which was surprising. Given the intensive nature of EMA protocols, participant burden has often been viewed as a major factor contributing to noncompliance [7,12]. One possibility is that other study design factors (eg, frequent contacts with participants to keep them motivated) are more important than load for continued engagement in EMA protocols, and these factors should be studied in future research. Additionally, despite the sizeable number of participants included in our analyses, IPD meta-analysis can be negatively influenced by a low statistical power at the highest (between-study) level of analysis, which may have limited our ability to detect effects based on study-level design features [37].

A limitation of our study is the potential selectivity bias: We were able to include 10 of 20 eligible datasets. Inclusion of >90% of the eligible studies in IPD meta-analyses has been suggested as an ideal target [38], although, in practice, many IPD meta-analyses include <80% of the eligible datasets [39]. To evaluate the potential for selection bias in the data available for the present analyses, we examined the pooled average completion rate reported for eligible studies that were not included in the analyses; of 10 studies, 7 provided average completion rates in the published reports. The weighted average completion rate in these studies was 78.2%, suggesting a potential upward bias of completion rates in the data that were available for our analyses.

When calculating completion rates, we relied on a fixed number of EMA prompts, unless studies employed a variable prompting schedule based on patients’ waking hours. As such, we assumed that each participant in those studies consistently received the same number of prompts. However, this assumption may have sometimes been violated due to the potential of malfunctioning of data-collection devices or limitations in their configuration capability (ie, prompting during sleep). Thus, some of the calculated daily completion rates may have underestimated participants’ actual completion rates.

Our study is also limited by the number and types of predictor variables that were consistently available across the different datasets. Data on additional predictors such as negative affect, disability status, and stress levels were not consistently available but are undoubtedly candidates for understanding EMA completion in chronic pain. Similarly, EMA data-capturing methods differ among many intricate dimensions that could be theoretically important predictors of EMA completion rates, including whether participants should be allowed to delay the assessment (ability to “snooze” or “suspend” an assessment), how fast a participant is expected to start an assessment (ie, the time window during which an assessment stays open for completion), the frequency of contact between the research team and the participant, the availability of reminders to complete the assessment (and the type and frequency of reminders), and the expectation of individualized feedback at the end of the study. Finally, our findings may not be generalizable to patients with other illnesses or healthy populations.

In summary, our IPD meta-analysis showed no evidence to suggest that EMA-completion rates in chronic pain differ by medical diagnoses; gender; EMA study design features related to participant burden; or variations in pain levels across days, patients, or studies. These findings support the use of EMA data-collection methods for careful assessment of patients’ pain and other experiences. Future EMA research in chronic pain should note that study length and young age can affect the quality of the momentary data and devise strategies to maximize EMA-completion rates across different age groups and study days.