Full details of Health@NUS are available elsewhere [34]. Briefly, Health@NUS uses traditional and digital strategies to capture health-related behaviors and related factors over a 2-year period as students complete their university education and as many of them transition into postuniversity work and life. Throughout the 2-year study, participants repeat traditional questionnaires and biometric assessments (baseline and 1- and 2-y follow-up). Movement behaviors (ie, physical activity, sedentary behavior, and sleep [35]) were monitored continuously using a wearable device (Fitbit Versa Lite), and a smartphone app tracked dietary intake and delivered up to 5 bursts of EMA surveys per year. These EMA data extend and contextualize the information from the traditional and digital assessments with questions covering movement (sleep, physical activity, and screen time) and diet (whether food was eaten, where it was eaten, what was eaten, activities while eating, and satiety after eating) and stress, fatigue, and mood. By combining bursts of EMA surveys with other digital technologies, Health@NUS will capture and describe the temporality of experiences within and between days, and their effects on health over time, at a level of granularity not previously possible.

The schedule for the first burst of Health@NUS EMA surveys was designed based on the available literature [11,32], the need to capture multiple constructs as succinctly as possible, and our experiences implementing EMA in the local context [22,36,37]. Despite this approach, the response rate to the first burst was low. On average, participants completed only 26% of the surveys they received, well below the 70% or higher response rate reported in other studies with comparable populations [11,32,38] and the recommended acceptable threshold of 80% [2,39]. This low response rate prompted the study team to carefully review the EMA implementation strategies and schedule with the overall aim to increase the response rate in future bursts.

To evaluate these changes, we implemented 2 randomized controlled trials (RCTs) within the ongoing Health@NUS study. The practical nature of these nested RCTs (ie, to act to improve overall EMA survey response rate in an ongoing study) meant that publishing a protocol before starting the study was not feasible. The outcome variable (response rate) was specified a priori. The flow of participants through these RCTs is shown in Figure 1.

Participants were recruited to Health@NUS via email, campus posters, and word of mouth. To be eligible, participants had to (1) be a full-time student at the NUS, (2) be aged 18 to 26 years, (3) be a citizen or permanent resident of Singapore, and (4) own a smartphone compatible with the study app (ie, minimum iOS 10 or Android 7). Recruitment was ongoing at the time of writing this paper.

There were no additional eligibility criteria for this study. A total of 384 students who joined Health@NUS during the first wave of recruitment between October 2020 and March 2021 were included. During study enrollment, participants provided written informed consent to receive short surveys (EMAs, <10 min each) via the study app (HiSG app). Participants were advised that the EMA surveys were optional but highly encouraged and that they would be compensated for answering them. No specific details of the survey timing, frequency, or the compensation were provided during the consent process.

At baseline, participants self-reported their age (date of birth); sex (male or female); ethnicity (Chinese, Indian, Malay, or Other); marital status (single or never married, currently married, separated but not divorced, divorced, widowed, or refuse to answer); monthly household income (<SG $2000 [US $1466], SG $2000-SG $3999 [US $1466-US $2932], SG $4000-SG $5999 [US $2933-US $4398], SG $6000-SG $9999 [US $4399-US $7331], >SG $10,000 [>US $7332]), refuse to answer, or do not know); whether they are an undergraduate or postgraduate student; and the faculty they are studying in.

Biometric assessments (height in cm and weight in kg) were taken by trained study personnel. BMI was calculated from height and weight measurements and classification recommendations for Asian populations [44] were followed (<18.4 kg/m²=underweight; 18.5-22.9 kg/m²=normal; 23-27.4 kg/m²=overweight; and >27.5 kg/m²=obese).

EMA surveys were administered via the HiSG app, and responses were automatically captured by the app and uploaded to a study server in real time.

The primary outcome measure was the response rate (ie, percentage of completed EMA surveys from all sent EMA surveys) for each burst of EMA surveys.

Baseline characteristics were analyzed descriptively. Separate 1-way ANOVAs were used to estimate the effect of changing the reward structure (aim 1, reward RCT) or the schedule length (aim 2, schedule length RCT) on the response rate at burst 2 and burst 3, respectively. A sensitivity analysis was conducted using analysis of covariance to adjust for the response rate at burst 1 (ie, baseline).

Secondary aim 3 was first analyzed using linear mixed model analysis with restricted maximum likelihood estimation to compare the overall response rate across the 3 bursts of EMA surveys following changes to the EMA implementation protocol (illustrated in Figure 2). The model was adjusted for group allocation at burst 2 and burst 3. Pairwise comparisons, with Bonferroni correction for multiple comparisons, were conducted to identify which bursts had significantly different response rates. We also conducted further subgroup analysis for secondary aim 3 with participants who were allocated to the control group for each EMA burst. The subgroup for this analysis was specified as all participants from burst 1 (N=384), data from group-A (control group) participants at burst 2 (n=96), and data from participants in the 4 groups that received the 14-day schedule (control group) at burst 3 (n=96; 24 of these participants also contributed data in burst 2). All analyses were conducted in R software (version 4.1; R Foundation for Statistical Computing).

Ethics approval was obtained from the National Healthcare Group in Singapore (reference 2019/00285). All participants provided written informed consent before commencing the study and consented for their deidentified data to be used for research purposes. For this study, the maximum compensation available to participants ranged from SG $21 (US $15) to SG $34.66 (US $25) depending on group allocation during the reward RCT and on the number of EMA surveys they completed.

Between October 2020 and March 2021, 384 participants were recruited and enrolled into this study, and data collection was complete by October 2021.

Following burst 1 and before burst 2, the participants were randomized to group A (n=96), group B (n=95), group C (n=96), or group D (n=97). Following burst 2 and before burst 3, participants were randomized to the intervention (7-d EMA schedule, n=288) or control (14-d EMA schedule, n=96) group. Figure 1 presents the details of participant flow through the trial.

The study participants were predominantly undergraduate students (376/384, 97.9%), female (232/384, 60.4%), of Chinese ethnicity (366/384, 95.3%), and reported as being single or never married (382/384, 99.5%). The mean age of the participants was 23.37 (SD 1.25) years, and most of the participants had a BMI that was classified as either normal weight (203/384, 52.9%) or moderately overweight (114/384, 29.7%). Details of the participant characteristics at baseline for all participants and for each group allocation in the 2 RCTs are presented in Table 2.

For the reward RCT, group A had a slightly higher proportion of participants with a monthly household income of <SG $2000 (<US $1466; compared with the other 3 reward groups). For the schedule length RCT, the 7-day schedule group had a slightly higher proportion of male participants and of participants with a reported monthly household income of <SG $2000 (<US $1466) as compared with the 14-day schedule group. The 7-day schedule group also had fewer participants who reported a BMI in the healthy range as compared with the 14-day schedule group (Table 2).

The first aim was to evaluate whether changing the reward structure for completing EMA surveys would lead to an increase in the response rate.

The average response rates for the 4 reward groups at burst 1 and burst 2 are presented in Table 3. The response rate at burst 2 increased for all groups (compared with that at burst 1). However, the differences in the burst 2 response rate between the groups were not significant (Table 3).

The second aim was to evaluate whether implementing a shortened 7-day EMA schedule would improve the response rate.

The average response rates for the 2 schedule length groups are shown in Table 4.

On average, participants in the 14-day group (control) completed 48.3% (SD 33.2%) of EMA surveys at burst 3 compared with 38.5% (SD 33.4%) in the 7-day group (intervention); this difference was significant (F_1,382=6.23; P=.01) and remained so after adjusting for the response rate at burst 1 (baseline; F_1,380=4.63; P=.03; Table 4).

At baseline (ie, burst 1), the average response rate per participant was 25.8% (SD 30.1%). At burst 2, the average response rate across groups was 46.6% (SD 34.6%) and 41% (SD 33.6%) at burst 3, and this difference was statistically significant (F_2,766=93.83; P<.001). Pairwise comparisons indicated that the difference between burst 1 and burst 2 and burst 1 and burst 3 were both significant (P<.001), whereas the difference between burst 2 and burst 3 was not significant (P=.05).

Subgroup analysis with control group participants was used to compare the overall response rates across the 3 bursts of EMA surveys following changes to the EMA implementation protocol. Table 5 shows the average response rate for all participants at burst 1 (N=384) and for participants allocated to the control conditions at burst 2 (group A, n=96) and burst 3 (14-d schedule group, n=96; note that 24 participants contributed data to all 3 bursts). The average response rate was higher at burst 2 (50.6%, SD 33.6%) and burst 3 (48.3%, SD 33.2%) than at burst 1 (25.8%, SD 30.1%), but the difference was not statistically significant (F_4,215=0.72; P=.58).

Multimedia Appendix 3 shows details of how the response rate varied across each burst of EMA surveys.

This study experimentally evaluated the effect of rewards and schedule length on EMA response rates within the context of an ongoing study implementing repeated bursts of EMA surveys. Reducing the number of days of EMA surveys led to a significantly lower response rate. However, changing the available rewards did not significantly change the response rate. Overall, for all groups the response rate was lowest at baseline (burst 1) as compared with the subsequent bursts of EMAs and the difference was statistically significant. However, our subgroup analysis that was intended to further explore whether this was because of changes in how participants were notified of the relevant burst of EMA surveys found no significant difference in the response rate at each burst.

The response rate to burst 1 was very low, prompting this study, and increased substantially in burst 2 and burst 3 in all groups. It is possible that the initial low response rate was because the upcoming EMA burst was not well communicated to the participants. Future bursts included more frequent communication delivered directly to all participants (via SMS text messages and push notifications). These simple changes may have contributed to the increased response rate. However, we did not experimentally evaluate the effects of these communication changes. Our secondary analysis partially supported this finding, as there were significant differences between the response rate at each burst. However, in the subgroup analysis of the control group participants, the differences were no longer significant.

The finding that neither offering greater reward amounts nor reducing the schedule length led to an increase in the response rate is broadly consistent with systematic reviews of factors associated with EMA compliance [8,32]. However, these reviews highlight the inconsistencies in how response rates are reported (eg, of studies in nonclinical populations, only 22% reported average response rate/person [8]), which makes direct comparisons challenging. Greater uptake of EMA study reporting guidelines [20,38] would be useful in this regard. Our study extends the currently available evidence by providing an experimental evaluation of the role of rewards and schedule length.

Rewards were selected as the first intervention target as the rewards available in burst 1 were low compared with other studies; participants could receive a total of approximately US $5 for completing all the EMA surveys. In other studies with a similar number of EMA days (between 10 and 14 d) and surveys (between 35 and 50 surveys), the lowest incentive was approximately US $25 [14]. In burst 2, the total available rewards increased for some groups (up to US $15) but remained lower than comparable studies and there was only a US $10 difference between the lowest and highest value reward group. In our study, very small rewards were directly tied to the completion of each individual EMA survey (ie, 25-50 HP, approximately US $0.12-US $0.24/completed EMA survey). In the context of Singapore, small incentives (in the form of HP and supermarket vouchers) have been used to promote compliance to interventions [45,46], although in these instances, the relationship between intervention compliance and the incentives available may have been clearer to participants. In contrast, Health@NUS participants have a range of different study requirements that are tied to different incentives; over the course of the 2-year study, participants can receive up to about US $313 (plus keep their study Fitbit). In addition, participants were likely aware that completing the EMA surveys was optional (but strongly encouraged). Taken together, immediate rewards may have seemed small for all reward groups, and cost-benefit reasoning may have resulted in a decision to not complete the EMA surveys [47].

In our study, contrary to our expectations, the burst 2 response rate was significantly higher in the 14-day schedule group than in the shorter 7-day schedule group. It seems intuitive that fewer days of EMA surveys would be less intrusive and therefore preferable to participants, particularly in the context of repeated bursts of EMA surveys. However, our findings indicate the opposite. More research is needed in this area as systematic reviews currently report inconsistent relationships between EMA schedule length and response rate [8,32].

Although the average response rate in EMA studies has been reported to be >70% [11,38], these studies implement a single burst of surveys rather than repeated bursts. Our results compare favorably with those of 2 other studies that have used burst designs. In the SPARC study [48], 4 bursts of EMA surveys (8 surveys/d for 4 d) were implemented over a 7-month period (September, October, February, and March) and reported an average per participant response rate of 41% across the 4 bursts [49]. Similarly, Howland et al [50] asked participants to complete a 30-day daily diary annually for 4 years. The retention rate across the 4 years was reported as 73%; however, this was after participants who did not meet the minimum reporting threshold of 15 days per year (out of 30 days) were excluded from the study. It is important to note that these 2 studies [48,50] only implement repeated EMA and traditional questionnaires (no continuous or in-person assessments). As such, in these studies, participants would likely have specifically signed up to an EMA study, as compared with Health@NUS participants who have numerous other elements of data collection to fulfill, with the EMA secondary to this and optional. This highlights potential challenges with repeatedly administering EMA (in or outside the context of a larger study) and that researchers should carefully consider the likelihood of missed EMA surveys (and missing data) when using burst designs [20,21]. However, further research is required to confirm our findings. Two recent reviews of EMA compliance found no evidence of a significant relationship between schedule length and compliance [8,32], although these data were obtained from observational rather than experimental studies. There are few experimental studies of study design features to minimize missing EMA data [33] and few longitudinal EMA studies [23-25].

The strengths and limitations of this study should be considered. Our RCTs were embedded within an ongoing cohort study that required participants to fulfill several mandatory requirements (eg, minimum Fitbit wear time and food diary logging/mo), whereas completing the EMA surveys was optional but highly encouraged, and participants may have decided to opt out of this study component. Furthermore, although our sample size was larger than that of many other EMA studies [11,38], no a priori sample size was calculated and instead all participants who enrolled during the first waves of recruitment (October 2020 to March 2021) were included. Our study is one of the first to experimentally evaluate the impact of EMA protocol features on overall response rates, and we purposefully chose protocol features that could be manipulated and evaluated within an RCT. However, as the first few bursts of EMA were organized and scheduled in advance, the research team had to rapidly analyze the previous bursts’ response rate data and select a suitable intervention strategy for the upcoming burst. Given more time, we may have selected alternative variables to manipulate, such as time-varying factors (eg, time of day or weekend vs weekday [51]); number of EMA surveys per day (eg, our varied number of EMA surveys/d vs a consistent number); or whether the response rate can be predicted based on the question type (eg, Likert scale or multiple choice) or content (eg, dietary intake or stress). Future studies should explore the role of time-varying factors, the number of EMA surveys, and the type and content of questions on the overall response rate.

Our pragmatic approach also meant that our secondary aim, to explore temporal changes in the response rate, was exploratory in nature. Future studies specifically designed to experimentally evaluate the effect of altering the announcement and communication strategy for each EMA burst (ie, the number of emails, SMS text messages, and push notifications that were sent to provide details of what to expect in the upcoming burst of EMA surveys) are needed. Studies evaluating the effects of other temporal variables such as holidays and key periods in the academic calendar (eg, exams) are also needed. As is typical of behavioral research, it was not possible to blind participants to their intervention condition, and we also cannot comment on whether participants received all of the EMA surveys that were sent. Furthermore, in our study, the number of EMA surveys per day varied (3-6/d), which may have lowered the response rate as participants did not know when to expect a survey. Finally, our sample consists of university students who may be especially motivated to engage in health research and therefore may not be representative of the broader population of young adults. The extent to which our findings generalize beyond this group is unclear. However, as young adults are a key population studied in EMA studies [20,32,52-55], our findings are likely to be of considerable interest to the field.

This study is one of the first to experimentally evaluate the effect of incentives and schedule length on EMA response rates. It is also the first study to consider factors related to response in the context of an ongoing prospective cohort study administering repeated bursts of EMA over a 2-year period. By embedding RCTs within an ongoing study, it was possible to rapidly implement and evaluate whether altering the implementation strategy, incentives, or schedule length would increase the response rate. Our study therefore contributes to a small but growing body of literature on how to implement EMA. This knowledge is essential for collecting high-quality EMA data, which has a flow-on effect to the quality of conclusions that can be drawn from these data.