Data were collected from three vocational education schools in the Netherlands. Out of the 44 students who were approached to participate in the mEMA study, 30/44 students (68%) participated for 7 consecutive days (17/30, 57% were female), of which 17 students (aged 16-21 years) completed the online evaluation (regarding feasibility and usability of the mEMA app) after their 7 days of participation. Participatory incentives of ten €20 coupons were randomly distributed to the students at the end of the study. Those who participated for 7 days and filled in the online evaluation form doubled their chance of winning. All students consented to take part in the study and students aged ≤17 were provided with a passive consent form for their parents to complete. All procedures were approved by The Ethical Committee of Psychology (Maastricht University).

In a study conducted prior to the mEMA study (n=305 vocational education students), an assessment was made of the type of smartphone platforms that were most commonly used among the vocational education students. The majority of the students (244/305, 79.9%) indicated that they used a smartphone operating on BlackBerry OS, Android, or iOS (respectively 151/305, 49.5%, 55/305, 18.1%, and 37/305, 12.1%). Estimating an increased use of the iOS and Android platforms, the mEMA app was built on all three platforms.

In total, 44 students from three different vocational education schools in the Netherlands were approached by their teachers to participate in the present study, of which 30 students participated in the mEMA study. Students from the first two schools were allocated to the first group of participants (n=14) and were randomly assigned to the DI or PA condition (study design and participant distribution are illustrated in Figure 1). The students from the third school were allocated to the second group (n=16) and started using the mEMA app after several adjustments were made based on feedback from the first group (discussed in the Results section). Participants in the second group were encouraged to use the mEMA app similarly to those participating in the first group.

Prior to the use of the mEMA app, students completed an online questionnaire regarding their DI or PA during the preceding 7 days. Data from this online questionnaire were collected for other study purposes. After completion, students who possessed a smartphone that operated on iOS (iPhone version 5.0), BlackBerry OS (version 6.0 or 7.0), or Android (version 2.2 and over) systems were eligible for participation, and downloaded the mEMA app from the BlackBerry OS, iOS, or Android app store (n=30). Students who possessed a smartphone with other operating systems or who did not own a smartphone were exempted from further participation. The students were invited to use the mEMA app for 7 consecutive days and gave their consent for participation. During these 7 days, participants were asked to fill in the same short questionnaires regarding their DI or PA during the preceding 3.5 hours, 5 times a day. Participants were able to close the app at any time.

All students who participated in the mEMA study were asked to fill in an online evaluation form concerning the feasibility and usability (ie, functionality and interface) of the mEMA app. Additionally, after participants from the first group filled in the online evaluation form, group discussions took place per school, led by one of the researchers.

In order to upload content for the online questionnaire, mEMA app (both DI and PA conditions), and the evaluation, a content management system (CMS) was built. This CMS enabled researchers to monitor multiple complex health behaviors simultaneously (eg, DI and PA) and to adapt the content, text, and prompting sequence.

An interval-contingent schedule was used for the mEMA app, which initially prompted participants at 5 time periods per day (8:00am, 12:00pm, 3:30pm, 6:30pm, and 9:30pm with a range of 30 minutes each), tailored to the schedule of their schools. These prompts were formatted as auditory signals and were displayed on their smartphone screens. When a diary entry was missed or rejected, up to two reminder signals were sent, once after 30 minutes and the second after 60 minutes. After missing and/or rejecting the two reminders, the CMS noted the initial diary entry as “missed”.

First, participants received instructions about the study and the use of the mEMA app (ie, how to download and use the app). They were instructed to start with an online questionnaire regarding their DI or PA during the preceding week. Second, students were able to use the app during the first day to get familiar with it. From the second day, the mEMA app started prompting participants, asking them to fill in a short questionnaire. Students were instructed to respond to these prompts as much as possible.

During each diary prompting sequence, students who were allocated to the DI condition received a total of 14-16 questions regarding their mood, eating behavior, food cravings, self-evaluative emotions, location, activities, social context, soft and energy drinks, snack intention, hunger, and the availability of food, respectively (ie, 12-14 questions depending on their responses within the decision tree; see Figure 2 A). These questions were derived from Carels et al, White et al, Dijkstra and Buunk, Adriaanse et al, Thomas, Thomas et al, and Rijpstra et al [21-26]. Students in the PA condition received 12-14 questions concerning their mood, sedentary behavior, physical activity, need for physical activity, evaluative emotions, location, social context, behavior intention, active transport, possible barriers, and feelings of security, respectively (see Figure 2 B). The PA questionnaires were derived from Carels et al, Cranford et al, De Vries et al, Dunton et al, Dunton et al, Grow et al, and Prins et al [21,27-32].

Four different types of response categories were used: Visual Analogue Scale (VAS slider) (see Figure 3 A), multiple-choice bullets (see Figure 3 B), multiple-choice open field combined with VAS slider (see Figure 3 C), and binary (see Figure 3 D). In the DI condition, 5-point (VAS slider) scales were used to measure mood, hunger, availability, and intention (eg, mood: “At the moment, I’m feeling happy”). Multiple-choice bullets were used as response categories for daily meal, self-evaluative emotions, location, activity, and social context (eg, location: “Where were you when you were having a snack?”). Multiple-choice open field combined with VAS sliders were used to indicate the type and frequency of fruits and vegetables, snacks, and sodas they had consumed (eg, “In the last three and a half hours, what type of snack(s) did you eat? And how many?”). These sliders ranged from 0-100, but the open fields enabled participants to fill in amounts that were larger than 100. Finally, binary response categories were used for decisions within the tree and food craving (eg, “In the last three and a half hours, have you eaten fruit or vegetables?”). All examples are translated from Dutch to English.

Feasibility and usability measures were derived from previously completed studies [15,16]. These measures contained 17 statements about the students’ subjective experience with the mEMA app (eg, “In my opinion, it was boring to work with the mEMA app”), the level of difficulty (eg, “In my opinion, the mEMA app was easy to use”), prompting sequence (eg, “In my opinion, the number of prompts that were sent during one week (5 times a day) was annoying”), length (eg, “In my opinion, the trial took too long”), and understanding (eg, “In my opinion, the questions were understandable”). Responses were scaled 1 (totally disagree) to 5 (totally agree). The midpoint of the scale was considered as a criteria for further interpretation (eg, scores above the midpoint of the usability scale were interpreted as “the mEMA app was easy to use”). Also, three open-ended questions were added to enable students to provide suggestions for further improvement of the mEMA app: (1) “Why did you ignore or postpone a prompt?”, (2) “I would make the following change to the mEMA app…”, and (3) “I would add the following to the mEMA app…”).

Ecological validity was determined by assessing the degree to which the collected data represent the full-range of possible social and physical influences. The assessment for both DI and PA included two questions regarding the location and social context (eg, “Where were you while having a snack?” and “While you were being physically active, how many people were there with you?”). Response categories for both the location and social context were multiple choice and similar for DI and PA. Multiple-choice options included the following locations and social settings: at home, school, work, friend’s house, outside, restaurant, cafeteria, supermarket, sports club, or elsewhere; and a friend, colleague, classmate, team member, sibling, parent, partner, child, teacher, stranger, or other, respectively for the social setting.

SPSS version 20 was used for data analyses. Data from the mEMA app were transported from the smartphone into a secure computer system when participants had access to a Wi-Fi connection. Next, data from the computer system were converted to an SPSS database. With SPSS, descriptive statistics were generated (means and percentages) and compliance rates were calculated for the exploration of response patterns. Furthermore, a t test was done in order to examine gender differences in the perceived usability of the mEMA app. Prior to the analyses, normality was tested by means of Q-Q plots. Mean differences were analyzed using t tests when normally distributed and Mann-Whitney U test when they were non-normally distributed.

After the first group had participated, technical problems were fixed and both feasibility and usability were evaluated by means of the aforementioned measurement procedures and group discussions. Based on the students’ feedback, two adjustments were made: (1) the number of daily prompts was reduced from 5 prompts a day to 4 prompts a day, and (2) the time sequence was reduced from 8 to 7 days of participation, as the start and evaluation of the study took place during their physical activity classes (once a week). Also, students evaluated the VAS sliders (see Figure 3 A, C) as bothersome. However, due to practical constraints (ie, Likert scales did not fit within the boundaries of all smartphone screens), usage of VAS sliders was maintained. The second group made use of the same mEMA app as the first group, but they were prompted 4 times a day during 7 consecutive days, instead of 5 prompts a day during 8 consecutive days.

An independent sample t test was conducted to test whether the usability of the mEMA app increased after the number of prompts was decreased from 5 to 4 times a day and the time sequence was reduced from 8 to 7 consecutive days. The results showed a significant difference between the first group (n=11) and the second group (n=6) regarding the app’s usability (t ₁₄=–2.15, P=.05). The second group perceived the app as easier to use (mean 4.33, SD 0.82) compared to the first group (mean 3.36, SD 0.92). However, no significant differences between the first group and the second group were found regarding the duration of the study (P=.19) and their evaluation (ie, how annoying) of the amount of prompting (P=.86).

Students’ compliance was reported per day (n=30). Thirty participants started using the mEMA app at Day 1 (100%), and 7 days later, 14 students still participated (44%). Compliance decreased 56%. Further exploration of times of day did not indicate clear differences in compliance between morning, early afternoon, late afternoon, early evening, and late evening (56%, 55%, 61%, 55%, and 56% respectively). The response patterns per time of day over 7 days of participation are projected in Figure 4. Interestingly, the mEMA app was used 23 times without being prompted (ie, user based). The overall mean completion time per response was 138.7 seconds (SD 65.6; for DI [mean 132.3, SD 63.7] and PA [mean 146.8, SD 67.4]).

Total response rates were assessed by calculating the percentage of answered prompts per day. From Day 1 through 7, total response rates were 63%, 54%, 48%, 35%, 31%, 23%, and 23% respectively (n=128 prompts per day). Over time, the total response rate decreased 40%.

Of all participants (n=30), 17 students (57%) completed the online evaluation form (ie, regarding the feasibility and usability of mEMA). Normality was assessed by means of Q-Q plots. These plots showed all items to be normally distributed. The mean scores (and standard deviations) of all items are displayed in Table 1. Participants reported that the mEMA app was relatively easy to use (mean 3.88, SD 0.93), that during the study they carried their smartphone with them every day (mean 4.65, SD 0.61), and that, according to them, the mEMA app worked well (mean 3.76, SD 1.09). Participants who took more time to complete the mEMA app questionnaires were also more likely to report that the time required to complete them was too lengthy (r=.154, P=.04). No significant differences in the perceived usability of the mEMA app between boys (mean 3.29, SD 0.95) and girls (mean 3.90, SD 9.99) were found (P=.88).

Ecological validity analyses were performed to indicate whether response categories covered all types of students’ snacking in the DI condition and if response categories in the PA condition covered all types of physical activity. Table 2 shows that the consumed snacks could be categorized as cookies, pastries, fast food, sweets, chips, chocolate, nuts, cheese/sausage, popcorn, and kebab/pizza. Out of all prompts that participants responded to (128 out of 1020 prompts), 58 prompts were answered with “Yes, I have consumed an unhealthy snack during the past 3.5 hours”. In total, 104 snacks were reported by participants, indicating that per snacking episode an average number of 1.8 snacks were consumed.

The reported PA could be broken down into the categories of walking, exercising, working (standing), biking, cleaning the house, doing groceries, shopping, and something else (eg, internship at a kindergarten; see Table 2). Out of all prompts that participants responded to (n=128), 48 prompts were answered with “Yes, I have been physically active during the past 3.5 hours”. In total, 97 activities were reported, indicating that during several of these 48 time periods of 3.5 hours each, approximately 2 different activities were performed.

In addition to the ecological validity of response categories for DI and PA, ecological validity was analyzed for social context and location. These two context-related categories refer to the people that joined the participants and their exact location while they consumed a snack or were physically active. Results indicate that response categories for the social context fitted with 93% of all responses for DI and 98.3% for PA. Remaining responses were categorized as “Other” (including “family”, “people from school”, and “grandma” for DI, and “people from the gym” for PA). Regarding the location in which participants consumed snacks or were physically active, the developed response categories fitted with 90% for DI and 96% for PA. Remaining responses were categorized as “Elsewhere” (including “family”, “car”, “theater”, and “camping site” for DI, and “gym” and “in town” for PA).

Table 3 provides an overview of the categorized social contexts and locations. The overall amount of DI and PA by all participants appeared to be higher than the amount of reported social contexts and locations, indicating that participants might have eaten different snacks and have performed different types of PA in the same setting.

In line with technological developments over the past few years, the mEMA technique is gaining in popularity with claims of better accuracy in ambulatory data collection [7]. The mEMA approach offers the opportunity to assess complex health behaviors in real-time settings, in which specifically routinized behaviors are involved (eg, DI and PA) [1]. However, the mEMA app faced several challenges that needed to be overcome in order to improve its validity. The present study showed that the mEMA app is a usable and ecologically valid tool to measure DI and PA behaviors among vocational education students, but compliance is still limited.

Vocational education students’ self-rated compliance was 70.6% and is in accordance with compliance rates of previous electronic EMA studies [33]. However, the registered compliance rates of the in situ registration by the mEMA app indicated a compliance rate of only 43.8%. Such a discrepancy may be explained by an availability heuristic, indicating an overestimation of compliance when students evaluate their compliance retrospectively as with the online evaluation form used in this study. According to Stone et al (2007), noncompliance might be caused by several factors, eg forgetting and monitoring burden [5]. Up to 3 reminder signals were programmed in the mEMA app when participants did not respond, with time intervals of 30 minutes. Therefore, we assume that the chance of forgetting to respond to a mEMA prompt as a cause of noncompliance in the present study was small. Additionally, the number of prompts was reduced from 5 times a day to 4 times a day, based on feedback received from the first group. Accordingly, a significant decrease in monitoring burden was expected. However, the results from the online evaluation indicated that more than half of the students who filled in the online evaluation form (52.3%) still experienced the number of prompts as bothersome. Therefore, monitoring burden might be a cause of the students’ noncompliance in the present study. Noncompliance in the present study might also be attributable to the educational level of the students, as higher noncompliance rates have been more commonly reported among less educated students [34].

The usability of mEMA was rated as good. However, it should be mentioned that on the online evaluation form participants commented that the VAS-slider (see Figure 3 A, C) was “difficult and sometimes bothersome to use.” These comments are in line with experiences from adolescents who participated in an electronic chronic pain diary study [33]. These adolescents also perceived the VAS slider as “hard to control” and commented that when they dragged the slider, it did not move the way they wanted. However, they also mentioned that once they discovered how to use the VAS slider more effectively, they “got the hang of it” (page 300, [33]). Stinson et al suggested that VAS slider use could be improved by creating a thicker slider so it would move more easily, changing several sliders into radio buttons, and adding space for adolescents to enter their own responses. These adaptations were taken into account during the development of the mEMA app. But, despite these adaptations and rehearsal time prior to the study, the slider was still perceived as difficult and sometimes annoying to use. However, when the students received an explanation of the VAS sliders after they participated in the study (ie, due to practical limitations), support for the slider increased. Therefore, irritation might be prevented by providing a good rationale prior to study, including an explanation of ways to use the slider effectively. In addition to providing a good rationale, future studies are encouraged to develop new 5-point scale designs on smartphones, in order to find a viable alternative to VAS sliders.

To our knowledge, empirical research on the ecological validity of mEMA is still lacking. In the present study, the response categories used within the contexts of DI and PA appeared to cover (almost) all responses. This indicates that the collected data represents the full range of social and physical factors of influence. These categories could be used in future mEMA research on DI and PA behavior.

Besides the response categories used within the contexts of DI and PA, the mEMA app has potential use for future research on complex cognitions and (health-related) behaviors for behavioral research in general. The rationale for using mEMA rests on three central benefits of this methodology. First, retrospective recall (and the associated biases) is greatly reduced with mEMA because people report on current or recent states and events that occurred shortly prior to the received prompt. Second, mEMA occurs in natural settings, increasing external and ecological validity compared to clinical settings (eg, laboratory research). Third, multiple assessments occur over time, so temporal relationships among variables can be explored [3,35]. Fourth, the use of mobile-based EMA offers the opportunity to examine cognitions, affect, and behavior in the everyday context of people, using the natural handling and carrying of mobile phones. As such, mEMA offers a good opportunity for those behavioral, social, and health scientists and practitioners who aim to understand, intervene, and evaluate the effects of interventions on behavior (change), to examine individual differences as well as contextual differences.

Finally, important limitations of the present study need to be addressed. First, not all students were able to participate in the mEMA study, because not all students possessed a smartphone that operated on (recent versions of) BlackBerry, Android, or iOS. Consequently, 14 out of 44 students were exempted from further participation in the mEMA study. In order to include these exempted students in future research, a solution may be found in the (temporary) provision of eligible smartphones. Second, the mEMA app may reduce the likelihood of various biases (eg, recall bias, availability, and recency effects), but other biases may still threaten the validity of responses, such as social desirability. Social desirability bias might play an important role in health-related behavior as, for instance, participants are expected to underreport their dietary intake and overestimate their physical activity [6-8,36,37]. However, Crutzen et al (2010) indicated with their study on social desirability and self-reported health risk behaviors in Web-based research that there was no meaningful association between social desirability and self-reported health risk behaviors in Web-based research [34]. Because students in the present study used the mEMA app in real time in the absence of the researcher, it might be that a meaningful association between social desirability and self-reported DI and PA in mEMA studies is lacking too. Therefore, a comparable study on social desirability in future mEMA research might be interesting. Nevertheless, correction for these types of biases might still be necessary. Third, time-based prompting and prompting frequency were adjusted to the school’s schedule. However, in the Dutch vocational education system, students are required to work as interns for 2 days a week, which, according to our participants, prevented them from responding to all prompting sequences. Some students also mentioned their spring break as a reason for low compliance. Accordingly, flexible prompting (ie, adjustable by the participant) or encouragement of user-based entries could be taken into account in future deployments.

Overall, mEMA offers the opportunity to assess complex health behaviors (eg, DI and PA) in real-time settings, in which specifically routinized behaviors are involved. However, the study also revealed some challenges with regard to use of the mEMA app that need to be taken into account in order to improve its validity. In particular, compliance is a reason for concern.