To accurately estimate the Bland-Altman limits of agreement between 2 methods, a sample size of 50-100 is required [23]. Allowing for 20% dropout, 121 participants aged ≥18 years were recruited from the University of Reading, UK, via email and poster advertising. Participants were provided with a study information sheet before participation and were asked to sign an informed consent form. A participant information form, which included self-reported weight and height measurements, was used to assess suitability for the study. Individuals reporting health issues or ill health, self-reported or diagnosed food intolerances, or special nutritional requirements (eg, pregnancy or lactation) were ineligible to participate. Ethical approval for the study was obtained from the School of Chemistry, Food and Pharmacy Research Ethics Committee, University of Reading, UK (01-12-Lovegrove).

Reproducibility of the Food4Me FFQ was determined by asking participants to complete the questionnaire on 2 occasions 4 weeks apart, mimicking its application in the Food4Me study. To assess the validity of the FFQ against a 4-day WFR, half the sample (those recruited first) were asked to complete a 4-day WFR 1 week following the first administration of the Food4Me-FFQ. Participants who completed both the Food4Me FFQ and 4-day WFR were also asked to complete a dietary record usability-rating questionnaire on Survey Monkey (Survey Monkey Inc, Palo Alto, CA, USA) in the week following the completion of the second Food4Me FFQ. The usability-rating questionnaire included questions about ease of use and willingness to complete the records. Participants were asked not to change their diet during the study.

Participants were asked to record all foods and beverages consumed over a nonconsecutive 4-day period that included 3 weekdays (Monday to Thursday) and 1 weekend day (Saturday to Sunday). Before completing the WFR, participants were coached on how to describe food products by a dietitian and provided with weighing scales (Salter Disc Electronic Kitchen Scales SKU# 1036 WHSSDR). When participants were unable to provide weighed portion size information, this was estimated retrospectively within 1 week using the Ministry of Agriculture, Fisheries and Food Portion Size Atlas [24].

The self-administered Food4Me FFQ is an online, semiquantitative food frequency questionnaire (developed by University College Dublin and Crème Software Ltd). To complete the questionnaire, participants were provided with a website address and unique log-in details. On logging into the server, participants were directed to a webpage containing detailed instructions on how to complete the FFQ. The questionnaire contained questions on the average consumption of 157 food items over the previous month. The food items were divided into the following 11 categories: (1) cereal, (2) bread and savory biscuits, (3) potatoes, rice and pasta, (4) meat and fish, (5) dairy products and fat, (6) fats and spreads, (7) sweets and snacks, (8) soups, sauces and spreads, (9) drinks, (10) fruit, and (11) vegetables. During completion of the Food4Me FFQ, participants were required to provide information on frequency of consumption and portion size. Frequency of consumption was measured by selecting one of the following options: never or less than once a month, 1-3 times a month, once a week, 2-4 times a week, 5-6 times per week, once a day, 2-3 times per day, 5-6 times per day, and >6 times per day. Food portion size was estimated using photographs. Each food item had 3 photographs representing small, medium, and large portions and these descriptors were provided below the appropriate image. Participants could select one of the following options: very small, small, small/medium, medium, medium/large, large, or very large which were linked electronically to portion sizes (in grams) (see Figure 1). Food intake (g/day) was calculated by multiplying frequency of consumption by the specified portion size (see Forster et al for detailed methods [18]). Further screenshots of the online Food4Me FFQ are shown in Multimedia Appendix 1.

The dietary record usability-rating questionnaire was comprised of 5 questions about the completion of the Food4Me FFQ and the 4-day WFR. Participants who completed both the Food4Me FFQ and the 4-day WFR (n=49) were asked to select one of the following responses to indicate their level of agreement: strongly agree, agree, neither agree nor disagree, disagree, or strongly disagree to the following questions:

The Henry equation [25] was used to calculate basal metabolic rate (BMR), and BMR was multiplied by a physical activity level (PAL) of 1.1 to calculate the lowest possible estimated energy requirements (EER) for each participant [26]. Participants reporting energy intakes lower than their EER were classified as underreporters. Participants reporting a daily energy intake greater than 4500 kcal, which is considered implausibly high, were excluded from the analysis [27].

Estimated nutritional intake data from the Food4Me FFQ were generated automatically by the online Food4Me programmed system, as described by Forster et al [18]. Composition of the food items listed in the FFQ were derived from WISP (Tinuviel Software, Anglesey, UK) [28] and modified to include recipes of composite dishes, generic commercial foods, new foods on the market, and current manufacturers information. The 4-day WFR intakes were analyzed using WISP (Tinuviel Software, Anglesey, UK) [28]. For the purpose of the current study, consumption of dietary supplements was not included in the analyses.

Statistical analyses were performed using SPSS version 20.0 (IBM Corp, Armonk, NY, USA). Mean nutrient intakes and standard deviations were calculated for baseline characteristics, repeated Food4Me FFQ, and 4-day WFR. Differences in participant characteristics and energy intakes (kcal) were assessed using a paired 2-sample t test. Nutrient intakes were compared using general linear model (GLM) analysis controlling for energy and gender where there was significant interaction between gender and nutrient intake. Data were checked for normality using the Shapiro-Wilk test and, depending on the outcome, the association between dietary intake methods and repeated Food4Me FFQ were assessed using either Pearson product-moment correlation or Spearman correlation coefficient (SCC, rho). A P value of <.05 was considered statistically significant.

The relative agreement between the dietary intake methods and repeated Food4Me FFQ was assessed using cross-classification of nutrient intakes to estimate the percentage of participants classified into quartiles as follows: exact agreement (percentage of cases classified into the same quartile), exact agreement plus adjacent (percentage of cases cross-classified into the same or adjacent quartile), disagreement (percentage of cases cross-classified 2 quartiles apart), and extreme disagreement (percentage of cases cross-classified into extreme quartiles). For intakes of energy and macronutrients, the Bland-Altman method [29] was used to further assess the limits of agreement between the 2 methods (Food4Me FFQ and WFR) and between the repeated Food4Me FFQ. As per the Bland-Altman methodology, dietary records were considered comparable/ repeatable if greater than 95% of data plots lay within 2 standard deviations of the mean. GraphPad PRISM version 6 was used to produce the Bland-Altman plots (GraphPad Software, Inc, La Jolla, CA, USA).

Differences in food group intakes between the FFQ and WFR and repeated Food4Me FFQ were also examined. For this purpose, food items in the Food4Me FFQ and 4-day WFR were arranged into 35 food groups as per previous validation by Forster et al [18]. SCC were calculated to assess the strength of association between methods for estimated intakes of the 35 food groups. To assess the relative agreement between the dietary methods and repeated FFQ for daily food group intake, food groups were also cross-classified to estimate the percentage of participants classified by the 2 methods into quartiles of exact agreement, exact agreement plus adjacent, disagreement, and extreme disagreement.

A total of 121 participants were screened for inclusion in the study, of which 113 were deemed eligible. Reasons for exclusion included self-diagnosed food intolerance (n=7) and medication use (n=1). Before completion, 10 participants dropped out of the study and a further 3 were excluded from analysis due to a reported energy intake >4500 kcal [26]. The final dataset for analysis included 100 participants, of whom 49 had also completed the 4-day WFR, as illustrated in Figure 2. A total of 48 participants completed the Diet Record Usability-Rating Questionnaire.

Self-reported demographic characteristics of the participants in the reproducibility and validation study are shown in Table 1. No significant differences were observed between age and body mass index (BMI) for males and females. Participants who completed the WFR (validation study) were, on average, 4.6 years younger than the participant group as a whole.

Mean values and standard deviations for responses to the dietary record usability-rating questionnaire are shown in Table 8.

The Food4Me FFQ was considered significantly easier and less time consuming to complete than the 4-day WFR. However, the 4-day WFR was rated as significantly more interesting than the Food4Me FFQ, making participants reflect more on their food intake. Participants were more willing to complete further Food4Me FFQ than 4-day WFR.

Previous validation of the Food4Me FFQ has demonstrated good agreement with the printed EPIC-Norfolk FFQ for the estimation of food and nutrient intake. In this study, participants were asked to complete the Food4Me and EPIC-Norfolk FFQ in a random order, 4 weeks apart. Good agreement between cross-classifications of daily energy and nutrient intakes, estimated using the 2 FFQ, demonstrated the utility of the Food4Me FFQ for ranking individuals based on their nutrient intake. However, it was noted that further testing of the Food4Me FFQ was required to establish its wider utility [18]. The present study thus aimed to demonstrate the reproducibility of the Food4Me FFQ and its validity against a 4-day WFR.

Overall, the Food4Me FFQ demonstrated good reproducibility for the estimation of intakes of nutrients and food groups. Reported energy intakes were significantly lower with the second administration of the Food4Me FFQ, but correlations between energy intakes were high (r=.77). Correlation coefficients for nutrient intakes ranged from .65-.90, showing above-average performance compared with the range of .50-.80 proposed by Willet [15]. The mean unadjusted correlation coefficient (r=.75) for energy and nutrient intake compared well with previous studies on both computerized [21,30-31] and non-Web-based FFQ [23,32-35]. Associations between food group intakes were similarly strong with an average unadjusted SCC of .75; previous studies have reported correlations of .66 and .72 [36-37]. Cross-classification analysis of repeated measures of intakes of energy, nutrients, and food groups indicated a high level of reproducibility with classification into quartiles of exact agreement plus adjacent averaging 92% for energy and nutrient intake and for food group intakes. Cross-classifications were within the range reported by previous studies [21,32,38]. Bland-Altman plots demonstrated a good level of reproducibility for energy-controlled total fat, protein, and carbohydrate intake, which reinforces evidence for the reliability of the Food4Me FFQ.

Estimated energy and nutrient intakes were higher on the first administration of the Food4Me FFQ than on the second administration. This pattern has been observed in numerous other reproducibility studies [21,31-32], and is proposed to result from learning effects and questionnaire fatigue [39]. The above-average reproducibility of the Food4Me FFQ could be attributed to the addition of food photographs to the FFQ for the estimation of portion size intake. Use of tools that allow participants to report their own portion sizes tend to report higher correlation coefficients between repeat administrations [19]. The relatively short interval between repeat administrations of the FFQ is another factor likely to have contributed to the questionnaire’s good performance. Tsubono et al [40] found that correlation coefficients tended to be lower when FFQ were repeated after a long time interval (6 months to 1 year) compared with a shorter time interval (1 to 6 months), and proposed that the temporal difference may be due to changes in dietary habits which are more likely to occur with longer time intervals. In addition, it has also been suggested that for very short time intervals between administrations, respondents may remember and replicate their entries rather than reporting their diet intake accurately [23]. However, with a large FFQ containing 157 food items, as used in the present study, it is unlikely that many participants would be able to remember their earlier responses.

The degree of underreporting between the Food4Me FFQ and 4-day WFR varied, with 12 (24%) and 15 (15%) participants deemed to be underreporting in the 4-day WFR and Food4Me FFQ, respectively. Given that the WFR is described as the gold standard for assessing intake, our observation that estimates of energy intake were similar between the Food4Me FFQ and the 4-day WFR and that a smaller proportion of participants appeared to underreport with the Food4Me FFQ suggests that the Food4Me FFQ is a promising tool for estimating habitual food intake.

Overall, the results of the validation study showed moderate agreement between the Food4Me FFQ and 4-day WFR for the estimation of energy and nutrient intake. Ranks of energy and nutrient intake estimated using the Food4Me FFQ were highly comparable to the 4-day WFR with the percentage of individuals classified into quartiles of exact agreement and exact agreement plus adjacent averaging 40% and 80%, respectively. Previous studies comparing FFQ with food records have reported average exact agreement classifications between 34% and 49% [32,38,41] and exact agreement plus adjacent quartile classifications of 77% [21]. Estimates of intake showing disagreement between measurement tools in the present study were small and were comparable with the aforementioned studies. Cross-classifications of estimates of food group intake were similar to that of the nutrients, with classification into quartiles of exact agreement plus adjacent averaging 78% and Bland-Altman plots demonstrated good agreement between the 2 methods for estimates of energy and energy-adjusted macronutrient intakes.

In the present study, 22 of 30 nutrients assessed had a correlation coefficient greater than the .4 threshold that was proposed by Cade at al. [23], and 13 of 30 achieved a correlation greater than or equal to the “desirable” .5 proposed by Masson et al [42]. The average unadjusted correlation coefficient of .47 compared favorably with the range reported by similar validation studies comparing FFQ with food records: .34-.46 [31-32,34,41]. SCC for food group intakes were highly variable, ranging from .11 (soups, sauces and miscellaneous foods) to .73 (yogurts), with a mean value of .2. Similar studies have reported correlations ranging from .09 to .83 [37], .17 to .95 [36], and .09 to .58 [41] with mean values of .38, .63, and .58, respectively. However, it is difficult to compare results from these studies because the type of food records and time intervals between dietary assessments differed substantially and there may be substantial differences in the food items included in particular food groups in each of the studies. Variation between Food4Me FFQ and 4-day WFR estimates were greatest for soups, sauces and spreads, carrots, other fruit and vegetables, and tinned fruit and vegetables. It is possible that intakes of these foods might have been overestimated in the Food4Me FFQ, as has been observed previously when several food items within a food group are listed separately in a questionnaire (eg, carrots could be counted under both fresh and tinned carrots and under tinned vegetables) [43]. In addition, foods perceived as healthy, such as fruit and vegetables, are prone to overestimation in FFQ. Furthermore, because they refer to just 4 days’ intake, WFR provide a limited snapshot of an individual’s diet only and are less able to assess patterns of dietary intake than the Food4Me FFQ, which attempts to capture intakes over the previous month. It is thought that individuals may be able to more accurately estimate the consumption of some foods (eg, alcoholic beverages) than others, as was the case in the present study [21,44]. Alcohol is often considered a confounder in nutrition research given that it constitutes the difference between food and total dietary energy intake; therefore, it is important that it is estimated reliably using the Food4Me FFQ. It is also encouraging that estimates of fish products were well correlated, as these foods are eaten less frequently and may be prone to underrepresentation in 4-day WFR. However, it is surprising that some more commonly consumed foods such as breakfast cereals and porridge show much weaker correlation (r=.27).

Our observation that participants in the present study reported that the Food4Me FFQ was easier to use and less time consuming compared with the 4-day WFR, is promising given the movement of health service delivery toward Web-based interventions. Moreover, completion of the Food4Me FFQ was associated with less reflection by participants on their dietary intake, which is known to influence eating behavior. Minimizing the impact of a questionnaire on dietary behavior is beneficial in nutrition intervention studies to ensure that study outcomes are not biased by the methods used for dietary assessment.

Strengths of the present study include the comparison of the Food4Me FFQ with the gold standard, a WFR, and the use of multiple methods to assess the validation and reproducibility of the Food4Me FFQ. In addition, this validation study had an adequate sample size [15] similar to those used in previous studies [30,45-46]. It should be noted that the validation of the Food4Me FFQ was assessed in a convenient rather than a nationally representative sample of the population, although the inclusion/exclusion criteria used were the same as those intended for the Food4Me study [16]. The use of a convenient university population, with a potentially higher education level, may have implications on the ability of the wider population to complete the online Food4Me FFQ.

Limitations of the study include the use of those recruited first to complete the 4-day WFR because these individuals may have been more motivated to comply with the guidelines. A further limitation is the use of nonconsecutive days in the 4-day WFR, which may have resulted in participants making up food intake on the days they do not record (eg, eating healthy for the record days and overconsuming between record days). However, nonconsecutive recording does have the advantage of capturing a greater diversity of food intake over a week’s period.

A potential criticism of the assessment of reproducibility is the short duration between repeated measures. It has been suggested that for very short time intervals, respondents may remember and replicate their entries rather than accurately reporting their dietary intake [23], but this is unlikely to be a significant problem in the present study where the Food4Me FFQ contained 157 food items. Remembering their responses to such a long list of questions after a period of 4 weeks is a memory challenge beyond most people’s abilities. The average reported correlation coefficient for crude total fat intake using FFQs repeated after 1 month or less was .68 [19], which compares very favorably with the correlation coefficient of .81 in the present study, showing above-average performance for the Food4Me FFQ. Cade et al [23] suggested that the time interval between repeated measures using a dietary instrument should be chosen to minimize changes in dietary intake and our use of 4 weeks fits that criterion.

In conclusion, the self-administered online Food4Me FFQ demonstrates good reproducibility for the estimation of energy, nutrient, and food group intakes and moderate agreement for the assessment of energy and nutrient intakes when compared with a 4-day WFR in an adult population. Consequently, the online Food4Me FFQ was considered suitable for the assessment of dietary intake in healthy adults.