We contacted 322 people and included 200 participants older than 20 years who lived in Seoul, Korea. Over a quarter (56/200, 28%) answered “Yes” to the question of searching history for influenza and provided search queries (Table 1).

Query group 1 contained 14 queries that were specific to influenza, and query group 2 had 14 queries that were not specific to influenza (Table 2). A total of 210 combined queries were submitted to Daum. Full data of combined queries are presented in Multimedia Appendix 2.

Correlation analysis was performed between the Daum data for combined queries and the KCDC ILI data in development sets 1 and 2 (Table 3). In development set 1, “New flu (abbr.)” had the highest correlation coefficient (r=.894, P<.001), and 13 combined queries had correlation coefficient r values of ≥.7. Among these 13 combined queries, the number of the combined queries that had correlation coefficient r values of ≥.7 was reduced to 4 in validation set 1 and to 2 in validation set 2. In development set 2, “Bad cold + Symptom” had the highest correlation coefficient (r=.969, P<.001), and a total of 15 combined queries had an r value of ≥.7. Among these 15 combined queries, the number of the combined queries that had correlation coefficient r values of ≥.7 was reduced to 6 in validation set 2. Only “Tamiflu” and “New flu + Symptom” showed correlation coefficients r values of ≥.7 for 3 consecutive years (Figure 1). The change of correlation coefficients for all combined queries over time are presented in Multimedia Appendix 2.

A total of 13 cumulative query methods were created in development set 1 (see Table 4). In validation set 1, cumulative query methods 7, 8, 9, and 10 showed the highest correlation coefficients (r=.943, P<.001; see Multimedia Appendix 3). Eight of the 13 cumulative query methods were useful, which was defined as having higher correlation coefficients than the highest correlation coefficient of a single combined query in development set 1 (min=.916, max=.943). But only three of the cumulative query methods from development set 1 were useful in validation set 2 (min=.935, max=.953). The correlation did not increase by adding queries in cumulative query method 5, 6, 8, 9, 10, and 13 in validation set 1. In validation set 2, cumulative query method 5 from development set 2 had the highest correlation coefficient (r=.987, P<.001; see Figure 2 and Multimedia Appendix 3). Eight of the 15 cumulative query methods from development set 2 were useful (min=.975, max=.987). The correlation did not increase by adding queries in cumulative query method 3, 4, 7, 8, 10, 12, and 14 in validation set 2. Scatter plots between the KCDC ILI and other useful cumulative query methods are presented in Multimedia Appendix 4. Cumulative query methods for influenza virologic data are presented in Multimedia Appendix 5.

In each development set, cumulative query methods had a higher correlation coefficient than combined queries (see Tables 5 and 6). After 1 year, 11 of 13 cumulative query methods had an r value of ≥.7, but 4 of 13 combined queries had an r value of ≥.7 in validation set 1 (see Table 5 and Figure 3). All 15 cumulative query methods had an r value of ≥.7, but 6 of 15 combined queries had an r value of ≥.7 in validation set 2 (see Table 6 and Figure 4).

In this study, the cumulative query method showed relatively higher correlation with national influenza surveillance data than combined queries in the development and validation set.

Many people use Internet searches for health information before visiting a doctor [18,20,23,33]. Hence, search query trends can reflect actual disease progression earlier than conventional surveillance. Queries used prior to this study only reflected the authors’ opinions [13] or were obtained from databases [13,14,23,37]. To obtain population search queries, we carried out a study survey.

Search queries may vary from country to country. In Korea, “Bad cold (독감)” in Korean has two meanings: one is influenza and the other, a severe common cold. Since the 2009/10 epidemiological season, the Influenza A (H1N1) pandemic period, the media began to use “New flu (신종플루)” in order to distinguish H1N1 influenza and previous influenzas. In 2010, KCDC announced that the official term was “Influenza (인플루엔자)” [36]. But “New flu (신종플루)” and “Bad cold (독감)” are still more popular terms than “Flu (플루)” or “Influenza (인플루엔자)” in Korea (Table 1).

For the 2009/10 epidemiological year (development set 1), 13 combined queries had correlation coefficient r values ≥.7. However, only 4 of these combined queries (“New flu + Cough”, “Flu + Symptom”, “New flu + Symptom”, and “Tamiflu”) had correlation coefficient r values ≥.7 in the 2010/11 epidemiological year (validation set 1) (Table 3). But 11 of 13 cumulative query methods had an r value of ≥.7 in validation set 1 (see Table 5 and Figure 3). Among 15 combined queries of development set 2, the number of the combined queries that had correlation coefficient r values of ≥.7 reduced to 6 in validation set 2. But all 15 cumulative query methods had an r value of ≥.7 in validation set 2. We think that the cumulative query method is more robust with time, and this factor is helpful for improving surveillance performance using search queries. Since Internet searching behavior may change over time, this could have affected the performance of Web query-based surveillance model [31]. In this study, 20 out of 210 combined queries had correlation coefficients for all 3 years. And only “Tamiflu” and “New flu + Symptom” showed correlation coefficients r values of ≥.7 for 3 consecutive years (see Figure 1 and Multimedia Appendix 2). Recently, a study using Google Trends for influenza surveillance showed that Google Trends can be used as a complementary source of data [33]; however, its performance is insufficient for use as a model for prediction because its maximum correlation coefficient was .82 for only one query, “Fever”, in 2009, and the coefficient decreased to .64 in 2011 [33].

It is difficult to predict the change of search queries in the future. To reduce the effects from changes in search queries, we used a combination of queries and cumulation of combined queries to construct our method. Additionally, the method we wanted to develop was meaningful only when the cumulative query method had a higher correlation coefficient than the highest single combined query. In each validation set, 8 useful cumulative query methods were developed. The useful cumulative query methods in each validation set had a high correlation coefficient (Table 4). In validation set 2, the range of correlation coefficients of the useful cumulative query methods was from .975 to .987. These values are similar to or higher than those reported elsewhere [13,14,23,26,27,29,31]. In Europe, correlation coefficients of .716 to .940 were reported for GFT [27], and coefficients of .82 to .99 were reported in the United States [23,26,31]. In the 2009/10 epidemiological year, called the Influenza A (H1N1) pandemic period, the proportional data of queries were likely to have been different compared to the other epidemiological year. It might affect performance of cumulative query methods in set 1. The performance of the cumulative query method in set 1 was decreased with time (Table 4). It is thought to be related to the changes of queries (see Table 3). For some cumulative query methods, the correlation did not increase by adding queries. The added query did not give extra value in the cumulative query methods 6, 8, and 10 in validation set 1 (see Table 4 and Multimedia Appendix 2). Combined queries 6, 8, and 10 from development set 1 in validation set 1 have a constant value 0 (see Table 3 and Multimedia Appendix 2). The added queries were relatively too small compare to the previous queries in the cumulative query methods 5, 9, 10 in validation sets 1, 3, 4, 7, 8, 10, 12, and 14 in validation set 2 (see Table 4 and Multimedia Appendix 2).

We used proportional data from Daum, a non-dominant local search engine (approximately 25% of the market share) in South Korea [32]. Our cumulative query methods showed a strong correlation with KCDC ILI data. Generally, Google’s market share is dominant in countries where GFT is available [27,28]. Our study showed the possibility of developing a surveillance model using a non-dominant local search engine.

There are several limitations to this study. The survey of our study is not a representative sample. Because respondents were asked to provide typed queries without mention of the influenza pandemic of 2009/10, recent search queries were more likely to have been included in this study because the survey was conducted recently. This might affect performance of the cumulative query method. Further, the data from the influenza pandemic of 2009/10 might affect the outcome of this study. In this study, we did not combine queries from the same query group. Although important, the performance of using symptoms in the definition of KCDC ILI was not tested. The learning effect from the influenza pandemic of 2009/10, news reports, outbreak briefs, health information from the Internet, and changing search behavior stemming from the diffusion of smartphones might have affected the outcome of this study. We did not determine the extent to which these factors affected the searching behavior. More data for subsequent years are required in order to know the life of the cumulative query method.

We presented a cumulative query method using search engine data. We conducted a survey to obtain population search queries. To reduce the effects from changes in search queries, we used a combination of queries and cumulation of combined queries. Our method showed high correlation with national influenza surveillance data in South Korea. However, to further our method, additional research is needed.