Published on 13.12.12 in Vol 14, No 6 (2012): Nov-Dec
Searching for Truth: Internet Search Patterns as a Method of Investigating Online Responses to a Russian Illicit Drug Policy Debate
Background: This is a methodological study investigating the online responses to a national debate over an important health and social problem in Russia. Russia is the largest Internet market in Europe, exceeding Germany in the absolute number of users. However, Russia is unusual in that the main search provider is not Google, but Yandex.
Objective: This study had two main objectives. First, to validate Yandex search patterns against those provided by Google, and second, to test this method's adequacy for investigating online interest in a 2010 national debate over Russian illicit drug policy. We hoped to learn what search patterns and specific search terms could reveal about the relative importance and geographic distribution of interest in this debate.
Methods: A national drug debate, centering on the anti-drug campaigner Egor Bychkov, was one of the main Russian domestic news events of 2010. Public interest in this episode was accompanied by increased Internet search. First, we measured the search patterns for 13 search terms related to the Bychkov episode and concurrent domestic events by extracting data from Google Insights for Search (GIFS) and Yandex WordStat (YaW). We conducted Spearman Rank Correlation of GIFS and YaW search data series. Second, we coded all 420 primary posts from Bychkov's personal blog between March 2010 and March 2012 to identify the main themes. Third, we compared GIFS and Yandex policies concerning the public release of search volume data. Finally, we established the relationship between salient drug issues and the Bychkov episode.
Results: We found a consistent pattern of strong to moderate positive correlations between Google and Yandex for the terms "Egor Bychkov" (rs = 0.88, P < .001), “Bychkov” (rs = .78, P < .001) and “Khimki”(rs = 0.92, P < .001). Peak search volumes for the Bychkov episode were comparable to other prominent domestic political events during 2010. Monthly search counts were 146,689 for “Bychkov” and 48,084 for “Egor Bychkov”, compared to 53,403 for “Khimki” in Yandex. We found Google potentially provides timely search results, whereas Yandex provides more accurate geographic localization. The correlation was moderate to strong between search terms representing the Bychkov episode and terms representing salient drug issues in Yandex–“illicit drug treatment” (rs = .90, P < .001), "illicit drugs" (rs = .76, P < .001), and "drug addiction" (rs = .74, P < .001). Google correlations were weaker or absent–"illicit drug treatment" (rs = .12, P = .58), “illicit drugs ” (rs = -0.29, P = .17), and "drug addiction" (rs = .68, P < .001).
Conclusions: This study contributes to the methodological literature on the analysis of search patterns for public health. This paper investigated the relationship between Google and Yandex, and contributed to the broader methods literature by highlighting both the potential and limitations of these two search providers. We believe that Yandex Wordstat is a potentially valuable, and underused data source for researchers working on Russian-related illicit drug policy and other public health problems. The Russian Federation, with its large, geographically dispersed, and politically engaged online population presents unique opportunities for studying the evolving influence of the Internet on politics and policy, using low cost methods resilient against potential increases in censorship.
J Med Internet Res 2012;14(6):e165)
This is a methodological study investigating the online responses to a national debate regarding an important health and social problem in Russia. Russia has the largest Internet market in Europe, exceeding Germany in the number of users. However, Russia is unusual in that the main search provider is not Google, but Yandex. By exploring the relationship between Yandex and Google, this study contributes to the methodological literature on analysis of search patterns for public health policy.
Studies of Internet search patterns provide a low cost, rapidly accessible data source across a range of disciplines. Underpinning these studies is the principle that each Internet search is a behavioral measure of an issue’s importance to an individual . If individuals are concerned or interested in an issue, they are more likely to search for information related to that issue. The relative importance of an issue can thus be inferred from the volume of search queries for a specific term or terms representing that issue.
Infodemiology and Infoveillance
The first studies of Internet search patterns were related to medicine. These initial studies examined the quality of online information , searches for cancer related information [ ], and influenza surveillance use diverse data sources including Google advertisements [ ], and Yahoo search trends [ ]. However it was the release of Google Insights For Search (GIFS) [ ] and the publication of 3 influential articles in 2009 that provided an impetus to this emerging field. The 3 influential articles are Ginsberg's study of influenza surveillance using Google data in Nature [ ], Brownstein's review of online surveillance in the New England Journal of Medicine [ ], and Eysenbach's consolidation of infodemiology as a distinct field of medical inquiry [ ]. Eysenbach describes infodemiology as “the science of distribution and determinants of information ...specifically (on) the Internet, or in a population, with the ultimate aim to inform public health and public policy...(including) data on what people browse, buy, and read”. Where infodemiology methods are used for epidemiological surveillance, Eysenbach refers to this as infoveillance. Since 2009, Google data has been the main data source for infodemiology studies, across a wide range of health problems including dengue [ ], depression [ ], abortion [ ], tobacco control [ ], and the global Google Flu Trends site [ ].
Analysis of Search Patterns in Political Communications
Infodemiology methods have also been applied to the study of political communications and policy processes [, - ]. These studies are generally founded on agenda setting theory, and use Internet search patterns as a data source to complement opinion polling or traditional media (ie, television and print) coverage. Agenda setting theory suggests that issues prominently covered in traditional media are subsequently ranked as important (or salient) in public opinion polls [ , ]. The transfer of issue salience from reporting in the media to influence public opinion is an important concept in agenda setting theory [ ]. Agenda setting theory appears in health advocacy studies [ ], but has not yet been incorporated into studies of Internet search patterns for health policy processes.
Agenda Setting Online
Since the early 2000s, studies of issue salience have increasingly focused on the interplay between traditional, online media, and the public agenda. A study of online bulletin board discussions found that media reports were rapidly reflected in online discussions . Rather than several weeks, themes emerged in online discussions within days of traditional media reports. More recent studies have used GIFS to measure and analyse search patterns in responses to prominent media issues. Granka suggests that issue importance can be inferred from overall changes in search query volume, and that search volumes rise and fall rapidly with public interest [ ]. Similarly, Scharkow suggests search patterns are the behavioral effects of salience, and provide valid and reliable measures of the public agenda [ ].
Is the Issue Salience Applicable Outside High Income Liberal Democracies?
Most studies of issue salience using search have been conducted in the United States and Organization for Economic Cooperation and Development (OECD) countries. However, there is some uncertainty as to whether the transfer of salience from traditional or online media to public opinion is universally applicable in low and middle income countries with different institutional arrangements. For example McCombs suggested that agenda setting effects require a reasonably free political system and media . Other authors too, have noted the lack of research into how media shape public opinion in less-than-democratic nations. For example, Moy and colleagues point to "a glaring absence of (research about)... how citizens in these states respond to specific televised messages or their attitudes regarding certain political and social issues" [ ]. This means it is difficult to infer public opinion from findings of issue salience in online media outside of high income liberal democracies.
The State of Russian Traditional and Online Media
The Russian Federation is a middle income country with institutional arrangements that sharply contrast to those in the US or European Union (EU). Contemporary Russia has been described as a managed democracy  and an authoritarian state [ ]. Several studies have pointed to a complex relationship between Russian traditional media, online media, and public opinion. A study of Russian's reactions to news broadcasts in 2005 reported marked differences between Russian and US viewer reactions to television news [ ]. Russians were found to adopt a range of cognitive strategies, routinely reinterpreting the frames presented in television news stories using complex reasoning outcomes. These strategies were, the authors suggested, consistent with Soviet-era television viewing. Others have noted unique patterns of online media use. In 2009, Russians were the most engaged social media users globally [ ]. Further, Russians engaged in unusually heterogeneous debates ranging across the political spectrum, as distinct from the partisan “echo chambers” that characterize online debate in the US [ ]. These studies caution against a simple transposition of agenda setting and issue salience theories to traditional and online media. This suggests more complex process than that suggested by agenda setting and issue salience theories.
Importance of Online Search in the Russian Federation
In 2011, Russia overtook Germany as the European country with the highest number of unique visitors online . Russian Internet users grew from 43% of the population in 2010 [ ] to 55% in 2012 [ ]. In May 2011, Google provided 84% of Internet search queries globally [ ]. The structure of the Russian-language Internet market is unique. Yandex provided 60% of Russian Internet searches in 2010-2011, compared with Google's 25% [ ]. Further, Yandex offers the Wordstat (YaW) search pattern analysis tool as a direct competitor to GIFS ( ).
Russian Online Media
In response to political and media constraints, Russian political debate increasingly moved online during the 2000s, using platforms such as LiveJournal , and more recently, Twitter [ ]. RuNet has been described as a catalyst for social activism [ ] political mobilization [ ], as well as a channel for an alternative news agenda [ ]. As Internet use grew, corruption and abuses of government power emerged as important themes online. (See for example the anticorruption blog Rospil.net [ ]). By 2012, the proliferation of Russian social commentary blogs prompted a Harvard study to described Russian online media as a “transparency watchdog” [ ].
RuNet's rapid growth led Russian media commentators to speculate that the Internet had eclipsed television’s traditional agenda setting function in importance [, ]. However, other observers have cautioned against overstating the importance of Russian online media, or its distance from mainstream practices. Less optimistic observers have described pro-government blogging campaigns [ ], cyber attacks [ ], monitoring dissent [ ], and sophisticated security filtering through SORM 2 [ ]. National surveys too, suggest a more modest role for online media in shaping public opinion. A 2012 survey found 63% of Russians mostly or completely believe traditional media, while 43% mostly believe online sources [ ]. In summary, as elsewhere, online media provide Russians with an information source complementing traditional media.
This is a methodological study that makes use of the unique characteristics of the Russian Internet search market. Firstly, we aimed to validate Yandex search patterns against those provided by Google. Secondly, we tested this method's adequacy for investigating online interest in a 2010 national debate over Russian illicit drug policy. In order to achieve these two aims we sought answers to the following questions:
1) What is the relationship between Google and Yandex search results?
2) What do search intensities and patterns reveal about the relative importance of an event?
3) How timely and geographically precise are GIFS and YaW results?
4) How do the search patterns during a national debate relate to salient drug issues?
Researchers have devoted considerable effort to establishing the validity of search pattern studies. Validity is the extent to which a test measures what it claims to measure . The initial studies using GIFS established a correlation between search patterns and epidemiological surveillance data for influenza [ ]. Other studies focusing on issue salience, established a correlation between search patterns, traditional media [ ], and opinion polling [ ]. Studies with large data sets have commonly employed ARIMA tests [ ] vector auto regression and visual comparison [ ], and multivariate regression [ ]. Studies with smaller sample sizes have generally conducted bivariate analysis with little or no data preparation [ , ].
Several common warnings concerning validity recur in studies of search patterns. The unrepresentative demographic sampling of GIFS populations is the most frequently cited concern. The Internet user population is generally regarded as younger and wealthier than the overall population, although this cannot be elicited from search data directly . Scharkow et al expand on this concern, questioning whether survey populations from traditional surveys and search pattern studies are comparable [ ].
The second concern relates to the disambiguation of search terms. Individual search terms may return ambiguous search results. Care with selection of search terms, and an appropriate range of search terms is necessary to capture the breadth of potential search terms for a concept .
Third, several problems are associated with the limited transparency of Google's treatment of data. Google does not reveal the threshold search volume used to determine whether data is reported on a search term in GIFS . This can produce unexpected zero values in time series. Further, Google provides results as relative rather than absolute data (That is, GIFS results are provided as a percentage relative to 100% during the user-defined date range-eg, 30% during June 2010). In addition, GIFS data is normalized and scaled, making comparisons between countries, regions and time spans difficult. Despite these limitations, there is a general consensus in the scholarly literature cited above that GIFS is a valid, low cost and flexible field research method. By analyzing online response to a Russian illicit drug policy episode, we hope to develop further develop the methods of search pattern analysis.
The Bychkov Episode
Illicit drug use is a serious social and policy problem in Russia. Russian public opinion surveys have consistently rated illicit drug use among the most serious of domestic social problems [, ]. However, Russian policy responses to this problem are generally regarded as punitive, unsupported by scientific evidence, and ineffective [ ]. The punitive aspect of Russian drug policies is exemplified in 2010 by a law prohibiting dissemination of drug related health information [ ]. In this complex environment, Russian reformers have compared attempts at influencing drug policy to "throwing spaghetti against a wall, and seeing what will stick” [ ].
We selected the public debate surrounding the court case against socially conservative drug policy reformer Egor Bychkov as our case study. Bychkov was the president of an NGO operating a non-medical drug rehabilitation center in the provincial Urals city of Nizhny Tagil, 1900 kilometers east of Moscow. In October 2010, a local court convicted Bychkov of holding several rehabilitation clients hostage. His subsequent imprisonment sparked widespread coverage in the Russian national media. Although Bychkov was outspoken in expressing reactionary social attitudes  and his belief in harsh, unscientific treatment methods, he won support from socially liberal as well as conservative commentators for bringing the fight against corrupt local courts and police into the open [ ]. In November 2010, following national media support, and presidential intervention, Bychkov was released on parole [ ]. In June 2011, all criminal charges against Bychkov were finally annulled [ ]. In 2011, having achieved national prominence, Bychkov collaborated with leading liberal bloggers and opposition politicians in Moscow [ ]. In summary, the Bychkov episode was one of several concurrent episodes of opposition to Russian government policies. In each case, traditional media sparked and spread popular outrage, leaving patterns of online search.
What is the Relationship between Google and Yandex Search Results?
We took the approach that this was an initial investigation with a small data set, and following earlier studies, did not cleanse data . This approach had the added advantage of allowing us to quantify threshold and relative data problems in GIFS. Our data collection and analysis involved the following steps.
First, we used the terms "Bychkov" and "Egor Bychkov" to represent the Bychkov episode. We used additional terms unrelated to the Bychkov episode to provide additional context for the Bychkov episode, and to test for the validity of correlation between GIFS and YaW. We identified the main Russian domestic news events of 2010 from end-of-year compilations on government and non-government media organizations (). Further detail about this process appears below.
Second, we extracted search data series for terms representing the Bychkov episode and concurrent events from GIFS and YaW. Studies demonstrating online responses to media events typically use weekly or daily GIFS data (eg, ). Weekly GIFS data and monthly YaW data were available. Daily GIFS data was unavailable, and most weekly GIFS data series recorded zero values for one or more weeks during this date range. We managed this by combining the weekly GIFS ratios, including zero values, to produce a GIFS value for each month.
This produced one time series of monthly GIF values, and another of YaW monthly values. We used monthly data in the date range between March 2010 and March 2012, as these were the maximum data points available in YaW. While this produced fewer data points than some previous studies, these were sufficient to conduct a correlation analysis. We were aware monthly data was not sufficiently frequent to establish relationships between salient media issues and online search patterns.
Third, in the absence of daily or weekly data, we corroborated events during the peak period of interest in the Bychkov episode with other available measures. We created a graph plotting GIFS daily searches against relevant media reports identified in Bychkov’s blog (). We anticipated that the one month time range between data points would make it difficult to distinguish changes in the relationship between individual news events and increased search. We therefore turned to a weekly GIFS series to provide visual corroboration of the relationship between traditional media and search patterns over the period of maximum public and media attention to the Bychkov episode.
Fourth, we conducted bivariate analysis of GIFS and YaW data. We plotted peak monthly GIFS and YaW monthly search values to produce two time series for each search term (). Given the diversity of approaches in previous studies and limitations in available data, we chose to restrict our statistical analysis to Spearman Rank Correlations only, with the aim of establishing convergent validity between GIFS and YaW results. In the case of the GIFS time series, the zero GIFS values were artifices of censoring. This meant that variations in the GIFS searches were restricted, and therefore the correlation was biased downwards. The true correlations were likely to be stronger than the estimated statistic.
Consistent with our study aim to minimize treatment of data, we did not adjust for seasonality. Moreover, we did not assess stationarity or autocorrelation in the data, or conduct formal time-series analyses, partly because this was consistent with the study aims, and also in response to the threshold issues in the GIFS data, which would bias any attempts to stationarize the data or measure autocorrelation.
What do Search Patterns Reveal about the Relative Importance of an Event?
The Bychkov episode was one of a series of important domestic Russian political events in the second half of 2010. Other concurrent events included the mismanagement of nation-wide forest fires, the Khimki forest clearance protests , and the "blue buckets" protests against the abuse of road privileges by economic elites [ ]. We identified seven protest-oriented Russian domestic events concurrent with the Bychkov episode from end-of-year compilations on government and non-government media organizations ( ). We selected a single term to represent each event from the terms contained in the end-of-year compilations. We used these additional search terms to provide additional context for the Bychkov episode in a complex political and media environment, and as an additional source of data with which to test the correlation between GIFS and YaW.
We then conducted limited corroboration of search patterns against opinion polls. We incorporated data from the openly accessible FOMnibus weekly national opinion poll  for each concurrent event. Please refer to . From FOMnibus, we used single peak values only, as an indicative measure of peak public awareness.
How Timely and Geographically Precise are GIFS and YaW Results?
In order for search pattern data to complement existing data sources, it should offer advantages in cost, timeliness or access to data. It was the availability of valid search data in advance of traditional surveillance that initially generated interest in this field. There are several differences related to the timing and geographic precision of data released by Google and Yandex. A comparative overview of the properties of public data released by these two search providers is outlined in. In , we compared the GIFS and YaW search term results for the Bychkov episode and concurrent events for Moscow, and separately for the remainder of Russia. Importantly, neither Google nor Yandex release all their data. For example, YaW can determine the physical location of individual users to their postcode, through Internet Service Provider (ISP) hardware locations [ ]. However, YaW only reveals public location data aggregated to the sub-provincial level, after two months. Google enforces similar restrictions on data availability. Commercial considerations by both search providers selling online advertising, rather than data availability, limit the release of precise and timely geographic location data.
How do the Search Patterns during a National Debate Relate to Salient Drug Issues?
Illicit drug use and treatment are important Russian social and policy problems. We examined search patterns in order to establish the relationship between online responses to the Bychkov episode and illicit drug issues. Firstly, we identified the two main themes associated with the Bychkov episode. These were corruption (police and judicial), and drug issues. In order to identify these high level themes, we hand coded all 420 primary posts published on Bychkov’s personal blog  in the date range March 2010 to March 2012. In order to identify these themes, two Russian-speaking researchers coded the primary and secondary themes in the body of each blog post, but excluded comments from readers. This resulted in 28.6% of blog posts coded as drugs (including addiction, illicit drugs, drug treatment, alcohol and tobacco), 18.3% as corruption (all sources including police and judicial) and 53.1% covering 17 other codes (including the Bychkov court case, Russian politics, the Orthodox Church, nationalism, pollution, sport, disability and philanthropic services). We assessed intercoder reliability on a random third of the total primary posts across the three themes (Kappa = 0.78). Consistent with the aims of this study, we then focused on the drug theme only.
Second, from Bychkov’s blog we identified the main drug themes in media reports about the Bychkov episode. We identified 57 separate national media reports referring to the Bychkov episode on Bychkov's blog within the study date range. These media reports were coded, and aggregated to two main themes, drugs and /or corruption. From the reports covering drugs, we then identified three main drug sub-themes. These were "addiction" (narkomania), "illicit drugs" (narkotiki), and "drug treatment" (lechenie narkomanii), which resulted in 29.2% coded for drugs (including sale and purchase), 30.1% for addiction (including use and dependence), and 39.8% for drug treatment (including medical and non-medical rehabilitation). We then assessed intercoder reliability on a random one-third sample of the total coded articles (Kappa = 0.75). All events referred to in Bychkov’s blog posts were corroborated using the websites of government news agency RIA-Novosti and non-government news sources.
Third, we established the relationship between search patterns for drug themes and the Bychkov episode. We did this by defining search terms, date ranges and minimizing confounding between search providers. We used the three drug themes (addiction, illicit drugs, and drug treatment) as search terms representing salient drug issues. We then investigated the relationship between these terms and the Bychkov episode, through Spearman Rank Correlations. To increase sensitivity and minimize ambiguity, we restricted search to “Egor Bychkov” to represent the Bychkov episode. Further, we restricted the date range to June 2011- June 2011. This date range search coincided with the period of Bychkov's maximum national media exposure. Finally, we conducted separate correlations within GIFS and within YaW, to prevent confounding between search providers.
We were able to gather data to achieve the two aims of this study. We gathered data with which to validate Yandex search patterns against those provided by Google. Secondly, we gathered data to test the adequacy search pattern analysis for investigating online interest in a 2010 national debate over Russian illicit drug policy. Our results are discussed in more detail below.
What is the Relationship between Google and Yandex Search Results?
Google and Yandex search results were positively correlated overall. We found a consistent pattern of strong to moderate positive correlations between the two search indices for the same term, both for the Bychkov episode and concurrent political events (). However, the relationship was weaker than anticipated. For example, “illicit drugs” has a weak negative relationship (r=-0.15). This was likely the result of GIFS returning zero values over several weeks during the specified data range. For example, GIFS searches for the term “Bychkov” displayed zero values in April, June and August 2010. During this period, YaW consistently recorded 54,000-68,000 searches per month. This is an example of the GIFS “threshold value” problem identified in earlier studies [ , ]. Within any selected date range, GIFS scales search results relative to a 100% value within that range. This scaling produces different results depending on the date range selected. This effect on GIFS data is only evident when compared with YaW data. For example, GIFS search for the term “illicit drug addiction” returned 100% in October 2010 (date range March 2010-March 2012), and 100% in November 2011 (date range November-December 2011). YaW values for these peaks GIFS dates were 132,000 and 167,102, respectively ( ). In summary, we were able to quantify patterns of missing data identified in earlier studies based on the use of GIFS data.
Visual examination of weekly results () suggests weekly peaks in GIFS indices corresponded to major milestones during the Bychkov episode. This provided further non-statistical corroboration, and face validity to the relationship between media reports and GIFS searches during the Bychkov episode.
What do Search Patterns Reveal about the Relative Importance of an Event?
We found search volumes for the Bychkov episode were comparable to other prominent domestic political events during 2010 (and ). GIFS values provide indicative comparisons of the relative importance of an event. However, YaW provides detailed raw numbers, allowing direct comparison of search patterns for an event across regions and across time. These tables provide comparative measures of the search volumes for the Bychkov episode and other concurrent events.
How Timely and Geographically Precise are GIFS and YaW Results?
Google potentially provides timely search results, whereas Yandex provides more accurate geographic localization. However, both GIFS and YaW place restrictions on the data made available to the public. These restrictions are outlined in. Whereas GIFS potentially provides detailed, near real time daily data, we found gaps in actual data availability. For example, only weekly data was available for the search terms "Bychkov" and "Egor Bychkov". We described these gaps above in relation to GIFS threshold, scaling and normalization policies.
GIFS does not provide detailed subnational geographic location data in Russia. However, YaW provides disaggregated search data to the level of individual Russian provincial cities. This differentiation is important, as several of the most important domestic political events during 2010 were associated with events around the Russian capital Moscow. For example, the widely-reported Khimki forest and blue buckets protests, revealed an average of 65% and 64% of searches originating in Moscow (). This result suggests the protests were relatively more important to Moscow residents, even while Russian and international commentators ascribed national significance to these events [ , ]. By contrast, only 26% of searches for the term "Egor Bychkov" came from Moscow, and the remainder from other parts of Russia. The Bychkov episode also generated greater absolute search volumes. These two results suggest the Bychkov episode was more important across Russia than the metropolitan protests.
How do the Search Patterns during a National Debate Relate to Salient Drug Issues?
We found moderate to strong positive correlations between search terms representing the Bychkov episode and terms representing salient drug issues in Russian media. In YaW, we found a moderate positive correlation of the term “Egor Bychkov” with the terms "illicit drugs" (rs = .77), "drug addiction" (rs = .74), and a strong correlation with “illicit drug treatment” (rs = .90). These correlations suggest searches for “Egor Bychkov” were positively correlated to salient Russian drug issues (). GIFS produced weaker or absent correlations. We attribute this to missing data as described earlier.
This study contributes to the methodological literature on the analysis of search patterns for public health policy. Firstly, we aimed to validate Yandex search patterns against those provided by Google. GIFS results have been validated against relevant offline measures across a range of scholarly domains, and across a range of countries. We were able to establish strong to moderate correlations for most search terms between GIFS and YaW. This suggests the use of YaW is a valid measure of online behavior in Russia.
We tested this method's adequacy for investigating online interest in a 2010 national debate over Russian illicit drug policy. Our use of available monthly data was insufficient to establish a statistical relationship between media reporting and search patterns for the Bychkov episode. However, we corroborated the relationship between media reporting and the Bychkov episode through the use of GIFS data within a restricted date range, opinion polling data, and media coverage. We were able to establish face validity. This suggests that media reporting influenced online behavior during the Bychkov episode. These findings are discussed in more detail below.
Google and Yandex Search Results are Positively Correlated.
Previous studies have established the validity of GIFS data through relevant offline measures. This is the first study to validate GIFS results through YaW. We believe this approach has several advantages in the Russian-language context. By exploring the relationship between Google and Yandex search patterns in response to a drug policy debate, we were able to quantify several previously identified limitations of GIFS. We demonstrated shifting GIFS threshold values, and the extent of GIFS scaling and normalization of data through reference to YaW search results.
The presence of zero values in GIFS results merits additional discussion. The zeroes are artifices of censoring and that this means that variation in GIFS has been restricted, and therefore the correlation is biased downwards. The true correlation is likely to be stronger than the estimated statistic. The logic for the expected downward bias is that GIFS and YaW appear not to be substitutes, based upon the positive data observed.
Although time series in nature, the data were not adjusted for seasonality or non-stationarity, and autocorrelation was not assessed. This might have biased the results towards stronger, but spurious, correlation. However, this cannot currently be assessed due to the effect of the threshold used in reporting the GIFs data. If GIFS and YaW were substitutes this would bias the results in favor of a stronger negative correlation, which was not observed; if GIFS and YaW were complements (or if seasonality was strong), this would bias the results in favor of a stronger positive correlation. These issues will need to be explored in further analyses using larger and more transparent data.
At the same time, this result highlights some of the current limitations of publicly available search tools as a data source. Search pattern studies have emerged as an opportunistic response to the availability of GIFS and YaW marketing tools. The data Google and Yandex make available through these tools is only a small portion of that collected. Most of the limitations on data availability described in this paper are in fact constraints on data release imposed by the search providers themselves. Further, search providers routinely make changes to their services. For example, in September 2012, Google merged GIFS into the Trends online analysis service, incorporating GIFS capabilities into the latter .
The Bychkov Episode was a Relatively Important Domestic Political Event.
Several studies have deployed GIFS to determine the importance of political episodes, as a low cost and rapid alternative to opinion polling [, ]. We analyzed domestic events unrelated to the Bychkov episode to provide additional context for the Bychkov episode. We found search volumes for the Bychkov episode were broadly comparable to other concurrent domestic news events. For example, the controversy surrounding the jailed oligarch Mikhail Khodorkovsky was both reported in the domestic media, and produced high search volumes, suggesting this was an important issue. Conversely, the Khimki Forest and Blue Buckets protests produced low search volumes. While all three of these issues attracted international media coverage, our results suggest the last two of these were not important to Russians nationally. We did not set out to examine constraints on traditional media on Internet search. Future search pattern studies in Russia should account for the influence of mainstream media constraints on issue salience and Internet search.
Google Potentially Provides Timely Search Results, whereas Yandex Provides More Precise Geographic Results.
Previous studies have described GIFS potential to complement existing public health data sources by providing timely, geographically precise [, ], and otherwise inaccessible data [ ]. Our results suggest timely GIFS data may not always be available in Russia. If GIFS data is missing, researchers will need to wait two months before YaW results are made available. While GIFS may not provide timely data useful for analyzing unfolding events, YaW is certain to provide delayed and detailed data. Geography is especially important in the Russian context. It is a large country, with many provincial cities, and considerable demographic variation. By comparing YaW raw data across specific regions, analysts may discern changes in search patterns for specific search terms across regions and across time. Based on our findings, we believe that YaW offers a potentially valuable tool to Russian drug policy researchers and advocates.
The Bychkov Episode was Positively Correlated with Salient Drug Issues.
Illicit drug use has long been among one of the most important social problems troubling Russians . We demonstrated a positive correlation between searches for Egor Bychkov and drug terms appearing in media reports associated with the Bychkov episode. This relationship merits further analysis. First, based on these results we are not able to distinguish between personal or sociotropic motivations for search [ ]. That is, we were unable to determine whether the concurrent increase in searches for the terms "drug use", "addiction", and "drug treatment" were motivated by individual's health problems, or an interest in drug policy issues. This is consistent with Reis and Brownstein's observations concerning searches for US abortion information [ ]. Second, unlike opinion polls, search patterns do not provide valency information. That is, search patterns offer no insight as to whether individuals support or oppose a specific issue. Further search terms to differentiate personal and sociotropic motivations, and to gauge valence should be considered in future search studies.
In conclusion, the Bychkov episode provides an opportunity to advance the science of search patterns. This paper investigated the relationship between Google and Yandex, and contributed to the broader methods literature by highlighting both the potential and limitations of these two search providers. We believe that YaW is a potentially valuable and underused data source for researchers working on Russian-related illicit drug policy and other public health problems. The Russian Federation, with its large, geographically dispersed, and politically engaged online population presents unique opportunities for studying the evolving influence of the Internet on politics and policy, using low cost methods resilient against potential increases in censorship. As online use grows further, primary sources of available online data will also grow. Adapting and refining research methods to best take advantage of these constantly evolving primary data sources is likely to present researchers in health policy and political communications with ongoing challenges.
We wish to acknowledge the assistance of Svetlana Chernova, Research Assistant, and Anya Sarang, President of the Andrey Rylkov Foundation for Health and Social Justice, Moscow.
Conflicts of Interest
There were no external sponsors involved in funding this research, reviewing or approving the manuscript for publication. The authors have no personal financial interests related to the subject matters discussed in the manuscript.
- Granka L. Inferring the Public Agenda from Implicit Query Data. In: Laura.granka.com. 2009 Presented at: ACM SIGIR Conference; July 19-23 2009; Boston URL: http://laura.granka.com/publications/granka_SIGIR09paper.pdf [WebCite Cache]
- Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 9;324(7337):573-577 [FREE Full text] [Medline]
- Cooper CP, Mallon KP, Leadbetter S, Pollack LA, Peipins LA. Cancer Internet search activity on a major search engine, United States 2001-2003. J Med Internet Res 2005 Jul 1;7(3):e36 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. Infodemiology: tracking flu-related searches on the web for syndromic surveillance. AMIA Annu Symp Proc 2006:244-248 [FREE Full text] [Medline]
- Polgreen PM, Chen Y, Pennock DM, Nelson FD. Using internet searches for influenza surveillance. Clin Infect Dis 2008 Dec 1;47(11):1443-1448 [FREE Full text] [CrossRef] [Medline]
- Google Inc.. Google.com. Google Insights for Search URL: http://www.google.com/insights/search/ [accessed 2012-06-15] [WebCite Cache]
- Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using search engine query data. Nature 2009 Feb 19;457(7232):1012-1014. [CrossRef] [Medline]
- Brownstein JS, Freifeld CC, Madoff LC. Digital disease detection--harnessing the Web for public health surveillance. N Engl J Med 2009 May 21;360(21):2153-5, 2157 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. Infodemiology and infoveillance: framework for an emerging set of public health informatics methods to analyze search, communication and publication behavior on the Internet. J Med Internet Res 2009;11(1):e11 [FREE Full text] [CrossRef] [Medline]
- Althouse BM, Ng YY, Cummings DA. Prediction of dengue incidence using search query surveillance. PLoS Negl Trop Dis 2011 Aug;5(8):e1258 [FREE Full text] [CrossRef] [Medline]
- Yang AC, Huang NE, Peng CK, Tsai SJ. Do seasons have an influence on the incidence of depression? The use of an internet search engine query data as a proxy of human affect. PLoS One 2010;5(10):e13728 [FREE Full text] [CrossRef] [Medline]
- Reis BY, Brownstein JS. Measuring the impact of health policies using Internet search patterns: the case of abortion. BMC Public Health 2010;10:514 [FREE Full text] [CrossRef] [Medline]
- Ayers JW, Althouse BM, Allem JP, Ford DE, Ribisl KM, Cohen JE. A novel evaluation of World No Tobacco day in Latin America. J Med Internet Res 2012;14(3):e77 [FREE Full text] [CrossRef] [Medline]
- Google Inc.. Google.com. Flu Trends URL: http://www.google.org/flutrends/ [accessed 2012-09-11] [WebCite Cache]
- Granka L. Inferring the Public Agenda from Implicit Query Data. In: Laura.granka.com. 2009 Presented at: ACM SIGIR Conference; July 19-23 2009; Boston URL: http://laura.granka.com/publications/granka_SIGIR09paper.pdf [WebCite Cache]
- Granka L. Measuring Agenda setting with online search traffic: Influences of online and traditional media. In: SSRN.: American Political Science Association; 2010 Presented at: Annual Meeting of the American Political Science Association; September 2-5, 2010; Washington D.C URL: http://ssrn.com/abstract=1658172 [WebCite Cache]
- Mellon J. http://www.sociology.ox.ac.uk. Oxford U.K: Department of Sociology, University of Oxford; 2011. Search Indices and Issue Salience: the properties of Google Trends as a measure of issue salience URL: http://www.sociology.ox.ac.uk/documents/working-papers/2011/swp1101.pdf [accessed 2012-06-17] [WebCite Cache]
- Scharkow M, Vogelgesang J. Measuring the Public Agenda using Search Engine Queries. International Journal of Public Opinion Research 2011;23(1):104. [CrossRef]
- Chay S, Sasaki N. Using Online Tools to Assess Public Responses to Climate Change Mitigation Policies in Japan. Future Internet 2011;3(2):117. [CrossRef]
- Koehler-Derrick G, Goldstein J. Combatting Terrorism Center at West Point. United States Military Academy, West Point, New York; 2011. Using Google Insights to Assess Egypt's Jasmine Revolution URL: http://www.ctc.usma.edu/posts/using-google-insights-to-assess-egypt%E2%80%99s-jasmine-revolution [accessed 2012-06-15] [WebCite Cache]
- McCombs M, Shaw D. The agenda setting function of mass media. Public Opinion Quarterly 1972;36(2):176-187. [CrossRef]
- Dearing J, Rogers E. Agenda-setting. In: Agenda-setting. Thousand Oaks, Calif: Sage; 1996.
- Kiousis S. Explicating Media Salience: A Factor Analysis of New York Times Issue Coverage During the 2000 US Presidential Election. Journal of Communication 2004;54(1):71-87. [CrossRef]
- Wallack L, Dorfman L. Media advocacy: a strategy for advancing policy and promoting health. Health Educ Q 1996 Aug;23(3):293-317. [Medline]
- Roberts M, Wanta W, Dzwo T. Agenda Setting and Issue Salience Online. Communication Research 2002 Aug;29(4):452-465. [CrossRef]
- McCombs M. http://sticerd.lse.ac.uk. London: LSE; 2002. The Agenda-setting Role of the Mass Media in the Shaping of Public Opinion URL: http://sticerd.lse.ac.uk/dps/extra/McCombs.Pdf [accessed 2012-09-11] [WebCite Cache]
- Moy P, Bimber A, Rojecki A, Xenos M, Iyengar S. Shifting Contours in Political Communication Research. International Journal of Communication ;Feature 247-20 2012 Feb 27;6 [FREE Full text] [CrossRef]
- Petrov N, McFaul M. Carnegie Endowment for International Peace. Moscow; 2005. The Essence of Putin's Managed Democracy URL: http://carnegieendowment.org/2005/10/18/essence-of-putin-s-managed-democracy/2a3 [accessed 2012-09-11] [WebCite Cache]
- Gessen M. Guardian.co.uk. London; 2011 Dec 26. Vladimir Putin's world is falling apart URL: http://www.guardian.co.uk/commentisfree/2011/dec/26/vladimir-putin-world-falling-apart [accessed 2012-09-11] [WebCite Cache]
- Mickiewicz E. Excavating Concealed Tradeoffs: How Russians Watch the News. Political Communication 2005;22(3):355-380. [CrossRef]
- ComScore Inc.. Comscore.com. 2009. Russia has World's Most Engaged Social Networking Audience URL: http://www.comscore.com/Press_Events/Press_Releases/2009/7/Russia_has_World_s_Most_Engaged_Social_Networking_Audience [accessed 2012-06-15] [WebCite Cache]
- Sunstein C. Bostonreview.net. Boston; 2001. The Daily We URL: http://bostonreview.net/BR26.3/sunstein.php [accessed 2012-06-15] [WebCite Cache]
- Comscore Inc.. Comscore.com. 2012. European Internet Usage by Country URL: http://www.comscore.com/Press_Events/Press_Releases/2012/4/Nearly_Half_of_European_Online_Audience_Visit_Sports_Sites [accessed 2012-06-15] [WebCite Cache]
- ITU. International Telecommunications Union. 2011. Percentage of Individuals using the Internet URL: http://www.itu.int/ITU-D/ict/statistics/material/excel/2010/IndividualsUsingInternet_00-10.xls [accessed 2012-06-15] [WebCite Cache]
- WCIOM. Russian Public Opinion Research Centre. Moscow: WCIOM; 2012 Feb 13. Russians networking: rating of social media URL: http://wciom.com/index.php?id=61&uid=637 [WebCite Cache]
- Netmarketshare. Netmarketshare.com. 2011. Search Engine Market Share URL: http://www.netmarketshare.com/report.aspx?qprid=4&qptimeframe=M&qpsp=148 [WebCite Cache]
- Crisp T. comScore Voices. 2011. Yandex: From Russia with Love URL: http://blog.comscore.com/2011/06/yandex_russia_with_love.html [accessed 2012-10-25] [WebCite Cache]
- Kovalev A. Wired.co.uk. London; 2011 Feb 05. Why is LiveJournal still massive in Russia? URL: http://www.wired.co.uk/news/archive/2011-02/04/livejournal-in-russia [accessed 2012-06-21] [WebCite Cache]
- Barash V, Kelly J. Berkman Center Research Publication No. 2012-9. Harvard University Cambridge: Berkman Center For Internet & Society; 2012. Salience vs. Commitment: Dynamics of Political Hashtags in Russian Twitter URL: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2034506 [accessed 2012-06-17] [WebCite Cache]
- Machleder J, Asmolov G. Internews.org. 2010 Aug. Social Change and the Russian Network Society. Redefining development priorities in new information environments URL: http://www.internews.org/sites/default/files/resources/Internews_Research_RussiaNetworkSociety1.pdf [accessed 2012-06-16] [WebCite Cache]
- Miroshnichenko A. slon.ru. Moscow; 2009 Dec 02. The Internet as a viral editor part 2- a hidden threat to the state URL: http://slon.ru/future/internet_kak_virusnyy_redaktor_ii_skrytaya_ugroza-202915.xhtml [accessed 2012-06-16] [WebCite Cache]
- Etling B. Global Voices Online. 2011 May 13. Russia: Quantitative Research Proves Blogs Set an Alternative News Agenda URL: http://globalvoicesonline.org/2011/05/13/russia-quantitative-research-proves-blogs-set-an-alternative-news-agenda/ [accessed 2012-10-17] [WebCite Cache]
- Petrov N. Carnegie Endowment for International Peace. Moscow; 2010 Dec 08. The Navalny Effect - Rospil URL: http://carnegieendowment.org/2010/12/08/navalny-effect-rospil.net/21ux [accessed 2012-06-16] [WebCite Cache]
- Nocetti J. Opendemocracy.net. 2011 Dec 14. Russia's virtual: the new reality? Open Democracy URL: http://www.opendemocracy.net/od-russia/julien-nocetti/russia%E2%80%99s-virtual-new-reality [accessed 2012-06-16] [WebCite Cache]
- Vinokurova E. Gazeta.ru. Moscow; 2011 Dec 02. Censorship - it's a matter of choice URL: http://www.gazeta.ru/politics/elections2011/2011/11/28_a_3850322.shtml [accessed 2012-06-13] [WebCite Cache]
- OpenNet.net. OpenNet Initiative. Harvard University, Cambridge: Berkman Center for Internet & Society; 2010 Dec 19. Research profile - Russia URL: http://opennet.net/research/profiles/russia [accessed 2012-06-16] [WebCite Cache]
- Borogan I, Soldatov A. Opendemocracy.net. 2012 Apr 25. The Kremlin and hackers: partners in crime? URL: http://www.opendemocracy.net/od-russia/irina-borogan-andrei-soldatov/kremlin-and-hackers-partners-in-crime [accessed 2012-06-21] [WebCite Cache]
- Soldatov A, Borogan I. Opendemocracy.net. 2011 Oct 25. The Russian state and surveillance technology URL: http://www.opendemocracy.net/od-russia/andrei-soldatov-irina-borogan/russian-state-and-surveillance-technology [accessed 2012-06-16] [WebCite Cache]
- Agentura.ru. Agentura.ru. 1998. SORM 2 URL: http://www.agentura.ru/timeline/1998/sorm/ [accessed 2012-09-11] [WebCite Cache]
- Levada Centre. Levada.ru. Moscow; 2012 Mar 20. The number of Internet users is growing URL: http://www.levada.ru/20-03-2012/chislo-polzovatelei-interneta-rastet [accessed 2012-06-16] [WebCite Cache]
- Gregory R. Psychological Testing: History, Principles, and Applications (6th Edition). London: Pearson; 2010.
- Weeks B, Southwell B. The symbiosis of news coverage and aggregate online search behavior: Obama, rumors, and presidential politics. Mass Communication and Society 2010;13(4):341-360. [CrossRef]
- Ripberger J. Capturing Curiosity: Using Internet Search Trends to Measure Public Attentiveness. Policy Studies Journal 2011;39(2):239-259. [CrossRef]
- Scheitle C. Google's Insights for Search: A Note Evaluating the Use of Search Engine Data in Social Research. Social Science Quarterly 2011;92(1):285-295. [CrossRef]
- Yang AC, Tsai SJ, Huang NE, Peng CK. Association of Internet search trends with suicide death in Taipei City, Taiwan, 2004-2009. J Affect Disord 2011 Jul;132(1-2):179-184. [CrossRef] [Medline]
- Google.com. http://support.google.com. 2012. How is the data derived? URL: http://support.google.com/insights/bin/answer.py?hl=en&answer=92768.Archived [WebCite Cache]
- Levada Centre. Which of the following social problems disturb Russians and complicate their lives?. Moscow; 2011 Aug 30. Levada.ru URL: http://www.levada.ru/30-08-2011/kakie-problemy-trevozhat-rossiyan-i-oslozhnyayut-zhizn [accessed 2012-06-16] [WebCite Cache]
- Echo-Moscow. Echo.msk.ru. Moscow; 2009 Sep 08. Dmitry Medvedev: Drug addiction in Russia presents a national security and demographic threat URL: http://www.echo.msk.ru/news/618545-echo.html [accessed 2012-06-15] [WebCite Cache]
- Elovich R, Drucker E. On drug treatment and social control: Russian narcology's great leap backwards. Harm Reduct J 2008;5:23 [FREE Full text] [CrossRef] [Medline]
- Stratgap.ru. Stratgap. Moscow; 2009. Strategy for the Implementation of the National Anti-Drug Policy of the Russian Federation in the Period Until 2020 URL: http://stratgap.ru/pages/strategy/3662/4434/4437/index.shtml [accessed 2012-09-11] [WebCite Cache]
- Zheluk A. Public Health across frontiers - A Russian Federation case study. In: unpublished PhD Thesis in progress. Sydney: University of Sydney Australia; 2012.
- Kuzina A. Mk.ru. Moscow; 2010 Nov 16. Bychkov says I hate junkies, pedophiles and queers URL: http://www.mk.ru/social/article/2010/11/15/544249-egor-byichkov-ya-ih-terpet-ne-mogu-narkomanov-pedofilov-pidsov.html [accessed 2012-06-22] [WebCite Cache]
- Vlassov V. When the official drug policy failed: self-organization of the people to defend public health interest. Eur J Public Health 2011 Jun;21(3):270 [FREE Full text] [CrossRef] [Medline]
- Salenkov M. RFERL. 2010 Nov 03. Egor Bychkov walks free from prison URL: http://www.svobodanews.ru/content/article/2209835.html [accessed 2012-06-16] [WebCite Cache]
- Bychkov E. Egor Bychkov's blog. Nizhniy Tagil; 2010. URL: http://egor-bychkov.livejournal.com/ [accessed 2012-10-17] [WebCite Cache]
- Chernyavskaya V. Tagilvariant.ru. Nizhniy Tagil; 2011 Jun 16. We are happy for you, Egor URL: http://www.tagilvariant.ru/?p=2545 [accessed 2012-06-15] [WebCite Cache]
- Latynina Y. Echo Moscow. Moscow; 2010 Sep 25. Access Code URL: http://rutube.ru/tracks/3615882.html?v=0efdb7981d9d7e32f025ce52fdae1d60 [accessed 2012-06-23] [WebCite Cache]
- Bychkov E. Egor Bychkov's blog. Nizhniy Tagil; 2010 Sep 30. My video message to all honest people! URL: http://egor-bychkov.livejournal.com/2010/09/30/ [accessed 2012-10-15] [WebCite Cache]
- Latynina Y. Echo Moscow. Moscow; 2010 Oct 02. Access Code URL: http://www.echo.msk.ru/programs/code/714906-echo/ [accessed 2012-06-23] [WebCite Cache]
- RIAN. Ria.ru. Moscow; 2010 Oct 12. Nizhny Tagil Court sentences Bychkov to 3.5 year prison term URL: http://www.ria.ru/justice/20101012/284882489.html [accessed 2012-10-15] [WebCite Cache]
- Kommersant.ru. Kommersant 190 (4490). Moscow; 2010 Oct 13. The president appears with a rock group URL: http://www.kommersant.ru/doc/1520881 [accessed 2012-06-23] [WebCite Cache]
- 1tv.ru. Let them talk. Moscow; 2010 Oct 14. In the city without drugs URL: http://www.1tv.ru/sprojects_edition/si5685/fi5791 [accessed 2012-06-23] [WebCite Cache]
- 1tv.ru. Judge for yourself. Moscow; 2010 Oct 21. Drug addiction - how to defeat evil URL: http://www.1tv.ru/sprojects_edition/si5691/fi5900 [accessed 2012-10-17] [WebCite Cache]
- NTV.ru. Profession - reporter. Moscow; 2010 Oct 30. Beating out stupidity URL: http://rutube.ru/video/43a0b7a35924311e93190a7fe3aed4b7/ [accessed 2012-10-30] [WebCite Cache]
- 1tv. 1tv.ru. Moscow; 2010 Nov 01. Special investigation - handcuffs as a treatment URL: http://www.1tv.ru/documentary/fi5246/sn18/fd201011012230 [accessed 2012-10-30] [WebCite Cache]
- Masterova G. Daily Telegraph. 2010 Sep 30. U2 and Greenpeace protest helps to save Russian woodland Khimki forest URL: http://www.telegraph.co.uk/sponsored/russianow/politics/8035211/U2-and-Greenpeace-protest-helps-to-save-Russian-woodland-Khimki-forest.html [accessed 2012-06-16] [WebCite Cache]
- Cyber.law.harvard.edu. Cyber.law.harvard.edu. Cambridge; 2011. Impact of the Internet on Russian Politics, Media, and Society URL: http://cyber.law.harvard.edu/research/russia [accessed 2012-06-17] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). 2011. FOMnibus survey URL: http://bd.fom.ru/ [accessed 2012-06-15] [WebCite Cache]
- Wciom.com. Wciom. Moscow; 2010 Dec 23. 2010: Persons and events of the year URL: http://wciom.com/index.php?id=61&uid=147.Archived [WebCite Cache]
- RIAN. Ria.ru. 2010 Dec 23. Main events of the year: fires, drought, terrorism in the metro and START URL: http://ria.ru/politics/20101223/312450378.html [accessed 2012-06-22] [WebCite Cache]
- RT.com. Russia Today. Moscow; 2010 Dec 3`. Russia's ups and downs the final cut URL: http://www.rt.com/news/main-events-russia-2010/ [accessed 2012-06-22] [WebCite Cache]
- Gazeta.ru. Gazeta.ru. Moscow; 2010 Dec 31. Between fire and ice - main political events of the year URL: http://www.gazeta.ru/politics/2010/12/30_a_3483194.shtml [accessed 2012-06-22] [WebCite Cache]
- Digital Element Inc.. Digitalelement.com. 2010 Oct 19. Yandex, Russia's Top Internet Company, Chooses Digital Element's Geolocation Technology to Improve Precision of Location-Based Information URL: http://www.digitalelement.com/press_room/yandex-russias-top-internet-company-chooses-digital-elements-geolocation.html [accessed 2012-09-11] [WebCite Cache]
- RFERL. Referl.org. 2010 Nov 04. Jailed Russian Antidrugs Activist Released URL: http://www.rferl.org/content/Jailed_Russian_Antidrugs_Activist_Released/2210932.html [accessed 2012-06-23] [WebCite Cache]
- FOM.ru. Fomnibus survey 6-7 November. Moscow; 2010. Public Opinion Foundation (FOM) URL: http://bd.fom.ru/pdf/d44sobned10.pdf [accessed 2012-06-23] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). Moscow; 2010. Fomnibus survey 18-19 December URL: http://bd.fom.ru/pdf/d50sobned10.pdf [accessed 2012-06-23] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). Moscow; 2011. Fomnibus survey 15-16 January URL: http://bd.fom.ru/pdf/d03sobned11.pdf [accessed 2012-03-26] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). Moscow; 2010. Fomnibus survey 25-26 September URL: http://bd.fom.ru/pdf/d38sobned10.pdf [accessed 2012-06-23] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). Moscow; 2010. Fomnibus survey 4-5 September URL: http://bd.fom.ru/pdf/d35sobned10.pdf [accessed 2012-06-23] [WebCite Cache]
- FOM.ru. Public Opinion Foundation (FOM). Moscow; 2010. Fomnibus survey 18-19 September URL: http://bd.fom.ru/pdf/d37sobned10.pdf [accessed 2012-06-23] [WebCite Cache]
- Heritage T. Reuters.com. 2012 Feb 20. Anti-Putin protests spur civil society in Russia URL: http://www.reuters.com/article/2012/02/29/us-russia-election-society-idUSTRE81S1AJ20120229 [accessed 2012-06-15] [WebCite Cache]
- Prokhorova I, Malashenko A, Petrov N, Getmanchuk A. Carnegie Endowment for International Peace. Moscow; 2012 Mar 14. Russia's 2012 Presidential Elections: Prospects for Russia and the Region URL: http://carnegieendowment.org/2012/03/14/russia-s-2012-presidential-elections-prospects-for-russia-and-region/a6mu [accessed 2012-06-16] [WebCite Cache]
- Google.com. insidesearch.blogspot.com.au. 2012 Sep 27. Insights into what the world is searching for - the new Google Trends URL: http://insidesearch.blogspot.com.au/2012/09/insights-into-what-world-is-searching.html [accessed 2012-10-25] [WebCite Cache]
- Min Y, Ghanem S, Evatt D. Using a Split-ballot Survey to Explore the Robustness of the 'MIP'question in Agenda-setting Research: A Methodological Study. International Journal of Public Opinion Research 2007;19(2):221-236. [CrossRef]
|EU: European Union|
|FOM: Public Opinion Foundation|
|GIFS: Google Insights For Search|
|NGO: Non Government Organization|
|OECD: Organization for Economic Cooperation and Development|
|RuNet: Russian Internet|
|SORM: 2 Russian federal government internet and telecommunications monitoring service|
|YaW: Yandex Wordstat|
Edited by G Eysenbach; submitted 16.07.12; peer-reviewed by V Vlassov, R Badgett, J Vogelgesang; comments to author 07.08.12; revised version received 14.09.12; accepted 08.11.12; published 13.12.12
©Andrey Zheluk, James A. Gillespie, Casey Quinn. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 13.12.2012.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.