Published on in Vol 14, No 6 (2012): Nov-Dec

Crowdsourcing Malaria Parasite Quantification: An Online Game for Analyzing Images of Infected Thick Blood Smears

Crowdsourcing Malaria Parasite Quantification: An Online Game for Analyzing Images of Infected Thick Blood Smears

Crowdsourcing Malaria Parasite Quantification: An Online Game for Analyzing Images of Infected Thick Blood Smears

Original Paper

1Biomedical Image Technologies group, DIE, ETSI Telecomunicación, Universidad Politécnica de Madrid, CEI Moncloa UPM-UCM, Madrid, Spain

2Biomedical Research Networking Center in Bioengineering, Biomaterials and Nanomedicine, CIBER-BBN, Madrid, Spain

3Nebutek Soluciones SL, Vizcaya, Spain

4National Institute for Communicable Diseases, National Health Laboratory Service, Johannesburg, South Africa

5School of Pathology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa

Corresponding Author:

Miguel Angel Luengo-Oroz, PhD

Biomedical Image Technologies group

DIE, ETSI Telecomunicación

Universidad Politécnica de Madrid, CEI Moncloa UPM-UCM

ETSIT, Av. Complutense 30

Madrid, 28040


Phone: 34 913366827

Fax:34 913367323


Background: There are 600,000 new malaria cases daily worldwide. The gold standard for estimating the parasite burden and the corresponding severity of the disease consists in manually counting the number of parasites in blood smears through a microscope, a process that can take more than 20 minutes of an expert microscopist’s time.

Objective: This research tests the feasibility of a crowdsourced approach to malaria image analysis. In particular, we investigated whether anonymous volunteers with no prior experience would be able to count malaria parasites in digitized images of thick blood smears by playing a Web-based game.

Methods: The experimental system consisted of a Web-based game where online volunteers were tasked with detecting parasites in digitized blood sample images coupled with a decision algorithm that combined the analyses from several players to produce an improved collective detection outcome. Data were collected through the MalariaSpot website. Random images of thick blood films containing Plasmodium falciparum at medium to low parasitemias, acquired by conventional optical microscopy, were presented to players. In the game, players had to find and tag as many parasites as possible in 1 minute. In the event that players found all the parasites present in the image, they were presented with a new image. In order to combine the choices of different players into a single crowd decision, we implemented an image processing pipeline and a quorum algorithm that judged a parasite tagged when a group of players agreed on its position.

Results: Over 1 month, anonymous players from 95 countries played more than 12,000 games and generated a database of more than 270,000 clicks on the test images. Results revealed that combining 22 games from nonexpert players achieved a parasite counting accuracy higher than 99%. This performance could be obtained also by combining 13 games from players trained for 1 minute. Exhaustive computations measured the parasite counting accuracy for all players as a function of the number of games considered and the experience of the players. In addition, we propose a mathematical equation that accurately models the collective parasite counting performance.

Conclusions: This research validates the online gaming approach for crowdsourced counting of malaria parasites in images of thick blood films. The findings support the conclusion that nonexperts are able to rapidly learn how to identify the typical features of malaria parasites in digitized thick blood samples and that combining the analyses of several users provides similar parasite counting accuracy rates as those of expert microscopists. This experiment illustrates the potential of the crowdsourced gaming approach for performing routine malaria parasite quantification, and more generally for solving biomedical image analysis problems, with future potential for telediagnosis related to global health challenges.

J Med Internet Res 2012;14(6):e167



Crowdsourcing methodologies leveraging the contributions of citizen scientists connected via the Internet have recently proved to be of great value to solve certain scientific challenges involving “big data” analysis that cannot be entirely automated [1]. In the GalaxyZoo project, citizen scientists classified imagery of hundreds of thousands of galaxies drawn from the Sloan Digital Sky Survey and the Hubble Space Telescope archive [2]. Crowdsourced contributions can be achieved with different motivation strategies, such as micropayments or games. The “serious games” concept refers to an intention not only to entertain users, but also to train or educate them [3]. The “gamification” [4] of the crowdsourcing approach enables a higher motivation of the participants and, using the Internet as a vehicle, untaps an underexploited resource for scientific research [5,6]: it is estimated that 3 billion hours per week are spent playing computer and videogames worldwide [7]. For instance, Fold-It, an online game where players solve 3-dimensional puzzles by folding protein structures, has resulted in several breakthrough scientific discoveries [8-10]. Another recent growing trend is the use of crowdsourcing techniques for participatory health research studies in which individuals report in real time a variety of health conditions [11], providing a promising complement to traditional clinical trials. Considering crowdsourced image analysis, collective processing has been recently explored for earthquake damage assessment from remote sensing imagery [12]. However, this methodology has not yet been mainstreamed for biomedical image analysis.

In this context, analysis of microscopic images of malaria-infected blood samples is an appealing goal. Worldwide, there are more than 200 million malaria cases and approximately 800,000 deaths annually, mainly in children [13,14]. Careful optical microscopic examination of a well-stained blood film remains the gold standard for malaria diagnosis [15]. Confirmation of a negative diagnosis is ultimately dependent on the technician’s expertise and can take up to 20 minutes. In addition, as malaria prevalence decreases in one specific place over time, microscopy technician skills may now be needed in other regions. Fast, cheap, ubiquitous, and accurate diagnosis is a priority in the Agenda for Malaria Eradication [16]. Although automated processing methodologies have been used extensively for the analysis of digitized blood smears [17,18], currently there are no completely automated image processing systems that can achieve perfect parasite recognition [19-24]. The main problem in computer-aided malaria diagnosis is that algorithms are usually not very robust with respect to the variable appearance of the parasites and changing image acquisition conditions.

The goal of this research was to test the feasibility of a crowdcomputing approach for malaria parasite quantification in which nonexperts count parasites in digitized thick blood smears through an online game (crowdsourcing) and a decision algorithm combines the data generated by several players in order to achieve a collective detection with a higher accuracy rate than an individual analysis. This idea—gaming for distributed malaria image analysis—has been also explored in a recent study by Mavandadi et al [25], in parallel to and independently of this study. These researchers designed a video game and a processing pipeline to investigate whether nonexperts can assess if a single-cell image extracted from a digitized thin blood sample is infected with malaria or not. Although this study and the present research share a similar vision and goal, the research questions posed and solutions adopted differ substantially in terms of the data analyzed, the nature of the participants, the main task required of them, and the processing methodologies.

The proposed system in this study provides a new tool for parasite counting, but not malaria diagnosis, which is a more complex problem [26]. For this purpose, the microscopist protocol will need to be translated completely into a gaming protocol, including assessing the presence or absence of parasites, the parasite species, and growth stages and prognostic markers, such as schizonts or gametocytes, or pigment load. In the long run, crowdsourced remote telediagnosis from images acquired with optical microscopy and distributed worldwide through the Internet and possibly with systems that integrate the microscope into mobile phones [27,28], might have a potential impact for malaria-endemic countries because diagnosis availability and its cost could be optimized. However, in addition to the need for conventional laboratory processing and imaging equipment to prepare the material to a sufficiently high standard, this kind of analysis will require a communications infrastructure with enough bandwidth to distribute the images over the Internet and a critical number of online participants in order to ensure timely analysis of the images.

This work presents a proof-of-concept system that explores the feasibility of an online game-based, crowdsourced solution for malaria parasite quantitation in digitized images of thick blood smears.

We selected an image database of malaria-positive blood films that had been previously analyzed by experts to generate gold standards. These images were then incorporated into an online game. The player’s task was to click on the parasites. When a player found all the parasites present in 1 image (constituting a level) within a limited amount of time, the game continued by presenting a new image. Otherwise, the game was over. All the players’ clicks were registered in a database. After 1 month, all the collected data was preprocessed in order to group all the clicks that players placed around the different objects in the image: parasites, white blood cells (leukocytes), and background noise. Finally, an algorithm that combined the different games to increase accuracy was developed and evaluated.

Ethics Statement

The malaria images used in this research were previously used to evaluate automated image analysis methods [20]. Original blood samples and resultant test images were collected and used with ethical approval from the Human Research Ethics Committee (Medical), University of the Witwatersrand, Johannesburg, South Africa (protocol number M051126). No new ethical review board approval was required since the digital images used in our work were not linked to any patient data or diagnosis and were digitally shared for microscopic training evaluation purposes. The data analyzed in this research were anonymously produced by online volunteers who agreed to play an Internet game. The participants were informed of the research purposes of the game on the game webpage.

Image Database

The image database was compiled from 28 Giemsa-stained thick films made from blood infected with malaria (Plasmodium falciparum) parasites, acquired using a 50× objective in a conventional laboratory optical microscope. Medium to low parasitemia images were selected for the game because of its design (1-minute games) and the fact that discrepancies between automatic counting methodologies and manual expert counting tend to be greater in low parasitemia cases. A gold standard mask image was generated for each of the 28 images to evaluate player performance.

Game Architecture

The objective of the MalariaSpot game was to tag as many parasites as possible in an image in 1 minute. The instructions—what is a parasite and what it is not—were briefly explained in the splash screen of the game website (Figure 1a). During the game, if the player found all the parasites in 1 image in the allowed time, a new image was presented (Figure 1b). Therefore, a player could analyze several images (levels) in a single game. In order to reinforce the game’s addictive nature, the players were given continuous feedback: each click was compared with the gold standard and an icon was placed immediately at the tag position to indicate a correct or incorrect selection. In addition, if the player misidentified an object and clicked in a wrong location (eg, on a leukocyte), the player was penalized by reducing the remaining time available to solve the level. Players were confronted with different, randomly selected test images. The difficulty of the levels increased as the time penalty for wrong tags grew with each level. As a motivation strategy, at the end of the game players were invited to register and provide their name, email address, and country in order to be included in the table of high scorers depicting the top daily, weekly, and monthly players.

Figure 1. MalariaSpot example screens and player distribution. (a) Splash screen of MalariaSpot game website showing the game instructions. (b) Example of MalariaSpot game screen. (c) Map showing the geographic distribution of players during the evaluation period (the darkness of the country is proportional to the number of visits).
View this figure

Data Collection

The MalariaSpot game webpage was launched on April 25, 2012 (World Malaria Day). During the following month, more than 6000 players from 95 different countries (Figure 1c) visited the game webpage according to the number of Internet Protocol (IP) addresses reported by Google Analytics, although the actual number of players was probably larger because those connecting from big institutions, such as universities, share the same IP address and other players may have blocked the Google Analytics script. Online volunteers played a total of 12,105 games that resulted in the analyses of 20,049 images and generated a database of 270,207 tags. Social media was the main traffic source; approximately 30% of the players originated from a Facebook link and 30% came from a Twitter reference. Most of the remaining visits were through links in digital newspapers and blogs, especially from Spanish-speaking countries.

Data Preprocessing

All the players’ clicks were saved into a database containing the user identification number, image identification number, x-position and y-position on the game screen, time of the click (from the start of the level), and whether the click was on a true parasite or not (see Figures 2a-d and Multimedia Appendix 1). In a preprocessing step, we generated a binary matrix, In(g,p), for each test image n, where each row g contains a different game and each column p corresponds to a parasite. A value of 1 at a certain position, In(gi,pj) = 1, means that the parasite with index j has been clicked in the game i. Otherwise, In(gi,pj) = 0. The number of rows is the number of games that have been played at each test image n. The number of columns corresponds to the number of parasites for a given level in the gold standard plus the number of phantom parasites. We defined a phantom parasite as an object in the image that is not a parasite and that has been tagged by ≥ 1 players. The phantom parasites were defined in order to group together all the clicks that were around the same position, but not on the identical pixel (eg, all the clicks that were inside a leukocyte were considered to be pointing at the same phantom parasite). An image processing pipeline that grouped together clicks that were at a distance of less than the typical parasite size was implemented in order to generate all the connected components in each image that corresponded to phantom parasites (Figure 2c). Therefore, the output of this preprocessing stage consisted of 1 binary matrix per test image that characterized the performance of all the games played for each image. Additionally, filtered versions of these matrices were created by selecting only the data from games in which at least 1 level was completed, 2 levels were completed, and so on.

Figure 2. Crowdsourced image analysis of thick blood film infected with malaria. (a) Test image analyzed in the game. (b) Gold standard image in which each label corresponds to a parasite. (c) Aggregation of gamer’s clicks where green regions correspond to correctly tagged parasites and red regions to players’ mistakes. (d) Gamers’ clicks superimposed on raw image.
View this figure

Collective Parasite Detection: Quorum Algorithm

A critical aim of this research was to show how individual nonexpert analysis can be combined to achieve higher accuracy rates. In order to combine the games of several players and produce a single “detection,” we implemented a quorum algorithm. The output of the quorum collective detection features all the image objects (both true parasites and phantom parasites) in one image that have been tagged in at least X individual games out of a larger group of Y games (Figure 3). The idea is simple: an object is considered in the collective detection if it has been tagged (“voted”) in at least X out of Y (X ≤ Y) games. Typically, when the quorum value increases, there are fewer true positives and false negatives. In order to evaluate the performance of different group sizes and quorum values, we randomly selected 1000 subsets of games per (X,Y) couple with a maximum group size of Y = 30 games. For each individual subset of Y random games in 1 image, the collective detection performance was measured for all quorum values of X ≤ Y. Performance evaluation was also measured taking the subset of games that passed at least level 1, level 2, level 3, and level 4.

Figure 3. Illustrative example of the quorum algorithm. Parasite detection results on test image obtained from the combination of 4 games processed with different quorum values.
View this figure

Out of a total of 270,207 clicks, 78.65% tagged a true parasite. Analysis of the levels reached by the players reveals that approximately one-third of the players were able to find all the parasites in an image, independent of the level (see Figure 4d). Additionally, once players successfully completed level 5, they became game experts and no longer followed the 1 in 3 chance of passing to the next level—they could complete as many as 22 levels (achieved by the best player so far). Interestingly, the overall number of clicks on each of the parasites in 1 image was similar, meaning that although 2 of 3 players usually did not complete the level, all the parasites were equally difficult to identify (Figures 4a and b). This fact was corroborated in a special case for image ID6, where the probability of tagging one particular phantom parasite was as high as the typical probability for a true parasite. A further look into the gold standard revealed that, in fact, this phantom parasite was a true parasite that was not included in the gold standard by mistake (Figure 4c).

We performed an exhaustive evaluation of the collective gamers performance using the quorum algorithm evaluated 1000 times for all group sizes ranging from 1 to 30 games over each of the test images under the different training conditions (completing 1 level can be considered as a 1-minute training) (Figure 5a). Results show a monotone smooth behavior for the true positive (TP) and false positive (FP) rates depending on the group size and quorum value: the bigger the group size or the smaller the quorum required, the more true parasites were tagged and the higher the TP rate, but also more phantom parasites were collectively tagged, increasing the FP rate (Figure 5b). Analysis of the discrimination index (DI) function (DI = TP – FP) revealed that there was an optimal quorum number that maximized the DI for each group size (Figure 5c). For instance, the optimal quorum value was 3 for a group size of 7 games (randomly chosen among all games) achieving a DI = 90%, whereas the optimal quorum for a group of 10 games was 4, providing a mean DI = 95%. When comparing the performance of the collective analysis based on the training time (levels completed), we observed a clear dependence between training and DI (Figure 5d). The number of games needed to be combined in order to achieve a DI = 99% was 22, 13, 10, 9, and 4, respectively, for the subset of games that successfully completed 0, 1, 2, 3, and 4 game levels.

The maximum DI for each group size (obtained with its optimal quorum value) at all training levels was fitted to a model equation DI = f(group size, training time) using the scientific data mining software Eureqa [29]. A multivariate optimization process was used to find the following collective detection equation:

DI(group,training) = 1 - e–(alpha + beta group + gamma group training)

where group size ranged from 1 to 30 games and the coefficient of determination (R2) goodness of fit with (alpha = 0.69, beta = 0.24, and gamma = 0.13) is greater than 0.97 for each training value {0,1,2,3,4} levels (or minutes) (Figure 6). This equation highlights the product group⋅training, meaning that the accuracy increase provided by adding 1 game to the group size can almost be compensated (the term beta⋅group varies when increasing the group size) by 1 minute training or vice versa.

We also evaluated the collective performance detection against the automated image recognition methodology presented by Frean [20]. For each image, we calculated the minimum number of gamers needed to perform as well as the automatic system and we found that it was required to combine 7.2, 4.6, 3.9, 3.0, and 2.3 games, respectively, from the subset of players that successfully completed 0, 1, 2, 3, and 4 game levels (see Multimedia Appendix 2).

Figure 4. Individual gamer’s performance. (a) Tagging probability for true parasites and phantom parasites on image ID1 based on all games. (b) Tagging probability field superimposed on raw image. True parasites (gold standard) are signaled by white squares. (c) Aggregated tagging probability for true parasites and phantom parasites on image ID6. Note that the probability of the phantom parasite (tagged in yellow) is as high as the true parasites. Detailed analysis showed that it was a mistake on the gold standard. (d) Number of games played at each level.
View this figure
Figure 5. Collective parasite detection. (a) Accuracy results in 1000 random groups run over image ID1 with a group of 10 and 15 games and a quorum of 3 and 6 votes. (b) Mean results of true positives (TP), false positives (FP), and TP – FP for 1000 experiments of all the group sizes and quorum values with different experience. (c) Quorum values that maximize the TP – FP rate for all group sizes and training levels. (d) Maximum recognition score for each group size and training level. Values represent the mean and standard deviation among the regular test images.
View this figure
Figure 6. Model of the collective detection of malaria parasites. Curve fitting of the mean accuracy rate for all group sizes and training levels.
View this figure

In this study, a crowdcomputing image analysis system was developed that identifies malaria parasites in digitized images of thick blood smears using an online game played by nonexpert volunteers. Results for the images analyzed showed that the performance of the quorum algorithm that combines the games from different players can be as high as both human expert counting and automated processing methodologies. Indeed, the results showed that, on average, the combination of 22 games or more, regardless of the players’ experience, was enough to obtain almost perfect parasite counting (99%) in the tested images. This performance could also be obtained by combining 13 games from players trained for 1 minute. However, no conclusions could be drawn about detecting extremely low parasitemias (as low as 1 parasite per 30 or more images).

Feedback from several players stressed the need for clearer game instructions. Although our strategy was learning by doing, commenters suggested that an explanation screen and/or an initial training level would lead to better results, at least for those players who did not complete the first level. The collective detection equation allowed us to model the system performance in terms of the number of games and the training of the players. An important question arising from this model is whether any crowdsourced image analysis system that roughly consists of detecting spots in images will have a similar behavior. In the affirmative case, the model could be used in the future to design and evaluate new crowdsourced biomedical image analysis applications.

The overall results endorse the online gaming approach to the task of counting malaria parasites in thick blood films using a crowdsourcing methodology, validated “in the wild” by thousands of anonymous online players. This conclusion extends the findings of Mavandadi et al [25], but from a different perspective. The methodology of the present research involves finding parasites in images from thick blood samples whereas in the study by Mavandadi et al the main task was to make binary decisions (infected versus uninfected) of single-cell images extracted from thin blood samples [25]. Both thin and thick blood films are used in malaria microscopy [26]. The thick film, consisting of many layers of red and white blood cells, is used to search for malaria parasites and to count them as an indicator of the severity of the disease. The thin film, a single layer of red and white blood cells, is mainly used to confirm the malaria parasite species and sometimes to enumerate parasites and evaluate other prognostic features. In addition to the type of data analyzed (thin versus thick blood film) and the task required of the participants (binary decisions in single-cell images versus parasite detection and location), the nature and number of participants varies, from 31 controlled volunteers [25] versus > 6000 anonymous online contributors in the current research. The studies can be considered complementary and directed toward the same goal; the different methodologies and experiments lead to the global conclusion that nonexperts are able to rapidly learn and identify the typical features of malaria parasites in digitized thin and thick blood films, and that the combination of the analyses of several users can provide similar accuracy rates for parasite quantification as expert microscopists.

Future developments of the current research should include the exploration of new algorithms that combine the games of several players in a more complex way than the quorum algorithm. For instance, if players’ identities are logged and tracked, it would be possible to adapt algorithms to differentially weight the analysis of players depending on their profile, experience, past performance, or gaming strategies [30-33]. Automated processing methodologies report accuracy rates that are high, but still not as good as human visual inspection; therefore, we expect that combined man-machine diagnosis systems will be the most effective strategy. Note that, for instance, in the present study the players’ detection performance was similar for all the images, whereas the automated detection algorithm [20] had heterogeneous performances for different images. Therefore, in hybrid systems, humans could be used first to train the recognition algorithm and later to analyze the more complex cases (supplementing the automatic processing methodologies), whereas the easy cases would be automatically processed.

In summary, this proof-of-concept research has shown that malaria image analysis for parasite quantification, obtained by combining the detection of several online nonexperts with minimal training, can be as good as the results provided by an expert microscopist. Although the game score is generated by comparing the user tags with previously analyzed images, in future, the observation protocol from expert microscopists could theoretically be translated into a game and images that have not yet been assessed by professionals could be introduced into that game. This raises the possibility of establishing a global specialized task force of remote gamers-workers able to perform online malaria parasite detection and quantitation. The validity of this approach for malaria diagnosis is still unclear and will depend on the method’s speed, efficiency, robustness, cost, and above all, accuracy. Constraints related to production of the high-quality images required for malaria species identification will have to be addressed. Specifically, the performance should be compared to current diagnostic tools and trends, such as rapid immunochromatographic diagnostic tests that offer a cost-efficient solution. However, rapid tests have limitations, such as restricted malaria species recognition and an inability to quantitate parasite load and monitor parasitological response to treatment. In general, we suggest that the methodology presented in this research could be applied to other biomedical image analysis tasks with potential impact on global health challenges, such as enumeration of acid-fast bacilli in sputum smears for tuberculosis diagnosis. An inherent benefit from this distributed telediagnosis system is that it is scalable and resilient. Among other positive externalities of this research, there is a clear educational impact because more than 6000 players have learned how malaria parasites appear in thick blood films. In addition, as we allowed players to introduce their nicknames into the table of high scorers, we could identify approximately 100 players who now can be considered as experts in parasite counting, within the system’s limitations. Citizen science projects of this kind could impact future educational paradigms: they are a clear opportunity for engaging with young people and offer a hands-on experience that could be used in online learning platforms [34,35].

Concerning the evolution of the MalariaSpot platform, next steps might explore the feasibility of developing a new game version that mimics, if possible, all the relevant steps of the microscopist protocol in real-life conditions (eg, decisions about presence or absence of malaria parasites, parasite stages and species, and quantitation), but this is a much more complex and challenging process. Assuming image quality concerns can be addressed, this system could potentially be completed by integrating the online platform for rapid diagnosis with the recently developed cellphone-microscope systems [36] that allow data transfer directly from field workers and health centers, distributing the data worldwide through the Internet.


This research was partially funded by the Picata program from the Moncloa Campus of International Excellence, Universidad Politécnica de Madrid and Universidad Complutense de Madrid, Spain; and the project TEC2010-21619-C04-03 from the Spanish Ministry of Science and Innovation.

The authors would like to thank Enrique Mendoza for setting up the Web database and Jacobo Gomez for the Web illustrations. Thanks for the support in the launch of the game to Anoush Tatevossian, Natalia Rodriguez, Maria Fernandez, Maria Luengo-Oroz, Antonio Blanco, Jose L Rubio-Guivernau, Nati Luengo-Oroz, and Cesar Martin. Thanks for comments and feedback on the manuscript to Carlos Castro-Gonzalez, David Pastor-Escuredo, Gert Wollny, Juan Ortuño, Pedro Guerra, Patricia Arroba, Thierry Savy, Ana Tajadura-Jimenez, Maria J Ledesma-Carbayo, Andres Santos, and Jonathan Platkiewicz.

Last, but not least, thanks to the more than 6000 anonymous participants who played the MalariaSpot game.

Authors' Contributions

Miguel A Luengo-Oroz conceived and directed this research, designed the experiment and the MalariaSpot game, performed data analysis, interpreted the results, and wrote the manuscript. Asier Arranz implemented the game software. John Frean provided the image data, advised on microscopic aspects of malaria diagnosis and parasite enumeration, and contributed to revising and editing the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Raw data collected during the experiment: Data_MalariaSpot.csv containing 270207 rows - 1 per parasite tag- in the format: [user id, image id, true/phantom parasite, x position, y position, time].

CSV File, 7MB

Multimedia Appendix 2

Comparison between crowdsourced parasite counting and automatic image analysis counting methodology.

PDF File (Adobe PDF File), 17KB

  1. Thompson K. All hands on deck. Sci Am 2012 Feb;306(2):56-59. [Medline]
  2. Lintott C, Schawinski K, Slosar A, Land K, Bamford S. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 2008;389:1179-1189. [CrossRef]
  3. Abt C. Serious Games. Lanham, MD: University Press of America; 1987.
  4. Deterding S, Dixon D, Khaled R, Nacke L. From game design elements to gamefulness: defining gamification. 2011 Sep 28 Presented at: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments; 2011; Tampere, Finland p. 9-15. [CrossRef]
  5. Good BM, Su AI. Games with a scientific purpose. Genome Biol 2011;12(12):135. [CrossRef] [Medline]
  6. Nielsen M. Reinventing Discovery: The New Era of Networked Science. Princeton, NJ: Princeton University Press; 2011.
  7. McGonigal J. Reality Is Broken: Why Games Make Us Better and How They Can Change the World. New York: Penguin (Non-Classics); 2011.
  8. Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online game. Nature 2010 Aug 5;466(7307):756-760 [FREE Full text] [CrossRef] [Medline]
  9. Eiben CB, Siegel JB, Bale JB, Cooper S, Khatib F, Shen BW, et al. Increased Diels-Alderase activity through backbone remodeling guided by Foldit players. Nat Biotechnol 2012 Feb;30(2):190-192. [CrossRef] [Medline]
  10. Lessl M, Bryans JS, Richards D, Asadullah K. Crowd sourcing in drug discovery. Nat Rev Drug Discov 2011 Apr;10(4):241-242. [CrossRef] [Medline]
  11. Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health research ecosystem. J Med Internet Res 2012;14(2):e46 [FREE Full text] [CrossRef] [Medline]
  12. Barrington L, Ghosh S, Greene M, Har-Noy S, Berger J. Crowdsourcing earthquake damage assessment using remote sensing imagery. Annals of Geophysics 2012;54(6). [CrossRef]
  13. WHO Global Malaria Programme. World Malaria Report 2011. Geneva, Switzerland: World Health Organization; 2011.   URL: [accessed 2012-11-27] [WebCite Cache]
  14. Murray CJ, Rosenfeld LC, Lim SS, Andrews KG, Foreman KJ, Haring D, et al. Global malaria mortality between 1980 and 2010: a systematic analysis. Lancet 2012 Feb 4;379(9814):413-431. [CrossRef] [Medline]
  15. Shute G. The microscopic diagnosis of malaria. In: Wernsdorfer WH, McGregor I, editors. Malaria: Principles and Practice of Malariology. Edinburgh: Churchill Livingstone; 1988:781-814.
  16. malERA Consultative Group on Diagnoses and Diagnostics. A research agenda for malaria eradication: diagnoses and diagnostics. PLoS Med 2011;8(1):e1000396 [FREE Full text] [CrossRef] [Medline]
  17. Angulo J, Flandrin G. Automated detection of working area of peripheral blood smears using mathematical morphology. Anal Cell Pathol 2003;25(1):37-49. [Medline]
  18. Luengo-Oroz M, Angulo J, Flandrin G, Klossa J. Mathematical morphology in polar-logarithmic coordinates application to erythrocyte shape analysis. In: Marques JS, editor. Pattern Recognition and Image Analysis: Second Iberian Conference, IbPRIA 2005, Estoril, Portugal, June 7-9, 2005, Proceedings, Part 1 (Lecture Notes in ... Pattern Recognition, and Graphics) (Pt. 1). Berlin: Springer Berlin Heidelberg; 2005:199-206. [CrossRef]
  19. Prescott WR, Jordan RG, Grobusch MP, Chinchilli VM, Kleinschmidt I, Borovsky J, et al. Performance of a malaria microscopy image analysis slide reading device. Malar J 2012;11:155 [FREE Full text] [CrossRef] [Medline]
  20. Frean JA. Reliable enumeration of malaria parasites in thick blood films using digital image analysis. Malar J 2009;8:218 [FREE Full text] [CrossRef] [Medline]
  21. Purwar Y, Shah SL, Clarke G, Almugairi A, Muehlenbachs A. Automated and unsupervised detection of malarial parasites in microscopic images. Malar J 2011;10:364 [FREE Full text] [CrossRef] [Medline]
  22. Sio SW, Sun W, Kumar S, Bin WZ, Tan SS, Ong SH, et al. MalariaCount: an image analysis-based program for the accurate determination of parasitemia. J Microbiol Methods 2007 Jan;68(1):11-18. [CrossRef] [Medline]
  23. Tek FB, Dempster AG, Kale I. Computer vision for microscopy diagnosis of malaria. Malar J 2009;8:153 [FREE Full text] [CrossRef] [Medline]
  24. Tek F, Dempster A, Kale I. Parasite detection and identification for automated thin blood film malaria diagnosis. Computer Vision and Image Understanding 2010;114:21-32. [CrossRef]
  25. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One 2012;7(5):e37245 [FREE Full text] [CrossRef] [Medline]
  26. World Health Organization. Basic Malaria Microscopy: Part I. Learner's Guide. Geneva: World Health Organization; 2010.
  27. Miller AR, Davis GL, Oden ZM, Razavi MR, Fateh A, Ghazanfari M, et al. Portable, battery-operated, low-cost, bright field and fluorescence microscope. PLoS One 2010;5(8):e11890 [FREE Full text] [CrossRef] [Medline]
  28. Breslauer DN, Maamari RN, Switz NA, Lam WA, Fletcher DA. Mobile phone based clinical microscopy for global health applications. PLoS One 2009;4(7):e6320 [FREE Full text] [CrossRef] [Medline]
  29. Schmidt M, Lipson H. Distilling free-form natural laws from experimental data. Science 2009 Apr 3;324(5923):81-85 [FREE Full text] [CrossRef] [Medline]
  30. Barrington L, Turnbull D, Lanckriet G. Game-powered machine learning. Proc Natl Acad Sci U S A 2012 Apr 24;109(17):6411-6416 [FREE Full text] [CrossRef] [Medline]
  31. Raykar V, Yu S, Zhao L, Valadez G, Florin C. Learning from crowds. The Journal of Machine Learning Research 2010;11:1297-1322 [FREE Full text]
  32. Stein JA, Asman AJ, Landman BA. Characterizing and Optimizing Rater Performance for Internet-based Collaborative Labeling. Proc Soc Photo Opt Instrum Eng 2011 Mar 3;7966 [FREE Full text] [CrossRef] [Medline]
  33. Khatib F, Cooper S, Tyka MD, Xu K, Makedon I, Popovic Z, et al. Algorithm discovery by protein folding game players. Proc Natl Acad Sci U S A 2011 Nov 22;108(47):18949-18953 [FREE Full text] [CrossRef] [Medline]
  34. edX.   URL: [accessed 2012-09-03] [WebCite Cache]
  35. Taddei F, Sasco LR. Proposing measures to promote the education of creative and collaborative knowledge-builders. In: Measuring Creativity: The Book. Brussels: European Commission; 2010:369-389.
  36. Tuijn CJ, Hoefman BJ, van Beijma H, Oskam L, Chevrollier N. Data and image transfer using mobile phones to strengthen microscopy-based diagnostic services in low and middle income country laboratories. PLoS One 2011;6(12):e28348 [FREE Full text] [CrossRef] [Medline]

DI: discrimination index
FP: false positive
TP: true positive

Edited by G Eysenbach; submitted 04.09.12; peer-reviewed by M Grobusch; comments to author 25.09.12; revised version received 08.10.12; accepted 24.10.12; published 29.11.12


©Miguel Angel Luengo-Oroz, Asier Arranz, John Frean. Originally published in the Journal of Medical Internet Research (, 29.11.2012.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.