Crowdsourcing methodologies leveraging the contributions of citizen scientists connected via the Internet have recently proved to be of great value to solve certain scientific challenges involving “big data” analysis that cannot be entirely automated [1]. In the GalaxyZoo project, citizen scientists classified imagery of hundreds of thousands of galaxies drawn from the Sloan Digital Sky Survey and the Hubble Space Telescope archive [2]. Crowdsourced contributions can be achieved with different motivation strategies, such as micropayments or games. The “serious games” concept refers to an intention not only to entertain users, but also to train or educate them [3]. The “gamification” [4] of the crowdsourcing approach enables a higher motivation of the participants and, using the Internet as a vehicle, untaps an underexploited resource for scientific research [5,6]: it is estimated that 3 billion hours per week are spent playing computer and videogames worldwide [7]. For instance, Fold-It, an online game where players solve 3-dimensional puzzles by folding protein structures, has resulted in several breakthrough scientific discoveries [8-10]. Another recent growing trend is the use of crowdsourcing techniques for participatory health research studies in which individuals report in real time a variety of health conditions [11], providing a promising complement to traditional clinical trials. Considering crowdsourced image analysis, collective processing has been recently explored for earthquake damage assessment from remote sensing imagery [12]. However, this methodology has not yet been mainstreamed for biomedical image analysis.

In this context, analysis of microscopic images of malaria-infected blood samples is an appealing goal. Worldwide, there are more than 200 million malaria cases and approximately 800,000 deaths annually, mainly in children [13,14]. Careful optical microscopic examination of a well-stained blood film remains the gold standard for malaria diagnosis [15]. Confirmation of a negative diagnosis is ultimately dependent on the technician’s expertise and can take up to 20 minutes. In addition, as malaria prevalence decreases in one specific place over time, microscopy technician skills may now be needed in other regions. Fast, cheap, ubiquitous, and accurate diagnosis is a priority in the Agenda for Malaria Eradication [16]. Although automated processing methodologies have been used extensively for the analysis of digitized blood smears [17,18], currently there are no completely automated image processing systems that can achieve perfect parasite recognition [19-24]. The main problem in computer-aided malaria diagnosis is that algorithms are usually not very robust with respect to the variable appearance of the parasites and changing image acquisition conditions.

The goal of this research was to test the feasibility of a crowdcomputing approach for malaria parasite quantification in which nonexperts count parasites in digitized thick blood smears through an online game (crowdsourcing) and a decision algorithm combines the data generated by several players in order to achieve a collective detection with a higher accuracy rate than an individual analysis. This idea—gaming for distributed malaria image analysis—has been also explored in a recent study by Mavandadi et al [25], in parallel to and independently of this study. These researchers designed a video game and a processing pipeline to investigate whether nonexperts can assess if a single-cell image extracted from a digitized thin blood sample is infected with malaria or not. Although this study and the present research share a similar vision and goal, the research questions posed and solutions adopted differ substantially in terms of the data analyzed, the nature of the participants, the main task required of them, and the processing methodologies.

The proposed system in this study provides a new tool for parasite counting, but not malaria diagnosis, which is a more complex problem [26]. For this purpose, the microscopist protocol will need to be translated completely into a gaming protocol, including assessing the presence or absence of parasites, the parasite species, and growth stages and prognostic markers, such as schizonts or gametocytes, or pigment load. In the long run, crowdsourced remote telediagnosis from images acquired with optical microscopy and distributed worldwide through the Internet and possibly with systems that integrate the microscope into mobile phones [27,28], might have a potential impact for malaria-endemic countries because diagnosis availability and its cost could be optimized. However, in addition to the need for conventional laboratory processing and imaging equipment to prepare the material to a sufficiently high standard, this kind of analysis will require a communications infrastructure with enough bandwidth to distribute the images over the Internet and a critical number of online participants in order to ensure timely analysis of the images.

This work presents a proof-of-concept system that explores the feasibility of an online game-based, crowdsourced solution for malaria parasite quantitation in digitized images of thick blood smears.

We selected an image database of malaria-positive blood films that had been previously analyzed by experts to generate gold standards. These images were then incorporated into an online game. The player’s task was to click on the parasites. When a player found all the parasites present in 1 image (constituting a level) within a limited amount of time, the game continued by presenting a new image. Otherwise, the game was over. All the players’ clicks were registered in a database. After 1 month, all the collected data was preprocessed in order to group all the clicks that players placed around the different objects in the image: parasites, white blood cells (leukocytes), and background noise. Finally, an algorithm that combined the different games to increase accuracy was developed and evaluated.

Out of a total of 270,207 clicks, 78.65% tagged a true parasite. Analysis of the levels reached by the players reveals that approximately one-third of the players were able to find all the parasites in an image, independent of the level (see Figure 4d). Additionally, once players successfully completed level 5, they became game experts and no longer followed the 1 in 3 chance of passing to the next level—they could complete as many as 22 levels (achieved by the best player so far). Interestingly, the overall number of clicks on each of the parasites in 1 image was similar, meaning that although 2 of 3 players usually did not complete the level, all the parasites were equally difficult to identify (Figures 4a and b). This fact was corroborated in a special case for image ID6, where the probability of tagging one particular phantom parasite was as high as the typical probability for a true parasite. A further look into the gold standard revealed that, in fact, this phantom parasite was a true parasite that was not included in the gold standard by mistake (Figure 4c).

We performed an exhaustive evaluation of the collective gamers performance using the quorum algorithm evaluated 1000 times for all group sizes ranging from 1 to 30 games over each of the test images under the different training conditions (completing 1 level can be considered as a 1-minute training) (Figure 5a). Results show a monotone smooth behavior for the true positive (TP) and false positive (FP) rates depending on the group size and quorum value: the bigger the group size or the smaller the quorum required, the more true parasites were tagged and the higher the TP rate, but also more phantom parasites were collectively tagged, increasing the FP rate (Figure 5b). Analysis of the discrimination index (DI) function (DI = TP – FP) revealed that there was an optimal quorum number that maximized the DI for each group size (Figure 5c). For instance, the optimal quorum value was 3 for a group size of 7 games (randomly chosen among all games) achieving a DI = 90%, whereas the optimal quorum for a group of 10 games was 4, providing a mean DI = 95%. When comparing the performance of the collective analysis based on the training time (levels completed), we observed a clear dependence between training and DI (Figure 5d). The number of games needed to be combined in order to achieve a DI = 99% was 22, 13, 10, 9, and 4, respectively, for the subset of games that successfully completed 0, 1, 2, 3, and 4 game levels.

The maximum DI for each group size (obtained with its optimal quorum value) at all training levels was fitted to a model equation DI = f(group size, training time) using the scientific data mining software Eureqa [29]. A multivariate optimization process was used to find the following collective detection equation:

DI(group,training) = 1 - e^{–(alpha + beta group + gamma group training)}

where group size ranged from 1 to 30 games and the coefficient of determination (R²) goodness of fit with (alpha = 0.69, beta = 0.24, and gamma = 0.13) is greater than 0.97 for each training value {0,1,2,3,4} levels (or minutes) (Figure 6). This equation highlights the product group⋅training, meaning that the accuracy increase provided by adding 1 game to the group size can almost be compensated (the term beta⋅group varies when increasing the group size) by 1 minute training or vice versa.

We also evaluated the collective performance detection against the automated image recognition methodology presented by Frean [20]. For each image, we calculated the minimum number of gamers needed to perform as well as the automatic system and we found that it was required to combine 7.2, 4.6, 3.9, 3.0, and 2.3 games, respectively, from the subset of players that successfully completed 0, 1, 2, 3, and 4 game levels (see Multimedia Appendix 2).

In this study, a crowdcomputing image analysis system was developed that identifies malaria parasites in digitized images of thick blood smears using an online game played by nonexpert volunteers. Results for the images analyzed showed that the performance of the quorum algorithm that combines the games from different players can be as high as both human expert counting and automated processing methodologies. Indeed, the results showed that, on average, the combination of 22 games or more, regardless of the players’ experience, was enough to obtain almost perfect parasite counting (99%) in the tested images. This performance could also be obtained by combining 13 games from players trained for 1 minute. However, no conclusions could be drawn about detecting extremely low parasitemias (as low as 1 parasite per 30 or more images).

Feedback from several players stressed the need for clearer game instructions. Although our strategy was learning by doing, commenters suggested that an explanation screen and/or an initial training level would lead to better results, at least for those players who did not complete the first level. The collective detection equation allowed us to model the system performance in terms of the number of games and the training of the players. An important question arising from this model is whether any crowdsourced image analysis system that roughly consists of detecting spots in images will have a similar behavior. In the affirmative case, the model could be used in the future to design and evaluate new crowdsourced biomedical image analysis applications.

The overall results endorse the online gaming approach to the task of counting malaria parasites in thick blood films using a crowdsourcing methodology, validated “in the wild” by thousands of anonymous online players. This conclusion extends the findings of Mavandadi et al [25], but from a different perspective. The methodology of the present research involves finding parasites in images from thick blood samples whereas in the study by Mavandadi et al the main task was to make binary decisions (infected versus uninfected) of single-cell images extracted from thin blood samples [25]. Both thin and thick blood films are used in malaria microscopy [26]. The thick film, consisting of many layers of red and white blood cells, is used to search for malaria parasites and to count them as an indicator of the severity of the disease. The thin film, a single layer of red and white blood cells, is mainly used to confirm the malaria parasite species and sometimes to enumerate parasites and evaluate other prognostic features. In addition to the type of data analyzed (thin versus thick blood film) and the task required of the participants (binary decisions in single-cell images versus parasite detection and location), the nature and number of participants varies, from 31 controlled volunteers [25] versus > 6000 anonymous online contributors in the current research. The studies can be considered complementary and directed toward the same goal; the different methodologies and experiments lead to the global conclusion that nonexperts are able to rapidly learn and identify the typical features of malaria parasites in digitized thin and thick blood films, and that the combination of the analyses of several users can provide similar accuracy rates for parasite quantification as expert microscopists.

Future developments of the current research should include the exploration of new algorithms that combine the games of several players in a more complex way than the quorum algorithm. For instance, if players’ identities are logged and tracked, it would be possible to adapt algorithms to differentially weight the analysis of players depending on their profile, experience, past performance, or gaming strategies [30-33]. Automated processing methodologies report accuracy rates that are high, but still not as good as human visual inspection; therefore, we expect that combined man-machine diagnosis systems will be the most effective strategy. Note that, for instance, in the present study the players’ detection performance was similar for all the images, whereas the automated detection algorithm [20] had heterogeneous performances for different images. Therefore, in hybrid systems, humans could be used first to train the recognition algorithm and later to analyze the more complex cases (supplementing the automatic processing methodologies), whereas the easy cases would be automatically processed.

In summary, this proof-of-concept research has shown that malaria image analysis for parasite quantification, obtained by combining the detection of several online nonexperts with minimal training, can be as good as the results provided by an expert microscopist. Although the game score is generated by comparing the user tags with previously analyzed images, in future, the observation protocol from expert microscopists could theoretically be translated into a game and images that have not yet been assessed by professionals could be introduced into that game. This raises the possibility of establishing a global specialized task force of remote gamers-workers able to perform online malaria parasite detection and quantitation. The validity of this approach for malaria diagnosis is still unclear and will depend on the method’s speed, efficiency, robustness, cost, and above all, accuracy. Constraints related to production of the high-quality images required for malaria species identification will have to be addressed. Specifically, the performance should be compared to current diagnostic tools and trends, such as rapid immunochromatographic diagnostic tests that offer a cost-efficient solution. However, rapid tests have limitations, such as restricted malaria species recognition and an inability to quantitate parasite load and monitor parasitological response to treatment. In general, we suggest that the methodology presented in this research could be applied to other biomedical image analysis tasks with potential impact on global health challenges, such as enumeration of acid-fast bacilli in sputum smears for tuberculosis diagnosis. An inherent benefit from this distributed telediagnosis system is that it is scalable and resilient. Among other positive externalities of this research, there is a clear educational impact because more than 6000 players have learned how malaria parasites appear in thick blood films. In addition, as we allowed players to introduce their nicknames into the table of high scorers, we could identify approximately 100 players who now can be considered as experts in parasite counting, within the system’s limitations. Citizen science projects of this kind could impact future educational paradigms: they are a clear opportunity for engaging with young people and offer a hands-on experience that could be used in online learning platforms [34,35].

Concerning the evolution of the MalariaSpot platform, next steps might explore the feasibility of developing a new game version that mimics, if possible, all the relevant steps of the microscopist protocol in real-life conditions (eg, decisions about presence or absence of malaria parasites, parasite stages and species, and quantitation), but this is a much more complex and challenging process. Assuming image quality concerns can be addressed, this system could potentially be completed by integrating the online platform for rapid diagnosis with the recently developed cellphone-microscope systems [36] that allow data transfer directly from field workers and health centers, distributing the data worldwide through the Internet.