We designed a multiphase, prize-based, open innovation competition in collaboration with the commercial platform provider HeroX. The challenge page on HeroX [44] served as a content hub throughout the challenge. It featured an explanatory video and a text-based description of the technology gap and its background. On HeroX, we published the judging scorecard and all formal and legal requirements. Moreover, the challenge hub included a chat room for discussions and questions for the sponsor team. The whole of the financial funds previously collected in a crowdfunding campaign were used for the crowdsourcing campaign [45]. First, expenses for using the HeroX platform, production of informative media content (eg, the video), and organization of the HORAO conference, as well as the travel costs of the participants and the jury, were paid. The remaining US $50,000 served as the monetary incentive for putting forward existing solutions. We decided to divide this prize money among several finalists to increase the likelihood of any innovator winning a prize and further motivate innovators to participate. The final share was set at US $35,000 for the winner, US $12,000 for the runner-up, and US $1000 each for the third to fifth places.

After 3 months of content creation (video and text), on April 23, 2018, the challenge went on the internet with a prelaunch in the categories of engineering, health care, and technology. The aim of the prelaunch phase was to raise awareness about the ensuing competition and give solvers a first opportunity to evaluate the challenge. HeroX’s service included advertisements on Facebook (2 weeks) and Twitter (1 week) as channels for recruiting potential participants and a one-on-one outreach campaign (targeted outreach: 2108 emails). The target audience for the one-on-one outreach included individuals, companies, and organizations involved in medical imaging, medical technology, radiology, surgical technology, clinical engineering, neurological societies, imaging science, health science, and microscopy. Finally, HeroX published the HORAO challenge in its newsletter as part of the media service. Figure 1 provides an overview of the challenge timeline.

The submission phase started on June 12, 2018, and ended on November 16, 2018 (after 22 weeks and 4 days). Proposals had to be presented using the official submission form and to address the questions asked thoughtfully. The form had space for a technical report with the option to embed a link to a video or website. The complete proposal had to be uploaded as a PDF to HeroX. Submissions needed to comply with all the terms of the challenge defined in the challenge-specific agreement, which specified, for example, that competitors retain all intellectual property rights to their technology. The challenge was open to all adult individuals or teams, requiring no specific qualifications. We considered, for further evaluation, only submissions satisfying the criteria of the judging scorecard (Table 1). We made the judging scorecard and the challenge guidelines publicly accessible from the very beginning. The judging scorecard narrowed down and specified the scope of the possible solution. The challenge guidelines and the challenge-specific agreement are reproduced in Multimedia Appendices 1 and 2.

After termination of the submission phase, the challenge team launched an individual project website serving as a new content hub independent of HeroX. The sponsor team also published regular updates about the challenge on the hospital and departmental websites and via social media channels (Facebook, Twitter, and Instagram).

The evaluation phase consisted of the following 3 consecutive rounds: the preround, the judging round, and the finals. Each round had its own panel of judges.

We refrained from asking for background information about the solvers themselves to avoid discouraging those with a lower level of academic attainment from submitting solutions and to avoid selection bias during the judging round. Using a web-based survey (SurveyMonkey [46]), sent to the individuals and teams, we obtained this information after the finals. We asked team leaders to forward the survey to their team members to capture information about as many of them as possible. The questions in the survey covered place of residence, type of employment and place of work, academic degrees obtained and field of education, number of prior challenges joined, and how the solver found out about the HORAO challenge. We used descriptive statistics to evaluate the diversity of the solvers crowd. In addition, HeroX performed an analysis of the reach of advertising on social media channels (Facebook and Twitter) for challenge visibility as part of their service.

We did not use any health-related data for this work. Participants shared the information about educational background and demographics voluntarily. We treated all personal data disclosed with the utmost care. The project does not fall under the jurisdiction of the local ethics committee, so we did not need to obtain their approval.

The 5 finalists proposed solutions based on multispectral time-resolved fluorescence spectroscopy, polarization-sensitive optical coherence tractography, machine-learned interpretation of red-green-blue images, using polarized light based on Mueller polarimetry, and wide-field Mueller polarimetry based on Lu-Chipman decomposition. Of these, wide-field Mueller polarimetry based on Lu-Chipman decomposition received the highest score, both from the conference audience and the expert jury, and won our crowdsourcing ideation campaign. The solution uses a series of liquid crystals to polarize white light from a xenon light source and captures the polarization details of the tissue in reflection transfiguration. Following the conference, we initiated an in-depth scientific collaboration between the sponsor team and the winning research team. For their preliminary results, the collaborators were awarded an industry grant. Following a series of ex-vivo experiments on cadaveric animal brain tissue and on fresh human tumor tissue with a prototype of the Mueller polarization microscope [47,48], we created a new interdisciplinary research unit, which comprises neurosurgeons, optical physicists, neuropathologists, and experts on artificial intelligence. The group recently launched a multiyear, in-depth clinical project, which has been awarded a Swiss National Science Foundation Sinergia grant 205904. After 3 years of the HORAO conference, we have already published a series of promising preclinical results [49-53].

With the crowdsourcing challenge HORAO, a technology-gap-type problem in neurosurgery was presented for the first time to the public, with the conviction that a solution already existed somewhere, albeit developed for a different use. The challenge proved successful, producing a handful of very innovative and promising proposals, leading to new scientific collaborations.

In recent years, simplified access for patients, participants, scientists, and biomedical staff through Web 2.0 has opened up a new world for collaborating on a variety of tasks. Consequently, various types of crowdsourcing have evolved in health care. A well-represented use case described in the recent literature concerns projects searching for new biomarkers, like the Anti-PD-1 Response Challenge, Prostate Cancer DREAM challenge, and the Multiple Myeloma DREAM challenge, to name but a few [24,35,36]. In those challenges, the initiators share data, usually from a large set of patients, on an open platform and mobilize groups of people with same interests around the world to analyze the data. Leveraging worldwide expertise and the power of the mass has speeded up the identification of novel biomarkers tremendously compared to the classical approaches used in research. Another, completely different approach, evaluated the Berlin Institute of Health, was the OPEN project [33]. In response to the slow progress being made with artificial pancreas systems for people with diabetes, the patient community has taken the problem into its own hands. Under the hashtag #wearenotwaiting, patients and their families are building their own systems and making the algorithms publicly available (do-it-yourself artificial pancreas systems, OpenAPS) [37]. The OPEN project examines what academia, industry, and individuals with diabetes can learn from each other by establishing empirical evidence of the impact of do-it-yourself artificial pancreas systems. The initiators of the OPEN project are convinced that such an interdisciplinary and collaborative approach will have a profound impact, not only on the patients but also on the health care system and on society in general. With the rise of machine learning in medicine, labeled data sets are in high demand. Hence, crowdsourcing for data labeling has become popular. A recent example is the NuCLS study [23]. This study used a crowdsourcing approach for nucleus classification, localization, and segmentation in hematoxylin- and eosin-stained slides of breast carcinomas using a preannotated data set elaborated in a previous crowdsourcing study [38]. The organizers specifically addressed medical students and graduates in pathology by searching interest groups on social media (Facebook and LinkedIn) and assigning the tasks depending on experience. The mixed crowd of experts and undergraduates produced the final NuCLS data set containing more than 220,000 annotations of cell nuclei. Although it was successful, the initiators of the project pointed out that the context-dependency of data set curation makes transfer of the approach to other problems difficult. The crowdsourcing approach described here focuses on the ideation process and therefore differs considerably from the abovementioned crowdsourcing applications.

The review by Nguyen et al [42] discusses the various methods of collective intelligence applied in clinical research and proposes a framework to implement them with respect to the domains shown in Figure 3. In comparison to other reviews that address crowdsourcing in medicine more generally [19,20,41,54], Nguyen et al [42] specifically addressed crowdsourcing that involved intellectual thinking on the part of the crowd and excluded other approaches like data collection or the performance of simple tasks (eg, classifying images or transcribing data). Information on ideation based on crowdsourcing is still scarce or underreported in the literature [42]. Only a pilot study by NASA [39] stands out. Since we were not able to identify other comparable projects, we discuss in detail the HORAO project in relation to the pilot projects of NASA, using the framework proposed by Nguyen et al [42].

HORAO was launched to overcome a technology gap in neuro-oncological surgery, specifically the inability to identify white matter tracts in real time during surgery. The rationale for turning to the public was that a solution already existed somewhere in another field of research, albeit having initially been developed to solve a different technological problem. The motivation to seek a solution, which many research groups have so far been unable to find, however, goes beyond simple ideation. The challenge organizers, all neurosurgeons themselves, face is the consequences of the lack of an appropriate technology almost daily, creating a strong personal desire to have a solution at hand. NASA faced a strategic challenge caused by a 45% reduction in its research and development budget in 2005. Formerly famous for its track record in research and innovation achieved by its own researchers, NASA decided to reach out to the crowd to solve space exploration problems, running a pilot program of challenges on InnoCentive [55] (NASA innovation pavilion). In NASA’s case, as in ours, the problem-owners identified a technology gap and believed that solutions for closing the gap would be accessible through open innovation.

HORAO—like the NASA pilot challenges—was open to the public with no restrictions or specific requirements for solvers. In both projects, the monetary incentives were large enough to motivate solvers to apply known solutions without having to finance scientific investigations (average NASA: US $7500-$30,000; HORAO: CHF 1000-35,000 [US $1089.60-$38,136]). For recruitment and challenge execution, HORAO collaborated with HeroX, whereas NASA worked with InnoCentive. InnoCentive, launched in 2001, is the pioneer and longtime leader in its field. Its hallmark is the wide network of registered experts with various academic backgrounds (about 390,000 solvers, 60% with a master’s degree or higher) and its longstanding experience in crowdsourcing innovative research. HeroX was founded in 2013 with the intention of opening up access to the public, enabling them to participate in and contribute to innovation challenges (about 170,000 solvers). Both platform providers offer a range of services, from challenge conceptualization to pre-evaluation of submissions. The services offered by HeroX and InnoCentive are comparable. One of the major benefits for HORAO was the targeted outreach offered by HeroX, which sent announcement emails to potential participants specifically to raise their awareness of the challenge. Whereas InnoCentive charges a challenge fee of about CHF 75,000 (US $81,720), HeroX charges 18% of the prize money with the security of a 50% refund if no winning idea can be identified and of a 100% refund if no idea at all is submitted, making the platform attractive to first-time users.

The number of solvers attracted by the HORAO challenge was about half the number of followers recorded for the single NASA pilot challenges (HORAO: n=220; NASA: n=419). Based on the report of NASA’s pilot program by InnoCentive [56], an average challenge usually attracts about 300 followers. What is interesting, however, is that the final number of submissions was about the same (HORAO: n=45; NASA: n=11-108). HORAO solvers were from 26 different countries, while NASA pilot challenge participants were from 5-20 different countries for a single challenge and for all 7 pilot challenges, from 30 different countries. Most submissions came from the same continents—North America, Europe, and Asia. Although more than half (7/13, 53%) of the submissions that reached the judging round (semifinals) of the HORAO challenge came from North America, all 5 finalists originated from a different country on 2 continents (Europe and North America). The NASA pilot challenge reports diversity of solvers with regard to education and expertise on the level of all followers, whereas HORAO evaluated participants from the solvers to the finalists. Thus, the numbers are not directly comparable. Moreover, different categorizations of the fields of expertise were used, which makes direct comparison difficult. Solvers attracted by the HORAO challenge were educated or employed in the fields of engineering, natural sciences, or technology, with engineering as the leading discipline. Likewise, for the NASA challenges, engineering was also the major field of employment of the solvers, followed by computer and physical sciences. Of the NASA solvers, 30% (147/490) reported having expertise within the challenge’s discipline.

In these domains too, the crowdsourcing approach applied for HORAO matches the approach used for the NASA pilot challenges. Participants contributed by providing their ideas or solutions in a competitive manner, with repeated interactions with the challenge organizers and only minor interaction with each other. A limitation reported for the challenge design of the NASA pilot program was the lack of a user template for submitting a proposal. HORAO provided a predefined format (submission form) (Multimedia Appendix 3) with questions concerning the issues relevant for evaluation.

Both the NASA pilot challenges and HORAO applied a stepwise evaluation system, with a first prescreening and consecutive evaluation round(s) conducted by an expert panel. In contrast to HORAO, in the NASA pilot challenges, the platform provider performed the prescreening. NASA itself reported this approach to be prone to inappropriate rating by the platform providers. The organizers of the NASA challenge resolved this issue by jointly defining clear rating criteria. The benefit of letting the platform providers do the pre-evaluation is that it lifts the burden of evaluating masses of low-quality proposals from the challenge owners, but it necessitates very accurately defined evaluation criteria. For HORAO, the judging scorecard was developed prior to the challenge, and the sponsor team performed a pre-evaluation, which prevented inappropriate rating and rejection of promising proposals.

In each case, pre-evaluation removed low-quality submissions and avoided the jury being overwhelmed in the consecutive evaluation round(s) by too many proposals, thus allowing them to focus on the promising solutions. For HORAO, an additional benefit of reducing the number of proposals passing the pre-evaluation stage was that it enabled an effective feedback system during the consecutive challenge phases.

Most of the potential difficulties listed by Nguyen et al [42] (Figure 3) did not apply to HORAO (integration of open innovation in traditional processes and protection of data privacy) or were avoided by the choice of the challenge design (risk of unqualified solvers or low-quality proposals, inappropriate incentives, lack of platform, unclear task description, and risk of dominant voices). Nevertheless, selective participation and bias in decision-making are important risks to consider. Although open to the public, the targeted outreach by HeroX probably led to a partial selection of the participants. We do not consider this a major disadvantage since it raised awareness among the crowd addressed without excluding anyone from participating. In a future crowdsourcing ideation campaign, we would again focus the targeted outreach on the scientific crowd, possibly by advertising the campaign at specific conferences. The provision of predefined criteria was intended to avoid bias in decision-making or evaluation. Nevertheless, some criteria were more objectively assessable than others, creating a possible source of bias. Overall, the sponsor team greatly benefited from working with a preexisting commercial crowdsourcing company, especially as it was the team’s first such campaign.

Our strategy of crowdsourcing ideation relied on prize money as an incentive, to which the usual, especially governmental, funding agencies are unlikely to contribute. Collecting the prize money beforehand in a crowdfunding campaign, as done for the HORAO project, requires time and effort.

Running the crowdsourcing campaign also involved time-consuming tasks, such as producing explanatory videos, press releases, and daily responses to solvers’ questions. The performance of these tasks by the challenge owners themselves, who were laypersons in the case of HORAO, increased the time and effort expended because they first had to acquire the essential skills. Alternatively—and as done for the video in our project—some of these tasks may be outsourced, in which case they incur additional costs. The pre-evaluation performed by the sponsor team was another time-consuming task, limiting the future usability of the approach applied.

The HORAO campaign was the first of its kind to crowdsource for new ideas on contemporary problems in neurosurgery. The campaign was successful in raising awareness of a longstanding neurosurgical problem; the lack of a means of intraoperative real-time visualization of fibers to delineate tumor tissue from surrounding healthy tissue. It allowed us to gain access to a multitude of outside-the-box potential solutions, and the team of experts was able to rapidly assess them in parallel for their capacity to solve our problem. Ultimately, the crowdsourcing campaign led to the creation of a very successful interdisciplinary research unit, which is now funded by traditional governmental funds.