For a small fee, the CrowdMed website allows undiagnosed patients to submit their clinical information and obtain potential diagnoses expeditiously. Patients anonymously answer a comprehensive set of medical questions and upload relevant test results and images related to their cases (Figure 1).

Patients also decide how long they want their cases open and whether they wish to compensate the case solvers. Anyone (including nonmedical persons) can sign up to be a case solver and select cases they think they can help solve (Figure 2).

While the cases are open, patients and case solvers can discuss details online about potential diagnoses, further work-up that should be done, and newly obtained test results and/or appointments completed with the patients’ physicians. Thus, case details can unfold online while the case is still open. All diagnostic suggestions and all case discussions are available to all case solvers as they are suggested and discussed throughout the open period. This enables the entire group of case solvers to work in concert to solve each case.

When a patient’s case is closed, the patient receives a detailed report containing the entire list of diagnostic suggestions made by the case solvers and suggested next steps, so that they can discuss them with their physicians. Diagnoses are ranked in decreasing order of “relative popularity.” The relative popularity of diagnoses is determined by case solvers’ “bets” on each diagnosis in terms of their beliefs that the diagnosis is the most specific, accurate, root cause of the symptoms presented. CrowdMed takes these bets and assigns points to each diagnosis using a prediction market algorithm, thereby determining the “relative popularity” of each diagnosis suggested. Finally, patients are provided with case solvers’ reasoning for choosing particular diagnoses. Patients choose which case solver(s) to compensate based on whose answers they found helpful. If the patient decides to reward multiple solvers, they also decide how to divvy up the compensation. Afterward, patients are invited to fill out surveys about their outcomes.

We independently analyzed all CrowdMed data collected from May 2013 to April 2015. Specifically, we analyzed data on patients’ demographic and case characteristics; case solvers’ demographic and performance characteristics; and preliminary case outcomes. Outcomes included whether patients would recommend CrowdMed, if the program provided insights leading them closer to correct diagnoses, and estimated improvements in patients’ productivity and medical expenses. Data were summarized using descriptive statistics and independent samples t tests using IBM SPSS Statistics 22.

During the study period, 397 cases were completed (350 from the United States). Patients’ self-reported mean (SD) age was 47.8 (18.8) years (age range 2-90) and 182 were males (45.8%).

Before case submission, patients reported visiting a median of 5 physicians (interquartile range [IQR] 3-10; range 0-99), incurred a median of US $10,000 in medical expenses (IQR US $2500-US $50,000; range US $0-US $5,000,000) including payments by both patients and payers, spent a median of 50 hours (IQR 15-150; range 0-12,000) researching their illnesses online, and had symptoms for a median of 2.6 years (IQR 1.1-6.9; range 0.0-70.6). Online case activity lasted a median of 60 days (IQR 30-90; range 2-150) and case solvers were offered a median of US $100 in compensation (IQR US $0-US $200; range US $0-US $2700) for diagnostic suggestions. A total of 59.7% (237/397) of the cases were compensated with a median compensation of US $200 (IQR US $100-US $300; range US $15-US $2700).

During the study period, CrowdMed had 357 active case solvers; of which 37.9% (132/348) were male, 76.7% (264/344) were from the US, and 58.3% (208/357) worked or studied in the medical industry; including 36 physicians and 56 medical students. Mean (SD) age was 39.6 (13.8) years (range 17-77 years).

Solvers participated in a median of 3 cases (IQR 1.0-12.8; range 0-415), earned a median of US $0 (IQR US $0-US $1.18; range US $0-US $3952) and a mean (SD) of US $93.97 (US $364.72; the majority earned US $0). Median solver rating was 3 (out of 10; IQR 3-6; range 1-10) and significantly higher (P=.006) for medical industry-based solvers (mean [SD] 4.8 [2.5]; range 1-10) than for others (mean [SD] 4.1 [2.2]; range 1-10).

At completion, 50.9% (202/397) of patients were likely to recommend CrowdMed to a friend, 59.6% (233/391) reported that the process gave insights leading them closer to correct diagnoses, 57% (52/92) reported estimated decreases in medical expenses, and 38% (29/77) reported estimated improvements in school or work productivity (Table 1).

Patients reporting helpful insights from CrowdMed saw fewer doctors (mean [SD] 7.2 [7.3]; range 0-99) before participating than those who did not report receiving helpful insights (mean [SD] 9.2 [10.7]; range 0-50), P=.047. The 14 most common diagnoses suggested as the most popular diagnosis for a case are presented in Table 2.

In addition, some patients informally reported to CrowdMed that the program helped them find diagnoses that their physicians previously were unable to determine, including Sjögren’s syndrome and chorda tympani dysfunction.

Our independent evaluation suggests that at least some patients with undiagnosed illnesses reported receiving helpful guidance from crowdsourcing their diagnoses during their difficult diagnostic journeys. Several of the conditions most commonly suggested by case solvers are conditions well known to represent diagnostic challenges. The crowdsourcing strategy enabled dynamic interaction between patients and case solvers as more case details unfolded over time.

Novel approaches are needed to help patients who experience difficulties in obtaining a correct and timely diagnosis. In that regard, advantages of using “wisdom of the crowd” could include low cost, increased program accessibility for patients, and relatively quick opinions. Although the data we obtained were useful for understanding this program, there were several limitations of our study. The postparticipation survey was rather limited in scope as it was designed for business purposes and not for research. In addition, there was no way to verify patient-reported data and some patient-reported data might be outside of realistic boundaries (eg, 1 patient reported spending 12,000 hours researching illnesses online). Furthermore, downstream outcomes of patients were not systematically collected, so it is not known what their eventual diagnoses were or if the program identified them accurately. Further development and use of crowdsourcing methods to facilitate diagnosis requires long-term evaluation as well as validation to account for patients’ ultimate correct diagnoses.

Although crowdsourcing appears to have potential, it is important to identify factors that lead to successful crowdsourcing to improve the process and help improve patient care. Multidisciplinary research is needed to gain both technical and nontechnical insights into how this can be done. For example, previous researchers have identified the importance of both finding crowd members with the appropriate skills to the relevant problem and providing adequate motivation to the crowd for the successful use of crowdsourcing for problem solving [23]. Finally, the potential legal ramifications of giving individuals without medical degrees (who make up a substantial portion of the case solvers) the ability to render diagnostic opinions would need to be considered [24].

In conclusion, our independent evaluation suggests that some patients with undiagnosed illnesses report receiving helpful guidance from crowdsourcing their diagnosis. Further development and use of crowdsourcing methods to facilitate diagnosis require multidisciplinary research and long-term evaluation that includes validation to account for patients’ ultimate correct diagnoses.