One approach to assessing the efficacy of symptomatic drugs when randomized controlled trials are lacking is through a survey instrument fielded to clinicians, patients, or caregivers. In 2003, Forshew and Bromberg [9] reported the results of a national survey sent out to 61 ALS clinics in North America, to which 39 clinics replied. Clinicians were asked to name the three most frequent interventions they would use for each symptom and their respective rankings for their perception of efficacy in relieving the symptom (ranging from 1, rarely effective, to 5, nearly complete relief). The 14 symptoms included in the survey were selected based on a literature search. The article reported the four top interventions for each of the predetermined symptoms.

Forshew and Bromberg mentioned in their article that a mirror study giving the patients’ perspective on the same topic was underway, though the results of that study have not been published yet. In the absence of such a study, it is still possible to contrast clinicians’ and patients’ perspectives by looking into rapidly increasing and publicly available databases of online communities of patients. PatientsLikeMe is an online community that allows members to track their progress with clinical scales, share information, and learn more about their condition. It has established data-sharing partnerships with doctors, pharmaceutical and medical device companies, research organizations, and nonprofits. Starting with an ALS patient community, PatientsLikeMe now hosts communities for more than 100,000 patients with any medical condition. Currently, the US ALS user base represents about 5% of the US ALS population and is an almost perfect match in terms of demographics, but with a bias toward long-term survivors. By definition, a bias toward patients with computer and Web access is indeed also present.

Each community member is asked to track 11 to 12 primary symptoms of his or her condition. The primary symptom list was generated for the ALS community with input from health care practitioners and the literature. Members can also report, in free-text form, any additional symptoms they are experiencing. The result is a semistructured list, which patients can use as an aid for future symptom reporting. It also permits comparison with symptoms reported by other patients in the system. Additional symptoms entered by a member lead to a prompt (“did you mean...”) to attempt to merge their description with existing symptom entries; if they wish to add a new symptom term, this is reviewed by a nurse curator.

Comparing data from very different sources is challenging, as the many adjustments that datasets must go through may induce further biases. However, in the absence of robust evidence from randomized controlled trials (which also have their own limitations and weaknesses), the data from surveys and online communities of patients become a viable resource in the search for a better understanding of clinicians’ and patients’ perceptions.

We used the Forshew and Bromberg study as our primary data source and adjusted the data from PatientsLikeMe to allow for comparisons. We first extracted the 14 symptoms and associated top four treatments listed by Forshew and Bromberg. We then searched the PatientsLikeMe database for the same symptom–treatment pairs. Members of PatientsLikeMe also mentioned 13 of the 14 symptoms listed by Forshew and Bromberg. The only nonoverlapping symptom was thick phlegm, which had no corresponding item. The PatientsLikeMe data are structured and thus no preprocessing of the data was required.

In this study we proposed to compare clinicians’ and patients’ perceptions of the efficacy of different treatments to alleviate ALS symptoms. There is one important distinction between the perceptions of these two groups. Patients’ perception of treatment efficacy is direct and personal. That is, patients evaluate treatment efficacy based on how well they responded to the treatment. Clinicians’ perception, on the other hand, is an aggregate judgment, but presumably is based on how well the treatment worked for their patient population.

To enable a fair comparison between the two datasets, we made a few adjustments and transformations. PatientsLikeMe provided a dataset for analysis. We started by searching for efficacy data that would match the 56 symptom–treatment pairs from the Forshew and Bromberg study. We then discarded data on any symptom–treatment pair for which patient sample size was smaller than 6 patients. Given that each pair had a different sample size, we converted the frequencies to percentages to facilitate a visual comparison. Finally, we adjusted the rating scales from both sources. The rating scales from the Forshew and Bromberg study and from PatientsLikeMe were composed of five levels. Items rated as “Can’t tell” from the PatientsLikeMe dataset were not included in our comparative analysis because (1) they were not part of an ordinal scale like the 4 other items (ie, none, mild, moderate, or major), and (2) they referred to cases in which patients had not yet formed an opinion. We combined the two top levels in the Forshew and Bromberg study because the last item in the scale, nearly complete relief, was quite small (corresponding to less than 3% of the ratings), and it interrupted the continuity of the scale. The first 4 items in the scale explicitly used the word effective. Further, the first two qualifiers (ie, rarely and sometimes) strongly suggested a rating scale based on frequency. The last item in the scale, however, suggested a rating based on the intensity rather than on the frequency of the effect by using the qualifier complete. Figure 1 summarizes the adjustments made to both scales to facilitate a comparative analysis.

After filtering out the symptoms with fewer then 10 respondents from the list of 13 overlapping symptoms, 10 symptoms and 15 symptom–treatment pairs remained. Table 1 shows the number of patients who responded to the question on the treatments prescribed or received for the 10 specific symptoms. Table 2 shows the 15 symptom–treatment pairs. None of the 15 pairs was used by either group more than 50% of the time. In 7 of the 15 pairs, the discrepancy in usage between the two groups was above 10%. In 3 pairs the differences were over 20%: amitriptyline for labile emotional effect; oxybutynin for urinary urgency; and amitriptyline for disturbed sleep.

Of the symptom–treatment pairs, 13 corresponded to US Food and Drug Administration-approved (indicated) uses of the drugs; 2 pairs corresponded to uses for which an indication had not been granted (off-label use): amitriptyline for labile emotional effect and gabapentin for fasciculations.

After eliminating symptom–treatment pairs for which the number of efficacy evaluations was smaller than 10, we were left with 5 pairs. In the following paragraphs we present the results for the perceived efficacy of amitriptyline to treat sialorrhea and baclofen to treat spasticity and stiffness. Results for the remaining 3 pairs are provided in Multimedia Appendix 1.

While providing valuable data, Forshew and Bromberg also duly acknowledge one limitation in the design of their survey: respondents ranked treatments based on an average ALS patient, which oversimplifies the real issue, given that response to treatment may vary according to the disease stage.

Both clinician and patient cohorts are expected to be heavily skewed by self-selection bias. That is, the sample is less likely to be a good representation of a larger population when it is not completely random. In fact, complete sample randomization is a constant issue in any study that deals with human participants. We do not know the demographics or clinical characteristics of patients treated by the physician participants in the Forshew and Bromberg study. We can assume that the patient population was roughly representative of the general patient population, since Forshew and Bromberg conducted a national survey. Similarly, we have little knowledge of the PatientsLikeMe population’s clinical characteristics. The size of the ALS community on PatientsLikeMe, however, is fairly large, representing roughly 5% of the US ALS population. The age and gender of the PatientsLikeMe ALS population do resemble those of the general ALS population.

Given the very small sample sizes for both datasets, the outcomes regarding treatment efficacy are not as robust as one would wish. However, given that the patient pool of PatientsLikeMe grows daily, we expect that such inaccuracies due to sample size will become increasingly smaller in the case of patient-reported data.

In terms of relative sample size, cohorts of clinicians and patients do not lend themselves to a direct comparison. Patients’ observations are based on their personal experiences and thus are direct and singular. Clinicians’ observations, on the other hand, are indirect and aggregate—that is, they are formed based on data coming from a collection of patients. Further, clinicians obviously possess more extensive clinical knowledge than patients do, which makes clinicians’ observations more systematic. Consequently, studies based on patients’ perspectives are likely to require larger sample sizes than studies based on clinicians’ perspectives.

In this study we compared data from only one survey and one online community of patients. Future studies should consider including more instances of each type of data source, which would compensate for the specific limitations and weaknesses of each data source.

Several authors have called attention to the fact that self-selection bias is a common threat to the internal and external validity in studies involving human participants [12,13]. This threat is further magnified in studies that use measurements based on participants’ perception.

Our comparative analysis detected significant discrepancies between patients’ and clinicians’ perceptions of treatment prevalence and efficacy. Both data sets have their own limitations, which were partially described in the previous sections. However, if they are properly conjugated, their potential to offer a more complete and accurate picture of ALS symptomatic treatment is likely to increase. As previously mentioned, data validity issues related to sample size in PatientsLikeMe are likely to gradually self-correct as the online community grows.

Assessing and comparing the efficacy of the symptomatic treatment of a complex and rare disease such as ALS is not easy. It is also important that patients’ as well as clinicians’ views be taken into consideration. Drawing a reliable profile of treatment efficacy requires taking into consideration many interacting aspects (eg, disease stage and severity of symptoms) that were not covered in the present study. Nevertheless, pilot studies such as the one described here can pave the way for more robust studies by helping researchers anticipate and compensate for limitations in their data sources and study design.