This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The advent of Internet-based self-help systems for common mental disorders has generated a need for quick ways to triage would-be users to systems appropriate for their disorders. This need can be met by using brief online screening questionnaires, which can also be quickly used to screen patients prior to consultation with a GP.
To test and enhance the validity of the Web Screening Questionnaire (WSQ) to screen for: depressive disorder, alcohol abuse/dependence, GAD, PTSD, social phobia, panic disorder, agoraphobia, specific phobia, and OCD.
A total of 502 subjects (aged 18 - 80) answered the WSQ and 9 other questionnaires on the Internet. Of these 502, 157 were assessed for DSM-IV-disorders by phone in a WHO Composite International Diagnostic Interview with a CIDI-trained interviewer.
Positive WSQ “diagnosis” had significantly (
The WSQ screens appropriately for common mental disorders. While the WSQ screens out negatives well, it also yields a high number of false positives.
The thriving development of Internet-based self-help aids [
The screening must be brief, as subjects will undergo screening more readily if it is short, quick [
Participants were recruited (between May and December 2007) from the general Dutch population by using Internet banners (eg, Google and Dutch Internet sites on mental health issues). The advertisements linked to a Web page containing information about common mental disorders, Internet treatment and this study, an application form, and a link to the questionnaires. Subjects were asked to input their name and email address, so they could be identified and added to the data pool only once.
We specifically targeted adults (18 years of age or older) with Internet access and who felt anxious, depressed, or thought of themselves as drinking too much alcohol. We targeted a population with a high rate of common mental disorders as the kind likely to use the WSQ in the future. Since this population can only illuminate false negative and true positive rates, we needed controls to test those rates. Therefore, we also recruited 20 undergraduate psychology students who were not required to have symptoms, using banners at the VU University’s students’ Web page seeking participants for VU studies.
We excluded people reporting a high suicide risk (ie, a score of 3 on Q15 of the WSQ); they were advised to contact their GP. To raise the response rate, participants were told in advance that completers of the screening questionnaires would be offered a self-help book for common mental problems. Students received academic credit for participating. The study protocol was approved by the Medical Ethics Committee at the VU Medical Centre in Amsterdam, Netherlands.
Our study tested the WSQ’s validity and consisted of two parts (
Completion of 10 sets of questions: Internet demographic questions, the WSQ, and other questionnaires for common mental disorders: Center for Epidemiological Studies Depression scale (CES-D [
A DSM-IV-diagnostic phone interview with a Composite International Diagnostic Interview (CIDI)-trained interviewer (CIDI lifetime, World Health Organization (WHO) version 2.1 [
In all, 687 people applied for the study, of whom 185 (27%) were excluded because they represented a high suicide risk (n = 5); there was no written informed consent (n = 22); or they refused to participate (n = 158). This left 502 participants, of whom 389 consented to a diagnostic phone interview, but 232 (60%) of those 389 either could not be contacted (n = 227) or refused (n = 5), leaving 157 participants who were phoned by a CIDI-trained interviewer within a mean of 13 days.
If participants had never experienced a traumatic event, they skipped the IES; if they had never drunk alcohol, they skipped the AUDIT; and if they had never suffered a panic attack, they skipped the PDSS-SR. Those who completed the screening questionnaires and gave informed consent entered the study.
Flowchart of participants (WSQ)
The WSQ for common mental disorders [
WSQ Q2 for depression, from CIDI [
WSQ Q4, 8, 9, 10, and 12 (for panic, social phobia, PTSD, and OCD from Mini-International Neuropsychiatric Interview (M.I.N.I. [
WSQ Q13 and 14 (for alcohol, from AUDIT [
Three questions of the original WSQ reached either low specificity or low sensitivity. To enhance validity, we used logistic regression analysis to determine whether other items from appropriate questionnaires could replace these WSQ-items. We amended three questions using items for GAD (WSQ Q3, from GAD-7 [
The Dutch version of the CES-D [
We translated the GAD-7 [
The Dutch version of the PDSS [
The Dutch version of the FQ [
The IES [
We used the Dutch 10-item severity subscale of the YBOCS [
The Dutch version of WHO’s self-rated AUDIT [
We used the Lifetime version 2.1 of the CIDI [
To establish whether WSQ scores differed significantly between subjects with positive and with negative screen results, we conducted t-tests on the mean and standard deviation of each screening instrument separately. In the sub-sample that had a diagnostic interview, we performed chi-square tests to ascertain whether WSQ scores differed between subjects with and without DSM-IV disorders.
We calculated sensitivity and specificity, and positive and negative predictive values, for each WSQ subscale regarding its corresponding DSM-IV disorder (predictive validity). Sensitivity is the probability that a person who has a disorder is screen positive. Specificity is the probability that a person not suffering from a disorder is screen negative. There is no consensus of what levels of sensitivity and specificity are acceptable, as they depend on the test’s aim, costs, and benefits [
For WSQ questions which turned out to have unacceptable sensitivity or specificity, we replaced them with relevant items from the appropriate screening questionnaire. To find which items best predicted the chance of detecting a diagnosis, we used logistic regression analyses (Forward Likelihood Ratio method). We replaced items only if they improved validity. We calculated the Area Under the Curve (AUC) for the WSQ’s scaled and dichotomous response options and its appropriate screening questionnaires. The AUC (the sum of sensitivity versus [1 – ] specificity) measures a scale’s accuracy; it equals the probability that a randomly chosen case will score higher than a randomly chosen non-case [
Our analyses used diagnoses reached within the last 6 months. MDD, Dyst, and MinD were combined into the category depressive disorder. For all analyses we used SPSS version 15.0 for Windows.
The total sample (N = 502) had a mean age of 43 years (SD 13, range 18 - 80), and 285 (57%) of the subjects were female. Of the 157 subjects who had a CIDI interview, the mean age was 43 (SD 15, range 18 - 80). Of these, 89 (57%) were female, and 107 (68%) subjects met DSM-IV criteria for any current (ie, within the past 6 months) depressive disorder, anxiety disorder, and/or alcohol abuse/dependence. A total of 67 (43%) subjects had more than one diagnosis (
Demographic characteristics and prevalence of diagnosis
N (%) | |||
Complete sample | CIDI sub-sample | ||
Completed all questionnaires on Internet | 502 (100) | 157 | |
|
|||
Male | 217 (43) | 68 (43) | |
Female | 285 (57) | 89 (57) | |
|
43 (13) | 43 (15) | |
(Range) | (18 - 80) | (18 - 80) | |
|
|||
Lowa | 99 (20) | 27 (17) | |
Mediumb | 217 (43) | 73 (47) | |
Highc | 186 (37) | 57 (36) | |
|
|||
Netherlands | 474 (94) | 146 (94) | |
Other | 28 (6) | 11 (6) | |
|
|||
Single | 180 (36) | 65 (41) | |
Married or cohabiting | 241 (48) | 67 (43) | |
Divorced/widowed | 81 (16) | 25 (16) | |
DSM-IV diagnosis within last 6 months, on CIDI phone interview | 157 | ||
Any depressive disorder | 52 (33) | ||
Major depressive disorder | 46 (29) | ||
Dysthymia | 9 (6) | ||
Minor depression | 8 (5) | ||
Any anxiety disorder | 94 (60) | ||
Social phobia | 32 (20) | ||
GAD | 30 (19) | ||
Panic disorder | 10 (6) | ||
Panic with agoraphobia | 22 (14) | ||
Agoraphobia | 10 (6) | ||
Specific phobia | 40 (26) | ||
Obsessive-compulsive disorder | 10 (6) | ||
PTSD | 12 (8) | ||
Alcohol abuse/dependence | 23 (15) | ||
Any disorder | 107 (68) | ||
> one diagnosis | 67 (43) |
aLow education: primary and lower general secondary education.
bMedium education: Intermediate Vocational Training,school of higher general secondary education or pre-university education.
cHigh education: higher vocational education or university.
WSQ and screening questionnaires: means, standard deviations (SDs) and prevalence (N = 502)
Other screening questionnaires: | “Diagnosis” on WSQ (Web Screening Questionnaire) | |||||
Yes | No | t (d.f. = 500) | ||||
N(%) | M (SD) | N (%) | M (SD) | |||
|
||||||
CES-D |
296 (59.0) | 32.2 (7.1) | 206 (41.0) | 18.1 (10.3) | 15.2a | |
|
||||||
GAD-7 |
320 (63.8) | 13.6 (3.9) | 182 (36.3) | 5.5 (3.1) | 24.3 a | |
|
||||||
PDSS-SR |
278 (55.4) | 9.3 (5.1) | 224 (44.6) | 0.6 (1.7) | 24.2 a | |
|
||||||
PDSS-SR | 153 (30.5) | 11.2 (5.1) | 349 (69.5) | 2.9 (4.1) | 19.3 a | |
|
||||||
FQ-agoraphobia |
205 (40.8) | 12.7 (10.9) | 297 (59.2) | 2.9 (4.5) | 14.0 a | |
|
||||||
FQ-social phobia |
226 (45.0) | 16.6 (8.7) | 276 (55.0) | 7.0 (6.0) | 14.6a | |
|
||||||
FQ-specific phobiab |
290 (57.8) | 7.6 (7.7) | 212 (42.2) | 2.3 (3.5) | 9.2 a | |
|
||||||
YBOCS |
182 (36.3) | 11.0 (6.3) | 320 (63.8) | 0.8 (2.3) | 26.2 a | |
|
||||||
IES |
273 (54.4) | 33.5 (20.1) | 229 (45.6) | 0.0c | 25.3 a | |
|
||||||
AUDIT |
198 (39.4) | 19.6 (6.2) | 260 (60.6) | 6.3 (5.5) | 24.4 a |
aSignificant at
bAdditional specific phobia questions were dichotomous, so their means and standard deviations could not be calculated.
cIf participants had never experienced a traumatic event then they skipped the IES.
For the three WSQ subscales, GAD, OCD, and panic, validity was below threshold levels of 0.70 for sensitivity and 0.40 for specificity, so we replaced those (based on logistic regression analysis) with relevant items from the appropriate screening questionnaires (GAD-7, YBOCS, and PDSS-SR, respectively). This improved sensitivity or specificity. The WSQ subscale-specific phobia had an unacceptably low sensitivity (0.60), but we did not replace it with an item from the appropriate screening questionnaire as that did not improve sensitivity or specificity.
Based on the log-likelihood ratio statistic, using logistic regression analyses, we added three categories of the specific phobia question, “Are you scared of …?”. These categories were (1) animals, (2) specific situations, and (3) medical issues, which improved the sensitivity of the WSQ subscale for specific phobia but not for specificity (sensitivity: from 0.60 to 0.80; specificity: from 0.77 to 0.47).
WSQ vs CIDI diagnoses: prevalence, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) (n = 157)
WSQ “Diagnosis” | CIDI DSM-IV Diagnosis | |||||||
No | Yes | χ² (d.f. = 1) | Sensitivity | Specificity | PPV | NPV | ||
n | n | |||||||
|
||||||||
WSQ-depression | No | 62 | 8 | 26.8a | 0.85 | 0.59 | 0.51 | 0.89 |
Yes | 43 | 44 | ||||||
|
||||||||
WSQ-GAD | No | 57 | 2 | b | 0.93 | 0.45 | 0.29 | 0.97 |
Yes | 70 | 28 | ||||||
|
||||||||
WSQ-social phobia | No | 91 | 9 | 22.0a | 0.72 | 0.73 | 0.40 | 0.91 |
Yes | 34 | 23 | ||||||
|
||||||||
WSQ-panic | No | 65 | 1 | b | 0.90 | 0.44 | 0.10 | 0.98 |
Yes | 82 | 9 | ||||||
|
||||||||
WSQ-panic+agoraphobia | No | 104 | 3 | b | 0.86 | 0.77 | 0.38 | 0.97 |
Yes | 31 | 19 | ||||||
|
||||||||
WSQ-agoraphobia | No | 92 | 0 | b | 1.00 | 0.63 | 0.15 | 1.00 |
Yes | 55 | 10 | ||||||
|
||||||||
WSQ-specific phobia | No | 55 | 8 | 9.1c | 0.80 | 0.47 | 0.34 | 0.87 |
Yes | 62 | 32 | ||||||
|
||||||||
WSQ-OCD | No | 102 | 2 | b | 0.80 | 0.69 | 0.15 | 0.98 |
Yes | 45 | 8 | ||||||
|
||||||||
WSQ-PTSD | No | 68 | 2 | b | 0.83 | 0.47 | 0.11 | 0.99 |
Yes | 77 | 10 | ||||||
|
||||||||
WSQ-alcohol | No | 97 | 4 | b | 0.83 | 0.72 | 0.34 | 0.96 |
Yes | 37 | 19 |
aSignificant at
bNot able to calculate χ² due to small numbers (< 5) in cells.
cSignificant at
Compared to the corresponding CIDI DSM-IV diagnoses, the AUC for the WSQ subscales with scaled responses (WSQ subscales GAD, OCD, alcohol, and panic) were similar to the AUC of the longer questionnaires, ranging from an AUC of 0.76 for the WSQ subscale panic versus an AUC of 0.70 of the PDSS-SR, to an AUC of 0.81 for the WSQ subscale OCD versus an AUC of 0.85 for the YBOCS. The AUC for the dichotomous WSQ’s subscales of panic with agoraphobia and agoraphobia were similar to the AUC of the longer, scaled questionnaires (PDSS: AUC of 0.79 versus WSQ panic with agoraphobia: AUC of 0.82; both WSQ and FQ subscale agoraphobia: AUC of 0.81), but not for the WSQ dichotomous subscales of depression, social phobia, and PTSD (ranging from WSQ subscale depression: AUC of 0.72 versus CES-D: AUC of 0.84 to WSQ subscale PTSD: AUC of 0.65 versus IES: AUC of 0.82) (
WSQ and screening questionnaires versus CIDI diagnoses: Area Under the Curve (AUC) and 95% CI for scaled and dichotomous response options
WSQ “Diagnosis” | CIDI DSM-IV Diagnosis | |
AUC | 95% C.I. | |
|
||
WSQ-depression | 0.72 | 0.64 - 0.80 |
CES-D | 0.84 | 0.77 - 0.90 |
|
||
WSQ-GAD | 0.78 | 0.69 - 0.86 |
GAD-7 | 0.77 | 0.68 - 0.85 |
|
||
WSQ-social phobia | 0.72 | 0.62 - 0.82 |
FQ-social phobia | 0.82 | 0.74 - 0.89 |
|
||
WSQ-panic | 0.76 | 0.59 - 0.93 |
PDSS-SR | 0.70 | 0.57 - 0.88 |
|
||
WSQ-panic+agoraphobia | 0.82 | 0.72 - 0.91 |
PDSS-SR | 0.79 | 0.69 - 0.89 |
|
||
WSQ-agoraphobia | 0.81 | 0.73 - 0.90 |
FQ-agoraphobia | 0.81 | 0.70 - 0.91 |
|
||
WSQ-OCD | 0.81 | 0.65 - 0.97 |
YBOCS | 0.86 | 0.72 - 0.99 |
|
||
WSQ-PTSD | 0.65 | 0.51 - 0.80 |
IES | 0.82 | 0.67 - 0.97 |
|
||
WSQ-alcohol | 0.77 | 0.68 - 0.86 |
AUDIT | 0.75 | 0.66 - 0.84 |
As expected, students compared to non-students had significantly lower scores on the WSQ subscales for depression (P = .004), alcohol (P < .001), GAD (P < .001), OCD (P < .001), panic (P < .001), and panic with agoraphobia (P = .004).
Demographic variables did not differ significantly between subjects who had a CIDI diagnostic interview and those who did not. However, those who had a CIDI interview scored significantly lower on one WSQ subscale (social phobia; P = .009), on the CES-D (P = .05), and on the FQ social-phobia subscale (P = .03).
It takes about two minutes to complete the WSQ to detect common mental disorders. The WSQ quickly detects clinically-relevant mood, anxiety, and alcohol-related problems and so can guide Internet users to Internet-self-help modules appropriate for their problem, or quickly screen patients prior to consultation with a GP. This measure can also be used in more homogeneous samples to screen out people with co-morbid disorders. The WSQ turned out to be a valid screener for social phobia, panic disorder with agoraphobia, agoraphobia, OCD, and alcohol abuse/dependence (sensitivity: 0.72 - 1.00; specificity: 0.63 - 0.80), and appropriate for depressive disorder, GAD, PTSD, specific phobia, and panic disorder (without agoraphobia) (sensitivity: 0.80 - 0.93; specificity: 0.44 - 0.51) in our study population. Interestingly, the AUC’s of the WSQ’s scaled single items, and some of the dichotomous items, were comparable to the AUC’s of the longer questionnaires, supporting our conclusion that short questionnaires, sometimes with just one item, can be as valid as longer ones. This is in line with previous studies [
Compared to psychometric properties of other online screening questionnaires [
Although WSQ’s false positives do not have a diagnosis, they might have symptoms of depression, anxiety, or alcohol problems, since they have elevated scores on the relevant screening questionnaires.
One limitation of our study is that the CIDI-diagnosis live phone interviews were not taped, so inter-rater reliability could not be calculated. Second, subjects always completed the WSQ on the Internet before the other screening questionnaires, so order effects could not be ruled out. Third, though sensitivity and specificity do not depend on prevalence of the disorders in the population, the PPV and NPV do; consequently, the values we found might not generalize to situations where prevalence is different. Fourth, it is not known how representative our self-recruited participants are of Internet self-help applicants. Fifth, subjects who had a CIDI interview had significantly less social phobia on that WSQ-subscale than those who did not, so the WSQ-social-phobia results might be less generalizable to other populations. Sixth, as described earlier, 6-month prevalence rates of DSM-IV diagnoses were used, whereas the WSQ assesses current symptoms. Ideally, the WSQ should be validated against concurrent DSM-IV diagnoses. Seventh, norms are unavailable for acceptable levels of sensitivity and specificity which depend on the test’s aim, costs, and benefits [
Despite its limitations, the WSQ is a useful and quick Internet screening tool to detect people likely to have common mental disorders.
Many false positives were found for WSQ subscales GAD, panic, specific phobia, and PTSD, while far fewer false positives were found for alcohol abuse/dependence, social phobia, panic disorder with agoraphobia, and OCD. The high rate of false positives may, for some questions, be due to a lack of clarity or classification criteria. Future research which enhances clarity of questions and classification criteria is needed to improve the predictive power of the WSQ.
This study is funded by the Faculty of Psychology and Education of the VU University, Amsterdam.
None declared.
WSQ
area under the curve
alcohol use disorders identification test
Center for Epidemiological Studies Depression scaleCIDI:composite international diagnostic interview
dysthymia
Diagnostic Statistical Manual, 4thedition
fear questionnaire
generalized anxiety disorder
generalized anxiety disorder - 7
general practitioners
impact of events scale
Internet-based self-assessment program for depression
mean
minor depression
major depressive disorder
mini-international neuropsychiatric interview
negative predictive value
obsessive compulsive disorder
panic disorder severity scale self-report
positive predictive value
Post-Traumatic Stress Disorder
standard deviation
screening questionnaire
Vrije Universiteit
Web-based depression and anxiety test
World Health Organization
Web screening questionnaire
Yale-Brown Obsessive Compulsive Scale