The diagnosis of mental disorders has many important roles in clinical practice, research, and administration (eg, communication, treatment planning and evaluation, decision making, classification, policy development) [1]. However, there are various issues that limit the practice and utility of diagnostic assessment in traditional face-to-face settings [2-5]. For example, clinicians typically favor unstructured interviewing despite being prone to bias and error [4], whereas the more reliable structured interviewing format is often overlooked for being cumbersome and costly to administer in everyday practice [5].

The Internet offers various benefits to assist the assessment of mental disorders [6,7]. Internet-based questionnaires can incorporate complex branching and scoring rules, as well as seamlessly present items and feedback in a standardized manner. The Internet also offers minimal ongoing delivery costs, accessibility across diverse population groups, and efficient data collection. Consumer accessibility is typically better than for traditional face-to-face services because it is usually associated with lower cost and greater convenience. Furthermore, the potential anonymity of online assessment facilitates self-awareness and self-disclosure, potentially enabling more valid outcomes [8].

Given these advantages, numerous and diverse online diagnostic assessment tools have been made available. However, published psychometric properties regarding diagnostic outcomes are only available for a small proportion of these. Furthermore, performance varies widely across these reported programs (eg, [9-12]), probably due to differences in program characteristics and study methodologies. For example, Farvolden et al [9] reported on the validity of the Web-Based Depression and Anxiety Test (WB-DAT), a diagnostic screener for depression and anxiety disorders that functions similarly to a structured diagnostic interview based on Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition) (DSM-IV) criteria. With a clinician-administered Structured Clinical Interview for DSM-IV (SCID-IV) as the gold standard, the WB-DAT displayed a high level of diagnostic accuracy in terms of sensitivity (0.71-0.95) and specificity (0.87-0.97). However, results were limited in that participants were recruited from face-to-face clinical trials and may not have represented typical online consumers of the program. Furthermore, the study involved generally low diagnostic base rates that could have biased classification statistics. Nevertheless, the results for the WB-DAT suggest that an online program can achieve a high level of diagnostic sensitivity and specificity.

More recently, Donker et al [10] evaluated the Web Screening Questionnaire (WSQ), which also diagnostically screens for multiple DSM-IV disorders (eg, depression, anxiety, and alcohol-related disorders). Unlike the WB-DAT, the WSQ is very brief, with only 1 to 2 items assigned to each disorder and 15 items in total to promote access and completion [10]. In contrast to Farvolden et al’s study, participants (N=502) were recruited online and subsequently completed the WSQ remotely to better represent potential program usage. Compared against a telephone-based Composite International Diagnostic Interview (CIDI) as the gold standard, a refined version of the WSQ displayed generally high sensitivity (0.72-1.00). However, the WSQ demonstrated relatively poor specificity (0.44-0.77) and low positive predictive values (PPV=0.11-0.51) with many false positives, probably due to the small item set. Hence, although the WSQ may be diagnostically sensitive and quick to complete, it does so at the expense of specificity when contrasted to a more comprehensive program such as the WB-DAT.

Within the psychometric literature of online diagnostic programs, test-retest reliability seems to be an important, yet underinvestigated, type of reliability given that numerous factors (eg, changes in test-taking attitudes and lack of control in test environment) could vary online performance and subsequent results between sittings [13]. Only one known study has examined the test-retest reliability of an online diagnostic assessment tool. In Lin et al’s study, participants comprising Taiwanese visitors to an online mental health website repeated the Internet-based Self-assessment Program for Depression (ISP-D), a 9- to 24-item measure of 3 different depressive presentations [11]. The ISP-D was found to have excellent test-retest reliability within 2 weeks (weighted κ=.80), although performance dropped over longer durations (eg, weighted κ=.45 for 2-4 weeks). Although Lin et al’s results are promising, it is unclear whether they can be generalized to programs targeting other disorders and with different population groups.

Given their practical benefits and psychometric evidence, Internet-based diagnostic assessments have been implemented and trialed in “virtual clinics” as a means of rapid assessment and referral to appropriate online interventions [10,14]. One example is the electronic Psychological Assessment and Screening System (e-PASS), which is the focus of this study. Appearing within the Anxiety Online virtual clinic [14] (now renamed as Mental Health Online [15]), the e-PASS predominantly functions as a diagnostic and referral tool for registered users and is the starting point for accessing online treatment programs [14]. For example, a user identified by the e-PASS as having panic disorder would be recommended to complete an online treatment program for panic disorder [14].

Unlike many other diagnostic assessment programs, the e-PASS aims to produce an accurate diagnostic result by incorporating items reflecting diagnostic criteria and severity. The e-PASS also assesses a considerably wider diagnostic breadth, including 21 DSM-IV (Text Revision; DSM-IV-TR) disorders, compared with most publically available programs, to help accommodate comorbid and lower prevalence disorders. Another distinct attribute of the e-PASS is that it distinguishes the primary diagnosis (ie, the disorder deemed of greatest severity in a presentation) from any secondary disorders. This feature helps users identify their main mental health issue and prioritize treatment recommendations. Finally, the e-PASS focuses on clinical disorders as well as “subclinical” presentations that represent significant symptoms, but do not meet full criteria and severity of a clinical disorder.

Preliminary evaluation has indicated high diagnostic agreement between the e-PASS and community sources (eg, psychologist, counselor, or medical doctor), although results were based on limited survey data [14]. The e-PASS has also undergone usability testing suggesting it offers distinct benefits and advantages (eg, convenience, anonymity, comprehensiveness) compared to a clinician-administered interview (D Nguyen, unpublished PhD thesis, Victoria: Swinburne University, 2013). In particular, the e-PASS has proven to be highly accessible with more than 22,620 completions between October 2009 and June 2014.

As with any diagnostic assessment tool, it is crucial to formally clarify the psychometric properties of the e-PASS. This need is particularly apparent given the e-PASS’ high usage and explicit role in diagnosis and treatment referral as well as outcome measurement in a “virtual” clinic (eg, [14]). Although psychometric evidence for several online assessment programs exist (eg, [9,10]), their findings are limited in reflecting the potential performance of the e-PASS. For example, the e-PASS differs from previously examined programs in terms of identifying a broader range of disorders (including less common disorders such as bulimia nervosa and body dysmorphic disorder) as well as subclinical diagnostic presentations.

Therefore, this study aimed to examine the diagnostic criterion validity and test-retest reliability of the e-PASS involving prospective users completing the e-PASS under relatively naturalistic conditions. This is the first study known to the authors to evaluate both the criterion validity and test-retest reliability of an online multidisorder diagnostic assessment program. This study is also distinct in examining an online diagnostic program that is central to an internationally available open-access “virtual” clinic for mental health disorders. The findings will help facilitate more informed and appropriate use of the e-PASS and further development of the e-PASS and similar online assessment tools.

The e-PASS is a free, internationally available, online diagnostic assessment (and referral) program for numerous mental disorders. As with any diagnostic tool, particularly one that is highly accessible and can be independently undertaken, there is a need to ensure the e-PASS is valid and reliable. Hence, this study evaluated the psychometric properties of the e-PASS, focusing on its diagnostic criterion validity and test-retest reliability. To enhance the ecological validity of the study findings, participants were recruited online and represented prospective e-PASS users completing the program under generally naturalistic conditions.

The e-PASS was found to have mixed diagnostic agreement with the semistructured clinical interview (ie, the gold standard), varying from fair (eg, OCD) to substantial (eg, panic disorder) agreement. Compared to previously evaluated programs, the e-PASS’ diagnostic sensitivity generally exceeded some (eg, Internet-administered CIDI-Short Form [12]), but not other programs (eg, WB-DAT [9], WSQ [10]). In contrast, the e-PASS’ specificity was generally high, resulting in far less false-positive results than certain programs (eg, WSQ [10]). Predictive statistics suggest that a positive e-PASS result had at least a 45% probability of accurately reflecting an actual disorder, whereas a negative e-PASS result for most disorders was correct in more than 90% of cases. The latter suggests a general strength of the e-PASS is its ability to rule out a disorder, which could be beneficial in minimizing burden associated with false-positive clinical diagnoses (eg, stigma, unnecessary follow-up assessment, and treatment).

Among previously reported programs, the e-PASS most closely resembles the WB-DAT [9]. When considering mutual disorders, the e-PASS produced similar psychometrics to the WB-DAT, except in the cases of OCD and PTSD, where the e-PASS clinical result was noticeably less sensitive. It is worth remembering that psychometric results of the WB-DAT [9] were based on a sample recruited from a face-to-face clinic population consisting of generally lower diagnostic base rates compared to those seen in this study. Furthermore, the e-PASS assesses a wider range of disorders than the WB-DAT and most other programs. To the best of the authors’ knowledge, this is the first study that has reported the psychometric performance of an online program that identifies BDD and bulimia nervosa.

Although the e-PASS screens particularly well for certain disorders (eg, panic disorder, MDD), it seems lacking for others (eg, OCD) when considering the combination of low sensitivity and diagnostic agreement with the clinical interview. Various factors could help explain these mixed classification statistics (eg, imprecise wording of some e-PASS items or unreliable diagnostic criteria for certain disorders). Given that e-PASS specificity often exceeded sensitivity values, one likely explanation is that the e-PASS’ diagnostic threshold was too high for particular disorders. In support of this, additional analyses found that sensitivity values consistently improved and exceeded 90% for some disorders (while maintaining reasonable specificity) when considering an e-PASS “subclinical” or “clinical” result as predictive of an actual clinical disorder. This suggests that the majority of actual clinical disorder cases at least received an e-PASS diagnosis of subclinical, if not clinical severity, which provides some reassurance in terms of notifying e-PASS users of potential mental health issues. Furthermore, the e-PASS is designed so that a subclinical result also prompts access to associated online treatment programs or recommendations of further assessment (eg, face-to-face consultation with a health professional) for follow-up.

Nevertheless, the results of this study suggest one way of improving the e-PASS’ screening properties in terms of maximizing sensitivity would be to reduce the diagnostic threshold (eg, so that a subclinical result is identified as a clinical disorder). However, this in turn would increase false-positive results, decreasing specificity. The extent to which diagnostic thresholds should be reduced will depend on the impact on the respective sensitivity and specificity properties, determined using receiver operating characteristic (ROC) analyses (D Nguyen, unpublished PhD thesis, Victoria: Swinburne University, 2013). A further consideration is the broader impact of accurate/inaccurate results (eg, potential burden of diagnosis including financial costs, stigma, and access of ineffective treatment) which further contributes to the overall utility of the e-PASS.

The e-PASS also demonstrates strong test-retest reliability for identifying a clinical disorder (particularly for panic disorder and bulimia nervosa) over an average of approximately 1 week and a maximum of 25 days. Compared to the ISP-D online screener for MDD [11], the e-PASS produced comparable consistency in identifying MDD. The results of this study are the first to document the test-retest reliability of an online diagnostic assessment program for the other reported disorders (eg, anxiety disorders, insomnia, bulimia nervosa). In general, the e-PASS’ test-retest reliability measures are comparable to those of a computer-assisted administration of the CIDI [31] and a clinician-administered MINI [32].

In this study, the few e-PASS cases with test-retest discrepancies were just as likely to reflect a diagnostic change from clinical to nonclinical compared with nonclinical to clinical. However, this result may have stemmed from underpowered statistical testing given the smaller than expected sample size. On closer inspection, test-retest discrepancies were generally subtle and tended to involve changes from clinical to subclinical results (and vice versa). This may have reflected actual symptom changes given the instability of certain disorders (eg, MDD) over the retesting period of up to 25 days after initial completion. Unfortunately, the reliability sample was too small to limit the analysis to those with shorter test-retest intervals (eg, 1 week). Overall, e-PASS results appear to be generally stable over the short term, which suggests that the potential variability of the online experience does not pose a significant risk to test-retest reliability.

Several limitations should be considered when interpreting the current findings. Firstly, insufficient clinical interviews were recorded to analyze interrater reliability. Also, the administration order of the e-PASS and clinical interview was not counterbalanced and participants’ viewing of e-PASS results in particular may have biased subsequent interview responses. The period between e-PASS and clinical interview completion (mean approximately 10 days) as well as between test and retest of the e-PASS (mean approximately 8 days) may have led to actual symptom changes in some cases. Therefore, the reported validity and reliability statistics could be conservative estimates. Furthermore, the limited number of participants repeating the e-PASS prompts the need for further reliability testing with a larger sample, while also possibly indicating that the e-PASS has low acceptability to some users. Indeed, separate research (D Nguyen, unpublished PhD thesis, Victoria: Swinburne University, 2013) has suggested that some of the e-PASS users were deterred from further use due to certain factors (eg, length, perceived repetition, lack of immediate assistance and support).

Participant recruitment targeted prospective e-PASS users to enhance the ecological validity of findings. Although not reported in this study, the sociodemographic characteristics (eg, gender, employment and marital status, education level) of the approximately 13,000 individuals who completed the e-PASS between October 2009 and October 2012 largely resemble those of this study sample. Nevertheless, the extent to which results based on this study’s sample can be generalized to all e-PASS users requires a more detailed analysis of participant characteristics as well as their potential relationship with psychometric properties. For example, it may be that certain individual characteristics (eg, education level) could be more conducive for e-PASS diagnostic validity or reliability.

With the introduction of DSM-5 [33], there is a need to revise the e-PASS in-line with new criteria and reevaluate its psychometric properties. Program changes will be minor for most disorder modules (eg, for MDD), although some will require substantial changes (eg, PTSD). Interestingly, the best performing e-PASS diagnoses (eg, MDD and panic disorder) are also those with relatively little criteria change from DSM-IV-TR to DSM-5. The e-PASS targets 21 disorders, but many of these (eg, anorexia nervosa, pathological gambling, substance disorders) were not examined due to very low diagnostic base rates in the sample. Therefore, further evaluation could involve specific population groups to clarify the e-PASS psychometric properties for these disorders. Additional psychometric evaluation could also consider properties such as the internal reliability of individual e-PASS items, although this would require a much larger sample size as well as modifications to the e-PASS form (eg, removing branching rules) to provide a suitable dataset for analysis.

New means of online diagnostic screening raises the issue of whether to replace, adapt, or supplement Internet-based programs such as the e-PASS. There is potential, for example, to incorporate audiovisual content (eg, [34]) that could enhance accessibility and acceptability. In light of its mixed diagnostic performance, Internet-based screening could also be followed up with clinician interviewing via videoconferencing (eg, [35,36]) or Web chat (eg, [37,38]). Online assessment could also be complemented with mobile-based applications measuring in-the-moment symptoms via questionnaires [39] or audiovisual cues (eg, speech and body language) of the respondent [40].

In contrast to diagnostic screeners, the use of online clinical scales focusing on dimensional measures may prove to offer greater utility in the assessment of mental health disorders [41]. Such programs extend beyond Internet administrations of standard paper-and-pencil measures and are becoming increasingly sophisticated. For example, Batterham et al [42] proposed a hierarchical system commencing with brief online prescreening (eg, K6) followed by an administration of relevant disorder-specific scales. Computer adaptive testing based on item response theory also shows promise in terms of efficiently screening latent traits underlying mental disorders (eg, [43,44]).

In the meantime, given the utility of a diagnosis in clinical practice [1], there is still arguable value in offering Internet-based questionnaires that produce diagnostic results and directly query diagnostic criteria as similar to the approach of gold standard structured clinical interview schedules [5]. As this study shows, an Internet-based diagnostic assessment program can produce diagnostic results that have high test-retest reliability and, at least for certain disorders, high criterion validity. Despite their potential psychometric limitations, these programs could be incorporated into traditional clinical practice alongside other imperfect assessment means (eg, unstructured interviewing) to broaden assessment information and improve overall diagnostic accuracy [3,5]. For many consumers who are unable or unwilling to access traditional services, Internet-based programs could offer a “good enough” alternative for identifying mental health disorders.

In conclusion, this study suggests that the e-PASS has potential for assisting in the diagnosis of mental health disorders and, in doing so, facilitating access to appropriate interventions among other benefits of identifying mental disorders. Nevertheless, further development and evaluation is needed to clarify the full scope of its clinical utility.