This paper is in the following e-collection/theme issue:

Quality/Credibility of eHealth Information and Trust Issues 

Advertisement: Preregister now for the Medicine 2.0 Congress

Article Thumbnail
Members can download this full article for Adobe PDF Format.
Membership provides unlimited access to all PDF files.
Or, ask your department head to become an institutional member.

For tax purposes please select your country and if applicable state/province of residence:
Buy Now (Pay-per-download for non-members):
Download Price (USD): $22.00

Original Paper

An Evaluation of Web-Based Clinical Practice Guidelines for Managing Problems Associated with Cannabis Use

Melissa M Norberg1, BA(Hons), MS, PhD; Michael W Turner2, MBChB; Sally E Rooke1, BA(Hons), PhD; Julia M Langton3, BPsych (Hons), MPsych(Clinical), Ph.D; Peter J Gates1, B.Sc

1University of New South Wales, National Cannabis Prevention and Information Centre, Randwick, Australia
2National Health Service, Aberdeen, United Kingdom
3University of Sydney, Faculty of Pharmacy, Sydney, Australia

Corresponding Author:
Melissa M Norberg, BA(Hons), MS, PhD

University of New South Wales
National Cannabis Prevention and Information Centre
PO Box 684
Randwick, 2031
Phone: 61 293850260
Fax: 61 267730201


Background: Cannabis is the most widely used illicit substance, and multiple treatment options and avenues exist for managing its use. There has been an increase in the development of clinical practice guidelines (CPGs) to improve standards of care in this area, many of which are disseminated online. However, little is known about the quality and accessibility of these online CPGs.
Objective: The purpose of study 1 was to determine the extent to which cannabis-related CPGs disseminated online adhere to established methodological standards. The purpose of study 2 was to determine if treatment providers are familiar with these guidelines and to assess their perceived quality of these guidelines.
Methods: Study 1 involved a systematic search using the Google Scholar search engine and the National Drugs Sector Information Service (NDSIS) website of the Alcohol and Other Drugs Council of Australia (ADCA) to identify CPGs disseminated online. To be included in the current study, CPGs needed to be free of charge and provide guidance on psychological interventions for reducing cannabis use. Four trained reviewers independently assessed the quality of the 7 identified guidelines using the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool. Study 2 assessed 166 Australian cannabis-use treatment providers’ (mean age = 45.47 years, SD 12.14) familiarity with and opinions of these 7 guidelines using an online survey. Treatment providers were recruited using online advertisements that directed volunteers to a link to complete the survey, which was posted online for 6 months (January to June 2012). Primary study outcomes included quality scores and rates of guideline familiarity, guideline use, and discovery methods.
Results: Based on the AGREE II, the quality of CPGs varied considerably. Across different reporting domains, adherence to methodological standards ranged from 0% to 92%. Quality was lowest in the domains of rigor of development (50%), applicability (46%), and editorial independence (30%). Although examination of AGREE II domain scores demonstrated that the quality of the 7 guidelines could be divided into 3 categories (high quality, acceptable to low quality, and very low quality), review of treatment providers’ quality perceptions indicated all guidelines fell into 1 category (acceptable quality). Based on treatment providers’ familiarity with and usage rates of the CPGs, a combination of peer/colleagues, senior professionals, workshops, and Internet dissemination was deemed to be most effective for promoting cannabis use CPGs. Lack of time, guideline length, conflicts with theoretical orientation, and prior content knowledge were identified as barriers to guideline uptake.
Conclusions: Developers of CPGs should improve their reporting of development processes, conflicts of interest, and CPGs’ applicability to practice, while remaining cognizant that long guidelines may deter implementation. Treatment providers need to be aware that the quality of cannabis-related CPGs varies substantially.

(J Med Internet Res 2012;14(6):e169)


Cannabis; Marijuana Abuse; Addiction; Psychotherapy; Standards; Information Dissemination; Health Plan Implementation; Internet


Clinical practice guidelines (CPGs) can facilitate appropriate clinical decision making and improve standards of care [1], but their effectiveness relies upon their quality [2]. Unfortunately, early research found that most CPGs published in the peer-reviewed medical literature, including those guidelines developed by specialty societies, were of poor quality [3,4]. Flawed guidelines may provide inaccurate scientific and clinical advice to treatment providers, thereby harming patients because of suboptimal or ineffective treatment delivery [2]. Therefore, the benefits of CPGs are contingent upon their development process. Although efforts have been made to improve the quality of peer-reviewed CPGs [5-7], many continue to lack rigorous development, editorial independence, and applicability to practice, or at least fail to adequately report on these issues [8-10]. The poor quality of CPGs may be perpetuated by the growing trend of publishing CPGs online that do not require peer review or the documentation of a systematic literature review unless they are indexed in a CPG database.

Although Internet dissemination is purported to increase accessibility by making CPGs freely available and by reducing publication delays associated with peer-reviewed journal submission, it may not have a corresponding effect on implementation [11,12]. A systematic review of dissemination strategies found that passive dissemination of educational material alone was not very effective for improving professional practice, but that the impact of educational material was enhanced when it was delivered through interactive educational meetings [13]. Other reviews have found that incorporating a combination of different activities is usually the most effective approach for getting health practitioners to change their behavior [14,15]. Based on these findings, the success of CPGs is not only dependent upon their quality, but it may also depend on which or how many dissemination strategies are used.

Treatment providers should choose which CPG to adopt based on a rigorous review process. Unfortunately, this may not be possible for those who do not have the training and/or time to scrutinize the methods by which guidelines were developed. Given that cannabis is the most frequently used illicit substance [16] and that multiple treatment options (eg, motivational enhancement therapy, cognitive behavior therapy, and family therapy) [17] and avenues (eg, inpatient, outpatient, rehabilitation, and primary care) [18] exist for managing cannabis use, there is a clear need for qualified individuals to evaluate the quality of cannabis-related CPGs, especially those disseminated online.

The purpose of study 1 was to demonstrate the extent to which cannabis-related CPGs adhere to established methodological standards using the validated Appraisal of Guidelines for Research and Evaluation (AGREE II) tool [19]. Its predecessor, the AGREE, is the most promising critical appraisal tool for CPGs and has been used to evaluate numerous CPGs across a variety of health issues [20]. The AGREE II was developed to improve upon the AGREE’s reliability and usability. The purpose of study 2 was to determine treatment providers’ familiarity with and views about the CPGs identified in study 1. Study findings will assist treatment providers in identifying high quality CPGs and assist guideline developers in improving their reporting and dissemination practices.


Study 1

Selection Criteria

To be included in the current study, CPGs needed to provide guidance on psychological interventions for reducing cannabis use. Interventions needed to target cannabis use broadly rather than one specific facet of reducing/ceasing use (eg, withdrawal). In addition, CPGs needed to be developed for professionals whose primary role is to provide counseling (ie, psychologists or counselors) and be available free of charge via the Internet. Further, CPGs were included only when the word cannabis (or a similar term) was in the title or in the table of contents (or similar summary list). Guidelines that primarily targeted professionals whose secondary role may include counseling (eg, nurses or general practitioners) or guidelines that were published in any language other than English were excluded. Client population was not an exclusion criteria.

Search Strategy

Two authors (MMN and SER) independently conducted a search of cannabis treatment guidelines using the Google Scholar search engine and the National Drugs Sector Information Service (NDSIS) website of the Alcohol and Other Drugs Council of Australia (ADCA), a service that provides direct and indirect access to guidelines through links [21]. The NDSIS contained 11 links to websites containing guidelines, but did not have a search facility. All websites accessed via links on this site were searched using the terms marijuana, cannabis, and guidelines if they had a search facility, otherwise links were followed to appropriate guidelines. Four websites contained eligible guidelines or contained links to other websites that had eligible guidelines: the Medical Observer [22], the National Guideline Clearinghouse [23], the National Institute for Health and Clinical Excellence website [24], and the Trip database [25]. The Medical Observer, the National Guidelines Clearinghouse, and the National Institute for Health and Clinical Excellence websites each contained 1 eligible CPG, whereas the Trip database contained 2 eligible CPGs for this study. In addition, the National Guideline Clearinghouse had a page with links to complementary websites. These links were explored and led to the discovery of 1 eligible guideline via the Guidelines International Network [26]. In total, 6 CPGs were sourced via NDSIS links to other websites following these methods. Next, Google Scholar was searched using the terms guidelines, cannabis, and marijuana. This returned 1290 hits, with titles revealing 6 potentially eligible CPGs. Only 1 of the 6 CPGs was eligible. Both authors agreed on the eligibility/ineligibility of each CPG (see Figure 1).

[view this figure]
Figure 1. Selection of Web-based guidelines.

Quality Assessment

All 5 authors read the AGREE II manual and completed the online AGREE II training [19,27]. One author (MMN) selected 4 non-cannabis-related guidelines for pilot testing. Pilot testing allowed all authors to become familiar with the AGREE II assessment procedure and clarify any discrepancies in subjective quality ratings. The AGREE II appraisals were completed independently by 4 authors (the reviewers MWT, SER, JML, and PJG) and then reviewed together with MMN who served as an intermediary. After all reviewers agreed upon a final score for all items for all pilot CPGs, the 4 reviewers independently assessed each of the 7 eligible CPGs using the AGREE II. After review by the intermediary, reviewers were asked to re-evaluate items for which their scores differed by more than 2 points from the other reviewers. The intermediary provided no indication if scores were higher or lower than the other reviewers’ scores and did not require reviewers to change their scores.

Agree II

The AGREE II instrument consists of 23 items across 6 domains: (1) scope and purpose, (2) stakeholder involvement, (3) rigor of development, (4) clarity of presentation, (5) applicability, and (6) editorial independence [19]. A 24th item (overall guideline assessment) assesses a reviewer’s overall impression of a guideline. All items are rated on a 7-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Domain scores range from 0% to 100% and enable scores to be compared across different domains because domains vary in the number of items they contain. Research on the AGREE II demonstrates that its domain scores have acceptable to good internal consistency (alpha range .64 to .89) and interrater reliability (alpha range .63 to .84) when 4 reviewers are used [28]. Importantly, the AGREE II is able to differentiate content designed to be of high and low quality [29]. For the purposes of this study, a domain cut-off score of 65% was used to determine guidelines of moderate to high quality. A cut-off score of 65% was chosen based on prior research identifying that guidelines approved by the Australian Government’s National Health and Medical Research Council typically achieve domain scores above 65% [30]. The AGREE II manual advises that domain scores should be interpreted within the context of a project; therefore, quality judgments were based also on inspection of an error bar graph.

Statistical Analysis

First, mean AGREE II item scores were calculated using the 4 reviewers’ item scores. Next, domain percentage scores were derived by summing all mean individual AGREE II item scores and standardizing the total as a percentage of the maximum possible score for that domain. Finally, the interrater reliability of the 4 reviewers was assessed using intraclass correlations (ICCs) for each guideline. A 2-way random model for absolute agreement was used. The ICCs were computed before and after arbitration by the intermediary.

Study 2


A convenience sample of health care professionals were recruited via Google advertisements restricted to Australia, as well as advertisements on the websites, newsletters, and email list servers of organizations whose members commonly provide substance use counseling. Advertisements specified that we were conducting a 30-minute online survey about cannabis use guidelines and that we were seeking health professionals who were involved in counseling individuals for cannabis use. The advertisements contained a hyperlink to the study information form. The information form specified that the purpose of the survey was to ascertain health professionals’ familiarity with guidelines for managing cannabis use and their opinions of them. The form also specified that if participants entered their email address at the end of the survey they would be entered into a draw to win 1 of 10 Aus $100 prizes via PayPal. Before initiating the online survey, participants were required to provide consent by clicking on the following option: “Yes, I have read the information and consent form and I am ready to participate.” Both the information form and survey were stored on the University of New South Wales website using Key Survey Enterprise. At the start of survey, individuals were asked whether or not they had counseled someone for cannabis use in the past 12 months. Only individuals who answered positively were allowed to continue. After 6 months of data collection (January to June 2012), the survey was closed and 10 participants were randomly chosen for the $100 prizes using a random number generator. At this point, data were downloaded from Key Survey Enterprise and stored in an intranet folder that only University of New South Wales study authors can access. Study 2 was approved by the Medical and Community Human Research Ethics Advisory Panel at the University of New South Wales.


The open survey contained demographic items that assessed participants’ gender, age, country of birth, educational background, profession, and practice area. Additional items assessed experience in counseling individuals who use cannabis and information sources participants used to inform their provision of cannabis use treatment. The remainder of the survey asked specific questions about the 7 CPGs identified during study 1 and used branch logic questioning in order to only ask questions which were relevant to the participant. For example, if a participant was not familiar with a particular CPG, no questions were asked about that guideline and the survey moved on to the next CPG. When a participant reported being familiar with a CPG, they were asked if they had read it and, if so, if they had used it. If participants had not read a CPG after encountering it, they were asked to report if it was because they were too busy, the CPG was too long, the CPG contained things they already knew, or if it was because the CPG conflicted with their theoretical orientation. When participants had read a CPG they were asked to rate the CPG on 9 Likert-scale items from 1 (strongly disagree) to 5 (strongly agree), where 3 represented neutral. Items were presented in the same order for each participant and were mandatory in that participants could not progress unless an item was answered. In addition, participants could only review items for the currently displayed items (ie, there was no “back” button). The usability and functioning of the survey was tested by the first and second authors before its public launch.


Study 1

Clinical Practice Guideline Characteristics

Seven eligible guidelines met inclusion criteria (see Table 1). Two guidelines were specific to cannabis use: Management of Cannabis Use Disorder and Related Issues: A Clinician’s Guide published by the National Cannabis Prevention and Information Centre (NCPIC) [31] and Clinical Practice Guidelines for Management of Cannabis Dependence published by the Indian Psychiatric Society (IPS) [32]. The other CPGs, published by the National Institute for Health and Clinical Excellence (NICE), the American Psychiatric Association (APA), the New South Wales Department of Health, and New Zealand’s National Health Committee (NHC), were related to substance use treatment in general and only included sections specifically related to cannabis use. The New South Wales Department of Health published two guidelines: (1) Drug and Alcohol Psychosocial Interventions Professional Practice Guidelines [33] hereafter referred to as NSWD (New South Wales-Drug), and (2) National Clinical Guidelines for the Management of Drug Use During Pregnancy, Birth and the Early Development Years of the Newborn [34] hereafter referred to as NSWP (New South Wales-Pregnancy). The guidelines were developed in 5 different countries. Guideline length ranged from 12 to 338 pages.

[view this table]
Table 1. Clinical practice guidelines characteristics.

Interrater Reliability

Before arbitration, the ICCs ranged from 0.80 to 0.94. After the intermediary notified reviewers of scores that differed by more than 2 points from other reviewers, 50 of 672 item scores (7.4%) were changed. These changes led to ICCs between 0.89 and 0.96, demonstrating high interrater reliability.

Agree II

The 3 items that scored the lowest assessed whether the views of the target population had been sought, if the CPG had been externally reviewed by experts, and if the competing interests of the CPG development group had been recorded and addressed (see Table 2). The 3 highest scores were from the scope and purpose domain. These items assessed the quality of descriptions for the overall objective of the CPG, the health question(s) covered by the CPG, and the population for whom the CPG was intended. Across CPGs, average domain scores were 65% or greater in 2 instances: (1) scope and purpose and (2) clarity of presentation. The most variable domain scores across CPGs were for rigor of development (9% to 89%), followed by editorial independence (0% to 77%). These 2 domains also received the lowest mean domain scores, along with applicability.

[view this table]
Table 2. AGREE II mean item scores and domain percentage scores for each guideline.

The CPGs showed great variability across the domains, ranging from 0% to 92% adherence (Figure 2). The NICE guideline was the most consistent performing CPG and the only CPG to score above 65% in all domains (indicating high quality). This CPG also received the highest scores in all domains, except for clarity of presentation, which was obtained by the NCPIC guideline. The IPS guideline did not achieve over 65% on any domain and achieved the lowest scores across all domains compared to other CPGs (indicating very low quality). Guidelines other than the NICE and the IPS guidelines achieved average domain scores between 46% and 64% (indicating acceptable to low quality).

[view this figure]
Figure 2. Mean AGREE II domain scores for the 7 eligible clinical practice guidelines. NCPIC: National Cannabis Prevention and Information Centre; IPS: Indian Psychiatric Society; NICE: National Institute for Health and Clinical Excellence; APA: American Psychiatric Association; NSWD: New South Wales-Drug; NSWP: New South Wales-Pregnancy; and NHC: National Health Committee.

Study 2

Participant Characteristics

A total of 190 individuals provided informed consent to participate in the study; however, only 166 of these individuals reported that they had counseled someone for cannabis use in the past 12 months, meeting the criteria to complete the survey. The average age of these 166 individuals was 45.47 years (SD 12.14). Of these, 64.5% (107/166) were female and 77.7% (129/166) were born in Australia. The next most common country of origin was England (7.2%, 12/166). Educational and professional details for the participants are presented in Table 3. Most participants had received some type of tertiary education, worked for an organization, practiced in a metropolitan area, were a drug and alcohol worker/specialist, and had over 3 years’ experience counseling individuals who use cannabis (Table 3). Slightly more of the study sample reported being from regional or remote areas compared to the entire Australian population [38]. Guidelines, websites, colleagues, and workshops were the most frequently identified sources for informing participants’ treatment of cannabis use.

[view this table]
Table 3. Educational and professional characteristics of the participants (N = 166).

Familiarity with Clinical Practice Guidelines

Participants were most familiar with the NCPIC, NSWD, and NICE guidelines; however, less than half of the study participants (44.6%, 74/166) were aware of the NCPIC guideline, the most frequently encountered CPG (Table 4). Peers/colleagues and websites were the top reasons cited for knowing about all 3 of these CPGs. Professional development workshops also were key in disseminating the NCPIC guideline, as were journal articles for the NICE guideline, and senior professionals for the NSWD guideline. The 3 least frequently encountered guidelines (APA, NSWP, and NHC) also were often discovered via these routes with the addition of education programs.

In most cases, approximately three-fourths of individuals had read the CPGs they had encountered; however, over 90% of individuals had read the NSWD and NSWP guidelines after they heard about them. These 2 CPGs were encountered through senior professionals more so than the other guidelines. The most common reasons acknowledged for not reading a CPG was being too busy (40% to 83% of respondents for all CPGs), believing a CPG was at conflict with one’s theoretical orientation (0% to 50% for all CPGs), believing a CPG was too long (0% to 40% for all CPGs), and already knowing the content (0% to 33% for all CPGs). In many cases, after a participant had read a CPG they were likely to use it in their practice. This was most often the case for the NCPIC and APA guidelines. Both of these CPGs were encountered through workshops more so than any other guideline.

Examination of mean scores demonstrates that participants tended to agree or feel neutral toward the quality statements about the selected CPGs (Table 5). Thus, little variability in item scores existed between the CPGs. For 6 of the 9 items, participants scored the NSWP guideline slightly higher. In regards to the 3 other items, participants reported that the NSWD guideline was the easiest to follow, the NCPIC guideline was the most clearly presented, and the NCPIC and NSWD guidelines were the most applicable to their practice. On average, the NHC guideline scored slightly lower across 6 items. In 2 cases, the NICE guideline was rated the lowest. Participants were less supportive of routinely using the NICE guideline and in less agreement that they were easy to follow compared to the other CPGs. Lastly, participants reported that the NCPIC guideline was based on patient preferences the least.

[view this table]
Table 4. Familiarity with clinical practice guidelines (CPGs).

[view this table]
Table 5. Treatment providers’ opinions regarding the clinical practice guidelines.


Publishing CPGs online is intended to facilitate the dissemination of evidence-based treatment information; however, the provision of these resources through the Internet alone does not guarantee practitioner uptake [13-15]. This caveat combined with the lack of quality control associated with online resources necessitated an evaluation of treatment providers’ access to and options of CPGs for managing cannabis use, as well as an evaluation of their scientific soundness. Utilizing 4 trained CPG reviewers, study 1 found a high amount of variability between CPGs for managing cannabis use, whereas treatment providers in study 2 reported much less variability between the CPGs. In addition, these studies identified areas of improvement for guideline developers and a potential dissemination combination that may lead to greater CPG familiarity and implementation.

Clinical Practice Guideline Quality

Based on AGREE II domain scores and examination of an error bar graph, the CPGs fell into 3 broad categories: high quality, acceptable to low quality, and very low quality. The NICE guidelines had the highest overall quality; the NCPIC, APA, NSWD, NHC, and NSWP guidelines were deemed acceptable to low quality; and the IPS guidelines were rated as needing substantial improvement. Inspection of domain scores demonstrated that consistent with prior research, CPGs performed the poorest in the areas of rigor of development, applicability, and editorial independence [8-10]. As such, CPG developers can enhance the quality of revisions to these CPGs by substantially improving reporting in these 3 areas.

Interestingly, findings from studies 1 and 2 were somewhat incongruent. In general, treatment providers assessed during study 2 reported substantially less variability in CPG quality as compared with the reviewers from study 1. All 9 areas of interest examined during study 2 received average scores. Although the clinical significance between mean item scores is likely low, examination of these scores further highlights the quality discrepancies reported by treatment providers and trained reviewers. For example, 3 of the 9 items assessed during study 2 are directly comparable to AGREE II domain scores (rigor of development, clarity of presentation, and applicability). In all of these areas, the trained reviewers rated the IPS guideline the lowest, whereas the treatment providers rated the NHC guideline the lowest. Treatment providers rated the IPS guideline second- to fourth-best in these areas. Examination of study 2 items that were similar to AGREE II items (defining target users, seeking patient preferences, and overall quality) demonstrated a similar discrepancy. In all cases, the IPS guideline was rated the lowest by the trained reviewers, whereas treatment providers reported that the NHC guideline was the poorest in terms of clarity about who should use the guideline and overall quality. Treatment providers also reported that the NCPIC guideline was based less on patient preferences than the other guidelines. The discrepancies in ratings between reviewers and treatment providers indicate that treatment providers may not be able to differentiate between good and poor CPG reporting quality. This finding is important because it may suggest that treatment providers are at risk for adopting CPGs that are not based on the best available evidence.

Although validity testing has demonstrated that the AGREE II is able to differentiate higher quality reporting from lower quality reporting [29], it is possible that well-reported CPGs contain flawed recommendations and poorly reported CPGs contain sound recommendations. A comparison of reports and research protocols for randomized controlled trials conducted by the Radiation Therapy Oncology Group showed that the methodological quality of studies was often substantially better than that reported [39]. For example, allocation concealment and sample size calculations were only reported in 42% and 16% of the reports, but reported in 100% and 76% of the research protocols. If these results are transferable to CPGs, it may mean that the scientific soundness of the poorest performing CPGs according to AGREE II ratings may be substantially better than thought. It is important to note, however, that all research protocols produced by the Radiation Therapy Oncology Group must pass through a rigorous peer review process and be approved through its own committee system and the National Cancer Institute before a randomized controlled trial can progress. Thus, CPGs that do not undergo peer review during development or afterwards may not be better than what AGREE II scores suggest.

Dissemination and Impact

Study 2 demonstrates that the effectiveness of current CPG dissemination methods is suboptimal. Slightly less than half of the study population was familiar with the NCPIC guideline, the most well-known guideline, whereas only a third of the sample had heard of the second most commonly identified guideline (NSWD). Prior research has found that 59% to 98% of substance abuse treatment providers are familiar with motivational and cognitive-behavioral treatment approaches [40-42]. Because these approaches have the most evidence base for treating cannabis use, unless CPGs reach the 49% of providers who do not have preexisting knowledge about these evidence-based approaches, current CPG access rates may not lead to an increase in standards of care. Examination of potential differences in dissemination efforts between the NCPIC guidelines and the other 6 CPGs may suggest methods for improvement because the NCPIC guideline was the most commonly encountered.

Peers/colleagues, websites, and workshops were the most common methods reported for discovery of the NCPIC guidelines. As peers/colleagues and websites were common methods for discovering all identified CPGs, the addition of workshop dissemination may lead to increased CPG familiarity. Previous research suggests that greater adoption of workshop materials is facilitated by greater relevancy of training (eg, information obtained is relevant to the needs of participants’ clients) and greater program support (eg, having enough time to implement the materials) [43]. As such, dissemination via workshops may only increase cannabis-related CPG uptake for participants who regularly provide psychosocial treatment for cannabis treatment and who have the time and support of their organization for CPG implementation. In support of this assumption, the NSWD and NSWP guidelines were the most frequently read once encountered, and were heard of more often via senior professionals than the other guidelines. CPG access through senior professionals may be a proxy indicator of organizational support. In addition, the NCPIC and APA guidelines were the most frequently used once read, and these CPGs were accessed via workshops more than the other guidelines. Based on these combined findings, successful dissemination and implementation may be facilitated by the combination of peers/colleagues, senior professionals, workshops, and websites.

To increase the uptake of guideline usage, CPGs should further take into account the needs of the treatment provider. Consistent with previous research on evidence-based treatment and CPG adoption [29,40,43], barriers commonly reported during study 2 for not reading a CPG included a lack of time, CPG length, conflict with theoretical orientation, and already being familiar with the content. Only a lack of time and CPG length can be addressed by guideline developers. Accordingly, finding ways to make CPGs more time efficient while not compromising on quality should be a primary objective of future guideline development. A method for achieving this goal may be for CPG developers to publish two documents online. One document could focus on the content of practice, whereas the other could be reserved for reporting development processes and conflicts of interest.

Strengths and Limitations

The current research has several strengths, including a systematic evaluation of CPGs conducted by multiple reviewers with a high level of consistency among the reviewers. Additionally, the evaluation was conducted using a psychometrically robust assessment tool. Finally, the systematic evaluation study was followed with data from users of the guidelines, enabling us to examine gaps between quality and real-world perceptions of the guidelines. These strengths must be considered in light of study limitations. First, the AGREE II instrument provided an indication of guideline reporting quality, rather than a direct indication of the appropriateness of the recommendations. Previous research cautions that cannabis use information for patients available online is not of a high standard [44]. Therefore, an important follow-up study would involve a content assessment of the identified CPGs. Second, study 2 did not provide a direct comparison of guidelines because this would have required every treatment provider to read and evaluate each CPG. This was not feasible, especially since this study identified that treatment providers did not have enough time to read the CPGs that they had heard of. Third, the type of data obtained through studies 1 and 2 prevented the use of inferential analyses for examining associations between the studies’ findings (eg, 4 data points for each AGREE II item because of 4 reviewers and a highly varied sample size for treatment provider opinions based on differential CPG familiarity rates). Lastly, the health provider survey we developed for this study utilized a 5-point response format. This response scale limited the variability of providers’ ratings and also prevented direct comparison with the AGREE II.


This research provided the first evaluation of online CPGs that address psychosocial treatments for reducing cannabis use. The findings provide an indication of the reporting quality of CPGs that are freely available to treatment providers, and highlight gaps between the quality of CPGs as assessed by a psychometrically validated assessment tool and treatment provider perceptions. The research also suggests possible methods for increasing the uptake of CPGs among treatment providers.


The authors gratefully acknowledge the Department of Health and Ageing of the Australian Government for providing funding for the National Cannabis Prevention and Information Centre and, subsequently, the present study. The Department of Health and Ageing had no role in study design, in data collection or analysis, or in the decision to submit the study for publication.

Conflicts of Interest

MM Norberg, SE Rooke, and PJ Gates are employed by the National Cannabis Prevention and Information Centre, a publisher of one of the guidelines reviewed in this paper. None of these authors were an author of that guideline.


  1. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993 Nov 27;342(8883):1317-1322. [Medline]
  2. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999 Feb 20;318(7182):527-530 [FREE Full text] [Medline]
  3. Grilli R, Magrini N, Penna A, Mura G, Liberati A. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000 Jan 8;355(9198):103-106. [CrossRef] [Medline]
  4. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999 May 26;281(20):1900-1905. [Medline]
  5. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer 2003 Aug;89 Suppl 1:S4-S8 [FREE Full text] [CrossRef] [Medline]
  6. Ricci S, Celani MG, Righetti E. Development of clinical guidelines: methodological and practical issues. Neurol Sci 2006 Jun;27 Suppl 3:S228-S230. [CrossRef] [Medline]
  7. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J, Deshpande AM. Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003 Sep 16;139(6):493-498. [Medline]
  8. Hegarty K, Gunn J, Blashki G, Griffiths F, Dowell T, Kendrick T. How could depression guidelines be made more relevant and applicable to primary care? A quantitative and qualitative review of national guidelines. Br J Gen Pract 2009 May;59(562):e149-e156 [FREE Full text] [CrossRef] [Medline]
  9. Santos F, Sola I, Rigau D, Arevalo-Rodriguez I, Seron P, Alonso-Coello P, et al. Quality assessment of clinical practice guidelines for the prescription of antidepressant drugs during pregnancy. Curr Clin Pharmacol 2012 Jan 25;7(1):7-14. [Medline]
  10. Tong A, Chapman JR, Wong G, de Bruijn J, Craig JC. Screening and follow-up of living kidney donors: a systematic review of clinical practice guidelines. Transplantation 2011 Nov 15;92(9):962-972. [CrossRef] [Medline]
  11. Fervers B, Carretier J, Bataillard A. Clinical practice guidelines. J Visc Surg 2010 Dec;147(6):e341-e349. [CrossRef] [Medline]
  12. Jeannot JG, Scherer F, Pittet V, Burnand B, Vader JP. Use of the World Wide Web to implement clinical practice guidelines: a feasibility study. J Med Internet Res 2003;5(2):e12 [FREE Full text] [CrossRef] [Medline]
  13. Bywood P, Lunnay B, Roche AM. The National Centre for Education and Training on Addiction (NCETA). Adelaide, Australia: National Centre for Education and Training on Addiction; 2008. Effective dissemination: A systematic review of implementation strategies for the AOD field   URL: http:/​/pandora.​​pan/​88746/​20080915-1622/​​documents/​NCETAReportFINALweb_optim.​pdf [accessed 2012-11-25] [WebCite Cache]
  14. Wensing M, van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract 1998 Feb;48(427):991-997 [FREE Full text] [Medline]
  15. Oxman AD, Thomson MA, Davis DA, Haynes RB. No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ 1995 Nov 15;153(10):1423-1431 [FREE Full text] [Medline]
  16. United Nations Office on Drugs and Crime. United Nations Office on Drugs and Crime (UNODC). Vienna, Austria: United Nations; 2011. World Drug Report 2011   URL: [accessed 2012-11-25] [WebCite Cache]
  17. Denis C, Lavie E, Fatséas M, Auriacombe M. Psychotherapeutic interventions for cannabis abuse and/or dependence in outpatient settings. Cochrane Database Syst Rev 2006(3):CD005336. [CrossRef] [Medline]
  18. Stinson FS, Ruan WJ, Pickering R, Grant BF. Cannabis use disorders in the USA: prevalence, correlates and co-morbidity. Psychol Med 2006 Oct;36(10):1447-1460. [CrossRef] [Medline]
  19. The AGREE Next Steps Consortium. AGREE Enterprise website.: The AGREE Research Trust; 2009. Appraisal of Guidelines for Research and Evaluation II: AGREE II Instrument   URL: [accessed 2012-11-25] [WebCite Cache]
  20. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005 Jun;17(3):235-242 [FREE Full text] [CrossRef] [Medline]
  21. National Drugs Sector Information Service (NDSIS). 2012. AOD Guidelines and Protocols   URL: [accessed 2012-02-29] [WebCite Cache]
  22. Medical Observer. Guide to Guidelines   URL: [accessed 2012-08-13] [WebCite Cache]
  23. Agency for Healthcare Research and Quality (AHRQ). US Department of Health and Human Services. 2012. National Guideline Clearinghouse   URL: [accessed 2012-03-02] [WebCite Cache]
  24. National Institute for Health and Clinical Excellence. Clinical Guidelines   URL: [accessed 2012-03-02] [WebCite Cache]
  25. Trip. Trip Database   URL: [accessed 2012-08-13] [WebCite Cache]
  26. Guidelines International Network. 2012. International Guidelines Database   URL: [accessed 2012-03-02] [WebCite Cache]
  27. AGREE II. 2012. Practice Exercise Overview   URL: [accessed 2012-07-17] [WebCite Cache]
  28. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, AGREE Next Steps Consortium. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010 Jul 13;182(10):1045-1052 [FREE Full text] [CrossRef] [Medline]
  29. Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, AGREE Next Steps Consortium. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010 Jul 13;182(10):E472-E478 [FREE Full text] [CrossRef] [Medline]
  30. Vitry AI, Zhang Y. Quality of Australian clinical guidelines and relevance to the care of older people with multiple comorbid conditions. Med J Aust 2008 Oct 6;189(7):360-365. [Medline]
  31. Copeland J, Frewen A, Elkins K. National Cannabis Prevention and Information Centre. Sydney, Australia: National Cannabis Prevention and Information Centre; 2009. Management of cannabis use disorder and related issues: A clinician's guide   URL: http:/​/ncpic.​​ncpic/​news/​ncpic-news/​pdf/​management-of-cannabis-use-disorder-and-related-issues-a-clinicians-guide [accessed 2012-11-25] [WebCite Cache]
  32. Gautam S, Gupta ID, Khan A. IPS Clinical Practice Guidelines.: Indian Journal of Psychiatry; 2006. Clinical Practice Guidelines for the Management of Cannabis Dependence   URL: [accessed 2012-07-19] [WebCite Cache]
  33. Mental Health and Drug and Alcohol Office. NSW Health. North Sydney, Australia: New South Wales Department of Health; 2008. Drug and Alcohol Psychosocial Interventions Professional Practice Guidelines   URL: [accessed 2012-11-25] [WebCite Cache]
  34. Ministerial Council on Drug Strategy under the Cost Shared Funding Model. National Clinical Guidelines for the Management of Drug Use During Pregnancy, Birth and the Early Development Years of the Newborn. North Sydney, Australia: NSW Department of Health; 2006.   URL: [accessed 2012-11-25] [WebCite Cache]
  35. National Collaborating Centre for Mental Health. National Institute for Health and Clinical Excellence. London: National Institute for Health and Clinical Excellence (NICE); 2007. Drug Misuse: Psychosocial interventions: NICE guideline   URL: [accessed 2012-11-25] [WebCite Cache]
  36. Work Group on Substance Use Disorders. Psychiatry Online.: American Psychiatric Association; 2006. Practice guideline for the treatment of patients with substance use disorders   URL: [WebCite Cache]
  37. National Health Committee. Wellington: National Health Committee; 1999 Jul. Guidelines for Recognising, Assessing and Treating Alcohol and Cannabis Abuse in Primary Care   URL: [accessed 2012-12-06] [WebCite Cache]
  38. Australian Bureau of Statistics. 2008. Australian Social Trends, 2008   URL: [accessed 2012-07-25] [WebCite Cache]
  39. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Radiation Therapy Oncology Group. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004 Jan 3;328(7430):22-24 [FREE Full text] [CrossRef] [Medline]
  40. Benishek LA, Kirby KC, Dugosh KL, Padovano A. Beliefs about the empirical support of drug abuse treatment interventions: a survey of outpatient treatment providers. Drug Alcohol Depend 2010 Mar 1;107(2-3):202-208. [CrossRef] [Medline]
  41. Herbeck DM, Hser YI, Teruya C. Empirically supported substance abuse treatment approaches: a survey of treatment providers' perspectives and practices. Addict Behav 2008 May;33(5):699-712 [FREE Full text] [CrossRef] [Medline]
  42. McGovern MP, Fox TS, Xie H, Drake RE. A survey of clinical practices and readiness to adopt evidence-based practices: Dissemination research in an addiction treatment system. J Subst Abuse Treat 2004 Jun;26(4):305-312. [CrossRef] [Medline]
  43. Bartholomew NG, Joe GW, Rowan-Szal GA, Simpson DD. Counselor assessments of training and adoption barriers. J Subst Abuse Treat 2007 Sep;33(2):193-199 [FREE Full text] [CrossRef] [Medline]
  44. Khazaal Y, Chatton A, Cochand S, Zullino D. Quality of web-based information on cannabis addiction. J Drug Educ 2008;38(2):97-107. [Medline]


ADCA: Alcohol and Other Drugs Council of Australia
AGREE II: Appraisal of Guidelines for Research and Evaluation
APA: American Psychiatric Association
CPG: clinical practice guideline
ICC: intraclass correlation
IPS: Indian Psychiatric Society
NCPIC: National Cannabis Prevention and Information Centre
NDSIS: National Drugs Sector Information Service
NHC: National Health Committee
NICE: National Institute for Health and Clinical Excellence
NSWD: New South Wales Department of Health (Drug and Alcohol Psychosocial Interventions Professional Practice Guideline)
NSWP: New South Wales Department of Health (National Clinical Guidelines for the Management of Drug Use During Pregnancy, Birth and the Early Development Years of the Newborn guideline)

Edited by G Eysenbach; submitted 26.08.12; peer-reviewed by B Kiluk; comments to author 20.09.12; revised version received 20.09.12; accepted 25.10.12; published 07.12.12

Please cite as:
Norberg MM, Turner MW, Rooke SE, Langton JM, Gates PJ
An Evaluation of Web-Based Clinical Practice Guidelines for Managing Problems Associated with Cannabis Use
J Med Internet Res 2012;14(6):e169
doi: 10.2196/jmir.2319
PMID: 23249447

Export Metadata:
END, compatible with Endnote
BibTeX, compatible with BibDesk, LaTeX
RIS, compatible with RefMan, Procite, Endnote, RefWorks

Add this article to your Mendeley library
Add this article to your CiteULike library


©Melissa M. Norberg, Michael W. Turner, Sally E. Rooke, Julia M. Langton, Peter J. Gates. Originally published in the Journal of Medical Internet Research (, 07.12.2012.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.