For risk assessment, an algorithm was written based on the 1991 Framingham risk equation to predict 5-year risk of a first CVD event (coronary heart disease, stroke, congestive heart failure, peripheral vascular disease) [25]. Recognizing that this equation might underestimate risk for certain clinical conditions and for specific ethnic groups, adjustments were made using recommendations from the New Zealand Guidelines Group and guidelines from the 2004 National Heart Foundation (NHF) of Australia [26,27]. The risk factor variables and adjustments are summarized in Textbox 1.

To define the risk management outputs of the tool, pharmacological treatment recommendations from six Australian CVD-related guidelines current in 2007 were consolidated into a single algorithm [26,28-31]. The thresholds and treatment targets for BP, lipid, and anti-platelet management are summarized in Textbox 2.

The risk assessment and management algorithms were programmed into a stand-alone software package (Igor Pro 6, WaveMetrics Inc, Portland, OR, USA) that produced a single-page output. If there was complete risk factor information available, a risk score was generated and plotted along a color spectrum bar and treatment recommendations were provided. If information required for absolute risk assessment was absent, the output identified the variables missing and the color bar was changed to greyscale. Because many Australian guidelines are not exclusively risk based, some treatment recommendations could still be made despite incomplete risk factor information. Examples of these two types of output are shown in Figure 1 and Figure 2.

Figure 1

Sample CDS output with complete information and color bar

De-identified data from all consecutive patients with complete risk factor information attending a specialist vascular clinic over a 3-month period (May to August 2008, n = 137) were entered into the tool by a trained research assistant to generate CDS outputs. The validity of these outputs was assessed in two parts. First, a researcher who was not involved with the algorithm development programmed the Framingham risk equation and adjustments into a second statistical software package, STATA version 9.2 (Stata Corporation, College Station, TX, USA). Correlation between risk scores generated from the CDS tool and the STATA program were assessed. Second, an experienced physician, blinded to the CDS tool management recommendations, reviewed the risk assessment data for each patient. She then performed a manual review of the guidelines and assessed whether anti-platelet, BP-lowering, and lipid-lowering medications were indicated or whether targets were being met for those patients already prescribed BP-lowering and lipid-lowering drugs. Agreement between the CDS tool and the physician’s recommendations was assessed.

The tool was field tested in two different Australian primary health care settings: eight teaching general practices in Sydney and three Aboriginal Medical Services (AMSs) in New South Wales. Sampling was purposive and sought GPs interested in research and training who might critically appraise the tool and provide recommendations for its future development. A diversity sample in terms of GP age, gender, and size of practice was sought. Consecutive, routinely attending patients (Aboriginal ≥ 35 years, non-Aboriginal ≥ 45 years) were invited from the waiting room to participate. The patient age range is based on Australian guideline recommendations for absolute risk assessment screening [32]. Each GP had outputs generated for around 10 patients. This number was considered sufficient to allow (1) adequate exposure to a variety of tool outputs, (2) an appreciation of the tool’s application in a typical working day, and (3) minimal administrative burden to the GP or the practice. Figure 3 provides a schema for how the study was conducted. Because the pilot version of the tool was built using stand-alone software, it lacked the ability to pre-populate with demographic and clinical data already existing in the EMR. Thus, the key role of the research assistant was to act as a proxy for this pre-populating feature by accessing the relevant risk factor information from the patient’s EMR. In essence, this simulated the situation that might occur if the tool was built into the GP’s practice software system. The resultant output was given to GPs prior to the consultation for review with their patients.

Figure 3

Study schema

A mixed methods evaluation was conducted following the methods outlined by Tashakkori and Teddlie [33]. Specifically, the quantitative and qualitative components were equally weighted and combined simultaneously to obtain an understanding of the effectiveness (quantitative), acceptability (quantitative and qualitative), and feasibility (qualitative) of a CDS tool for CVD risk management in primary care settings.

At the end of each consultation, GPs completed a short survey on their attitudes about the tool and management provided. At study completion, GPs completed a second survey on their practice characteristics. This survey adapted some questions from a previously published instrument [34]. GPs then participated in an in-depth interview evaluation. Interviews were semistructured and conducted by a GP researcher who had a practical working knowledge of the tool in clinical settings. Interviews covered three domains: (1) general attitudes about the tool and its impact on the consultation; (2) a review of specific tool outputs; (3) recommendations for future tool development. Full details of the survey instruments and interview guide are provided in Multimedia Appendix 1-3.

Descriptive statistics and quantitative analyses were conducted using SAS version 9.1 (SAS Institute Inc, Cary, NC, USA). Management decisions were assessed as to whether GPs acted on recommendations from the tool output. Interview recordings were professionally transcribed, and thematic content analysis was performed drawing on the methods outlined by Patton [35]. Interview transcripts were initially reviewed in their entirety to become familiar with the data. They were then open coded to core thematic categories and these analyses were conducted contemporaneously with data collection. At the end of study, the investigator team met on several occasions to determine how these open-coded categories would be relationally grouped to determine the final major themes. NVivo 8 (QSR International, Melbourne, Victoria, Australia) was used to help organize the data through this analysis process.

The study was approved by both the Sydney South West Area Health Service and Aboriginal Health and Medical Research Council ethics committees. Patients and GPs gave written informed consent to participate in the study. Signed agreements were obtained from the three participating AMSs.

The tool’s risk assessment algorithm showed near perfect correlation with the independently programmed algorithm used in STATA (intraclass correlation coefficient 0.999). The variation was wholly explained by different rounding methods used in each software program. For prescribing recommendations, agreement between the tool and the physician’s recommendations for initiation of anti-platelet and lipid treatment was 100%. Agreement on meeting guideline targets for those already prescribed BP- and lipid-lowering treatments was also 100%. Agreement on initiation of BP treatment was 97% (kappa 0.95). In both cases of disagreement, the BP was < 125/75 mmHg and the physician judged that treatment was not indicated, while the tool recommended that treatment could not be determined due to missing information on proteinuria.

Twenty-one GPs participated in the study. Practices varied greatly in size, ranging from a solo GP practice with minimal administrative support to a large practice with 23 GPs and 15 nurses. Table 1 outlines GP characteristics and their use of electronic practice management features. Table 2 shows the risk factor characteristics of the patient population by Aboriginal status and prescribing rates of preventive CVD medications.

For the 200 CDS outputs generated for review, GPs agreed or strongly agreed that the output was easy to understand (77% of outputs), that screening and prescribing recommendations were appropriate (72% and 64% of outputs, respectively), and that recommendations on treatment targets were appropriate (70% of outputs). Fifty-two (26%) patient records were updated with CVD-related information, most commonly family history, past history of CVD, and smoking status. Figure 4 highlights the changes in risk factor screening and management following the consultation. Ninety-five (48%) patients received changes to their management, of whom 49 (52%) received lifestyle advice on CVD risk factors. For people assessed at high CVD risk (n = 82), 10% and 9%, respectively, had lipid-lowering and BP-lowering medications commenced or dose adjustments made, while 7% newly commenced anti-platelet therapy.

Figure 4

CVD management practices before and after a consultation involving the CDS tool

All GPs participated in the interview evaluation, with interviews ranging from approximately 30 to 60 minutes duration. One interview was conducted with a pair of participants, two interviews were conducted over the telephone, and the remainder were individual face-to-face interviews. Three major themes arose from the interview content analysis that will be reported here. A fourth substantive theme was identified that related to how tools are used in general practice and the role of evidence-based medicine in decision making. As this issue extends beyond factors related to the CDS tool and was not a specific objective of the study, an in-depth analysis of this theme will be conducted separately.

A limitation of this preliminary evaluation was that changes in care provider practices were based on a single consultation, reducing the ability to assess the potential impact of the CDS tool over time. A second potential limitation was the sampling method. Rather than seek a representative sample, we sought GPs who might actively contribute to the future development of the tool. AMSs were considered important settings to assess whether the tool was acceptable for use in a population with high levels of health disadvantage. Despite this purposive sampling, the types of medical software used, the electronic features used within those software systems, and the rates of performing absolute risk assessments were broadly similar to those reported in the Australian literature [12,36].

The implications of a CDS tool for CVD risk management extend well beyond being a point-of-care clinical resource. Data from UK CVD risk programs have allowed for the generation of population-specific risk prediction equations that outperform Framingham-based algorithms [20]. The NZ decision support system, combined with linkage to mortality and hospital databases, is similarly allowing for rapid advances in CVD risk factor epidemiology. The combination of a centrally managed Internet-based system with local management of program specifics by primary health organizations allows for a “ground up” approach to incorporating population health aspects into such systems. Along with epidemiological advances, both the UK and NZ systems allow for the use of large-scale primary care data to monitor health system performance. In Australia, such systems will play an integral role in the broader eHealth strategies being proposed to reform the health care system [37-39]. Performance measures in CVD risk management are integral to the UK Quality and Outcomes Framework and are allowing for large-scale analyses of regional variation and progress in reducing health inequalities [40]. In Australia, this is especially pertinent to addressing Aboriginal health inequities where specific indicators for the measurement and reduction of CVD risk are proposed [41]. Awareness of these broader issues and incorporation of the major study findings into the next phase of the project will provide a strong foundation to develop, implement, and evaluate an integrated CVD risk management system in Australian primary health care.