Published on in Vol 24, No 1 (2022): January

Preprints (earlier versions) of this paper are available at, first published .
Implementation and Evaluation of a Digitally Enabled Precision Public Health Intervention to Reduce Inappropriate Gabapentinoid Prescription: Cluster Randomized Controlled Trial

Implementation and Evaluation of a Digitally Enabled Precision Public Health Intervention to Reduce Inappropriate Gabapentinoid Prescription: Cluster Randomized Controlled Trial

Implementation and Evaluation of a Digitally Enabled Precision Public Health Intervention to Reduce Inappropriate Gabapentinoid Prescription: Cluster Randomized Controlled Trial

Original Paper

1Quality Use of Medicines and Pharmacy Research Centre, UniSA Clinical and Medical Sciences, University of South Australia, Adelaide, Australia

2Australian Medicines Handbook Pty Ltd, Adelaide, Australia

Corresponding Author:

Andre Q Andrade, MD, PhD

Quality Use of Medicines and Pharmacy Research Centre

UniSA Clinical and Medical Sciences

University of South Australia

GPO Box 2471

Adelaide, 5001


Phone: 61 8 8302 2314


Background: Digital technologies can enable rapid targeted delivery of audit and feedback interventions at scale. Few studies have evaluated how mode of delivery affects clinical professional behavior change and none have assessed the feasibility of such an initiative at a national scale.

Objective: The aim of this study was to develop and evaluate the effect of audit and feedback by digital versus postal (letter) mode of delivery on primary care physician behavior.

Methods: This study was developed as part of the Veterans’ Medicines Advice and Therapeutics Education Services (MATES) program, an intervention funded by the Australian Government Department of Veterans’ Affairs that provides targeted education and patient-specific audit with feedback to Australian general practitioners, as well as educational material to veterans and other health professionals. We performed a cluster randomized controlled trial of a multifaceted intervention to reduce inappropriate gabapentinoid prescription, comparing digital and postal mode of delivery. All veteran patients targeted also received an educational intervention (postal delivery). Efficacy was measured using a linear mixed-effects model as the average number of gabapentinoid prescriptions standardized by defined daily dose (individual level), and number of veterans visiting a psychologist in the 6 and 12 months following the intervention.

Results: The trial involved 2552 general practitioners in Australia and took place in March 2020. Both intervention groups had a significant reduction in total gabapentinoid prescription by the end of the study period (digital: mean reduction of 11.2%, P=.004; postal: mean reduction of 11.2%, P=.001). We found no difference between digital and postal mode of delivery in reduction of gabapentinoid prescriptions at 12 months (digital: –0.058, postal: –0.058, P=.98). Digital delivery increased initiations to psychologists at 12 months (digital: 3.8%, postal: 2.0%, P=.02).

Conclusions: Our digitally delivered professional behavior change intervention was feasible, had comparable effectiveness to the postal intervention with regard to changes in medicine use, and had increased effectiveness with regard to referrals to a psychologist. Given the logistical benefits of digital delivery in nationwide programs, the results encourage exploration of this mode in future interventions.

J Med Internet Res 2022;24(1):e33873



Audit and feedback interventions can be effective tools to promote evidence translation through professional behavior change [1]. Audit and feedback interventions objectively measure professional performance and create benchmarks against professional standards. Development and dissemination of audit and feedback interventions have benefited from advances in information technology that have increased data availability, decreased costs, and improved automation. As a cost-effective and data-driven intervention, audit and feedback seems well suited for migrating to a fully electronic mode of delivery [2].

Despite the potential advantages of electronic delivery, there is a theoretical and evidence gap regarding the influence of changing the mode of delivery on the efficacy of behavior change interventions [3]. Previous studies of behavior change interventions suggest that the mode of delivery may influence the efficacy of behavior change techniques. The most likely mechanism is a fundamental change in user experience, which may elicit different responses [3]. Use of different modes of delivery changes users’ experiences by creating new contexts (eg, SMS text message sent at any given time versus scheduled educational sessions), creating more personal experiences (eg, face-to-face group sessions versus social media), and providing new modes of interaction (eg, interactive computer interventions versus printed material). A review on the use of behavior change techniques for smoking cessation found a positive effect of the techniques when delivered in person, but not when delivered in writing [4]. Another review on internet-delivered behavior change interventions found increased efficacy when using additional modes of delivery, such as SMS text messages and email communication [5]. However, further analysis of the same data set did not find a synergistic effect between any combination of mode of delivery and behavior change technique. The authors suggest that having additional channels of delivery may be beneficial, but were unable to recommend which modes to use for particular behavior change techniques [6].

Evidence on the influence of digital delivery in audit and feedback’s efficacy is also needed. A 2017 review of electronic audit and feedback interventions found heterogeneous results due to differences in the intervention implementation and the underlying theory and context [7].

Following the suggestions put forward by [8], the aim of this study was to evaluate the influence of delivering an audit and feedback intervention by secure digital delivery to the clinical desktop for integration to the patient care record, and compare it to the same intervention delivered by post. The behavior change goal was the reduction in gabapentinoid prescription. Gabapentinoids are a group of medicines that includes gabapentin and pregabalin. Evidence suggests these medicines are often incorrectly prescribed in nonneuropathic pain [9], with significant risk of serious side effects and potential for abuse and misuse [10]. To test the efficacy of the digital intervention, we (the authors) performed a cluster randomized trial of an intervention aimed at reducing inappropriate prescription of gabapentinoids by primary care providers.

The Veterans’ Medicines Advice and Therapeutics Education Services Program

The study is part of the Veterans’ Medicines Advice and Therapeutics Education Services (MATES) program [11], which is funded by the Australian Government Department of Veterans’ Affairs and provides medicines advice and promotes physician adoption of best practices. Since 2004, it has provided repeated multifaceted interventions, composed of an audit and feedback and educational component targeted at general practitioners (GPs), with supportive educational material provided to veterans, pharmacists, and other health professionals. The intervention is informed by social cognitive theory [12], the transtheoretical model [13], and the health promotion model PRECEDE-PROCEED [14]. Between 2004 and 2021, the program delivered 62 distinct interventions to GPs and veterans in all Australian states.

The intervention is developed in three sequential steps. The first is an epidemiological analysis performed on a comprehensive database containing administrative health claims data collected by the Australian Government Department of Veterans’ Affairs (DVA). The DVA claims database includes all health care services and medicines funded by DVA, including outpatient and hospital services, aged care, prescription medicines, allied health services, and other health coordination and support services.

The second step is the design of the educational component. It involves clinicians, researchers, and veterans, and results in two sets of educational materials. The first is targeted at GPs, and describes scientific updates and therapeutic recommendations. The second is targeted at veterans, and promotes general awareness and practical guidelines for patients.

The third step is the development of the audit and feedback component. The design process is also collaborative. The intervention adopts evidence-based strategies listed in [15] to improve effectiveness, such as authority (content endorsed by a clinical DVA committee), focus on problems with larger scope for improvement, and repeated feedback (topics are revisited after a few years). It also incorporates behavior change techniques such as heuristic techniques, goal setting, and prompts, which have been shown to improve perceived usefulness [16].

Digital Solution Design

The digital solution was conducted using a collaborative, pragmatic approach, influenced by Greenhalgh et al’s [17] Diffusion of Innovations Model, to develop a solution that could be implemented at a national scale. We used a series of stakeholder meetings scheduled as part of the Veterans’ MATES program to map out context and understand adopters’ (ie, GPs’) practices and preferences. The meetings involved funder (DVA) representatives, clinicians, veteran representatives, and information technology professionals.

To develop the solution, some particularities of the Australian health system and context were considered relevant, in particular:

  • Reliance on primary care providers: The GP is the gatekeeper of the Australian health care system. About 84% of Australians see a GP every year, and 77% of patients have a preferred GP [18].
  • Geographical distribution: GPs responsible for Australian veterans are located in all parts of the country. There are only a few GPs specialized in veteran care; most professionals have less than a handful of veterans under their care.
  • Technological readiness in primary care: Use of electronic health records in primary care has been widespread in Australia for at least 10 years [19]. Additionally, secure messaging infrastructure is well established for receiving laboratory test results.

The proposed solution is an adaptation of the 3 steps used by Veterans’ MATES interventions, suited for a digital medium. To identify individuals at risk of medication-related harm, the solution uses a set of algorithms to extract information from claims data (services and medicines) indicating phenotype (which conditions affect the patient based on the resources they use). These algorithms identify patients at risk, either due to long-term conditions, medicine use, or current events such as medicine discontinuation.

To create the electronic messages, patient information extracted from the claims database is embedded in a template to create an audit and feedback document designed to promote recognition of patient risk. The document uses behavior change techniques, including prompts, goal setting, discrepancy between current behavior and goal, information about health consequences, and feedback on behavior; all of these techniques have been shown to improve intervention usefulness [16].

Documents are created as PDF documents, encrypted, and embedded in a Health Level Seven (HL7) version 2 file using internally developed software. Audit and feedback documents may contain complex graphical elements and may change significantly according to patients’ conditions and suggested recommendations. Therefore, we chose to initially develop documents as HTML pages, which are then converted to PDF format.

Finally, our investigation suggested the suitability of using an existing secure message infrastructure to reach GPs. Encrypted HL7 messages are sent to GP offices using a third-party provider and then decrypted by the clinical software and incorporated into the GP workflow. This solution adheres to many determinants of innovation diffusion identified in [17], as shown in Table 1.

Table 1. Determinants of innovation diffusion and predicted advantages of proposed solution.
Determinant of innovationPredicted advantages of proposed solution
Relative advantageElectronic messages are easier to read and act upon and less cumbersome than other communication means, such as printed materials or telephone communication
CompatibilityThe solution uses communication infrastructure already being used to receive laboratory test results, with minimal additional impact on clinician workflow
ComplexityThe solution can be described by the three main processes (patient identification, message tailoring, and secure delivery), which are understood by all stakeholders
TrialabilityThe solution was trialed in 3 small pilots and 1 randomized controlled trial before large-scale adoption
RiskThe solution has a relatively low cost and builds upon a 15-year program, reducing risk
Task issuesThe solution is embedded in current workflow, with minimal task disruption
Augmentation/supportEach message is data driven, meaning it offers information related to a unique patient, also providing clear and unambiguous recommendations

Feasibility Studies

The most important implementation risk identified during the initial stakeholder meetings was that the intervention could be perceived as intrusive and disruptive to GP workflow. To mitigate this risk, the solution was trialed in 3 sequential small-scale pilots, taking place in April, July, and September 2019. The main goals of the pilots were the following:

  1. Evaluate the technical feasibility. We measured the proportion of messages acknowledged as successfully received.
  2. Reduce the risk of disrupting GP work practices. GPs involved in the pilot could get in touch via a support email, website, and telephone. Additionally, we sent an invitation to an online survey containing 16 questions about usability and satisfaction.

The first pilot was planned as an opt-in trial, and GPs were invited to participate by email (Multimedia Appendix 1). The second pilot was planned as an opt-out trial, and GPs were sent an email explaining the study and offering the opportunity to be removed from the list. The final trial was planned as usual service and preceded by a mailed information leaflet (Multimedia Appendix 2).

Trial Design

To test the influence of mode of delivery on the effectiveness of audit and feedback interventions, we performed a parallel, cluster randomized trial of a computer-delivered intervention to reduce inappropriate gabapentinoid prescription. The trial was designed to compare the post-delivered intervention (usual intervention as concurrent control) with the computer-delivered intervention. Since the intervention targets GPs who may have multiple veteran patients, we adopted a cluster design whereby a GP received information for all of their patients in the intervention by the same mode.

The intervention delivered via postal mode has been shown effective in translating evidence in different domains [20] including promoting medicine review [21], osteoporosis screening [22], uptake of health services [23], reducing inappropriate proton pump inhibitor use [24], and hypnotic use for insomnia [25].


To be eligible for participation, both veterans and their primary GP had to be eligible for the digital intervention. Eligible veterans comprised active DVA clients that had 2 or more gabapentinoid (either pregabalin or gabapentin) prescriptions in a 4-month period (October 2019 to January 2020). Veterans were also required to be resident in Australia, living in the community setting (ie, not residing in aged care or other long-term care facilities), and to not have previously requested exclusion from Veterans’ MATES interventions for any reason.

GPs were eligible if they were identified as the primary GP of one or more Australian veterans, and at least one of the veterans was eligible for the intervention. Participant GPs were excluded if they did not have installed capacity to receive secure electronic messages from our partner message provider (HealthLink Group Limited) or if they had previously requested exclusion from Veterans’ MATES interventions for any reason.

To determine the primary GP for a given veteran, we developed an algorithm based on prescriptions and outpatient services provided. Providers were scored based on the number of prescriptions and services provided, and weighted based on recency of services to account for veterans changing providers.


The trial was conducted across all Australian states and territories. We determined patient eligibility by querying the DVA claims database. Outcome data including service provision and medicine dispensing were collected from the DVA claims database.

The Intervention

GPs in the intervention arm received the intervention exclusively in a digitally delivered format. It was sent via secure message infrastructure directly to the GP’s clinic. Once received by the practice, it is reviewed by a practice manager of the GP and assigned to the appropriate patient. Once it is assigned, it can be accessed in the electronic health record alongside pathology reports and referral letters.

GPs in the usual care arm received the intervention by postal service. This delivery contains both the audit and feedback documents (for all selected veteran patients) and the educational materials (including a copy of the material targeted at veterans).

Both sets of materials contained the same theoretical content and personal information. Due to feedback from users, the digitally delivered intervention was slightly modified to user workflow. Since we could not deliver general educational documents to the health record, the audit and feedback document was enhanced to contain a link to the online educational material (see Figure 1). Additionally, the single letter containing multiple patients was segmented into one electronic document per patient. Finally, a color chart was added at the top of the electronic document to highlight different prescription patterns and help GPs prioritize patients when receiving multiple documents.

Veterans in both the intervention and usual care arms received educational material by post. The material can be found on the Veterans’ MATES web page [26].

Figure 1. Example of the intervention delivered to general practitioners (digital version).
View this figure

Enrolment and Randomization

Following the eligibility criteria, all GPs acting as the main care provider for an eligible patient (current gabapentinoid use) were considered for recruitment. We excluded all practitioners not found in the partner’s (Healthlink Group Limited) provider directory, as they would be unable to receive secure electronic messages. All eligible GP and patient pairs were included in the study sample.

GPs were randomized 1:1 to intervention or usual care. Randomization was block stratified by number of veterans under care. Randomization numbers for each GP were computer generated by a statistician who was not involved in enrolment. Due to the highly automated nature of the intervention and data collection (claims data), no further masking procedures were performed.

An ethics protocol for the study was approved by the University of South Australia Human Research Ethics Committee (ethics protocol P203/04) and the Australian Government Department of Defence and Veterans’ Affairs Human Research Ethics Committee (E016/007).


The primary outcomes were the change in average gabapentinoid prescription during the study period, standardized as multiples of the defined daily dose (DDD) per day, and the proportion of veterans visiting a psychologist for the first time. Primary outcomes were evaluated at 6 and 12 months.

Since the dosing of pregabalin and gabapentin are different, we calculated DDD for each medicine and summed the results. To remove the influence of extreme stockpiling and dispensing data errors, patients with a DDD over 10 (10 times the defined daily dose) were removed from analysis. The DDD was created by the World Health Organization (WHO) as a comparative unit of medicine use [27]. In this study, it allows the comparison of different gabapentinoids, such as gabapentin and pregabalin. The average daily DDD was calculated as per the following formula:

The total mass amount was determined according to all claimed prescriptions of gabapentin and pregabalin in the 3 months prior to the intervention (January 3, 2020, to April 2, 2020) and in the 6 months (July 3, 2020, to October 2, 2020) and 12 months (January 3, 2021, to April 2, 2021) following the intervention.

Secondary outcome was the time to the first visit (face-to-face, telephone, or video) with the primary provider. All outcome measures pertain to the individual (patient).

We conducted a secondary analysis that can be divided into two parts. First, we evaluated the overall intervention impact by measuring changes in average gabapentinoid DDD before and after the intervention. Furthermore, we evaluated whether the dose of gabapentinoid (high, medium, or low) or concurrent use of opioids influenced the efficacy of the different modes of delivery. Veterans were considered to be on a high dose if the average DDD in at least one month of the selection period was >2. Veterans were considered to be on a low dose if the average DDD in every month of the selection period was <0.25. Values between those two values were considered to be a medium dose.

Statistical Methods

We analyzed data from services and medicines claims for all enrolled patients who were alive at 12 months postintervention. To account for the cluster design, we analyzed the primary outcome using a linear mixed-effects model [28], with the GP as the grouping variable. The effect of mode of delivery on patients’ likelihood of visiting a psychologist was tested by logistic regression, also using GP as the grouping variable. The time to first GP visit was analyzed by survival analysis. Patients were considered to have the “event” if they had an appointment with the targeted GP. Events were right censored at 3 months (92 days). The relative effect of digital mode versus postal mode was evaluated by a Cox proportional hazards model, with the GP as cluster variable. Secondary analysis was performed by univariate linear mixed-effects model, with the GP as grouping variable. For all hypothesis tests, we considered a 95% CI (P≤.05). All analysis was performed in Python 3.7 (The Python Software Foundation). The main statistical libraries used were Statsmodels (version 0.12) [29] and Lifelines (version 0.25.11) [30].

Availability of Data and Material

The data that support the findings of this study are available from the Australian Government DVA but restrictions apply to the availability of these data, which were used under license for this study, and so are not publicly available.

Feasibility Studies

For the first pilot, a convenience sample of 75 GPs were sent an email invitation to participate (opt-in), and 5 GPs agreed to be included. For the second pilot, we selected a convenience sample of 20 GPs who could opt out of the pilot. For the third pilot, 189 messages were sent to GPs who had not participated previously. We received 6 survey responses, and all responders evaluated the usability as good (easy to read, correct information) and were either likely or very likely to continue to subscribe to future interventions. We received a single letter advising a patient had recently switched medical providers. Given the lack of negative feedback and positive survey responses, the project leadership considered the pilot successful and the intervention feasible, and approved the randomized controlled trial.

Cluster Randomized Controlled Trial

A total of 3271 veterans were considered eligible for the intervention, and 2552 GPs were identified as their main care providers (Figure 2). After randomization, the intervention was successfully delivered in March/April 2020. The postal intervention was sent to GPs on March 19, 2020. The computer-delivered intervention was delivered in three waves (on March 23, March 25, and April 2, 2020).

Veterans randomized to either intervention arm had a similar demographic profile (Table 2). The patterns of gabapentinoid and opioid use were also similar.

By the end of the study, both intervention groups had a significant reduction in gabapentinoid dispensing, as measured by the change in average daily DDD from baseline to 12 months (digital: mean reduction of 0.058, SD 0.38, or 11.2%, P=.004; postal: mean reduction of 0.058, SD 0.37, or 11.2%, P=.001). Figure 3 shows the trends in DDD before and after the intervention.

Figure 2. CONSORT (Consolidated Standards of Reporting Trials) flowchart. GP: general practitioner.
View this figure
Table 2. Clinical and demographic data at baseline.
Baseline dataPostal interventionDigital intervention
Number of participants14661519
Age (years), mean (SD)76.1 (14.6)76.1 (14.5)
Male, n (%)853 (58)883 (58)
Gabapentinoid dose at baseline, n (%)

High41 (3)34 (2)

Medium1188 (81)1213 (80)

Low237 (16)272 (18)
Concurrent opioid use, n (%)590 (40)636 (42)
Figure 3. Average daily DDD by intervention group. DDD: defined daily dose.
View this figure

We found no difference between digital and postal mode of delivery in reduction of gabapentinoid volume at 6 or 12 months (Table 3). A greater proportion of veterans in the digital intervention group saw a psychologist in the following 12 months (P=.02). Digital intervention promoted a small but statistically significant (P=.04) effect of earlier GP visits postintervention.

Veterans were segmented according to dose and concurrent opioid use. Consistent with the results of the primary analysis, no differences were found between the digital and postal interventions in any subgroup analysis. Dose reduction was more pronounced in high-dose gabapentinoid users, and there was no observed reduction in the average dose of low-dose users in either arm.

Table 3. Primary and secondary outcomes, by intervention arm.
OutcomesPostalDigitalP value
Average defined daily dose change (baseline to 6 months)–0.030–0.023.61
Average defined daily dose change (baseline to 12 months)–0.058-0.058.98
Percentage of new psychologist visits (baseline to 6 months)
Percentage of new psychologist visits (baseline to 12 months)
Hazard ratio for general practitioner visit within 90 days (95% CI)0.92 (0.85-0.99)1 (reference).04a


Principal Findings

In this study, we present the successful migration of a paper-based national behavior change intervention into a digital intervention. After a careful scaling of the intervention, ample communication, and stakeholder support, we were able to perform a large-scale randomized controlled trial covering all Australian states. The trial showed that both paper and digital versions of an intervention composed of education and audit and feedback was effective in reducing gabapentinoid prescriptions for an Australian population. Additionally, it showed that the digital intervention is equivalent to paper in changing prescription patterns.

This study is one of the first to test the effect of mode of delivery in a large-scale, precision public health intervention. The use of a digital medium of delivery has several advantages over conventional interventions, including the capacity for improved personalization and precision; improved automation; use of predictive analytics for targeting; data analytics; and improved interaction [31]. However, any new intervention may have unforeseen consequences, and requires testing as with any other new technology [32]. The digital media and paper version had similar effectiveness for affecting medicine use, but this study provides emerging evidence that digital intervention may be superior for services that require referral.

Digital delivery changes how participants interact with and experience the intervention. Integration with the patient electronic health record reduces the effort required to create new patient requests, such as actively inviting patients to a follow-up appointment. Therefore, the incorporation of the intervention in a clinician’s workflow may explain the increased number of GP visits and psychologist referrals after the digital intervention when compared to the usual intervention. Creating a request to follow up a patient is easy to implement and unlikely to cause significant disruption. In contrast, reducing the dose of gabapentinoids, commonly indicated for pain, requires careful consideration and close patient contact and participation. Our results suggest that both postal and digital interventions are effective in promoting dose change, but it is possible that the digital medium advantage lies in creating triggers that can be easily followed.

The timing of this study is an important limitation of this study, as intervention delivery coincided with the initial restrictions implemented in Australia in response to the COVID-19 pandemic in March 2020. The week of the intervention, several policies to restrict gatherings and reduce risk of contagion were enacted [33], which influenced some of the metrics used in this study. Medicine dispensing was likely affected, with stockpiling occurring and a temporary lack of access. Additionally, many clinics were closed to avoid waiting room risks, and appointments via telehealth were funded by the Department of Health. This may have influenced intervention effectiveness, as the opportunity to adjust therapy was reduced; however, it is unlikely to have affected the assessment of mode of delivery as both arms of the trial would have been equally affected by the COVID-19 restrictions. Postal mail services were fully maintained during restrictions.

This study also provides a foundation for further research aimed at improving the effectiveness of audit and feedback in public health digital interventions. The effect size of conventional and digital audit and feedback interventions is usually small [1,7], and a clear methodology to improve effect remains an open question. Effect may be influenced by factors related to the recipient (eg, GP), behavior, or content and delivery of the intervention [34]. Using digital media enables nationwide programs such as Veterans’ MATES to contribute to such research by creating repeated interventions at lower cost, with greater speed and precision.


This study showed a digitally delivered professional behavior change intervention had comparable effectiveness to a postal intervention and superior efficacy for referral services. Given the logistical benefits of digital delivery in nationwide programs (cost, speed, and precision), the results encourage exploration of this mode in future interventions.


This work was funded by the Australian Government Department of Veterans’ Affairs (DVA) as part of the Veterans’ Medicines Advice and Therapeutics Education Services (Veterans’ MATES) program. The DVA reviewed this manuscript before submission but played no role in the study design, study execution, analysis or interpretation of data, writing of the manuscript, or decision to submit the paper for publication.

Veterans’ MATES is provided by the University of South Australia, Quality Use of Medicines and Pharmacy Research Centre, in association with Discipline of General Practice, The University of Adelaide; Discipline of Public Health, The University of Adelaide; Repatriation General Hospital, Daw Park; National Prescribing Service (NPS) – Better choices, Better health; Australian Medicines Handbook; and the Drug and Therapeutics Information Service.

EER is supported by the National Health and Medical Research Council (GNT 1110139).

Editorial Notice

This randomized study was not prospectively registered. The authors explained that the trial was focused exclusively on its effect on providers, rather than patients. This trial was a modification of a previous existing intervention running continuously since 2004 (4 modules each year), so there was no additional enrollment for this particular trial. To reduce the bias in analysis, all analytic procedures (for this and all other interventions) were prospectively defined and submitted to the Australian Government Department of Veterans' Affairs for evaluation and approval. However, there is no formal registration number, and the plan is not made publicly available due to confidentiality reasons. The editor granted an exception from ICMJE rules mandating prospective registration of randomized trials because the risk of bias appears low, and the trial is targeted at providers. However, readers are advised to carefully assess the validity of any potential explicit or implicit claims related to primary outcomes or effectiveness, as the lack of registration means that authors could change their outcome measures retrospectively.

Authors' Contributions

AQA conceived of, designed, and conducted the final analysis for this study and drafted the manuscript. AQA, JPC, VTL, and EER developed the protocol and study approach. GMK, ER, LMKE, and NLP were involved in the data analysis. EER conceived of and designed the study, and critically revised the manuscript for important intellectual content. All authors made important contributions to the theoretical approach and interpretation of insights. All authors read and approved the final manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Opt-out letter.

DOCX File , 14 KB

Multimedia Appendix 2

General practitioner communication campaign to inform about digital intervention.

DOCX File , 248 KB

Multimedia Appendix 3

CONSORT-eHEALTH checklist (V 1.6.1).

PDF File (Adobe PDF File), 1291 KB

  1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012 Jan;6:CD000259. [CrossRef] [Medline]
  2. Landis-Lewis Z, Brehaut JC, Hochheiser H, Douglas GP, Jacobson RS. Computer-supported feedback message tailoring: theory-informed adaptation of clinical audit and feedback for learning and behavior change. Implement Sci 2015 Jan 21;10(1):12-12 [FREE Full text] [CrossRef] [Medline]
  3. Dombrowski SU, O'Carroll RE, Williams B. Form of delivery as a key 'active ingredient' in behaviour change interventions. Br J Health Psychol 2016 Nov;21(4):733-740. [CrossRef] [Medline]
  4. Black N, Eisma MC, Viechtbauer W, Johnston M, West R, Hartmann-Boyce J, et al. Variability and effectiveness of comparator group interventions in smoking cessation trials: a systematic review and meta-analysis. Addiction 2020 Sep 11;115(9):1607-1617 [FREE Full text] [CrossRef] [Medline]
  5. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010;12(1):e4 [FREE Full text] [CrossRef] [Medline]
  6. van Genugten L, Dusseldorp E, Webb TL, van EP. Which Combinations of Techniques and Modes of Delivery in Internet-Based Interventions Effectively Change Health Behavior? A Meta-Analysis. J Med Internet Res 2016;18(6):e155 [FREE Full text] [CrossRef] [Medline]
  7. Tuti T, Nzinga J, Njoroge M, Brown B, Peek N, English M, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci 2017 May 12;12(1):1-20 [FREE Full text] [CrossRef] [Medline]
  8. Grimshaw J, Ivers N, Linklater S, Foy R, Francis JJ, Gude WT, Audit and Feedback MetaLab. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf 2019 May 09;28(5):416-423 [FREE Full text] [CrossRef] [Medline]
  9. Morrison EE, Sandilands EA, Webb DJ. Gabapentin and pregabalin: do the benefits outweigh the harms? J R Coll Physicians Edinb 2017 Dec;47(4):310-313. [CrossRef] [Medline]
  10. Goodman CW, Brett AS. Gabapentin and Pregabalin for Pain - Is Increased Prescribing a Cause for Concern? N Engl J Med 2017 Aug 03;377(5):411-414. [CrossRef] [Medline]
  11. Quality Use of Medicines and Pharmacy Research Centre. Veterans' MATES. 2021.   URL: [accessed 2021-12-29]
  12. Bandura A. Human agency in social cognitive theory. Am Psychol 1989 Sep;44(9):1175-1184. [CrossRef] [Medline]
  13. Prochaska JO, Velicer WF, Rossi JS, Goldstein MG, Marcus BH, Rakowski W, et al. Stages of change and decisional balance for 12 problem behaviors. Health Psychol 1994 Jan;13(1):39-46. [CrossRef] [Medline]
  14. Green L, Kreuter M, Green L. Health program planning: an educational and ecological approach. New York: McGraw-Hill; 2005.
  15. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O'Brien MA, French SD, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014 Nov 26;29(11):1534-1541 [FREE Full text] [CrossRef] [Medline]
  16. Andrade AQ, LeBlanc VT, Kalisch-Ellett LM, Pratt NL, Moffat A, Blacker N, et al. Determinants of usefulness in professional behaviour change interventions: observational study of a 15-year national program. BMJ Open 2020 Oct 14;10(10):e038016 [FREE Full text] [CrossRef] [Medline]
  17. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82(4):581-629 [FREE Full text] [CrossRef] [Medline]
  18. The Royal Australian College of General Practitioners. General Practice: Health of the Nation. Melbourne, Victoria: RACGP; 2019.
  19. Jha AK, Doolan D, Grandt D, Scott T, Bates DW. The use of health information technology in seven nations. Int J Med Inform 2008 Dec;77(12):848-854. [CrossRef] [Medline]
  20. Roughead EE, Kalisch Ellett LM, Ramsay EN, Pratt NL, Barratt JD, LeBlanc VT, et al. Bridging evidence-practice gaps: improving use of medicines in elderly Australian veterans. BMC Health Serv Res 2013 Dec 12;13:514 [FREE Full text] [CrossRef] [Medline]
  21. Kalisch Ellett LM, Pratt NL, Sluggett JK, Ramsay EN, Kerr M, LeBlanc VT, et al. Sustaining practice change in health care: the impact of a national quality improvement program on the uptake of collaborative medicines reviews. J Pharm Pract Res 2018 Jun 15;48(3):222-230. [CrossRef]
  22. Kalisch Ellett LM, Pratt NL, Sluggett JK, Ramsay EN, Kerr M, LeBlanc VT, et al. Patient-specific prescriber feedback can increase the rate of osteoporosis screening and treatment: results from two national interventions. Arch Osteoporos 2017 Dec;12(1):17. [CrossRef] [Medline]
  23. Pratt NL, Kalisch Ellett LM, Sluggett JK, Ramsay EN, Kerr M, LeBlanc VT, et al. Commitment questions targeting patients promotes uptake of under-used health services: Findings from a national quality improvement program in Australia. Soc Sci Med 2015 Nov;145:1-6. [CrossRef] [Medline]
  24. Pratt NL, Kalisch Ellett LM, Sluggett JK, Gadzhanova SV, Ramsay EN, Kerr M, et al. Use of proton pump inhibitors among older Australians: national quality improvement programmes have led to sustained practice change. Int J Qual Health Care 2017 Feb 01;29(1):75-82. [CrossRef] [Medline]
  25. Kalisch Ellett LM, Lim R, Pratt NL, Kerr M, Ramsay EN, LeBlanc TV, et al. Reducing hypnotic use in insomnia management among Australian veterans: results from repeated national interventions. BMC Health Serv Res 2018 Aug 09;18(1):626-636 [FREE Full text] [CrossRef] [Medline]
  26. Quality Use of Medicines and Pharmacy Research Centre. Recovering from pain: strategies that can help. Veterans' MATES.   URL: [accessed 2021-12-29]
  27. Bergman U. The history of the Drug Utilization Research Group in Europe. Pharmacoepidemiol Drug Saf 2006 Feb;15(2):95-98. [CrossRef] [Medline]
  28. Candlish J, Teare MD, Dimairo M, Flight L, Mandefield L, Walters SJ. Appropriate statistical methods for analysing partially nested randomised controlled trials with continuous outcomes: a simulation study. BMC Med Res Methodol 2018 Oct 11;18(1):105-122 [FREE Full text] [CrossRef] [Medline]
  29. Seabold S, Perktold J. Statsmodels: Econometric and Statistical Modeling with Python. In: Proceedings of the 9th Python in Science Conference. 2010 Presented at: 9th Python in Science Conference; June 28-July 3, 2010; Austin, TX p. 92-96   URL:
  30. Davidson-Pilon C. lifelines: survival analysis in Python. JOSS 2019 Aug;4(40):1317-1311. [CrossRef]
  31. Odone A, Buttigieg S, Ricciardi W, Azzopardi-Muscat N, Staines A. Public health digitalization in Europe. Eur J Public Health 2019 Oct 01;29(Supplement_3):28-35 [FREE Full text] [CrossRef] [Medline]
  32. McKee M, van Schalkwyk MCI, Stuckler D. The second information revolution: digitalization brings opportunities and concerns for public health. Eur J Public Health 2019 Oct 01;29(Supplement_3):3-6 [FREE Full text] [CrossRef] [Medline]
  33. Desborough J, Hall Dykgraaf S, de Toca L, Davis S, Roberts L, Kelaher C, et al. Australia's national COVID-19 primary care response. Med J Aust 2020 Aug 04;213(3):104-106.e1 [FREE Full text] [CrossRef] [Medline]
  34. Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, et al. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci 2017 Sep 29;12(1):117-110 [FREE Full text] [CrossRef] [Medline]

DDD: defined daily dose
DVA: Department of Veterans’ Affairs
GP: general practitioner
HL7: Health Level Seven
MATES: Medicines Advice and Therapeutics Education Services
RCT: randomized controlled trial
WHO: World Health Organization

Edited by R Kukafka; submitted 27.09.21; peer-reviewed by MS Shafi, S Pesälä; comments to author 13.11.21; revised version received 15.11.21; accepted 21.11.21; published 10.01.22


©Andre Q Andrade, Jean-Pierre Calabretto, Nicole L Pratt, Lisa M Kalisch-Ellett, Gizat M Kassie, Vanessa T LeBlanc, Emmae Ramsay, Elizabeth E Roughead. Originally published in the Journal of Medical Internet Research (, 10.01.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.