Based on the scoping review methodological framework proposed by Levac et al [27], and the Joanna Briggs Institute (JBI) methodological guidance for scoping reviews [28], we developed a protocol with the following steps: (1) clarifying the purpose of the review and linking it to a research question, (2) identifying relevant studies and balancing feasibility with breadth and comprehensiveness, (3) working in a team to iteratively select studies and extract their data, (4) charting the extracted data, incorporating a numerical summary, (5) collating, summarizing, and reporting the results, and (6) consulting the results regularly with stakeholders throughout regarding emerging and final results. This protocol is registered and available on the JBI website and the Open Science Framework (OSF) websites. We completed this review as per the published protocol.

We formed a multidisciplinary committee of experts in public health, primary health care, AI and data science, knowledge translation, and implementation science, as well as a patient partner and an industry partner (with expertise in the AI-health domain) with whom we consulted during all the steps of the scoping review. This helped us to interpret the results. The screening process is shown in Figure 1. Our review is reported according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Scoping Reviews) reporting guideline for reporting the study [29] (see Multimedia Appendix 1). Studies that did not report their study design are categorized by methodology according to the classification outlined by the National Institute for Health and Care Excellence [30].

We used the Prediction Model Risk of Bias Assessment Tool (PROBAST) tool for assessing the risk of bias, which includes 20 signaling questions to facilitate structured judgment of risk of bias organized in four domains of potential biases related to the following: (1) participants (covers potential sources of bias related to participant selection methods and data sources); (2) predictor variables (covers potential sources of bias related to the definition and measurement of predictors evaluated for inclusion in the model); (3) outcomes (covers potential sources of bias related to the definition and measurement of the outcomes predicted by the model); and (4) analyses (covers potential sources of bias in the statistical analysis methods) [31]. Risk of bias was judged as low, high, or unclear. If one or more domains were judged as having high risk of bias, the overall judgment was “high risk” [31].

We defined our bibliographic database search strategy for peer- reviewed publications in English or French using the Population, Intervention, Comparison, Outcomes, Setting and Study (PICOS) design components [32].

Studies about any population that provides health care services, including nurses, social workers, pharmacists, dietitians, public health practitioners, physicians, and community-based workers (an unregulated type of provider) were included, as were those about any populations who receive CBPHC services. We adhered to the definition of CBPHC provided by the Canadian Institutes of Health Research (CIHR) (ie, the broad range of primary prevention measures including public health, and primary care services within the community, including health promotion and disease prevention; the diagnosis, treatment, and management of chronic and episodic illness; rehabilitation support; and end-of-life care) [4]. Studies that took place in any CBPHC points of care, including community health centers, primary care networks, clinics, and outpatient departments of hospitals, were also included. Studies conducted in emergency departments were excluded.

Only studies that “tested” or “implemented” or “tested and implemented” AI methods, such as computer heuristics, expert systems, fuzzy logic, knowledge representation, automated reasoning, data mining, and machine learning (eg, support vector machines, neural networks, and Bayesian networks) were included. Studies related to robot-assisted care were excluded.

No inclusion or exclusion criteria were considered.

The primary outcomes of interest were those related to individuals receiving care (eg, cognitive outcomes, health outcomes, behavioral outcomes), providers of care (eg, cognitive outcomes, health outcomes, behavioral outcomes), and health care systems (eg, process outcomes). Moreover, we analyzed the outcomes of the AI systems for their accuracy and impact on the outcomes of care.

All study designs using qualitative, quantitative, or mixed methods were eligible for inclusion. In particular, we included experimental and quasi-experimental studies (randomized controlled trials, quasi-randomized controlled trials, nonrandomized clinical trials, interrupted time series, and controlled before-and-after studies), and observational (cohort, case control, cross- sectional, and case series), qualitative (ethnography, narrative, phenomenological, grounded theory, and case studies), and mixed methods studies (sequential, convergent).

An information specialist with an epidemiologist, an AI-healthcare researcher, and a family doctor developed a comprehensive search strategy and Medical Subject Headings (MeSH) mediated by the National Library of Medicine. The systematic search was conducted from inception until February 2020 in seven bibliographic databases: Cochrane Library, MEDLINE, EMBASE, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), ScienceDirect, and IEEE Xplore. Retrieved records were managed with EndNote X9.2 (Clarivate) and imported into the DistillerSR review software (Evidence Partners, Ottawa, ON) to facilitate the selection process (see Multimedia Appendix 2 for the search strategies used on each database).

We used a data extraction form, approved by our consultative committee, that we designed based on the Cochrane Effective Practice and Organisation of Care Review Group (EPOC) data collection checklist [33]. Specifically, we extracted study characteristics (eg, design and country of the corresponding author); population characteristics (eg, number of participants and type of disease or treatment); intervention characteristics (eg, AI methods used); and outcome characteristics, including outcomes related to the patients (eg, cognitive outcomes, health outcomes, behavioral outcomes), providers of care (eg, cognitive outcomes, health outcomes, behavioral outcomes), and health care systems (eg, process outcomes).

Two reviewers independently appraised the included studies using the criteria outlined in PROBAST to evaluate the risk of bias in each included study that was eligible for evaluation using PROBAST [31]. A third reviewer verified their appraisals.

We performed a descriptive synthesis [34] to describe the studies in terms of their population (patient, primary care providers), interventions (AI systems, evaluated parameters), and outcomes. The results were arranged according to the PICOS format. The tools and techniques for developing a preliminary synthesis included textual descriptions of the studies, grouping and clustering, and tabulation.

Throughout the steps of the review, we regularly updated all members of the research team and requested their feedback. We also presented our preliminary results during a workshop at Université Laval, Québec, Canada, with a multidisciplinary group of experts (in public health, primary care, AI and data science, knowledge translation, implementation science, as well as a patient partner, and an industry partner) and collected their comments and feedback.

Using a patient-centered approach, our team co-developed the protocol, conducted the review, and reported the results of this study. We integrated patients’ priorities within our research questions, search strategy terms, and outcomes of interest. Our patient partner was involved in each step of the research process, including the definition of the objectives, main analysis, descriptive synthesis, interpretation of preliminary and final results, and dissemination of the results obtained in this study.

The included studies sought to describe and test or implement either a novel AI model in CBPHC (16/90, 18%) or an off-the-shelf AI model, which is a modified or improved version of existing AI models in CBPHC (74/90, 82%).

Among the 90 studies, 2 (2%) reported using a sociocognitive theoretical framework [35,36]. One of these used the I-change model [35], a model that evolved from several cognitive models, explores the process of behavioral change and the determinants that relate to the change, and focuses on individuals’ intentions for adopting innovations [35,37]. In the first study [35] using the I-change model, the authors investigated the cognitive determinants associated with Dutch general practitioners’ intention to adopt a smoking cessation expert AI system in their respective practices and found that workload and time constraints are important barriers.

The second study used a continuing medical education framework [38] and compared traditional expert-led training (control group) with an online multimedia-based training activity supplemented with an AI-driven simulation feedback system (treatment group) [36]. Diagnosis accuracy significantly improved in the treatment group when compared to the control group, providing evidence supporting the efficacy of AI medical training methods.

Among the included studies, 25% (23/90) used data collected over a period of 1 year or less, 20% (17/90) used data collected over a period between 1 and 5 years, 12% (11/20) used data collected over a period between 5 and 10 years, and 9% (8/90) used data collected during more than a 10-year period. One study (1%) used three data sets, collected data from three different sites with over three different time periods (<1 year, 1-5 years, >10 years) [39]. The remaining studies (30/90, 33%) did not specify the time frames of their data set collections.

In terms of evaluating the performance of AI models, we considered the following performance metrices: True positive (TP), True negative (TN), False positive (FP), False negative (FN), sensitivity, specificity, precision, F1 score (ie, the weighted average of precision and recall, and area under the curve [AUC]). Among the 90 included studies, 31 (34%) did not report the performance of their models. Among the 59 studies that reported model performance, 13 (22%) used 2 or more performance measures and the remaining 46 (78%) used one measure (see Multimedia Appendix 4 for detailed information on studies’ AI methods used in the included studies and their performance measures).

Most of the included studies (81/90, 91%) were either diagnosis- or prognosis-related or focused on surveillance, and the remaining involved operational aspects (eg, resource allocation, system- level decisions) (see Multimedia Appendix 4 for detailed information).

The majority of the 90 included studies (68/90, 76%) investigated the use of AI in relation to a specific medical condition. Conditions studied were vascular diseases including hypertension, hypercholesteremia, peripheral arterial disease, and congestive heart failure (10/90, 11%) [40-49]; infectious diseases including influenza, herpes zoster, tuberculosis, urinary tract infections, and subcutaneous infections (8/90, 9%) [50-57]; type 2 diabetes (5/90, 6%) [58-62]; respiratory disorders including chronic obstructive pulmonary disease and asthma (6/90, 8%) [63-69]; orthopedic disorders including rheumatoid arthritis, gout, and lower back pain (5/90, 5%) [36,39,70-72]; neurological disorders including stroke, Parkinson disease, Alzheimer disease [73-75], and cognitive impairments (6/90, 5%) [76,77]; cancer including colorectal cancer, and head and neck cancer (4/90, 4%) [78-81]; psychological disorders including depression and schizophrenia (3/90, 3%) [82-84]; diabetic retinopathy (3/90, 3%) [85-87]; suicidal ideations (2/90, 2%) [88,89]; tropical diseases including malaria (2/90, 2%) [90,91]; renal disorders (2/90, 2%) [92,93]; autism spectrum disorder (2/90, 2%) [94,95]; venous disorders including deep vein thrombosis and venous ulcers (2/90, 2%) [96,97]; and other health conditions (8/90, 8%) [98-105].

In this section, we briefly explain the training, testing, and validation of the data sets, and then present our results. The training data set is the subset of the data that are used to fit in the initial AI model and to train it. The testing data set is the subset of the data used to evaluate the model that fits the initial training data set. The validation data set is a subset of the data used to conduct an unbiased evaluation of the model that fits the training data set, while simultaneously optimizing the model's hyperparameters, namely the parameters whose values are used to control the learning process [106]. The evaluation of these parameters is important because it provides information about the accuracy of predictions made by the AI model, and the prospective effects of hyperparameter tuning [107].

Among the 90 included studies, 9 (10%) reported on all three data sets, 33 (36%) reported on the training and testing data sets, and 36 (40%) reported on the training and validation data sets. No descriptions of these data sets were provided in 49 (54%) of the included studies.

Legal information concerning privacy was mentioned in 4% (4/90) of the studies in our review. Although health care records were anonymized to protect participants’ information in all four of these studies, only one explicitly reported ensuring data collection, storage, and sharing security. The remining studies did not report on data privacy and other legal information.

Extraction of the data related to the benefits for patients, primary health care providers, and the health system explained in this section was conducted according to what the authors of the included studies clearly reported as specific benefits to each of these categories.

The included studies reported the following information regarding primary health care provider-related benefits of implementing AI in CBPHC: enhanced interprofessional communication and quality of primary care delivery, reduced workload of these providers, and facilitation of referrals and patient-centered care.

Other benefits included benefits with respect to use of AI as a reminder system, application of AI tools to inform commissioning health care priorities, the benefit of an AI system as a quality improvement intervention by generating warnings in electronic medical records and analyzing clinical reports, facilitating monitoring of the diseases, and using AI to reduce health risks.

Studies in our review found that AI can play a role in improving individual patient care and population-based surveillance, can be beneficial by providing predictions to inform and facilitate policy makers decisions regarding the effective management of hospitals, benefits to community-level care, cost-effectiveness, and reducing burden at the system level.

Only one study (1%) among the included 90 papers assessed the cost-effectiveness of the AI system studied. The Predicting Out-of-Office Blood Pressure in the Clinic [PROOF-BP] system that the study authors developed for the diagnosis of hypertension in primary care was found to be cost-effective compared to conventional blood pressure diagnostic options in primary care [49].

Our results suggest that challenges of using AI in CBPHC include complications related to the variability of patient data as well as barriers to use AI systems or to participate in AI research owing to the age or cognitive abilities of patients.

With respect to the health care system, our review found challenges related to how information is recorded (eg, the use of abbreviations in medical records), poor interprofessional communication between nurses and physicians, inconsistent medical tests, and a lack of event recording in cases of communication failures. The included studies also mentioned problems with respect to the restricted resources and administrative aspects such as legislations and administrative approvals, as well as challenges with respect to the lack of digital or computer literacy among the primary health care providers.

In the included studies, other challenges were reported at the level of the health care system such as the data available for use with AI as well as challenges at the level of AI itself (eg, complexity of the system and difficulty in interpretation). The following were identified as the main barriers regarding the data: (1) insufficient data to train, test, and validate AI systems, leading to negative impacts on the robustness of AI models and the accuracy of their predictions; (2) poor quality data, inaccuracies in the data, misclassifications, and lack of representative data; (3) deidentification of protected medical data; and (4) variability in the data sets and combining different data sets. Regarding AI, computational complexity and difficulties in interpreting or explaining some AI model compositions were among the barriers at the AI level.

We identified the studies that were eligible to be evaluated using PROBAST. Among our included studies, 54% (49/90) were eligible to be evaluaeted using the PROBAST tool and most (39/49, 80%) were at high risk of bias according to our assessment with PROBAST (Figure 4). With respect to risk of bias for each of the four domains assessed, few studies presented risks regarding participants, (2/49, 4%), whereas 45% (22/49) studies exhibited risks of bias regarding outcomes. See Multimedia Appendices 5 and 6 for details on common causes of risks in each study).

We conducted a comprehensive systematic scoping review that included 90 studies on the use of AI systems in CBPHC and provided a critical appraisal of the current studies in this area. Our results highlighted an explosion in the number of studies since 2015. We observed variabilities in reporting the participants, type of AI methods, analysis, and outcomes, and highlighted the large gap in the effective development and implementation of AI in CBPHC. Our review led us to make the following main observations.

ML, NLP, and expert systems were the most commonly used in CBPHC. Convolutional neural networks and abductive networks were the methods with the highest performance accuracy within the given data sets for the given task. We observed that a small number of studies reported on the development and testing or implementation of a new AI model in their study, and most of the included studies (74/90, 82%) reported on the usage and testing or implementation of an off-the-shelf AI model. Previous work has demonstrated how off-the-shelf models cannot be directly used in all clinical applications [109]. We observed a high risk of overall bias in the diagnosis- and prognosis-related studies. The highest risk of bias was in the outcome, predictor, and analysis categories of the included studies; validation of studies (external and internal) was poorly reported, and calibration was rarely assessed. A high risk of bias implies that the performance of these AI models in a new data set might not be as optimal as it was reported in these studies. Given the high risk of bias observed in the included studies, AI models used in other settings (ie, with other data) may not exhibit the same level of prediction accuracy as observed.

Primary health care providers are more likely to use AI systems for system-level support in administrative or health care tasks and for operational aspects, rather than for clinical making decisions [1]. However, our results show that few AI systems have been used for these purposes in CBPHC. Rather, the existing AI systems are mostly diagnosis- or prognosis-related, and used for disease detection, risk identification, or surveillance. Further studies in this regard are needed to evaluate the reason behind this tendency in addition to studies for proving the efficiency and accuracy of AI models for assisting in clinical decision making within CBPHC settings. In our review, we found that only 2 of the 90 studies used a (sociocognitive) theoretical framework. Future research needs to use knowledge, attitudes, and behavior theories to expand AI usage for clinical decision making, and more efforts are required to develop and validate frameworks guiding effective development and implementation of AI in CBPHC.

Our results show that AI-CBPHC research rarely considers sex, gender, age, and ethnicity. In general, the effect of age is rarely investigated in the AI field and ageism is often ignored in the analysis of discrimination. In health research, AI studies that have evaluated facial and expression recognition methods identified bias toward older adults [109]. This bias could negatively affect the accuracy of the predictions made by AI systems that are commonly used by health care providers.

Furthermore, sex and gender are sources of variations in clinical conditions, affecting different aspects including prognosis, symptomatology manifestation, and treatment effectiveness, among others [110,111]. Despite this importance, big data analytics research focusing on health through the sex and gender lens has shown that current data sets are biased given they are incomplete with respect to gender-relevant indicators with sex-disaggregated data. Indeed, less than 35% of the indicators in international databases have full disaggregation with respect to sex [112]. Our results are consistent with this observation, as we found just half of the AI-CBPHC research with patient participants and nearly one-third with health care provider participants described the sex distribution. Moreover, no AI-CBPHC research has reported on gender-relevant indicators. These are important aspects that need to be considered in the future AI-based CBPHC studies to avoid potential biases in the AI systems.

Less than one quarter of included studies have reported patient participants’ ethnicities, with no discussion on the ethnicities of participating health care providers. Moreover, for those studies that reported patient ethnicity, we observed that the collected data were related to causation populations, thus raising questions regarding the representativeness of the data set, leading to biases. Such biases could result in the AI system making predictions that discriminate against marginalized and vulnerable patient populations, ultimately leading to undesirable patient outcomes.

According to our results, most of the AI research in CBPHC has taken place in North American and Europe-centric settings. Several factors contribute to ethnoracial biases when using AI, including not accounting for ethnoracial information, thereby ignoring the different effects illnesses can have on different populations [113]. Consequently, studies can yield results with historical biases as well as biases related to over- or under-representation of population characteristics in data sets and in the knowledge, bases used to build AI systems. In turn, stereotypes and undesirable outcomes may be amplified. Ensuring ethnic diversity in study populations and accounting for this diversity in analyses is an imperative for developing AI systems that result in equitable CBPHC.

Despite the many potential benefits of AI to humans, the development of AI systems is often based on “technology-centered” design approaches instead of "human-centered" approaches [114]. Our results indicate that no AI-CBPHC study has involved any end users in the system development stage and involving primary health care professional users during the validation or testing stages has been rare. This results in AI systems that do not meet the needs of health care providers and patients; they suffer from poor usage scenarios and eventually fail during implementation in clinical practice. A recent assessment of the current user-centered design methods showed that most of the existing user-centered design methods were primarily created for non-AI systems and do not effectively address the unique issues in AI systems [115]. Further efforts are needed to include health care providers and patients as users of the developed AI systems in the design, development, validation, and implementation stages in CBPHC. Nevertheless, effectively involving these users in the development, testing, and validation of AI systems remains a challenge; further studies are required to overcome them.

Ethical and legal challenges related to the use of AI in health care include, but are not limited to, informed consent to use AI, safety and transparency of personal data, algorithmic fairness, influenced by the aforementioned biases, liability, data protection, and data privacy. Our results indicate that ethical and legal aspects have rarely been addressed in AI-CBPHC research, except with respect to privacy and data security issues. There is a need to address all legal and ethical aspects and considerations within AI-CBPHC studies to facilitate implementation of AI in CBPHC settings. For instance, to increase the use of AI systems by CBPHC providers, clarifying scenarios in which informed consent is required could be useful, as would clarifying providers’ responsibilities regarding the use of AI systems. To improve patient outcomes related to AI use in CBPHC, defining the responsibilities of providers and researchers regarding the development and implementation of AI-health literacy programs for patients may be necessary, together with gaining an understanding of how and when patients need to be informed about the results that AI systems yield.

AI systems can provide solutions to rising health care costs; however, only one (1%) AI-CBPHC study has addressed this issue by conducting a cost-effectiveness analysis of AI use. This is consistent with other study results showing that the cost-effectiveness of using AI in health care is rarely and inadequately reported [116,117]. Thus, further research analyzing cost-effectiveness is needed for identifying the economic benefits of AI in CBPHC in terms of treatment, time and resource management, and mitigation of human error; this would be valuable as it could influence decisions for or against implementing AI in CBPHC.

Our results show different barriers and facilitators for implementing AI in clinical practice. Aspects related to the data were among mostly mentioned ones. For instance, the lack of high amounts of quality data, specifically when using modern AI methods (eg, deep learning), is a challenge commonly faced when developing AI systems for use in CBPHC. The promotion of AI-driven innovation in any setting, including CBPHC, is closely linked to data governance, open data directives, and other data initiatives, as they help to establish trustworthy mechanisms and services for sharing, reusing, and pooling data [118] that are required for the development of high-quality data-driven AI systems.

In addition, some data security and privacy laws can create a bottleneck, limiting the use of AI systems in CBPHC and the sharing of health care information that is required for developing high- performance AI systems. To facilitate the implementation and adoption of high-quality AI systems in CBPHC and ensuring benefits to patients, providers and the health care system, research providing insights for addressing these implementation challenges is needed.

Our review has some limitations. Firstly, given that we used the Canadian Institute of Health Research’s definition of CBPHC to determine our inclusion criteria and given that the definition of CBPHC differs from one country to another, our search strategy may not have captured all relevant records. Secondly, we excluded studies conducted in emergency care settings. In many countries, emergency departments are the points of access to community-based care. The European Commission recently released a legal framework (risk-based approach) for broad AI governance among EU member states [118] and categorized emergency care and first aid services as “high risk.” Requirements of high-quality data, documentation and traceability, transparency, human oversight, and model accuracy and robustness are cited as being strictly necessary to mitigate the risks in these settings [118].

In this systematic scoping review, we have demonstrated the extent and variety of AI systems being tested and implemented in CBPHC, critically evaluated these AI systems, showed that this field is growing exponentially, and exposed knowledge gaps that remain and that should be prioritized in future studies.