The protocol of this scoping review was registered in Open Science Framework (OSF) [15].

Searches were conducted on August 13, 2020, in MEDLINE (PubMed), Web of Science, PsycInfo, and CINAHL databases. No time window was applied. Three search strings covering the topics “social robots,” “community setting,” and “elderly people” were constructed. For each database, reverent subject headings were adapted. For MEDLINE, we used the following strings and keywords: ((robotics[MeSH Terms] OR robot*)) AND ((humanoid OR companion OR social* OR “socially assistive” OR interact* OR android)), (((((((aging[MeSH Terms]) OR (aged[MeSH Terms])) OR (elderly[MeSH Terms])) OR (vulnerable population[MeSH Terms])) OR (senior)) OR (ageing)) OR (geriatric)) OR (old*), and (((((community health service[MeSH Terms]) OR (social support[MeSH Terms])) OR (residential facilities[MeSH Terms])) OR (independent living[MeSH Terms])) OR (social support[MeSH Major Topic])) OR (“community dwelling” OR “home dwelling” OR “care home” OR “in-home” OR “at home” OR “home-based” OR “home setting” OR “nursing home” OR home).

Publications potentially eligible for inclusion had to study a social robot that was physically embedded in an experimental or clinical study in people aged 65 or above. Studies were excluded if they were (1) conducted in an acute care setting; (2) conference abstracts, case studies, dissertations, books, or review papers; (3) published in a language other than English or Spanish; (4) solely reporting a robot design, development process, or theoretical models (5) stakeholder opinions on robots without any interaction; (6) involved in the implementation of new hardware or software or an assessment tool on a robot (such as assessing a fall detection sensor); and (7) involving telepresence robots with only video call features.

After the removal of duplicates, 2 reviewers (authors AM and MM) independently applied the inclusion and exclusion criteria in 3 steps, starting with screening titles, abstracts, and then full texts. The selections were compared, and in case of disagreement, discussed by the 2 reviewers. In cases where no consensus could be reached, a third reviewer was consulted (author HR).

The literature was mapped according to the following areas of interest: author and country, robot name, the aim of the robot, aim of the study, type of outcome measure, study design, study sample, study setting, methodology of data collection, interaction scenario, relevant outcome measures, measurement instruments, results, and reported limitations of the study. Information was synthesized descriptively, and findings were narratively summarized according to the areas of interest.

The quality of the included articles was appraised independently by 2 authors (AM and MM), through the quality assessment of digital health interventions within the Monitoring and Evaluating Digital Health Interventions framework established by the World Health Organization (WHO) [14]. The tool includes a list of methodological study criteria comprising (1) 23 essential criteria for all types of studies and (2) essential criteria for qualitative and quantitative studies (3 criteria each). The extent to which criteria were met by studies was rated by 3 independent researchers on a 3-point scale (0= poor, 1= fair, 2= good). We calculated the percentage of agreement between the ratings (Multimedia Appendix 1). We also applied this framework to categorize the studies according to their aims in 4 categories: (1) feasibility, (2) usability, (3) efficacy, and (4) effectiveness.

The search resulted in a total of 723 individual publications. After the screening process, 33 articles met the selection criteria (Figure 1). In 33 papers [11,16-45], 23 different social robots were evaluated among elderly adults and people with dementia in 13 different countries. Moreover, 19 studies specifically evaluated either feasibility, usability, efficacy, or effectiveness and were considered as single aim studies. The remaining studies (n=14) had multiple aims, evaluating 2 or 3 of the aforementioned study aims. Overall, feasibility was studied in 17 (51.5%) studies, usability in 13 (39.3%), effectiveness in 12 (36.3%), and efficacy in 10 (30.3%).

The quality appraisal identified that primary and secondary outcomes were clearly defined in all studies. Additionally, the methods of data collection were described well, but the eligibility of the participants was not reported in 12 (36.4%) papers. Moreover, 12 out of 33 (36.4%) papers did not present a clear description of the study design. Multimedia Appendices 2 and 3 show a summary of the characteristics, methodologies, and outcomes of the included studies.

Of the 23 different social robots, PARO (n=10, 30.3%), Nao (n=5, 15.1%), AIBO (n=2, 6%), and Hobbit (n=4, 12.1%) were the most often investigated.

Of the 33 identified studies, 16 (48.4%) focused on people with dementia [11,18,23,24,27,29-35,38,39,43,44], 9 (27.3%) were performed in samples of residents of long-term care facilities whose cognitive status was not mentioned [17,20,22,26,28,36,40,41,45], and the remaining 8 (24.2%) focused on elderly people living in the community whose cognitive status was also not clearly revealed [16,19,22,25,37,42,46,47]. Moreover, 4 (12.1%) studies additionally recruited care staff [22,24,26,38]. The age range of older adults was 65-98 years. Of the included studies, 3 (9.1%) did not report the number of participants [17,20,26]. The sample sizes used in the studies ranged from 5 to 139.

The social robots were studied in long-term facilities (n=17, 51.5%), private households (n=8, 24.2%), and laboratory settings (n=4, 12.1%). Additional settings were based in a care organization (n=1, 3%), a daycare center for dementia (n=1, 3%), and a health service facility (n=1, 3%). Four (12.1%) studies investigated the robots in 2 different settings [22,33,43,46].

Only 5 (27.8%) of the 18 studies aiming to evaluate feasibility and/or usability were performed in nursing home settings; 5 (27.8%) were performed in laboratory settings, and the remaining 8 (44.4%) were performed in private households. In 7 (38.9%) of the 18 studies, people with dementia and those with cognitive impairment were included. In the remaining 11 (61.1%) studies, the cognitive status of the participants was not clearly indicated.

Of the 22 (66.7%) studies that focused on efficacy or effectiveness, all but 4 (81.8%) [29,33,39,47] were performed in long-term care settings. These 4 (18.2%) were performed in private households and a daycare facility. Of these studies, 13 (59.1%) included cognitively impaired samples, only 1 (4.5%) study included community-dwelling elderly persons without disclosing their cognitive status, and the remaining 8 (36.4%) included long-term care residents.

Interaction between study participants and social robots was mostly investigated during individual sessions (n=18, 54.5%). In 12 (36.4%) studies, interactions were studied in group sessions. Only 3 (9.1%) studies applied both individual and group interactions [11,20,33], while 1 (3%) demonstrated the task performance of the robot without any close interaction with study participants [19]. Feasibility and/or usability (n=11, 33.3%) were mostly studied in individual settings (n = 8, 72.7%); 1 (9.1%) study was performed in a group, and 1 (9.1%) was applied to both individual and group settings. Individual (n=7, 21.2%) and group settings (n=6, 18.2%) were used most often to study efficacy and effectiveness (n=15, 45.5%); 2 (6.1%) studies applied the intervention individually and in a group. In studies with multiple aims, 4 (28.6%) individual and 2 (14.3%) group setting interventions were found. In 2 (6.1%) studies, social robots were available in the residents’ lounge in nursing homes, and participants were free to interact with the social robots during scheduled time slots [22,45]. In 8 (24.2%) studies, the robots were installed in participants’ private homes for a duration of 5 days to 3 months [16,27,29,33,37,39,44,47].

A total of 9 (27.3%) studies executed 1 or 2 interactive sessions [11,19,22-24,28,30,32,34], of which 6 (66.7%) investigated the usability and feasibility of the robot, 1 (11.1%) investigated effectiveness, and 2 (22.2%) investigated efficacy. Most of the studies conducted more than 2 interactive sessions: 5 (15.2%) studied feasibility and/or usability, 12 (36.4%) studied efficacy or effectiveness, and 8 (24.2%) were multiple aim studies. The interactive sessions ran from 10 to 90 minutes a day for a maximum of 4 months.

We identified 4 methods of data collection: (1) questionnaires (n=26, 78.8%), (2) observations (physical and videotape) (n=19, 57.6%), (3) interviews (n=13, 39.4%), and physiological measurements (n=3, 9.1%). Figures 2 and 3 show the data collection methods and the responsible administrator of data for the identified data collection methods, respectively.

Outcomes regarding feasibility were assessed with the Unified Theory of the Acceptance and Use of Technology (UTAUT) model adapted by Heerink [11,19,23,29,30,46,48], the Robot User Acceptance Scale, the Robot Attitude Scale, the Mind Perception Scale [21,47], and the Negative Attitudes Toward Robots Scale [16,37]. Of the included studies, 5 (15.2%) utilized questionnaires regarding robot functions and acceptance that were specifically developed for the study [19,22,23,25,46].

Studies exploring usability applied the System Usability Scale [22,25,28], a modification of the Usefulness, Satisfaction, and Ease of Use [26] scale, and the Technology Usage Inventory [39]. Two (6.1%) qualitative studies performed conversation and video analysis [27,28] to extract statements on acceptability and usability.

Efficacy and effectiveness outcomes were evaluated by a wide range of neuropsychosocial measurement instruments: (1) mood: Geriatric Depression Screening [49], the Cornell Scale For Depression in Dementia [50], Apparent Emotion Rating Instrument [51], University of California, Los Angeles (UCLA) Loneliness Scale [52], Observed Emotion Rating Scale [53]; (2) cognition: Montreal Cognitive Assessment [54], Mini-Mental State Examination [55]; (3) QoL: QoL Alzheimer Disease, Dementia QoL Questionnaire [56], QoL in Late-Stage Dementia [57]; (4) behavior: Neuropsychiatric Inventory [58], Gottfries-Bråne-Steen Scale [59], Apathy Evaluation Scale [60], Cohen-Mansfield Agitation Inventory [61], Apathy Inventory [62], Apathy Scale for Institutionalized People With Dementia Nursing Home Version [63]; (5) Participation and Interaction: Activity Participation Scale [40], and Assessments of Communication and Interaction Skills [64].

Among the studies applying questionnaires, 5 (19.2%) indirectly collected data via care staff and informal caregivers, and 13 (50%) directly collected data via the researchers.

Concerning social robots’ feasibility outcomes, almost all studies (n=16, 94.1%) deemed social robots acceptable. Nevertheless, 1 (5.9%) study reported mixed results on acceptability by care staff [22], and 1 (5.9%) did not find any significant results on quantitative measurements for acceptability, but qualitative results were positive [16]. In 3 (17.6%) studies, the perceived agency [21,47] and perceived enjoyment [46] were found to decrease over time.

The reported usability (n=12, 36.4%) was overall positive, except in 2 (16.7%) studies in which the usability was negatively affected by technical issues or lack of robustness of the robots [28,37]. Only 3 (9.1%) studies assessed affordability for Hobbit and Nao, in which the participants did not consider the social robots affordable and were skeptical of buying them [25,28,37].

Most of the findings endorse the use of social robots by older adults. Improvements were mostly found in emotion and mood [20,31,33-35,38,44], engagement [24,29,30], and participation and social interaction [20,31-33,40,45]. Increased job satisfaction of staff [22], self-report pain reduction [38], and improved global psychiatric symptoms [43] were the other positive study outcomes. There were findings of reduced challenging behavior [20,31,34], sense of loneliness [17], and stress levels [45]. However, dementia symptoms like agitation and other problematic symptoms did not improve in 1 (3%) study [33].

Meanwhile, 4 (12.1%) studies did not find a significant impact on QoL [19,21,43,47]. Only 1 (3%) study found a moderate-to-large positive effect on QoL of people with dementia [35]. Social robot interventions also failed to significantly improve depression [18,21,43,47], perceived social support [18], medication adherence [47], and cognition [24]. There were mixed results regarding physiological measures, such as urine tests measuring stress levels and blood pressure [33,45]. No author declared a proven negative effect of social robots on older adults.

Of the 33 studies, 7 (21.2%) did not report any study limitations [16,17,23,25,30,42,46]. A wide range of limitations was reported, and the most common barrier considered in 17 (51.5%) studies was the small sample size, which was mostly reported for efficacy and effectiveness studies. In the feasibility and/or usability studies, the limitations were mainly attributed to technical problems or interaction difficulties. The use of unvalidated questionnaires, homogeneity in the sex of the study sample, inadequate observation, and short duration of interventions were reported as other limitations in general.

The inter-rater agreement for the quality appraisal was 86.1%. Reports of the description of study design, bias, and enrollment procedure were mostly rated as “fair.” In most of the articles, the sampling methods, confounding factors, missing data in quantitative studies, and reflexivity of data interpretation in qualitative studies were poorly reported. Other criteria were mostly rated as “good” (Multimedia Appendix 1). The quality appraisal revealed unclear descriptions or insufficient details in 5 (15.2%) studies, especially those in disciplines other than health research [25,30,42,44,45].

The results of this scoping review revealed a variety of applied study methods in studies with social robots concerning study design, sample size, study setting, method of data collection, interaction scenario (the sequence, duration, and setting of the intervention), outcome measures, measurement instruments, study results, and reported limitations. Feasibility and usability were mainly studied on preprototype social robots in laboratory settings. Considering the relatively short history of the use of social robots in psychosocial interventions, it is crucial to determine the main features and functions of the robots to be considered in the design and development phase. Hence, usability, feasibility, and implementation should be strategic research aims. Fully developed robots such as PARO were evaluated in terms of effectiveness in real-world settings. Most of the identified studies aimed to determine the neuropsychosocial impact of social robots on older adults.

For the studies that explicitly fall within a feasibility and/or usability evaluation, researchers applied experimental, mixed method, and field trial designs, mostly applied outside nursing home care settings. This might imply that feasibility and/or usability for persons that are more severely cognitively impaired are currently understudied. Most of the studies verified the acceptability and usability of the robots within single or multiple interactive sessions in individual or group settings, and all these studies reported positive outcomes in varying degrees on the feasibility and/or usability of the social robots. The quantitative and qualitative data were collected mostly through questionnaires and interviews and a few by direct observation. Regarding this point, researchers should consider using the direct observational methodology to capture main factors of the interaction and emotional relationships fostered by robot use. Within the questionnaires and interview questions based on the UTAUT model, some concepts such as trust, anxiety, perceived enjoyment, and social support can change over time [37,46,47]. Therefore, longer use of the robots might reveal these changes and reduce the novelty effect over time [46].

Overall, efficacy and effectiveness studies were conducted on study populations either with cognitive impairment or residing in long-term care facilities. The studies with significant results [17,24,29-31,34-36,45] mostly employed experimental designs including RCTs and quasi-experimental designs with larger sample sizes and longer intervention periods compared to studies showing slight or no improvements. RCTs are likely to be the most appropriate design and a gold standard to confidently demonstrate that a specific intervention has resulted in a change in a process or a health outcome [14]. Biased assessment of outcomes and any confounding effects can be avoidable by large-scale RCTs. However, due to the degenerative character of dementia and personal differences in capacities of people with dementia, difficulties in randomizing subjects often arise [14]. Additionally, when using long study periods, the dropout rate might be high, as participants’ cognitive deterioration can hinder their continued participation in the study. On the other hand, when it is not feasible or ethical to conduct an RCT, a quasi-experimental design may serve best for collecting quantitative data. We also recommend randomized controlled block designs in the case of heterogeneous study samples. When, for instance, including people with dementia in studies with long intervention periods, the dementia levels alter. With a randomized controlled block design, some variables in different blocks can be controlled for, or comparable approaches can be applied within the blocks.

Studies targeting the efficacy and effectiveness of the robots delivered interventions diverged in format, duration, dosage, and location. Two (6.1%) studies [32,38] highlighted a need for individual intervention sessions tailored to users’ preferences and capacities, and the findings of another study confirmed this approach [65]. Additionally, individualized sessions could omit confounding factors by reducing the chance of interactions with the facilitator or other participants [66]. Group interventions may lower the odds of interaction between potential users and the robot, potentially lowering the effect of the intervention, especially when the intervention is delivered in a noisy setting with many participants [18]. The issues of small sample sizes and short interactions were considered major limitations in studies that failed to find significant results, and they are considered the toughest challenges for researchers in this field [66,67]. These limitations are often cojoined with the study setting. In nursing homes, a larger number of residents are often enrolled in a clinical trial, and the robots are not personalized but must be shared by the entire group. Whereas in private homes, studies are conducted with individuals or dyads, which creates better possibilities for personalization of the robot. Overall, the personalization of the intervention and alleviation of loneliness are 2 advantages of home-based clinical trials. However, there are some challenges to these types of studies, such as the need for several robots, implementation difficulty, and personalization, but it is nevertheless a step in the right direction. We observe a paradox, in that new or experimental robots are employed in research with low numbers of participants, whereas commercially available robots are tested on large study samples. Commercialization allows for better testing and evaluation, which is logical from an economical perspective. However, we urge that before robots are marketed, developers should study the feasibility and usability appropriately in the target group, as well as the effectiveness in a substantial study sample with sufficient power. After bringing the robot to the market, producers should continue to invest in studies to improve their product to tailor it optimally to their users. This should be a joined mission of producers and policy makers to improve the sustainability of health care systems.

Apart from the aforementioned limitations of the studies, some weak aspects of the study designs led to failure of the social robots’ impacts. For instance, a mismatch between the studied construct and the main aim of the robot may lead to the poor conclusion that the robot is not efficient. An example of this is the studies on PARO that failed to demonstrate significant results for cognition, as PARO is not developed to stimulate cognitive functioning [31,33,41,43]. Additionally, to capture significant results in constructs such as cognition, QoL, and depression, a long intervention period is necessary because these are constructs that do not change very quickly. In studies with people with dementia, it might also be useful to study stability of these constructs instead of improvement, since it is inherent to the disease that these constructs deteriorate over time. Regarding the broad concept of QoL ranging from physical health to psychological state and social relationships, the application of a suitable QoL measurement instrument that corresponds to the robot’s aim should be taken into account.

Appropriate RCTs with large sample sizes and individual interaction sessions running for longer than 1 month would serve best for such studies to draw relatively robust and reliable results.

The study methods are similar for both aims, so researchers could apply the same design. Experimental designs with mixed methods of data collection are recommended for these studies. Multiple interaction sessions might reveal the changes in feasibility and usability.

We recommend performing separate studies for multiple aims since the study designs for each aim are comparable.

We gathered further practical recommendations through which future work may address existing shortcomings (Table 1). Regarding the mixed methods of data collection, studies suggest a combination of qualitative and quantitative methods for data collection, which will enable the researcher to capture different details in users’ responses and address different aspects of the research question. A mixed methods approach was helpful in studies that could not derive positive results from quantitative data but did from qualitative data [16]. Regarding the difficulties of recruiting many users in case of availability of just a few robots, these mixed methods should be mandatory. Even though we did not find any negative results regarding the intervention dosage, there are shreds of evidence of highly intense intervention resulting in negative effects or exhaustion [18]. Hence, the dose response for specific measures remain an open question for future researchers.

Although the use of social robots is promising to support people with dementia, we did not include dementia specifically in the search strings, since this scoping review focused on elderly people in general. However, we believe that our search captured most of the studies executed in the field of dementia because many of the identified studies included people with dementia in either mixed or specific study samples. However, some relevant studies on elderly people with dementia may be missing in this review, as well as may studies that are only traceable in databases that were not taken into account in this review. The searches were conducted in scientific databases deemed the most viable to retrieve valid and reliable results for this scoping review. The exclusion of studies focusing only on the development phase of social robots can be considered a limitation of this study. Some information on the evaluation of the feasibility and usability executed in the development stage might have been missed. In addition, studies on telepresence robots were excluded due to their relatively simple features. Compared to pet robots and humanoid robots, telepresence robots are limited in interactions with users, which occur merely through a touch screen, making use of visual and audio stimuli but omitting other sensory stimulation. Although mainly used for medical visits, some telepresence robots might support social functioning. Information on studies performed on these robots might have been missed.

Our study is the first scoping review on the methodologies for studying social robots in elderly people and people with dementia. The existing reviews on this topic mostly focus on design, use, effectiveness, facilitators, and barriers to the implementation of social robots [12,66,67,68-73]. This study might support future researchers to design a research study on social robots in elderly adults and answer some study design queries.

This review narratively synthesizes information on the methodology of studying social robots in elderly adults and people with dementia. Relevant recommendations were formulated, directed for studies with specific aims that may aid future researchers in developing adequate study designs to evaluate social robots, allowing for more reliable information on study outcomes. Our research leads us to the conclusion that more studies with large sample sizes are needed on the effectiveness of social robots in private households of elderly adults and people with dementia to demonstrate the actual usefulness of social robots on delaying institutionalization by improving QoL, cognition, and social contact, and counteracting loneliness. Most of the identified studies focused on usability, and the robots appeared to be favorably accepted in most cases. It is time to encourage investigations in private homes to supplement existing knowledge about the effectiveness of robots and personalization of their functions. We expect that additional research will corroborate the impact of social robots on loneliness, mood, QoL, and social health in people with dementia and the elderly.