German Version of the Mobile Agnew Relationship Measure: Translation and Validation Study

Introduction

According to the World Health Organization, mental health conditions are increasing globally, with an estimated 20% of children and adolescents having a mental health condition and suicide being the second-leading cause of death among those aged 15-29 years [1]. In order to address this increasing need, the digitalization of psychotherapeutic interventions and mental health-related monitoring holds enormous potential to be implemented either in addition to clinical interventions or as a substitute altogether. Unlike traditional therapeutic methods, digital implementations may be realized on a much larger scale as they are not restricted by potentially limiting factors such as time constraints, availability of therapists or clinicians, monetary considerations, or fear of stigmatization. In addition, mobile mental health interventions can be applied in everyday situations [2].

Yet, despite the increase in commercial app development in the field of mental health, which has resulted in currently more than 10,000 mental health apps available in the market [3], there has been a striking lack of empirical studies investigating the effectiveness of these novel apps [4], leading to serious concern about their scientific credibility. More research is therefore required to evaluate the effectiveness and suitability of these mental health apps. One of the key aims in this context has been to identify factors contributing to the potential success and effectiveness of mental health apps.

Therapeutic alliance (TA) refers to the working relationship between client and therapist [5]. In face-to-face therapeutic settings, TA has been suggested to be an important contributor to therapeutic success [6]. For example, a recent systematic review by Baier et al [7] found TA to mediate positive outcomes by over 70%. Moreover, a longitudinal study reported patients’ and therapists’ positive perception of TA to predict clinical improvement in patients’ depressive outcomes [8], and another study reported TA to have a strong association with youth mental health and addiction treatment outcomes [9], which has also been found for drug abuse treatment altogether [10].

TA has been increasingly researched for digital therapy with video therapy [11,12] and text-based therapy [13]; however, questions still pertain as to whether it is possible to build a relationship between a user and an app in the context of a mobile health (mHealth) intervention, and if so, what would be the nature of such a rapport? The findings have so far been mixed. Some research found associations between digital TA and treatment outcomes using fully automated smartphone apps [14] while other studies reported a lack of such association [15]. As such, there is currently no consensus regarding the nature and potential impact of digital TA on treatment outcomes [16].

One of the limiting factors that has prevented adequate investigations in this regard has been the lack of suitable measures to assess digital TA [16]. Some authors used measures from traditional face-to-face therapy [15], whereas others used altered versions of these instruments [17]. However, several reviews noted the need to substantially revise and adopt new measures to adequately measure the nature of TA for app-based interventions [16,18]. To address this, the Agnew Relationship Measure (ARM)—a common tool used to assess TA in traditional face-to-face settings [19]—has been modified into the mobile Agnew Relationship Measure (mARM) to measure TA in digital settings [20]. The mARM consists of 25 items that are rated on a 7-point Likert scale, ranging from (1) strongly disagree to (7) strongly agree. Like the ARM, the mARM is built upon five key concepts of TA, which include (1) the bond between the app and user; (2) the partnership or collaboration between user and app; (3) the confidence the user holds in the competency of the app; (4) the openness or freedom the user feels to personally disclose; and (5) the client or user initiative, which refers to feelings of control and empowerment within the relationship.

The mARM has been discussed in several reviews regarding digital innovation in psychological practice [18,21-24], as well as being the focal point of a recent paper on human-computer interaction [25].

To date, the mARM is only available in English, and the aim of this study was to translate the mARM into German and consequently validate the new version so that it can be used for research in German-speaking countries.

Methods

Translation and Adaptation Process

The translation process was modeled after the guidelines by Beaton et al [26]. After having obtained permission to translate the mARM from its first author, the items of the mARM were forward translated from English into German separately by 2 researchers who are native German speakers and who speak English as their second language. These 2 versions of the German translation were then compared and harmonized by another researcher. The resulting version was then back translated into English by 2 native German speakers with a fluent knowledge of English. Following this, an impartial judge who is bilingual in German and English reviewed the versions and created a final mARM German version (mARM-G; Figure 1).

‎

Validation and Reliability Testing and Data Analysis

The mARM-G underwent a validation process that included testing for content validity, face validity, and reliability. Content validity aims to measure the relevancy of each item for TA. This was assessed according to the content validity index (CVI) [27]. The relevancy of each item of the mARM-G was measured with regard to the TA, conducted on a 4-point Likert scale ranging from 1 (not relevant) to 4 (very relevant). First, average ratings and SDs for all answers were computed to provide a general overview. Then, in order to calculate the CVI, all scores of 3 to 4 were categorized as relevant (1), and all scores of 1 and 2 were categorized as not relevant (0). The CVI was calculated for each item (item-CVI) and for the entire scale by averaging the scores of all items.

Face validity aims to measure the comprehensibility and clarity of each translated item of the mARM-G. This was done using a 4-point Likert scale ranging from 1 (not clear) to 4 (very clear). Average ratings for all answers as well as SDs were calculated. The face validity index (FVI) was then computed by coding all scores of 3 to 4 as clear (1), and all scores of 1 and 2 as not clear (0). The FVI was then calculated for each item (item-FVI) and for the entire scale by averaging the scores of all items.

Reliability testing was performed by measuring the internal consistency of the items of the mARM-G via Cronbach α. Sample size was estimated according to the method provided by Bonett [28] with the use of a web-based sample size calculator [29]. The Cronbach α was set to .7 with a precision of 0.1 and a 95% CI. The minimal required sample size was 77 participants. Finally, confirmatory factor analysis was conducted using the lavaan package in R (Ghent University) [30] and visualized using the semPlot function [31]. For this, the sample size was estimated for root mean square error of approximation (RMSEA) based on the resources by Preacher and Coffman [32]. Parameters set with α level of .05, df of 265, a null hypothesized RMSEA value of 0.05, and an alternative RMSEA value of 0.07. Based on this, a sample size of 130 participants was required to reach a power of 0.80. All analyses were performed using R in RStudio (version 4.1.1; R Core Team) [33].

Participants

The 15 experts who conducted the content and face validation were all native German speakers living in Germany and the German-speaking part of Switzerland. The group consisted of clinical psychologists (n=7, 47%) who conduct psychological research as well as scientists (n=8, 53%) specializing in psychology research. This included 6 (40%) psychologists at the doctoral level, 5 (33%) at the postdoctoral level, and 4 (27%) full professors. A total of 11 (73%) experts were female and 4 (27%) were male, with a mean age of 35 (SD 9.2) years.

The 15 nonexperts who conducted content and face validation were all native German speakers living in Germany (n=11, 73%), the German-speaking part of Switzerland (n=3, 20%), and the United Kingdom (n=1, 7%). A total of 5 (33%) were female, 2 (13%) preferred not to reveal their gender, and 8 (53%) were male. The mean age was 27.53 (SD 9.7) years.

During the study, participants were using the mental health app—ReApp [34]. ReApp is based on principles of cognitive behavioral therapy and reappraisal. Full development protocol can be found elsewhere [35]. The participants used the app for 21 days before completing the mARM-G.

Ethical Considerations

The reliability analysis was based on 144 participants. The Ethics Committee for the Faculty of Arts and Social Sciences of the University of Zurich approved the study proposal (21.2.12). All participants were students and spoke fluent German, and the exclusion criteria were having a mental illness as reported by the participant and attending psychotherapy or using other kinds of support from a qualified psychologist or psychiatrist. All participants provided written informed consent to participate in the study and agreed to data analysis by researchers from the University of Zurich. To ensure the privacy of the participants, they were provided with their own personal code to use throughout the study. Participants were remunerated up to 105 Swiss francs (around US $114) or 6 university credit points to compensate for their contribution to the study.

Results

For the expert group, the overall average of the relevancy rating was 3.52 (SD 0.29; range 1-4). The CVI average for the scale was 0.9 (SD 0.12). The overall average of face validity or clarity ratings was 3.6 (SD 0.35; range 1-4). The FVI average for the entire scale was 0.89 (SD 0.15). For all ratings, see Table 1. All items with scores below 0.78 for the FVI and CVI were further inspected, and adjustments were made to make the translated items clearer and more relevant (Multimedia Appendices 1 and 2).

For the nonexperts group, the overall average of the item relevancy rating was 3.15 (SD 0.22; range 1-4). The CVI average for the scale was 0.79 (SD 0.12). The overall average of face validity or clarity ratings was 3.44 (SD 0.25; range 1-4). The FVI average for the entire scale was 0.86 (SD 0.09). For all ratings, see Table 2. All items with scores below 0.78 for the FVI and CVI were further inspected, and adjustments were made to make the translated items clearer and more relevant. The full list of items of the original mARM and the translated mARM-G can be found in the Multimedia Appendix 1.

Cronbach α for the mARM-G was calculated first for the entire scale which was .91 (95% CI 0.88-0.94) and indicated excellent internal consistency. The value of Cronbach α for a whole scale, if an item is removed, remained highly consistent without any considerable difference (Table 3).

Cronbach α values were also computed for the specific subscales. These were taken from the description of the ARM [19] and were constructed of bond (items in the m-ARM-G: 2, 12, 14, 15, and 17; α=.76); confidence (items: 5, 6, 8, 11, 13, 16, and 19; α=.83); openness (items: 1, 4, 7, and 9; α=.61); partnership (items: 18, 21, 23, and 24; α=.77); and client initiative (items: 3, 10, 20, 22, and 25; α=.42) subscales. For the client initiative subscale, item 20 (“I am responsible for my recovery, not the app”) proved to be negatively correlated with the scale. The removal of this item from the mARM-G increased the α for the subscale to .55 (Table 4).

To further inspect the underlying structure of the translated measure, a confirmatory factor analysis was conducted. The chi-square statistic (χ²₂₆₅=440.78, P<.001) was highly significant. The chi-square statistic to df ratio was 1.66. The Comparative Fit Index (CFI) was 0.87 and the Tucker-Lewis Index (TLI) was 0.86. The RMSEA was 0.07. The factor loading for item 20 was negative, and as such, it was removed from the measure (Figure 2).

‎

Discussion

Principal Findings

TA refers to the working relationship between therapist and client [6] and has been found repeatedly to predict therapeutic outcomes [7-10]. Traditionally, TA has been widely assessed using the ARM [19]. With the digitalization of therapeutic practices, novel measures for TAs are required and there is an increased need to culturally adapt measures [21]. In this study, a German version of the mARM was developed and validated.

The results indicated high content validity with regard to relevance for the TA for the entire measure. Moreover, the translated items also proved to be easy to understand, with high ratings for the comprehensibility of the entire measure. Small adjustments were made upon suggestion, aiming to improve the comprehensibility of individual items. The study conducted reliability testing via Cronbach α. The entire scale had excellent internal consistency with α=.91. In addition, α values for the specific subscales were calculated as this is common practice for longer measures [36]. The subscales were taken from the original ARM [19]. We found good results for the bond, confidence, and partnership subscales (0.76-0.83), as well as close to acceptable results for the openness subscale (0.61). The client initiative subscale had poor results (0.42). These were further inspected, and item 20 (“I am responsible for my recovery, not the app”) was found to be negatively correlated with the overall scale and subscale. If removed, the α of the subscale increases to a value of .55, which equals the α results (.55) from the ARM [19]. The results are in line with the previous research using the ARM, which has reported ranges from good to acceptable results for the bond, openness, confidence, and partnership subscales; however, it has low internal consistency for the client initiative subscale [19,37].

To further inspect the underlying structure of the translated measure, a confirmatory factor analysis was conducted which revealed mixed results. The chi-square statistic (χ²₂₆₅=440.78, P<.001) was highly significant and indicative of poor fit. The chi-square statistic to df ratio was 1.66, which indicates a good fit [38]. The CFI was 0.87 and the TLI was 0.86, which are below the suggested value of 0.90 [39]. The RMSEA was 0.07, which is acceptable [40]. The factor loading for item 20 was negative, and thus, the item was removed from the measure. We advise future researchers who use the mARM to evaluate the relationship between this item, the concept of client initiative, and TA overall with caution. Moreover, 6 items had factor loadings of <0.5 (items: 3, 4, 9, 12, 16, and 25). These items were further inspected; however, we ultimately decided against their removal since it would result in the openness and client initiative subscales being composed of only 2 remaining items.

Limitations and Recommendations for Future Studies

This study had limitations. Reliability testing in this study was conducted with a group of students. Ratings for TA may thus be influenced by prior familiarity with mobile app use and overall technological familiarity. Moreover, this study used only 1 mobile app. Further research could therefore test the reliability of the measure with different apps, as well as using a sample of both younger and older participants with various socioeconomic backgrounds and various levels of experience with technology and mobile apps. This could further expand the literature on the generalization of TA measures in the broader population.

As there are still open questions concerning the nature and relation between digital TA and various treatment outcomes, future research could use the current scale to test associations between engagement and treatment outcomes, using both the overall scores of the mARM as well as the specific subscales. Further, obtaining more qualitative data concerning the nature of digital TA may yield novel insights.

Finally, additional studies comparing the effect of digital TA on treatment outcomes are needed for different therapeutic strategies, for example, for cognitive behavioral therapy, psychodynamic therapy, and acceptance-commitment therapy, among others, to gain a more nuanced understanding of the role of digital TA in various psychotherapy processes.

Conclusions

With an increasing number of mHealth apps being developed and the need for research into the effectiveness and suitability of these novel innovations rising, the mARM-G presents an accurate, easily accessible, scalable, and low-cost measure to conduct further research in this novel area. This translation may prove to be of use since there are estimated to be more than 100 million native German speakers living across Germany, Austria, Switzerland, Belgium, Luxembourg, and Lichtenstein for whom this measure can be used [41].

The mARM-G and instructions are presented in Multimedia Appendix 2. We encourage researchers to use it freely for research and noncommercial evaluation of the TA between humans and mental health mobile apps.