The translation process was modeled after the guidelines by Beaton et al [26]. After having obtained permission to translate the mARM from its first author, the items of the mARM were forward translated from English into German separately by 2 researchers who are native German speakers and who speak English as their second language. These 2 versions of the German translation were then compared and harmonized by another researcher. The resulting version was then back translated into English by 2 native German speakers with a fluent knowledge of English. Following this, an impartial judge who is bilingual in German and English reviewed the versions and created a final mARM German version (mARM-G; Figure 1).

The mARM-G underwent a validation process that included testing for content validity, face validity, and reliability. Content validity aims to measure the relevancy of each item for TA. This was assessed according to the content validity index (CVI) [27]. The relevancy of each item of the mARM-G was measured with regard to the TA, conducted on a 4-point Likert scale ranging from 1 (not relevant) to 4 (very relevant). First, average ratings and SDs for all answers were computed to provide a general overview. Then, in order to calculate the CVI, all scores of 3 to 4 were categorized as relevant (1), and all scores of 1 and 2 were categorized as not relevant (0). The CVI was calculated for each item (item-CVI) and for the entire scale by averaging the scores of all items.

Face validity aims to measure the comprehensibility and clarity of each translated item of the mARM-G. This was done using a 4-point Likert scale ranging from 1 (not clear) to 4 (very clear). Average ratings for all answers as well as SDs were calculated. The face validity index (FVI) was then computed by coding all scores of 3 to 4 as clear (1), and all scores of 1 and 2 as not clear (0). The FVI was then calculated for each item (item-FVI) and for the entire scale by averaging the scores of all items.

Reliability testing was performed by measuring the internal consistency of the items of the mARM-G via Cronbach α. Sample size was estimated according to the method provided by Bonett [28] with the use of a web-based sample size calculator [29]. The Cronbach α was set to .7 with a precision of 0.1 and a 95% CI. The minimal required sample size was 77 participants. Finally, confirmatory factor analysis was conducted using the lavaan package in R (Ghent University) [30] and visualized using the semPlot function [31]. For this, the sample size was estimated for root mean square error of approximation (RMSEA) based on the resources by Preacher and Coffman [32]. Parameters set with α level of .05, df of 265, a null hypothesized RMSEA value of 0.05, and an alternative RMSEA value of 0.07. Based on this, a sample size of 130 participants was required to reach a power of 0.80. All analyses were performed using R in RStudio (version 4.1.1; R Core Team) [33].

The 15 experts who conducted the content and face validation were all native German speakers living in Germany and the German-speaking part of Switzerland. The group consisted of clinical psychologists (n=7, 47%) who conduct psychological research as well as scientists (n=8, 53%) specializing in psychology research. This included 6 (40%) psychologists at the doctoral level, 5 (33%) at the postdoctoral level, and 4 (27%) full professors. A total of 11 (73%) experts were female and 4 (27%) were male, with a mean age of 35 (SD 9.2) years.

The 15 nonexperts who conducted content and face validation were all native German speakers living in Germany (n=11, 73%), the German-speaking part of Switzerland (n=3, 20%), and the United Kingdom (n=1, 7%). A total of 5 (33%) were female, 2 (13%) preferred not to reveal their gender, and 8 (53%) were male. The mean age was 27.53 (SD 9.7) years.

During the study, participants were using the mental health app—ReApp [34]. ReApp is based on principles of cognitive behavioral therapy and reappraisal. Full development protocol can be found elsewhere [35]. The participants used the app for 21 days before completing the mARM-G.

The reliability analysis was based on 144 participants. The Ethics Committee for the Faculty of Arts and Social Sciences of the University of Zurich approved the study proposal (21.2.12). All participants were students and spoke fluent German, and the exclusion criteria were having a mental illness as reported by the participant and attending psychotherapy or using other kinds of support from a qualified psychologist or psychiatrist. All participants provided written informed consent to participate in the study and agreed to data analysis by researchers from the University of Zurich. To ensure the privacy of the participants, they were provided with their own personal code to use throughout the study. Participants were remunerated up to 105 Swiss francs (around US $114) or 6 university credit points to compensate for their contribution to the study.

TA refers to the working relationship between therapist and client [6] and has been found repeatedly to predict therapeutic outcomes [7-10]. Traditionally, TA has been widely assessed using the ARM [19]. With the digitalization of therapeutic practices, novel measures for TAs are required and there is an increased need to culturally adapt measures [21]. In this study, a German version of the mARM was developed and validated.

The results indicated high content validity with regard to relevance for the TA for the entire measure. Moreover, the translated items also proved to be easy to understand, with high ratings for the comprehensibility of the entire measure. Small adjustments were made upon suggestion, aiming to improve the comprehensibility of individual items. The study conducted reliability testing via Cronbach α. The entire scale had excellent internal consistency with α=.91. In addition, α values for the specific subscales were calculated as this is common practice for longer measures [36]. The subscales were taken from the original ARM [19]. We found good results for the bond, confidence, and partnership subscales (0.76-0.83), as well as close to acceptable results for the openness subscale (0.61). The client initiative subscale had poor results (0.42). These were further inspected, and item 20 (“I am responsible for my recovery, not the app”) was found to be negatively correlated with the overall scale and subscale. If removed, the α of the subscale increases to a value of .55, which equals the α results (.55) from the ARM [19]. The results are in line with the previous research using the ARM, which has reported ranges from good to acceptable results for the bond, openness, confidence, and partnership subscales; however, it has low internal consistency for the client initiative subscale [19,37].

To further inspect the underlying structure of the translated measure, a confirmatory factor analysis was conducted which revealed mixed results. The chi-square statistic (χ²₂₆₅=440.78, P<.001) was highly significant and indicative of poor fit. The chi-square statistic to df ratio was 1.66, which indicates a good fit [38]. The CFI was 0.87 and the TLI was 0.86, which are below the suggested value of 0.90 [39]. The RMSEA was 0.07, which is acceptable [40]. The factor loading for item 20 was negative, and thus, the item was removed from the measure. We advise future researchers who use the mARM to evaluate the relationship between this item, the concept of client initiative, and TA overall with caution. Moreover, 6 items had factor loadings of <0.5 (items: 3, 4, 9, 12, 16, and 25). These items were further inspected; however, we ultimately decided against their removal since it would result in the openness and client initiative subscales being composed of only 2 remaining items.

This study had limitations. Reliability testing in this study was conducted with a group of students. Ratings for TA may thus be influenced by prior familiarity with mobile app use and overall technological familiarity. Moreover, this study used only 1 mobile app. Further research could therefore test the reliability of the measure with different apps, as well as using a sample of both younger and older participants with various socioeconomic backgrounds and various levels of experience with technology and mobile apps. This could further expand the literature on the generalization of TA measures in the broader population.

As there are still open questions concerning the nature and relation between digital TA and various treatment outcomes, future research could use the current scale to test associations between engagement and treatment outcomes, using both the overall scores of the mARM as well as the specific subscales. Further, obtaining more qualitative data concerning the nature of digital TA may yield novel insights.

Finally, additional studies comparing the effect of digital TA on treatment outcomes are needed for different therapeutic strategies, for example, for cognitive behavioral therapy, psychodynamic therapy, and acceptance-commitment therapy, among others, to gain a more nuanced understanding of the role of digital TA in various psychotherapy processes.

With an increasing number of mHealth apps being developed and the need for research into the effectiveness and suitability of these novel innovations rising, the mARM-G presents an accurate, easily accessible, scalable, and low-cost measure to conduct further research in this novel area. This translation may prove to be of use since there are estimated to be more than 100 million native German speakers living across Germany, Austria, Switzerland, Belgium, Luxembourg, and Lichtenstein for whom this measure can be used [41].

The mARM-G and instructions are presented in Multimedia Appendix 2. We encourage researchers to use it freely for research and noncommercial evaluation of the TA between humans and mental health mobile apps.