We used a quantitative study design to evaluate the implementation of myFertiCare according to the HOT-fit framework. This framework states that a fit between human, organization, and technology factors is needed to successfully implement an eHealth intervention [15]. The HOT-fit framework was applied by using validated questionnaires focusing on these 3 domains. Furthermore, we studied the effect of using myFertiCare on (1) couples’ knowledge about fertility treatment, (2) couples’ experiences of treatment burden, and (3) couples’ experiences of patient-centered care. We also included nonusers of myFertiCare to explore motivations for not using the app and to identify opportunities for improvement. Finally, we analyzed user data to obtain insight into various app-related variables, such as the number of users, visits, and page views and the frequency and duration of use.

Ethical approval was proposed but was not required according to the local research ethics committee (Commissie Mensgebonden Onderzoek Arnhem Nijmegen).

The study was established at a Dutch university medical center specializing in ART and surgical sperm retrieval for men with azoospermia. ICSI with surgical sperm retrieval is a multidisciplinary treatment involving a urologist, who is responsible for the andrological evaluation and surgical sperm retrieval; a gynecologist, who is responsible for the subsequent ICSI procedure; a psychologist, who is present for easily accessible mental support; and, if necessary, a clinical geneticist. In January 2016, the online app myFertiCare was launched and made available via the clinic’s website, the Apple App Store, and the Google Play Store.

All couples visiting the outpatient clinic for possible ICSI with surgical sperm retrieval between January 1, 2016, and July 1, 2017, were invited to use myFertiCare and to participate in the questionnaire study. Men undergoing surgical sperm retrieval for fertility preservation purposes and couples in which neither partner understood the Dutch language were excluded. All participating couples signed an informed consent form.

The study comprised 2 separate questionnaires: one was targeted at users of myFertiCare and one was targeted at nonusers. The questionnaires were available both on paper and digitally (using Castor electronic data capture). They were sent out in June 2017. Both questionnaires contained questions on the demographic and medical characteristics of the participants.

The user questionnaire was based on the principles of the HOT-fit framework [15]. The human domain consisted of system use and user satisfaction. Both were evaluated through validated questionnaires: the System Usability Scale (SUS) [16] and the End-User Computing Satisfaction (EUCS) instrument [17], respectively. The organization domain consisted of the structural and environmental context of fertility care and was evaluated through self-developed questions in the user questionnaire (a validated questionnaire was not used because the questions were unique to the context of the organization) and through short, face-to-face structured interviews with staff members of the department of reproductive medicine. The interview questions focused on the staff’s experiences with the organization and implementation of myFertiCare. These interviews were performed face-to-face because this revealed more subtle motivations. Relevant items regarding the implementation of myFertiCare were identified and reported as results. The technological domain included the quality of the system, information, and service (ie, the interface). This was studied through the information and interface quality domains of the validated Computer System Usability Questionnaire (CSUQ) [18]. All validated questionnaires were translated into Dutch by a member of the research team. The translations were checked by having another member translate them back into English. Both researchers were proficient in English and Dutch. Discrepancies were discussed until consensus was reached. To supplement the HOT-fit framework, questions were added concerning knowledge about infertility, the burden of infertility and fertility treatment, and the extent to which couples experienced patient-centered care. Couples were asked if using myFertiCare increased their knowledge about the causes of and treatments for infertility. Questions concerning the burden of infertility and infertility treatment were based on the main items described in the validated fertility quality of life (FertiQoL) questionnaire [19]. Questions about patient-centeredness were based on the 8 subheadings of the validated Patient-Centredness Questionnaire–Infertility [20]. The complete user questionnaire consisted of 72 questions with answer options being open, on a 5- or 7-point Likert scale, or on a unipolar verbal scale.

The questionnaire for nonusers of myFertiCare consisted of 4 questions regarding familiarity with the availability of myFertiCare, motivations for not using the app, and suggestions to increase its use and the use of other online sources of information. The questions were self-developed by the research team to match the specific context of the organization.

Both questionnaires were pilot-tested with 3 couples attending the outpatient clinic. The couples considered the questions clear and understandable, so no major changes were necessary. The questionnaires were sent out via postal mail by a member of the treatment team. One questionnaire was sent per couple, addressed to the couple, because a close connection and interaction between partners during the treatment was assumed. Each couple received both the user and nonuser questionnaires and had to decide which of the two was applicable. Use was defined as the minimum of one log-in to the app; nonuse was defined as never having logged in. Patients had to write down who completed the questionnaire: the male partner, the female partner, or both. Couples could return the questionnaire via postal mail or email. Nonresponders were sent a reminder 2 weeks after the initial invitation. Questionnaires were collected from July 1, 2017, until August 18, 2017, and assigned a code that was only available to the researchers. All data were analyzed confidentially.

User data were automatically transferred to an anonymized Microsoft Excel file. In this way, all app visits and page views could be logged by date and time and analyzed later. All patients were included who used myFertiCare between January 1, 2016, and July 1, 2017, whether they participated in the questionnaire study or not.

Statistical analysis was performed using Microsoft Excel and SPSS (version 22). Baseline characteristics of the study population, user data, and the results of the questionnaires were analyzed using descriptive statistics: median (range), mean (SD), or frequency. We also focused on finding possible differences in user data between men and women. P values were calculated using independent-sample 2-tailed t tests, Mann-Whitney U tests, and chi-square tests to find possible significant differences (P<.05) between users and nonusers and between men and women.

In total, 314 ICSI couples were invited to participate (Figure 1); 151 couples (48%) completed and returned a questionnaire, including 21 user questionnaires and 111 nonuser questionnaires. Additionally, 19 couples returned both versions of the questionnaire. If both versions were returned, the user data were consulted to classify these couples as users or nonusers and the appropriate questionnaire was included in the analysis. Nine nonuser questionnaires were excluded because no questions were filled in or because the couples actually were users, which was defined as 3 or more log-ins. Finally, the user group consisted of 35 couples and the nonuser group of 107 couples (Figure 1). Demographic and medical characteristics of the participants (users and nonusers) are provided in Tables 1 and 2.

We analyzed questionnaires from 35 couples who were myFertiCare users according to the HOT-fit framework.

Questionnaires from 107 couples who were nonusers of myFertiCare were analyzed. Although providing information about myFertiCare was part of routine care, only 25% (27/107) of nonusers remembered being informed about the availability of myFertiCare. The main reason for nonuse was a lack of need or interest. Of the couples that did not remember being informed about myFertiCare, half (54/107) said that they would have liked to use the app. The other half felt that they did not need the app because they obtained sufficient information via other sources. The majority of nonusers of myFertiCare used other sources of online information about infertility or treatment (71/107, 66%).

We analyzed user data as a supplement to the questionnaire study. In total, 163 individual patients used myFertiCare, who were included in 139 couples. In 21% (29/139) of couples, only the male partner used the app, while in 47% (66/139) of couples, only the female partner used the app, and in 32% (44/139) of couples, both partners used the app. When both partners were users, the female partners were the more frequent users, with a median of 9 visits, compared to a median of 3 visits for the male partners. All user data are shown in Table 5. Finally, it appeared that use of myFertiCare gradually declined during the treatment trajectory, as illustrated in Table 6.

In this quantitative study, the implementation of myFertiCare was evaluated with the HOT-fit framework. In the human and technology domains, myFertiCare showed good system usability, high user satisfaction, and high information and interface quality. In the organizational domain, based on high scores in the user questionnaire and positive feedback from the staff interviews, implementation was considered to be sufficient. App use could have been improved by creating more awareness among patients and staff. Use of the app increased knowledge about the treatment, improved coping with the treatment, and enhanced the experience of patient-centeredness of care.

The current study shows that women were the main app users. Either the female partner was the only user or was the main user when both partners used the app. This observation is in agreement with previous research [5,7,8,22]. It is attributable to sex differences in health-related internet use, the experience of infertility and fertility treatment, and strategies to cope with fertility-related issues [23]. However, we found that although men were less frequent myFertiCare users, the duration and number of page views per visit was equal for both sexes.

User data showed that use of the app was highest before surgical sperm retrieval and gradually declined thereafter. The observed gradual decline in app use contradicts previous research, which found that the highest user activity occurred in later treatment phases, namely between oocyte retrieval, embryo transfer, and the pregnancy test [24], which was attributed to high stress levels during this particular treatment phase [3,25]. This could be explained by the law of attrition, which is a phenomenon whereby participants discontinue use of eHealth interventions that are neither mandatory nor critical to their direct well-being [26]. However, a part of this decline was also caused by the expected treatment dropout that occurs with a negative result from the surgical sperm retrieval. It is known that in half of men with nonobstructive azoospermia, no sperm cells can be retrieved [27]. Therefore, these couples were not able to continue their fertility treatment with an ICSI procedure. Their treatment stopped, as did their use of myFertiCare. Furthermore, as a result of a relatively high rate of fertilization failure in our study related to the use of surgically retrieved sperm instead of the use of ejaculated sperm [28], a proportion of our participants probably did not proceed to embryo transfer and therefore did not use the app at this treatment stage. On the other hand, the couples that continued with ICSI treatment were possibly satisfied with the information they received, and the need for app use declined.

It is remarkable that the majority of nonusers of the app did not remember being informed about the availability of myFertiCare. All couples received this information as part of standard care during an informative group consultation conducted by a specialized nurse at the beginning of the treatment trajectory. It is known that patients’ memory for medical information is often poor and inaccurate, especially when the patient is anxious [29]. This underlines the necessity of repeating important information on several occasions and providing information in written form.

By studying the effects of app use, we found that users considered myFertiCare to be mainly a source of information, rather than a tool able to significantly decrease the burdens of infertility and treatment. However, the couples were outspoken that myFertiCare improved their experience of patient-centeredness of care, meeting the goal we set at the start of the study. Therefore, myFertiCare constitutes an innovative tool to help professionals provide patient-centered care. We hypothesize that myFertiCare could score higher for influence on the burdens of infertility and treatment if it were supplemented with functionalities targeting this effect. In the current study, we developed an online app for a pilot population and were not equipped with resources to add extra functionalities. Another possible way of improving app use would be to develop an app that also provides a benefit for the treatment team, such as by making it easier or more efficient to provide care to patients or by making that care better. Therefore, we call for other medical professionals to continue developing online interventions in collaboration with their patients and staff, so that patient-centered care can be improved from the perspective of the patient and the professional.

Numerous eHealth interventions with different functionalities targeted at a variety of patient categories have been reported. It is remarkable that most evaluations of these interventions are only performed at the end of the intervention, although the importance of conducting evaluations throughout an intervention is regularly discussed [30]. A study of eHealth evaluations made the striking finding that only 64% of studies evaluated clinical aspects, 48% evaluated human and social aspects, 20% evaluated technological aspects, and 16% evaluated organizational aspects [30]. A recent example of an eHealth intervention that did include human, organization, and technology factors in the evaluation used a qualitative study design with semistructured interviews to explore patients’ experiences and described these experiences on the basis of the 3 domains [31]. Therefore, we feel that the major strength of the current study is the study design. By using validated questionnaires on the human, organizational, and technological domains, we quantitatively studied implementation of the app. Furthermore, we studied the effects of app use and analyzed user data. Finally, we also included nonusers of the app in the study, to explore motivations for not using the app and identify opportunities for improvement. This way, we provided a complete framework for app design, development, implementation, evaluation, and improvement. We call for better evaluation of eHealth interventions to facilitate successful long-term implementation.

Our study also has limitations. There was a relatively low response rate of 25% (35 of 139 couples) to the user questionnaires. For the nonuser questionnaires, the response rate was 61% (107 of 175 couples). A possible explanation could be that we sent the questionnaires in June 2017 to all couples who were treated between January 2016 and July 2017. It could be that a significant proportion of these couples had already dropped out or finished treatment. Another hypothesis is that the response rate could have been affected by the length of the questionnaires. Because we used multiple validated questionnaires on different domains, the user questionnaire was quite extensive, whereas the nonuser questionnaire contained only 4 questions. However, there was good consistency in the data from the user questionnaires, which supports the reliability of the study data. Another limitation is that we studied the effects of app use (ie, knowledge about infertility and treatment, the burdens of infertility and treatment, and patient-centeredness of care) based on self-reported differences before and after app use, rather than measurements before and after app use. We chose this approach because we did not want patients to feel that they had to perform an exam instead of a questionnaire, and we did not want to make the questionnaire more extensive to study patients’ knowledge.

A multi-faceted online app, myFertiCare, has been successfully evaluated quantitatively for implementation with the HOT-fit framework. Use of the app increased knowledge about the treatment, improved coping with the treatment, and enhanced the experience of patient-centeredness of care.

In our previous study, we successfully designed, developed, and qualitatively evaluated myFertiCare for usability. In the current study, implementation of the app was positively and quantitatively evaluated based on the HOT-fit framework, and the effects of app use were studied. Through these consecutive studies, a framework has become available that can be used throughout the complete trajectory of app development, implementation, evaluation, and improvement, and which involves both patients and professionals in every study phase.

Providing myFertiCare encourages professionals in fertility care to guide patients through their treatment trajectory and to deliver patient-centered care. Furthermore, myFertiCare offers an opportunity to empower patients and help them manage their own treatment trajectories. It would be valuable for future research to improve the app based on the couples’ and professionals’ suggestions, so that more support is perceived and app use can be expanded to other patient categories and medical departments. We appeal to professionals in both fertility care and other medical departments to provide eHealth initiatives to their patients in which both patients and professionals are involved in every phase of design, development, implementation, and evaluation.