The prevalence of diabetes is increasing worldwide. Global prevalence in adults is estimated at 8.8% [1] and the Norwegian prevalence at 4.7% [2]. Around 245,000 people are diagnosed with diabetes in Norway. Of these, around 28,000 have type 1 diabetes (T1D) [2]. Patients represent a large proportion of health care contacts, and diabetes is a considerable burden on patients regarding morbidity and mortality [3]. Most patients do not reach the combined national treatment targets for prevention of complications [4-6].

Electronic health (eHealth) is defined as “the transfer of health resources and health care by electronic means” [7]. Most Norwegian households (98%) had Internet access and access to a smart phone in 2017 [8,9]. Use of the Internet for health purposes has increased rapidly in the past decades [10-12], and 75%-80% of Internet users in the United States and Europe report conducting health-related searches [10-13]. Our first study in this project found that 87% of Norwegians with T1D used eHealth in one or more forms, exceeding the use of about 78% in the general population [14,15]. Although T1D is a prevalent chronic and challenging disease, the consequences of patients seeking health information from the Internet have not been comprehensively explored. Medlock et al studied the consequences of health information seeking among seniors in the Netherlands, finding that 38% had felt anxiety and 56% had felt reassured. Some had changed their plans regarding doctor visits: 48% had decided to go to the doctor, and 24% had decided not to go based on health information from the Internet [16].

Norwegian health care is based on universal insurance. However, consultations for adults are co-paid through a small fee [14]. Primary health care, including a regular general practitioner (GP) and GP-based out-of-hours (OOH) services, is provided to all residents by the municipalities through the patient list system. This includes almost all Norwegian residents [17]. Specialist services are operated by regional and local health enterprises owned by the national government. Access to specialist care is usually through referral from GPs or OOH services (the gatekeeper role). People with T1D are recommended to visit specialist services at least annually. There are indications that many patients miss this check-up [14]. We know little about the extent to which people with T1D visit Norwegian OOH services, but we know that people with any type of diabetes and people with chronic disease constitute a significant proportion of visitors [18]. Regarding the general population, 27.4% of listed patients contacted OOH services in 2008 (all contact types) [19], and 17% of the population visited OOH services in 2017 [20]. Norway has a high OOH contact rate compared to other countries, and 75% of contacts have been classified as non-urgent [21].

Research on the association between the use of eHealth and provider-based health care is scarce [22,23]. However, frequent users of health services seem more likely to seek health information on the Internet compared with non-users [12]. This study is a part of the DIAcare project, aiming to investigate associations between eHealth and different provider-based health care services. The project also aims to investigate the associations between socioeconomic status and the use of eHealth, whether eHealth information is discussed in the clinical encounter, and whether the use of eHealth might lead to (or prevent) doctor visits [24]. Our first study in this project revealed no association between the use of eHealth and GP visits for T1D patients, whereas we found a positive association between the use of search engines and somatic specialist services [14]. This second study focuses on the association between the use of eHealth and the use of OOH services in persons with T1D. Understanding the associations between eHealth and the use of provider-based health services is important for patients as well as for health care providers, policy makers, and society, in order to enable evidence-based planning for future health care services in a society where eHealth is increasingly used.

The aim of this study was to investigate which eHealth services were used among OOH visitors with T1D, and the associations between the use of eHealth (ie, health apps in general, search engines, video services, and social media) and the use of OOH services. Furthermore, we aimed to investigate whether anxiety, reassurance, and change of doctor-seeking behavior because of the use of eHealth were associated with the use of OOH services.

This cross-sectional study used data obtained in 2018 from members of the Norwegian Diabetes Association (NDA). As of December 31, 2017, the organization had 33,908 members, of which about 30% have T1D [25]. The Norwegian Social Science Data Service (NSD) Web survey distributed the invitations to a randomly selected sample of 5971 individuals who had their email addresses recorded by NDA. Initially, we planned to use data from the seventh Tromsø Study, conducted in 2015/2016 [24]. However, the Tromsø Study could not give us access, and we developed a tailored questionnaire based on the specific objectives of our study, using relevant questions from other published surveys [26,27].

Information about the study was posted together with the invitation. The questionnaire (Multimedia Appendix 1) included questions about health status including specific questions about disease duration, severity and treatment of diabetes, use of and experiences with eHealth and health care services, as well as demographic and socioeconomic information. We reviewed and tested the questionnaire several times before distribution to the informants. Non-respondents were given one reminder, submitted by email 15 days after the first request.

The respondents could not fill in the questionnaire more than once. Starting from 1250 participants, we first excluded the 66 individuals who did not suffer from diabetes themselves (eg, family members, health personnel, and others). We also excluded those who failed to respond to most of the questions (n=5), those who did not give information about gender (n=93), and participants with diabetes types other than T1D (n=563). The analyzed sample consisted of 523 respondents with T1D. For a flowchart of the study population, we refer to our previous study [14].

The dependent variable in all analyses was the use of OOH services once or more during the previous 12 months. The use of eHealth was subdivided into 4 variables: apps for mobile phone or tablet computers (health apps in general, not necessarily diabetes self-care apps), search engines (eg, Google), social media (eg, Facebook), and video services (eg, YouTube). We dichotomized these variables by merging the original four response options into “never or once” and “sometimes or often,” in line with previous research [16].

The following questions were also asked: “Based on information from the Internet, have you felt anxious/felt reassured/decided to consult a doctor when you would otherwise not have consulted one/decided not to consult a doctor when you would otherwise have consulted one?” The answering options “no, once, sometimes, or often” were merged into “no” and “once, sometimes or often” for an easier interpretation of logistic regressions. Since we were interested in whether changes in decisions had ever taken place, we placed “no” in a separate category.

Age was grouped in 20-year age groups. The education categories were labeled low (primary/part of secondary school), middle (high school), high (college/university <4 years), and highest (college/university 4 years or more). Response options for self-rated health were excellent, good, fair, bad, and very bad. We merged the bad and very bad categories due to low numbers in the very bad category (4 respondents). Response options for self-reported degree of anxiety/depression were none, slight, moderate, severe, and extreme. We merged the severe and extreme categories due to only 1 respondent in the extreme category.

Since the use of health apps in general turned out to be an issue in this study, we wanted to inform the readers about the answers obtained from the following question in the questionnaire: “During the past 12 months, have you used apps for smartphone or tablet computer for follow-up of your own diabetes?” The answering options were “never, less than once a month, once a month, once a week, and every day.” According to the study protocol, this variable was not included in the regression analyses.

Due to the relatively low participation rate, we constructed a dichotomous response time variable in order to compare late respondents with early respondents, assuming that the late respondents were more similar to non-respondents [28]. All participants who responded initially were placed in one group, and those who responded after the reminder were placed in the other group.

Data were analyzed by means of descriptive statistics and logistic regressions. Correlations were tested with Spearman correlation test.

Use of OOH services was the dependent variable in all the analyses. In the first multivariate regression, the independent variables were the use of apps (general health apps), search engines, social media, and video services, gender, age, education, self-rated health, and self-reported degree of anxiety/depression. For the second set of analyses, we performed four univariate and four multivariate regression analyses. The independent variables in the four univariate analyses were “felt anxious,” “felt reassured,” “decided to consult a doctor when you would otherwise not have consulted one,” and “decided not to consult a doctor when you would otherwise have consulted one.” These were key independent variables in the multivariate analyses, which we adjusted for gender, age, education, and self-rated health. All the independent variables were introduced collectively into the multivariate models.

We compared those who did not respond to our initial inquiry, but eventually consented, with the early respondents. This was done by subsequently introducing the response time variable into the regression models. We used 95% confidence intervals (CI) throughout the study. All analyses were accomplished using Stata, version 14.2.

The Regional Committee for Medical and Health Research Ethics (REK) found that an application for this project was not required according to the Norwegian Health Research Act (ref 2015/1779/REK nord). The data protection officer (Personvernombudet) at the University Hospital of North-Norway approved the study (ref. 2017/6579). The NSD data bureau received no information about the participants other than the email addresses.

In total, 1250 persons aged 18-89 years answered the questionnaire, constituting a minimum response rate of 20.9%. We assume the real response rate to be higher since we had more than 400 bounce backs from email servers unable to deliver the invitation, and we do not know how many actually received the survey email. Eligible for analysis were the 523 persons who reported to have T1D.

Among the users of OOH services, the largest groups were men (66/129, 51.2%), people aged 40-59 years (53/129, 41.1%), married/cohabitants (87/99, 87.9%), people employed full-time or part-time (81/129, 62.8%), people with the highest education (41/128, 32.0%), the highest household income (63/126, 50.0%), good self-rated health (59/128, 46.1%), good self-rated regulation of diabetes (66/127, 51.9%), no anxiety/depression (71/128, 55.5%), and who had lived with diabetes for 30 years or more (43/129, 33.3%) (Table 1).

Based on information from the Internet, 44.5% (57/128) of OOH users reported that they had felt anxious, 55.9% (71/127) had felt reassured, 28.7% (37/129) had decided to consult a doctor when they would otherwise not have consulted one, whereas 21.9% (28/128) had decided not to consult a doctor when they would otherwise have consulted one (Table 1).

Mean age among the users of OOH services was 47.2 years, 44.3 years for women, and 50.0 years for men. Median age was 48 years. Mean disease duration was 21.9 years (median 20 years).

During the previous year, 26.7% (129/484) visited OOH services once or more. Men visited slightly more than women (29.1% versus 24.5%), and people aged 60 years and over visited slightly more than younger age groups (Table 2).

Among the OOH users, 63.0% (80/127) visited once and 26.8% (34/127) visited twice during the previous year. Only 2.4% (3/127) visited 5 times or more.

In the total sample as well as among the OOH users, search engines were the most widely used form of eHealth, followed by the use of health apps (all kinds). Among the OOH users, search engines were used by 86.7% (111/128), health apps (all kinds) by 63.6% (82/129), social media by 45.3% (58/128), and video services by 28.4% (36/127). Users of OOH services used apps slightly more than the total sample (Table 3).

Among the OOH users who reported having used health apps (all kinds) sometimes or often, the largest group (32/82, 39%) had never used apps for self-care of their diabetes, and the second largest group (28/82, 34%) had used apps for self-care less than once a month. Only 11% (9/82) had used apps for diabetes self-care every day (Table 4).

Visits to OOH services were positively associated with the use of health apps (odds ratio [OR] 1.73, 95% CI 1.05-2.85), but not associated with the use of any other type of eHealth. OOH services visits were positively associated with self-reported moderate or severe anxiety/depression, compared with no anxiety/depression (OR 2.18, CI 1.04-4.54, and OR 4.53, CI 1.43-14.32, respectively). Gender, age, education, and self-rated health were not associated with the use of OOH services (Table 5).

Those who had felt anxious based on information from the Internet were more than twice as likely to visit OOH services compared with those who had not felt anxious, both in univariate (OR 2.13, CI 1.40-3.25) and in multivariate regression models (OR 2.38, CI 1.50-3.78). People who had decided to consult a doctor based on Internet information (when they would otherwise not have consulted one) were almost three times as likely to visit OOH services (OR 2.70, CI 1.65-4.42, and OR 2.76, CI 1.64-4.66, for univariate and multivariate regressions, respectively), compared with those who had not changed their decision to consult a doctor.

Feeling reassured based on information from the Internet was not associated with the use of OOH services. Likewise, we found no association between deciding not to consult a doctor (when they would otherwise have consulted one) and the use of OOH services (Table 6).

All findings presented in Tables 5 and 6 persisted after including the response time variable in the regression analyses. There were no strong correlations (defined as Spearman rho >.5) between the independent variables in any of the models.

We found that 26.7% (129/484) of people with T1D visited OOH services once or more during the previous year. Search engines were used sometimes or often by 86.7% (111/128) of the OOH visitors, apps by 63.6% (82/129), social media by 45.3% (58/128), and video services by 28.4% (36/127). Visits to OOH services were positively associated with self-reported anxiety/depression and with the use of health apps, but not with other forms of eHealth. Among those who had used OOH services as well as health apps during the previous year, 39% (32/82) had never used apps for self-care of their diabetes. Those who had felt anxious based on information from the Internet were more than twice as likely to visit OOH services compared with those who had not felt anxious. People who had decided to consult a doctor based on information from the Internet (when they would otherwise not have consulted one) were almost three times as likely to visit OOH services compared with those who had not made a decision to consult a doctor based on information from the Internet.

Among people with T1D, 26.7% reported one or more visit to OOH services during the previous year. Sandvik et al reported 27.4 contacts with OOH services per 100 list patients in Norway (general population, 2008) [19]. Unlike our rate, this rate includes more than just visits (ie, house calls, telephone contacts, and simple contacts), indicating a higher visit rate among people with T1D compared with the general population. The National Centre for Emergency Primary Health Care reported 1,332,024 visits to OOH services in Norway in 2017 [29], indicating a rate of 25.3% in a population of 5,258,317 inhabitants (January 1, 2017). Children, adolescents, and tourists were included, as well as the number of visits for those who visited more than once, meaning that our rate solely for visits is higher. Statistics Norway found that 17.0% of the population (all ages, children included) consulted OOH services in 2017 [20]. Since children and adolescents are frequent OOH users [29], the adult general population rate is probably lower. This is in line with data collected by the Tromsø Study, suggesting a preliminary general population OOH visit rate of 13.5% (2740/20,294) among people aged 40 years and over [27]. Despite constraints regarding comparison with these figures, we conclude that our rate for OOH visits among people with T1D is higher than these general population rates. Our findings contribute to underpinning previous documentation that people with diabetes are frequent users of OOH services in Norway [18,29]. Research from other countries confirms that this is an international phenomenon. Diabetes was the most frequent chronic somatic disorder among frequent attenders in the Netherlands in 2007 [30], and patients with diabetes accounted for almost 1 out of 10 emergency department visits in the United States (2010) [31].

In the first paper in this project, we documented that people with T1D are frequent users of all the investigated four forms of eHealth [14]. OOH attenders with T1D used all forms of eHealth equally or more than the total T1D sample, and the largest difference was found for apps (Table 3). This extensive use is in line with the illness behavior model stating that people in poorer health more likely seek Web-based disease-related information [32]. It is also in line with a German study reporting that app users were more likely to report chronic conditions, such as diabetes [33].

Most of the combined OOH and app users had used apps for self-care of their diabetes; however, only 11.0% of them used diabetes self-care apps every day (Table 4). Only 5.9% of the total T1D sample used diabetes self-care apps every day, which is a low percentage compared to 24% in adults with T1D reported in a recent Australian study [34]. They found that the use of self-care apps was positively associated with female gender and lower age. More female participants and a lower mean age in the Australian study may explain some of the differences from our rate. However, the study methodology and question wording vary, and the figures are not directly comparable.

The possibility of visiting OOH services was higher for persons with T1D who used health apps (all kinds), compared with those who did not (Table 5). We found no other studies investigating this relation.

Moderate and severe self-reported anxiety/depression was more prevalent among the users of OOH services, compared with the total sample (Table 1). Severe anxiety/depression increased the possibility of visiting OOH services 4-5 times, compared with those who reported no anxiety or depression (Table 5). Extensive use of OOH services among people with mental problems and psychiatric disease is well documented in previous research [18,30,35,36].

In many countries, Norway included, there is concern about unnecessary use of emergency medical services [19]. Patients, as well as GPs, consider worry an important reason for contact [37,38]. From a medical perspective, most OOH use is non-urgent, and motives for contacting OOH services are primarily patient-related [21,38]. Self-reported anxiety/ depression and the use of health apps might be such patient-related factors. A common feature of apps, OOH services, and anxiety/depression is that they operate independently of opening hours. We consider this a possible explanation of the association between the use of health apps and the use of OOH services, as well as the association between anxiety/depression and the use of OOH services. Unsatisfactory design and functionality of many apps might also partly explain this finding [39]. Furthermore, a lack of GP availability or accessibility during daytime might contribute in some regions [19,38,40].

The extensive use of OOH services among people with T1D, and particularly among those reporting anxiety/depression, indicates that some patients do visit OOH services rather than waiting for a consultation with the regular physician. This might indicate that policy makers and GPs should consider increasing the regular GPs’ capacity and/or extending opening hours. In this regard, eHealth consultations could be a valuable supplement to face-to face consultations.

Many studies regarding mobile apps and worries, anxiety, or emotional distress focus on apps as possible tools in coping with these problems [41]. However, some have studied the potential of digital devices to create or intensify worries and anxiety in susceptible individuals. Recent research found that the Internet has the potential to reduce as well as exacerbate health anxiety [42-44] and that individuals with moderate to high levels of health anxiety experienced more anxiety during and after online symptom checking, whereas those with low illness anxiety experienced relief [44]. Thus, the use of eHealth might as a side effect intensify health anxiety and, used in the wrong context, cause harm instead of benefit to health. If this is the case, it might contribute to explaining the positive association between the use of health apps and the use of OOH services.

Many apps used for monitoring of disease may meet the definition of a medical device [45]. Still, most apps have not been evaluated by authorization authorities and are not CE-marked (Conformité Européenne, European Conformity) according to European Union directives, or regulated by the US Food and Drug Administration [45,46]. Possible risks related to the app itself have been focused [39]; however, little is known about the use of apps outside tightly controlled research settings [34]. A wider focus on effectiveness related to the use by different individuals in different settings has not been addressed thoroughly [42,43,47]. In our opinion, such possible side effects should be evaluated for all equipment considered as medical devices, apps included.

We found that 31.8% (115/487) of people with T1D and 44.5% (57/128) of the OOH users had felt anxious based on information from the Internet (Table 1). According to data collected by the Tromsø Study (general population), 25.8% had felt anxious [27], whereas 38.0% among elderly people in the Netherlands had felt anxious based on information from the Internet [16]. Our figures for people with T1D are placed between them, whereas figures for the OOH users exceeds the other rates.

The differences between the T1D sample and the OOH users among them were greater for feeling anxious (31.8% vs 44.5%) than for feeling reassured (54.6% vs 55.9%), which might indicate that using the Internet for health purposes could have more negative effects for the OOH users, compared with the T1D sample. Even if many people reported anxiety, more people had felt reassured based on Internet information in our study, as well as in the Tromsø Study and in the Dutch study (47.7% and 56.0%, respectively) [16,27].

We found that 17.5% (87/497) of the total T1D sample and 28.7% (37/129) of the OOH users among them decided to consult a doctor based on Internet information—when they would otherwise not have consulted one (Table 1). The similar rate from the Tromsø Study data was 23.6% [27]. Again, our rate for the OOH users exceeds the rates for the total T1D sample, as well as the general population rate obtained from the Tromsø Study. This might indicate that information from the Internet has a larger influence on the OOH users in this regard. Among elderly people in the Netherlands, almost half of the participants (48%) had decided to consult a doctor based on Internet information, which might be explained by cultural diversities, differences in the use and in trusting information from the Internet, as well as methodological differences [16].

Those who had felt anxious based on information from the Internet were 2-3 times more likely to visit OOH services than those who had not felt anxious. People who had decided to consult a doctor based on information from the Internet—when they would otherwise not have consulted one—were almost three times as likely to visit OOH services compared with those who had not changed their decision regarding consulting a doctor. A recent systematic review and meta-analyses found a positive correlation between health anxiety and online health information seeking [48]. As health anxiety levels increase, the relationship between health information seeking online and visiting a doctor based on information found online also increases [42,49]. Other studies asked whether use of the Internet had changed the frequency of doctor visits. Around 90% in these general population studies reported that use of the Internet for health purposes did not change their health care seeking behavior (United States 94%, Japan 88.9%, France 88.6%) [50-52]. Consequently, around 10% did change their health care seeking behavior, as they made either more visits or fewer visits due to information from the Internet. However, other available studies in different populations, most of them general populations, cannot be directly compared with the current study of people with T1D. Our findings may suggest that when decisions to consult a doctor are based on Internet information, people tend to consult as soon as possible. Since OOH services are available at any time, they might be a natural choice.

Strengths and limitations have been explored in detail in our first study in this project [14]. The strengths of this study are similar to the strengths discussed in the first paper in this project [14]. The most important strength is the focus on a scarcely investigated research field, which might contribute to evidence-based planning for future health care services in a society where eHealth is increasingly used. Other strengths are the detailed questionnaire specifically tailored to people with diabetes, the recruitment of participants from all of Norway, the inclusion of a wide age span of participants, and that we were able to analyze the data shortly after they were collected. Finally, the collection of data in cooperation with NDA enabled us to develop excellent user participation with a large and important group of health care users.

A limitation of this study is the low estimated participation rate. However, response rate must not be confused with response quality [28]. We found that older people dominated among the late respondents compared with the early respondents [14]. As late respondents might be more similar to non-respondents, younger individuals might be overrepresented.

Distribution of the questionnaire by email is another limitation, which excluded those who do not use the Internet or do not have an email address. Since 97% of Norwegian households have Internet access, we do not think that this affected our results significantly [14]. It is well known that women, healthier persons, higher socioeconomic groups, and middle-aged people are more likely to participate in surveys [14]. This suggests that women, people around 40 to 80 years, people in better health, and higher socioeconomic groups might be overrepresented in our study, thus tending to level out a possible skewness in the opposite directions.

Other relevant limitations were recall bias, the validity of self-reported data, and the cross-sectional study design, as reported and discussed in detail in the first study in the DIAcare project [14]. It is not possible to judge the magnitude or direction of a possible non-response bias, since different factors might pull the tendency in different directions or level each other out. The low response rate is in itself not an indication of low representativeness, as non-response bias may be a problem even if response rates are high [53]. We suggested that non-response bias posed a limited threat to our study’s validity; however, generalization must be made with caution.

Increasing travel distance is associated with reduced use of OOH services in Norway [54]. The lack of information about travel distance is thus a study limitation. However, we have no reasons to believe that the use of apps or other of the independent variables are influenced by travel distance and do not consider this a confounding threat to our results.

It should also be mentioned that worries/anxiety/ depression/emotional distress are not defined according to diagnostic manuals in this study and rely solely on self-report. We consider self-report to be interesting as such in this field and do not think that this has disturbed the validity of our results.

This study investigated the use of health apps in general and associations with the use of OOH services. For future research, the more specific use of diabetes self-care apps and associations with the use of different health care services would be beneficial. Nor did this study investigate the reasons for visiting OOH services, which would be interesting as well: were visits directly related to the use of eHealth for diabetes self-care, or were they due to other health care needs? Furthermore, the finding that information acquired from the Internet is associated with making decisions to attend OOH services merits further investigations.

We found that people with T1D were extensive users of OOH services and that OOH service users were extensive users of eHealth. There was a positive association between the use of OOH services and the use of health apps, as well as between the use of OOH services and self-reported anxiety or depression. Feeling anxious based on information from the Internet was positively associated with visiting OOH services. Likewise, there was a positive association between deciding to consult a doctor based on information from the Internet and the probability of visiting OOH services. The use of eHealth seems to have a large impact on people with T1D. This study investigated the use of health apps in general, and we think that the more specific use of diabetes self-care apps and associations with the use of different health care services would be of interest for future research. Decision making regarding doctor visits based on information from the Internet also merits further investigations, along with specific investigations regarding reasons for visiting.