We searched the literature but did not find any questionnaires that measured the use of e-messaging between home health care services and GPs. Therefore, we developed a questionnaire in two phases [32]. In the first phase, we explored the recommended guidelines for the use of e-messages [33], the description of the standards for the e-messaging system [30], and the results from previous contiguous research [34-37]. We then conducted focus groups and semistructured interviews with home health care nurses from two municipalities. The results from the interviews showed that both organizational and technical elements were assessed as important dimensions for the use of e-messaging [38].

The second phase consisted of the questionnaire development. The questionnaire was reviewed by 6 registered nurses with clinical, ICT, and research skills, and it was pilot tested by 15 registered nurses who worked in home health care services. These steps provided significant and valuable information that was used to revise the questionnaire. The final version of the questionnaire contained 62 items in six sections: (1) Demographic information, eight items, (2) Procedures for and amount of information exchange and communication with GPs, 19 items, (3) Information content, 12 items, (4) Expectations for electronic communication with GPs, 10 items, (5) Electronic communication, three items, and (6) Experiences of the use of electronic communication, 10 items (see Multimedia Appendix 1). Six items were negatively worded: Items 19 (a)-(d) and Items 20 (d) and (e). Here, we report the demographic, electronic communication, and experiences of the use of e-messaging results from Sections 1, 5, and 6, respectively.

The items selected for this study have been previously documented as important factors for health care providers’ assessments of health information systems [7,9,11,15,21,26-28]. These items can be grouped as individual, organizational, and technological elements. The individual element consists of Items 2-6 from Section 1 and Item 16 from Section 5 of the questionnaire. The organizational element consists of Items 20 (a) and (b) from Section 6 of the questionnaire. The technological element consists of Items 19 (b)-(d) and 20 (c) from Section 6 of the questionnaire.

The questionnaire items were rated on a 5-point Likert scale as follows: (1) strongly disagree, (2) disagree, (3) neither agree or disagree, (4) agree, and (5) strongly agree. The Cronbach alpha exceeded .7, indicating acceptable reliability [39].

To test the validity of the questionnaire, a selection of the items from the main study was summarized and correlated with the item “useful tool for communication with GPs”, which is the questionnaire’s concluding question. Pearson’s r was .57 (P<.001), indicating a fairly strong correlation between the overarching concept of usefulness of the e-messaging tool and the selected questionnaire items, but also that there may be items or facets of items that should have been addressed in the questionnaire. However, we wanted to keep the questionnaire short to achieve a high response rate. The pilot test showed that it took approximately 10 minutes to complete the questionnaire.

The questionnaire data were collected from home health care nurses in 12 out of 428 municipalities in Norway. The inclusion criterion for the municipalities required that the e-messaging system had been in use for more than 6 months. We assumed that system usage would be stable after this time frame [40]. The 12 municipalities were the first in Norway to introduce e-messaging to support communication between home health care nurses and GPs, and they were the only municipalities that met this inclusion criteria at the time of our study. All of the home health care nurses were employed in a 50% or greater full-time equivalent position, had been using e-messaging for at least 3 months, and were able to read and write Norwegian. The home health care nurses received the questionnaire, information letter, and return envelope at their workplace from a designated contact person in each municipality. The contacts were responsible for distributing collective reminders and for collecting the envelopes with the completed questionnaire. There were 681 nurses qualified to answer the questionnaire, and the questionnaire was distributed to 632 (92.8%) nurses.

The home health care nurses who were invited to answer the questionnaire received written information explaining that their participation was anonymous and that returning the questionnaire meant that they agreed to participate in the study. The methods for data collection and handling the interviews and the questionnaire were approved by the Norwegian Social Science Data Services, which is the official data protection agency for research at the University of Oslo (reference no. 26230).

We used a descriptive analysis to determine the characteristics of the sample and the nurses’ assessments of using e-messaging. Scores for negatively worded items were reversed. Two logistic regression models were developed to determine the extent to which specific elements influenced the odds that the nurses would report that e-messaging was a useful and easy tool for communication with GPs.

In the first logistic regression model, “the useful tool model”, we chose to include the independent variables of “easy to use” and the home health care nurses’ demographic characteristics of “age”, “gender”, “full-time equivalent percentages”, “years of experience as a registered nurse (RN)”, “years in current position”, and “number of months using e-messaging”. The reason for choosing these variables is based on results from previous research [3,8,21,26,38].

In the second logistic regression model, “the ease-of-use model”, the dependent variable of “easy to use” was combined with the independent variables of “received training”, “access to user support”, “hindered by poor functionality”, “hindered by low system performance”, and “hindered by software error”, and the home health care nurses’ characteristics. These variables have been shown to have an impact of the assessment and adoption of ICT systems [3,7-9,11,15,21,26-28,38].

In the pre-analysis of the data, we searched for outliers in continuous variables. The results showed that it was not necessary to transform any of the data. We used ±3.30 standard deviations (SD) to check for outliers, as suggested by Altman [41]. Two variables had outliers: “years in current position” with 24 years (0.9%) in the positive direction and “number of months using e-messaging” with 37 months (1.3%). However, working for an extended period of time in the same position or using the e-messaging system for 37 months is unlikely to affect the results.

Homoscedasticity in both logistic regression models was assessed by a one-way analysis of variance of the standardized residuals to establish that their variance was approximately the same for all values of the predictor variables. No homoscedasticity was found. The first model was used to predict if the e-messaging system was a useful tool for communication with GPs. The item “easy to use” did not show statistically significant differences (P=.76) between the prediction errors in the group that found the e-messaging system to be a useful tool and the group that did not find the tool to be useful. The second model was used to predict if the e-messaging system was easy to use. The items “received training” (P=.88), “hindered by poor functionality” (P=.84), and “full-time equivalent percentage” (P=.77) did not show statistically significant differences between the prediction errors in the group that found the e-messaging system easy to use and the group that did not find the tool easy to use.

We tested for possible multicollinearity among the independent variables in both models using the Pearson correlation to exclude that the independent variables were highly correlated. The highest correlation was .76 for age and years of experience as an RN; therefore, no significant multicollinearity was found [42].

In the logistic regression analysis, we dichotomized the variables because of their skewed distribution. The variable “useful tool” was dichotomized as strongly disagree, disagree, neutral, and agree (0=1-4) and strongly agree (1=5). The rest of the ordinal variables were dichotomized as strongly disagree, disagree, neutral (0=1-3), agree, and strongly agree (1=4-5).

The P value of the Hosmer and Lemeshow goodness of fit statistic for “the useful tool model” was .317, and for “the ease-of-use model” the P value was .650. The data were analyzed using IBM SPSS Statistics version 20.0.

A total of 425 (67.2%) of the 632 home health care nurses who received the questionnaire completed it. The demographic information is presented in Table 1.

The vast majority of the responding home health care nurses (388/399, 97.2%) agreed or strongly agreed that e-messaging was a useful tool for communication with GPs. Table 2 presents all responses in order to show the complete distribution of the responses.

For the “easy to use e-messaging” item, the majority of home health care nurses agreed or strongly agreed (357/398, 89.7%) that e-messaging was easy to use. A high proportion of the home health care nurses agreed or strongly agreed that they had received training (333/399, 83.5%), while the remaining nurses were neutral or disagreed. The home health care nurses agreed to a lower degree that they had access to user support (256/389, 65.8%).

A relatively high proportion of the home health care nurses were neutral or agreed that they were hindered when using e-messaging because of poor functionality (123/395, 31.1%), low system performance (123/396, 31.1%), or software errors (129/395, 32.7%).

The logistic regression model showed that the “easy to use” item gave a statistically significant contribution to the model (Table 3).

The odds of reporting that e-messaging was a useful tool were over five times higher if the nurses agreed or strongly agreed that e-messaging was easy to use. The “useful tool model” was controlled for the demographic variables of gender, age, years of experience as an RN, full-time equivalent percentage, and years in current position.

The “ease-of-use model” determined which elements influenced the assessment of e-messaging as easy to use. The results from the logistic regression model (Table 4) showed that the independent variables “hindered by poor functionality”, “training received”, and “full-time equivalent percentage” were statistically significant.

The odds that a home health care nurse would experience e-messaging as easy to use increased as the full-time equivalent percentage of the nurses increased. If the home health care nurses agreed that they were not hindered by poor e-messaging functionality, the odds of agreeing that e-messaging was easy to use were over 6.9 times higher than if they did not agree. The home health care nurses who agreed that they had received training had odds over 6.6 times higher of agreeing that e-messaging was easy to use. The home health care nurses’ gender, number of months using e-messaging, being hindered by low system performance or software errors, or having access to user support were not statistically significant predictors for the assessment of e-messaging as easy to use.

The results from this study showed that individual, organizational, and technological elements are interrelated and affect home health care nurses’ assessments of using e-messaging to communicate with GPs. This is in line with a sociotechnical perspective [20]. This study demonstrated that the majority of the home health care nurses assessed the e-messaging system as a useful tool for communication with GPs.

Several home health care nurses cooperated with the vendors in the development and implementation of the e-messaging system. The realization of an ICT system relies on the participation of the people who will ultimately use it [19]. This is important because the system must fit the needs and working practices of the users. ICT development and implementation projects in health care can be controversial because ICT systems change organizational routines and relationships between different health care professionals. To prevent resistance and non-utilization of the ICT systems, users need to be thoroughly and systematically involved at an early stage in the development and implementation process [20]. The involvement of home health care nurses in the development and implementation of the e-messaging system may have resulted in a system that was better aligned with nurses’ needs and working practices; therefore, a positive assessment of the e-messaging system could be anticipated.

The only statistically significant predictor of e-messaging as a useful tool for communication with GPs was that nurses assessed e-messaging as easy to use. This is supported by a study of health care professionals’ adoption and use of a clinical information system that found that the ease of use of the system was required for it to be considered as a beneficial tool for their clinical practice [24].

Ease of use is one of the most frequent elements reported among studies of facilitators and barriers of the adoption of ICT systems in health care [21,22,24]. Ease of use is related to individual characteristics of the people who are using ICT systems, technical features of the ICT system such as the software and hardware, and organizational implementation of the ICT system in terms of training, procedures, user support, and configuration of the system. We found that only one individual element, full-time equivalent percentage, was a statistically significant predictor for e-messaging being easy to use. Previous research has reported opposing evidence that multiple individual elements, such as age, gender, years of experience as a nurse, and full-time or part-time work, affect the assessment of ICT systems [3,8,43-45]. However, we found that the higher the full-time equivalent percentage of the home health care nurses, the more likely they were to assess e-messaging as easy to use. One explanation for this finding is that the more time home health care nurses spend at work, the more they have access to training, user support, and help from their colleagues; thus, the nurse is more familiar with the use of e-messaging, which may result in a positive experience and assessment of the e-messaging system. Another explanation could be that the more time spent at work, the more time is spent using the system, and the more adept the user becomes with the system.

Tools/technical concerns, such as functionality and system design, are important elements that can act as barriers and facilitators for the implementation and use of ICT systems in health care [28]. In our study, a lack of poor functionality was the strongest predictor for the home health care nurses’ assessments of e-messaging as easy to use. This finding is supported by another study on nurses’ assessments of health care technology, which revealed that poor system design was among the most common elements for negative attitudes [12]. Ease of use is also related to the technical environment and the integration between ICT systems. In a study on usability, the results showed that physicians rated ICT systems as low because the integration between the ICT systems was insufficient [11]. In this study, the technological environment was important in the way that the e-messaging system was integrated into the EHR systems that home health care nurses were already using. The user interface was well known, which lowered the threshold for implementing and adopting the e-messaging system [16].

Assessments of ICT systems are affected by organizational aspects, such as offering user support and training [11]. Our results show that a high proportion of the home health care nurses had received initial training in the use of the e-messaging system, which was a strong predictor for assessing the e-messaging system as easy to use. Training has previously been identified as a key element for the successful implementation of ICT systems in complex systems such as health care services [7,9,15,46,47]. Training is important in ensuring that the systems are used in the intended way and reducing the risk of incorrect use, which could jeopardize patient safety [7,15,27]. Nurses’ use of ICT systems is affected by training in both basic ICT and specific software [3,28,48]. Training at different levels should be offered by organizations and must be tailored to the individual needs of home health care nurses [49].

By applying a sociotechnical perspective, we were able to detect and predict which elements were important for the development and implementation of e-messaging. All of these elements are interrelated and shape the sociotechnical system; furthermore, these elements should be considered when developing and implementing new technology in home health care services [20].

This study has several limitations. The best methodology to develop a questionnaire is to include a pilot study with a smaller sample using all possible variables and subsequently identifying the relevant ones. However, we did not complete this type of study because of our limited time frame. As an alternative, we based the questionnaire on explanatory variables identified by the focus group interviews and existing research.

The questionnaire was not assessed across all dimensions of validity and reliability. However, the aim of the study did not include a complete psychometric testing of the questionnaire.

The mean full-time equivalent percentage may have been falsely high because one of the inclusion criteria for participation in the study was that the nurses had to be engaged in a 50% or greater full-time equivalent position. This was necessary to ensure that the nurses had enough experience in using e-messaging. According to the Norwegian Nurses Organization in 2011, 56% of the nurses in primary care in Norway did not work in a full-time positon, and 15% of those had less than a 50% full-time equivalent position [50]. The odds ratio of the full-time equivalent percentage predictor may be higher than our results suggest; thus, the results in our study may understate the odds of the full-time equivalent predictor in the general population. Another element that may have biased the results was that the home health care nurses who participated in this study worked in the first municipalities in Norway to implement e-messaging. Being among the early adopters may have caused the home health care nurses to be especially encouraged, enthusiastic, technologically optimistic, and positive toward e-messaging. These characteristics could differ in populations that were late to adopt the system.

The strength of this study is that we questioned home health care nurses in all of the municipalities that had used the e-messaging system for at least 6 months at the time of the study. The response rate of the study was 67.2%, which can be considered fairly good for a questionnaire study [51].

By identifying elements that affect home health care nurses’ assessments of e-messaging for communication with GPs, we are better able to determine which interventions are most important for the development and implementation of e-messaging. This study demonstrated that home health care nurses assessed e-messaging as a useful tool for communication with GPs. It also shows that ICT systems must be easy to use to be assessed as a useful tool and that a higher full-time equivalent percentage, having received training, and not being hindered by poor functionality are important predictors for the assessment of the e-messaging system as easy to use. Our results imply that users should be actively involved in the development and implementation of ICT systems. Future studies should use a sociotechnical approach to consideration the complete range of elements that can affect working practices and the outcomes for the patients and organizations involved. These insights may help to increase the understanding of effective strategies for developing and implementing ICT systems in home health care services.