This prospective observational study on agreement between EIMD measurements and patient-reported inhalations was conducted between April 18 and May 25, 2018.

This study was conducted by 21 students in the second year of their master’s program in pharmacy from the University of Leiden, the Netherlands, during their internships in community pharmacies. The students were asked to validate EIMD registrations for two patients each, for 3 weeks, and they received additional training on the use of the EIMD program. The individual monitoring of patients was time consuming because of the protocol-specific information requirements, and consequently, dedicated persons with enough time were needed in the pharmacy. Additionally, the acquaintance with an innovative device to be implemented into daily practice and individual patient coaching were relevant learning objectives for the students.

Dutch pharmacists have a professional and legal responsibility to provide drug treatment for their patients and, as most patients in the Netherlands attend one community pharmacy, pharmacists usually possess the complete medication histories of their patients [29].

The study protocol was approved by the Ethical Committee of the Radboudumc Nijmegen (approval number, 2018-4153). Written informed consent was obtained from all individual participants included in this study, prior to the study.

The EIMD to be assessed was the Turbu+ V2.1 (AstraZeneca UK Limited), consisting of three components: (1) the electronic device that could be attached to the inhaler of the corresponding product “Symbicort Turbuhaler,” (2) an app to be installed on the patient’s mobile phone, and (3) an online portal allowing the health care professional access to the same actuation data. During patient inhalation, the device was actuated and the date and time of the actuations were recorded. The data were synchronized with the app on the mobile phone by Bluetooth, and the app visualized the timeline of these data up to the previous month. All EIMD data for measured actuations were automatically and electronically linked to the online portal of the health care provider for his/her patients and an additional research database containing the same data but anonymized. The research database has been setup to study patterns of medication adherence and to evaluate the effectiveness on interventions for medication adherence across multiple studies. Actuation data from all health care professionals included in this study were aggregated into one central research database containing data on all included patients from the participating pharmacies.

Patients could only use the EIMD program after they were enrolled and trained by their health care provider. After entering the name, birth of date, sex, dose regimen settings, and email address, the patient automatically received an email with the link to download the app. The time(s) for a pop-up reminder in the app could be set individually by the patient at the time of scheduled inhalation(s). If patients did not take their medication, 30 minutes after the scheduled inhalation, a “missed-medication” motivational message was sent automatically. Patients received a weekly motivational push notification in the app (eg, “Great week. You’ve been following your prescription this week! Keep it up!”).

Patients were eligible to participate if they (1) were current users of budesonide/formoterol Turbuhaler (Anatomic Therapeutic Chemical code R03AK07) [30] with at least two prescriptions in the previous 12 months; (2) were of age ≥ 18 years; (3) were regular patients in the pharmacy (registered in the pharmacy system and receiving dispensings from this pharmacy only); (4) had access to a smartphone, and (5) were able to use the internet. We aimed to include current users who were familiar with the inhaler. At random, eligible patients were invited during regular pharmacy visits or by phone. If interested, patients received an information leaflet on the study and an informed consent form. Patients interested in participating were asked to provide informed consent, allowing the student to collect general data about gender, age, use of short corticosteroid courses (indicating exacerbations), and refills from the pharmacy database (to assess adherence rates). Eligible patients were invited to the pharmacy for an intake visit.

During the intake visit, the students provided the patient with an EIMD and (oral and written) instructions. The patients attached the device to the inhaler and installed the app during the visit or at home following the instructions. Additionally, the students verified the pharmacy’s information collected on drug use, patient’s refill adherence to inhaled corticosteroids, and the number of oral corticosteroid courses in the previous year. Subsequently, the patients completed the Beliefs About Medicines Questionnaire (BMQ), consisting of two parts: the BMQ Specific list, which measures perceptions of specific medicines, and the BMQ General list, which measures general beliefs about medicines. The BMQ Specific list comprises two scales—one assessing patients’ beliefs about the necessity of preventer medication for maintaining present and future health (Necessity scale) and the other for assessing their concerns about the potential adverse consequences of using such medication (Concerns scale) [31,32]. All items were rated on a 5-point Likert scale, with a range of 5-25 possible scores for both scales.

The students explained that they would call the patients 6 times within 14-21 days to check on their actual drug intake on the previous days and agreed on the most suitable days and time to call. If patients wished to, they were allowed to use a paper diary.

During the follow-up period of 14-21 days, the students phoned the patients 6 times to check on their actual drug intake and EIMD performance from the previous day, and if possible, up to the day of the earlier phone call. The student collected and reported information on the number and time of daily inhalations, and all particularities or other circumstances that the patient reported were registered by the student in an Excel file. These were subsequently centralized into one patient self-reported database.

To avoid socially desirable answers and to collect reliable data, patients were instructed that this study focused on the accuracy of the device (instead of their medication use), and they were asked to use the medication at their usual dose and frequency and to use only the EIMD prepared device during the study period and no other inhalers of this medication. Semistructured questions were used during the phone call; for example, “When did you use your medication for the last time?” and “Do you see any registrations in your app that do not represent real medication use?”

During the phone call, the student had immediate access to the EIMD data and compared the data from the research database with the information reported by the patient (and registered in the patient self-reported database). Any discrepancies were directly discussed with the patient and registered.

To evaluate patients’ usability, patients received two questionnaires at the end of the study, including the System Usability Scale (SUS), which is a validated instrument for evaluating the usability of a wide range of products and services [33,34]. The SUS score can range from 0 to 100. For products with a score less than 70, improvement options should be considered [34].

The second questionnaire addressed the experiences and acceptability of the program and the expectations on the pharmacists’ role in counselling. This part consisted of 12 statements, formulated positively or negatively, measured on a 5-point Likert Scale (1=strongly disagree, 5=strongly agree), followed by 3 multiple-choice questions regarding the frequency of using the app and whether the patient would like to continue the program and by 4 open questions on advantages, disadvantages, and targets for improvements.

The main outcome measure was the positive predictive value (PPV), calculated as (the number of correctly registered doses according to the patient self-reported database and registered database)/(the total number of registered doses [true and false positives])*100. Results of the questionnaires were analyzed using descriptive statistics. Responses to the open questions were coded and summarized for patients’ opinions, wishes, and barriers encountered using the EIMD program to identify key topics via the grounded theory approach [35].

Patient-specific characteristics were assessed with the scores for the BMQ Specific and BMQ General questionnaires, represented on a continuous scale. The BMQ necessity and concerns scores were split at the scale midpoints to distinguish between four subgroups: accepting (high necessity, low concerns), ambivalent (high necessity, high concerns), skeptical (low necessity, high concerns), and indifferent (low necessity, low concerns) [36]. Refill adherence was calculated as the proportion of days covered in the prior 12 months (due to prior dispensings up to 15 months).

A total of 32 patients were included by 18 students (Table 1). In 3 pharmacies, no patients eligible and willing to participate were available. The mean age of the included patients was 48.1 years (range, 20-69 years), and 17 patients were female (53.1%). Medication adherence in the previous year calculated from dispensing data was 81.3%. Most patients used their medication twice a day (n=26). Three patients (9%) used an additional diary to note their daily inhalations. Twenty-five patients (78%) completed the questionnaires and provided individual comments on the EIMD (Table 2 and Textbox 1).

Thirty patients (94%) completed the BMQ General and Specific questionnaires at the start of the study. The majority of them could be classified as acceptant (n=19) regarding their inhalation medication: 4 patients were ambivalent and 7 were indifferent. No patients were classified as skeptical.

Of the 32 patients, 28 completed all 6 phone calls. Overall, the 18 students verified 932 medication doses (mean=29.1 doses verified per patient; range=3-88). A total of 796 doses registered in the research database matched patient-reported inhalations captured in the patient self-reported database (true-positive, 85.41%). In addition, 33 inhalations were registered in the research database that did not match the actual drug intake, as reported by the patients in the patient self-reported database (false-positive, 3.5%). Further, 22 patients reported an average of 4.7 inhalations (range, 1-20) that were not recorded in the research database, accounting for a total of 103 inhalations (false-negative, 11.1%).

Information on registration of 6 of the 32 devices was in full agreement with the patient-reported inhalations. The PPV of all registrations in the research database was 96.0%.Some patients reported technical problems with EIMD data recording and synchronization of the EIMD data with their mobile phone. Delayed data synchronization resulted in two patients taking more medication than prescribed under the assumption that they had forgotten their dose.

The majority of patients indicated that the app was easy to use and not unnecessarily complex; the mean SUS score was 68.9 (SD 11.34; range, 52.5-90).

In the acceptability questionnaire, patients rated the EIMD generally as useful (76% neutral or agree) and 84% rated the EIMD program as not time-consuming (Table 2). With regard to recommending the EIMD to other patients, the majority rated this item as neutral (n=12, 48%). In addition, 80% of the patients indicated that they were not willing (n=15) or uncertain (“maybe,” n=5) about continuing the program themselves, and 70% (n=14) of these patients reported that they had only participated in this study because they wanted to help the students fulfil their assignment and facilitate the research. The five patients who were positive about continuation reported that they participated mainly to gain personal insight into their inhalation patterns. The patients generally felt positive about the pharmacists’ role in counselling.

Written feedback on the advantages, disadvantages, and targets for improvements was provided by 25 patients. This feedback was clustered into four themes: EIMD functionality, reminders, motivational messages, and attitude toward electronic monitoring (Textbox 1).

Some of the previously mentioned technical issues reported by some patients resulted in a level of frustration or confusion because of missed or unnecessary reminders due to missing data, which impacted their acceptance of the app (Textbox 1). The reminders and motivational messages were appreciated by 9 and 4 patients, respectively. Several patients suggested more individualization of the settings in the app, such as inclusion of a personal choice to share data with a specific health care provider and management of their dose regimen settings.

In this study, we found an acceptable PPV: 96.0% of the registered doses represented patient-reported drug intake. However, we found a high number of false-negative registrations: 11.1% of patient-reported inhalations were not recorded by the EIMD.

These unrecorded patient-reported inhalations could have been the result of a number of factors, either technical or user-related issues, that should be discussed. Although in earlier studies, loss of data or missing data were associated with technical issues such as battery drain, this was not likely to have occurred during the short duration of our study [7,24,26]. The high number of false-negative registrations could possibly be the result of overreporting by the patients or suboptimal use of the device (eg, Bluetooth not activated, EIMD not paired to the phone, or EIMD not within a 5-meter distance from the phone during inhaler actuation). Assessment of user experience revealed that the written instruction did not contain detailed information about the data synchronization protocol, which, combined with instructions at enrolment, may have led to some of the reported observations. In earlier research with the same EIMD, the researchers had presumed the possibility of false-negative as well as false-positive registrations, but they were unable to verify the registrations by patients’ actual inhalations or the data on user interaction with the system to interpret their findings [15].

This study demonstrated the importance of validating medication adherence data in real-world settings. Patients’ interaction with the device is critical for objective measurement of medication adherence in research and clinical settings. We emphasized on the importance of evaluating technical performance to identify technical/user issues and stressed the need of evaluating usability and acceptability across multiple components of the EIMD.

Previous studies on EIMD performance focused on accuracy and reliability in laboratory settings and lacked data collected in a real-life setting where patients interact with the EIMD. Furthermore, previous studies did not access patient acceptability and user experience [7,20,23,24,26,37,38], both of which are essential for successful implementation and sustained use in daily practice [39]. This is the first study to demonstrate the importance of evaluating and validating EIMDs in a real-world setting. The accuracy of inhalation measurement is essential from not only a research point of view, but also a clinical perspective, as false-negative registrations lead to unnecessary signals and reminder messages. Furthermore, it falsely reports patients as nonadherent, and this could lead to underestimation of adherence, incorrect clinical decision making, and overuse of medication when patients assume to have forgotten their dose.

Our findings on acceptability and user experiences further underline the importance of evaluating EIMDs on acceptability, preferably early in the implementation process. There is a growing body of evidence on electronic health apps, in general, that do not perform as expected in clinical practice, because the app turns out to be unacceptable or does not fit the users’ needs [40-43]. Although patients indicated that the app was easy to use and not complex, the majority did not intend to use the app in the future, apparently because there was no clear personal need to use the EIMD. They were primarily motivated to use the EIMD to facilitate the research rather than having an intrinsic motivation to gain personal insights into their inhalation patterns, and their medication adherence in the previous year calculated from dispensing data was already high. Patients with intrinsic motivation to improve their disease management (eg, based on low adherence or impaired disease control) would possibly benefit more from the EIMD. Therefore, understanding different types of patient segments is important to succeed in the implementation; the EIMD needs to match with the patient profile (eg, adherence and asthma control), needs, and preferences. In addition, the fact that the majority of patients did not intend to continue with the EIMD may have been the result of a suboptimal technical performance, for example, loss or troubles with the Bluetooth connection, which was regularly experienced; delay in the synchronization of data from the EIMD to the mobile app; and inhalations taken just before midnight not visible on the intended day. These technicalities can probably easily be improved and thereby increase the chance of acceptance and successful implementation. Moreover, health care providers and patients would benefit from further development of the EIMD, so that it can not only detect inhaler actuation, but also check the inhalation quality, breath force, and inhalation technique.

Our findings emphasized the need for clear patient selection and a more individualized, tailored device. In this study, half of the people found the reminders helpful in taking the medication on time, while the other half found them useless. We found similar results for the motivational messages. Hence, when designing a self-management intervention containing multiple self-management strategies (ie, motivational messaging, reminders, and audio-visual behavioral feedback), it is important to critically review each component on usability separately and preferably tailor the intervention to the needs of the patient.

This study has some limitations. First, the actual inhalations were self-reported by the patients, which could have led to bias by overreporting actual use (and thus also an overestimation of unrecorded actuations), as patients might tend to provide socially desirable answers. However, as a 24-hour patient observation was not feasible, this was the best way to collect data on patients’ actual drug intake within this setting. To prevent socially desirable answers, patients were instructed at the intake that this study was on verifying the registrations of the EIMD rather than their medication use or adherence, and during the phone calls, the students asked open questions. Additionally, patients were questioned about their actual inhalation mainly at the same and previous days, with a maximum of 5 days prior. Thus, the phone calls for data evaluation focused on the most recent moments of drug intake in order to reduce possible recall bias. In further analysis regarding the possible impact of recall bias, we did not observe any differences between measurements on day 5 and those on the earlier days, or between different age groups (data not shown). As earlier studies have shown that there is considerable variation in the accuracy of diaries to note medication intake [44], the use of a diary was not mandatory. As patients frequently reported technical issues as a possible explanation for both missing and extra registered inhalations, we do not expect much bias from this setting.

Second, all pharmacies were related to the master’s education program of pharmacy from the University of Leiden. Data were collected in different pharmacies and by different students, to prevent bias from specific settings. However, such bias could not be fully excluded. Although the students were not yet registered health care providers, they were quite motivated and technically skillful. The students selected a convenience sample of patients, although it seemed difficult to find patients with the original inhalation medication that was fit for the EIMD; in some pharmacies, no patients could be selected at all. Third, the short inclusion time of this study, due to the internship period, could have led to selection of patients who were more willing to help the students with their task rather than being interested in their own medication performance or adherence. This was reflected in patients’ individual comments: They regularly indicated that the program could be especially useful for other patients, but that it was of little value to them. Some patients reported that they did not need “help with the inhalation” or “an app for taking the medication.” With a mean medication adherence of >80%, this seems to be a group with relatively high adherence and could indicate selection bias of the more adherent patients, for which the device may be less useful. Further research is needed on how health care providers should preselect patients for an EIMD on the basis of their experiences.

A strength of this study was the intensive follow-up from students, with more than 900 validated measurements. Although the number of included patients was limited, the number of drug intake comparisons was sufficient to detect omissions in the recorded actuations. However, the small population may have affected the representativeness of the results of the acceptability questionnaire. The statements in this questionnaire were formulated positively or negatively to reduce the risk of positively biased answers. This questionnaire was adapted to this specific EIMD and not validated beforehand. As a consequence, some patients might have experienced difficulties in comprehending the language used or the variety in both positively and negatively formulated questions.

It is recommended that the discussed technical issues should be further elucidated and solved before using EIMD data as an objective adherence measurement, and medication overuse may also be of interest. In order to fully benefit from the EIMD and guarantee reliability and validity, an EIMD should be validated in a setting where the users interact with the system and can encounter technical or user issues. Furthermore, EIMDs and the accompanying self-management program should be evaluated on usability in daily practice. This study provided an example of how to do this. Validation of EIMDs in real-world settings is likely to improve usability in daily practice; the EIMD should be easy to use and measure all actuations correctly, even when the patient is not technically skilled. Future research should pay sufficient attention to different types of patient segments, as the EIMD needs to match with the patient profile (eg, adherence and asthma control), needs, and preferences.

Comparison of EIMD data with patient-reported inhalations showed that EIMD registrations represented patient inhalations to an acceptable degree, with a PPV of 96%, but these registrations were likely to underreport actual drug intake by 11%. Technical improvements should address the Bluetooth connection and data synchronization. Additionally, patient characteristics contribute to the validity of EIMD measurements, and larger sample sizes are needed to explore their influence. For the acceptance of a self-management program with an EIMD, patients who benefit from self-monitoring and reminders should be targeted by tailoring the possibilities to the needs of the individual user.