Published on 16.10.15 in Vol 17, No 10 (2015): October
Evaluating the Validity of an Automated Device for Asthma Monitoring for Adolescents: Correlational Design
Background: Symptom monitoring is a cornerstone of asthma self-management. Conventional methods of symptom monitoring have fallen short in producing objective data and eliciting patients’ consistent adherence, particularly in teen patients. We have recently developed an Automated Device for Asthma Monitoring (ADAM) using a consumer mobile device as a platform to facilitate continuous and objective symptom monitoring in adolescents in vivo.
Objective: The objectives of the study were to evaluate the validity of the device using spirometer data, fractional exhaled nitric oxide (FeNO), existing measures of asthma symptoms/control and health care utilization data, and to examine the sensitivity and specificity of the device in discriminating asthma cases from nonasthma cases.
Methods: A total of 84 teens (42 teens with a current asthma diagnosis; 42 without asthma) aged between 13 and 17 years participated in the study. All participants used ADAM for 7 consecutive days during which participants with asthma completed an asthma diary two times a day. ADAM recorded the frequency of coughing for 24 hours throughout the 7-day trial. Pearson correlation and multiple regression were used to examine the relationships between ADAM data and asthma control, quality of life, and health care utilization at the time of the 7-day trial and 3 months later. A receiver operating characteristic (ROC) curve analysis was conducted to examine sensitivity and specificity based on the area under the curve (AUC) as an indicator of the device’s capacity to discriminate between asthma versus nonasthma cases.
Results: ADAM data (cough counts) were negatively associated with forced expiratory volume in first second of expiration (FEV1) (r=–.26, P=.05), forced vital capacity (FVC) (r=–.31, P=.02), and overall asthma control (r=–.41, P=.009) and positively associated with daily activity limitation (r=.46, P=.01), nighttime (r=.40, P=.02) and daytime symptoms (r=.38, P=.02), and health care utilization (r=.61, P<.001). Device data were also a significant predictor of asthma control (β=–.48, P=.003), quality of life (β=–.55, P=.001), and health care utilization (β=.74, P=.004) after 3 months. The ROC curve analysis for the presence of asthma diagnosis had an AUC of 0.71 (95% CI 0.58-0.84), which was significantly different from chance (χ21=9.7, P=.002), indicating the device’s discriminating capacity. The optimal cutoff value of the device was 0.56 with a sensitivity of 51.3% and a specificity of 72.7%.
Conclusions: This study demonstrates validity of ADAM as a symptom-monitoring device in teens with asthma. ADAM data reflect the current status of asthma control and predict asthma morbidity and quality of life for the near future. A monitoring device such as ADAM can increase patients’ awareness of the patterns of cough for early detection of worsening asthma and has the potential for preventing serious and costly future consequences of asthma.
J Med Internet Res 2015;17(10):e234
Achieving acceptable asthma control in adolescents remains elusive despite the availability of efficacious treatment options. In 2010, nearly 11% of adolescents (2.7 million) aged 12-17 years in the United States reported a current diagnosis of asthma  and adolescents suffer greater asthma-related morbidity than other age groups [ ]. Adverse asthma outcomes in this age group are largely attributable to poor self-management [ - ]. Establishing daily routines of symptom monitoring is recognized as the initial step to successful asthma self-management [ , ] leading to better asthma outcomes [ ].
National Heart, Lung, and Blood Institute (NHLBI) Expert Panel Review 3 (EPR 3)  recommends that all patients with asthma learn to recognize symptom patterns related to inadequate asthma control. Currently, there are 2 basic types of home-based symptom-monitoring methods that patients can use to monitor symptom patterns: symptom-based and peak flow monitoring. However, many studies have consistently raised a concern about young people’s poor perception in recognizing asthma symptoms [ - ]. Adolescents, in particular, tend to be overly optimistic in rating their asthma control despite the presence of symptoms and activity limitations [ - ]. As a consequence, sole reliance on patients’ perception in symptom monitoring can be misleading.
Peak flow monitoring has been recommended to enhance the objectivity of symptom monitoring, particularly for those who suffer a high level of asthma severity because of their impaired symptom perception [- ]. However, confirming the reliability of peak expiratory flow rate values has been an ongoing challenge [ - ]. Moreover, its clinical usefulness in children and adolescents is hampered by users’ poor adherence and inadequate techniques, and the effort-dependent nature of the method. Concerns have been raised about questionable long-term sustainability of peak flow monitoring and inaccurate and/or fabricated readings [ - ].
Given the limitations of the existing symptom-monitoring methods, alternative strategies have been called for to mitigate the previously mentioned issues and enhance objectivity and sustainability for continuous symptom monitoring in children and adolescents [, , , ]. Recently, we developed an Automated Device for Asthma Monitoring (ADAM) to increase the objectivity of symptom monitoring and to facilitate adolescents’ adherence to continuous symptom monitoring in vivo. The device employed audio analysis technology to recognize symptoms, particularly coughs. The device uses a mobile system, iPod, as a platform. The methodological and technical details involving the development of the device and user acceptability are reported elsewhere [ , ]. This paper reports findings on the validity of the device as a monitoring tool. Specifically, this study examined (1) concurrent validity by correlating the data of the device with other measurements informing asthma control, including pulmonary function, fractional exhaled nitric oxide (FeNO), symptom-based monitoring (eg, daily asthma diary), asthma control questions, quality of life, and health care utilization; (2) predictive validity of the device by examining the extent to which the results of the device predict asthma control, quality of life, and health care utilization after 3 months; and (3) sensitivity and specificity of the device in discriminating between an asthma group and a nonasthma group.
Study Sample and Setting
Participant eligibility criteria for the asthma group were (1) age 13-17 years, (2) physician-diagnosed asthma for at least 1 year, and (3) ability to understand spoken and written English. The nonasthma group were age-matched adolescents with no current/past history of asthma and free of ongoing respiratory conditions. For both groups, we excluded those with other diagnoses producing respiratory symptoms (eg, upper respiratory infection, cardiac disease, cystic fibrosis) or significant cognitive impairment that could interfere with following the study protocol. Potential participants were recruited from the pediatric emergency department (ED) and outpatient clinics (primary practice and pediatric pulmonary practice) in a major university medical center located in the Northeastern United States. Of a total of 84 participants, most (73%, 61/84) were recruited from the ED and the remaining were from the study flyers (23%, 19/84) and clinician referrals from outpatient clinics (4%, 4/84). Unverifiable asthma diagnosis by medical records was the most common cause of ineligibility for the asthma group (n=52) followed by comorbidity with other respiratory diagnosis (n=14). For the nonasthma group, having an asthma diagnosis in the past (n=22) and presenting respiratory symptoms (n=7) were common reasons for ineligibility.
Study Measures: Both Groups
Automated Device for Asthma Monitoring
The ADAM device uses an iPod as a platform and was designed to continuously process audio data in real time to detect coughs. The device detected the number of cough events in 6-second intervals. It also provided a display of cough count data in a chart form on the device for users. Detailed descriptions of the device have been reported elsewhere [, ]. ADAM was used by all participants for at least 7 consecutive days.
Fractional Exhaled Nitric Oxide
FeNO is a noninvasive method of assessing asthmatic inflammation . Increasing FeNO levels have been found to be predictive of deteriorating asthma [ ] and correlates more closely with symptoms than does forced expiratory volume in the first second (FEV1) [ , ]. FeNO was measured before and after the 7-day trial in accordance with the American Thoracic Society (ATS) recommendations [ ] using NIOX MINO (Aerocrine AB, Stockholm, Sweden).
Pulmonary Function Test
To assess the degree of airway obstruction, the volume of air expired during the first second of a forced vital capacity maneuver (FEV1) and forced vital capacity (FVC) was measured using a KoKo spirometer (Pulmonary Data Service; Louisville, CO, USA) connected to a personal computer. Trained research staff performed spirometry for each participant two times, before and after the 7-day trial in accordance with the ATS standards .
Participant Demographic Form
Basic demographic information was collected, including gender, age, race, annual family income, years with asthma diagnosis (for the asthma group), types of health conditions that led to a clinic or ED visit (for the comparison group), and current medications (if applicable).
Study Measures: Asthma Group Only
Asthma Control Questions
The Asthma Control Questionnaire (ACQ) was developed based on the 2007 NHLBI National Guidelines’ asthma control classification criteria involving 4 areas of asthma impairment, including the frequency of daily activity limitations, asthma symptoms, nighttime symptoms, and use of short-acting beta agonists (SABA) in the past 4 weeks. The 4 questions were measured on a 5-point scale and higher total scores indicated better asthma control. The ACQ was administered at pretrial and at 6-month follow-up. Cronbach alpha of the scale was .79 in this study.
Pediatric Asthma Quality of Life Questionnaire
The 23-item Pediatric Asthma Quality of Life Questionnaire (PAQLQ) measures 3 subdomains pertaining to asthma-related quality of life in children with asthma aged 7-17 years, including symptoms (10 items), emotional function (8 items), and activity limitation (5 items) . Each item is measured on a 7-point scale (1=maximum impairment, 7=no impairment). Higher total scores indicate better levels of functioning. In this study, high internal consistency (Cronbach alpha) was found in all 3 subscales: .94, .95, and .88 for symptoms, emotional function, and activity limitation subdomains, respectively.
Health Care Utilization Form
The Health Care Utilization Form captured any asthma-related events including ED visits, hospitalization, office visits, and missed school days. This form assessed the frequency of events that occurred over the past 3 months (pretrial, 3-month follow-up) and the past 7 days for post-trial.
Visual Analog Scale
The Visual Analog Scale (VAS) is a line 100 mm long with 3 anchors dividing 3 zones (red, yellow, and green). For each symptom, there is a green zone (80-100 mm) labeled “no symptoms,” a yellow zone (79-50 mm) labeled “mild symptoms,” and a red zone (<50 mm) labeled “very bad symptoms” [, ]. Teens marked any point on the line according to their perception of asthma symptoms two times a day in the morning and evening during the 7-day trial. The distance between the 0-mm mark and the placement of the “X” was measured to provide a numeric interpretation of their symptom perception.
Asthma Control Diary
The Asthma Control Diary consisted of 6 items, each with scores ranging from 0 (no symptoms) to 6 (continual symptoms) . The device automatically sent diary reminders two times a day during the 7-day trial and allowed teens to conveniently complete the diary electronically using the touchscreen. The device automatically triggered diary reminders and made diary questions available only within the designated time window for am (6 am-noon) or pm (6 pm-midnight) to minimize recollection errors or the risk of data fabrication later. Morning questions pertained to nocturnal waking and morning symptoms, and evening questions assessed the degree of activity limitation, daytime symptoms (shortness of breath and wheeze), and SABA use in previous 24 hours. The mean score for each diary question was computed with higher scores indicating a greater degree of symptoms.
At enrollment, spirometry and FeNO tests were conducted for all participants followed by the measurement of asthma control and health care utilization for the asthma group. All participants used the device continuously for the next 7 days during which the asthma group completed daily the electronic asthma diary and the VAS in the morning and at bedtime. On completion of the 7-day trial, spirometry and FeNO tests were repeated for all participants. In addition, the asthma group completed a quality of life questionnaire and reported any health care utilization that occurred in the past week. Follow-up data on asthma control, health care utilization, and quality of life were collected at 3 months after the trial only from the asthma group. Only 2 of 42 participants in the asthma group were lost to the 3-month follow-up. Of the 40 follow-up cases, 30 were completed by mail and 10 were in-person with research staff. This study protocol was approved by the Institutional Review Board, the Research Subjects Review Board, located in the University of Rochester Medical Center, Rochester, NY, and informed consent and assent were obtained from parents and teens, respectively. The participants received a monetary incentive (US $130 for the asthma group; US $100 for the nonasthma group) for their participation.
All analyses were performed using SAS v9.3 (SAS Institute, Inc, Cary, NC, USA). Descriptive statistics were used to examine demographic and clinical characteristics. To assess concurrent validity, Pearson correlations were calculated between the device data from the asthma group with spirometer data (FEV1 and FVC), FeNO, and other measures of asthma conditions, including asthma control questions, daily symptom diaries, VAS, quality of life, and health care utilization. To assess predictive validity of the device, regression was used to examine the relationships between the device data and asthma control, health care utilization, and quality of life collected at 3 months after the 7-day trial. Age, gender, and race were adjusted for in the regression analyses. Sensitivity and specificity were evaluated by assessing the device’s capability to classify 2 distinctive groups. A receiver operating characteristic (ROC) curve analysis was conducted to calculate the area under the curve (AUC) as an indicator of the device’s capacity to discriminate between asthma and nonasthma participants. An AUC of 1 indicates perfect classification and an AUC of 0.5 indicates that the ability to correctly classify is no better than chance. Based on the ROC, an optimum cutoff value was chosen to jointly maximize the sensitivity and specificity of the device. Given the few studies looking at devices for symptom monitoring in adolescents, any noticeable statistical trend is valuable; therefore, the significance level used in the analysis was less than .10. These findings are useful for an initial understanding of the validity of ADAM and for providing direction for further studies. Although it is understood that these findings will require further investigation, the risk of rejecting important research hypotheses was judged more important than the risk of type I error.
Details of participant flow from screening to the 3-month follow-up and sociodemographic characteristics of each group in the sample are reported elsewhere . A total of 84 adolescents aged between 13 and 17 years (mean 15 years, SD 1.4) participated in the study. Of those, 61% (51/84) were females and 44% (37/84) were nonwhite adolescents. No significant differences were found between the asthma and nonasthma groups in age and gender. The asthma group included significantly more nonwhite adolescents, predominantly African American. At enrollment, the asthma group had slightly lower FEV1 (mean 88.3%, SD 16.3 vs mean 90.9%, SD 13.8) and elevated FeNO (mean 28.6 ppb, SD 38.6 vs mean 25.6 ppb, SD 24.8) than the nonasthma group, yet these differences were not statistically significant. Within the asthma group, 19 of 42 (45%) reported active asthma symptoms at enrollment. Mean years since asthma diagnosis was 10.4 (SD 4.9) years. SABA use was reported by 95% (40/42) of the asthma group and most (60.5%, 25/42) were on at least one controller medication. The most common medication was inhaled corticosteroids (ICS), which was reported by 38% (16/42) of the asthma group, followed by ICS and long-term beta-agonist combination (29%, 12/42), and leukotriene modifier (21%, 9/42). No significant difference in control medication use was found between the symptomatic and nonsymptomatic groups. Only 2 participants (5%, 2/42) reported oral steroids as a current medication.
Coughs in the Asthma Group Monitored by the Device
Descriptive analysis was conducted on data from the asthma group excluding 3 asthma teens for which no data were recorded in the device due to unknown mechanical issues. All 39 teens with asthma used the device for a mean 8.26 (SD 1.47) consecutive days (median 8, range 5-14 days) and each teen used the device for a mean 19.4 hours/day (SD 1.71; median 19.7, range 15-22 hours). When the number of coughs was compared for morning (6 am-11:59 am), afternoon/evening (noon-10 pm), and bedtime (10 pm-6 am), a greater number of coughs were registered during afternoon/evening compared to morning and bedtime (P=.01 and P=.004, respectively). The number of coughs was not significantly different by gender or age.
Cough Counts and Fractional Exhaled Nitric Oxide and Pulmonary Function
No significant correlations were found between the device’s cough data and FeNO. The number of coughs was negatively associated with FEV1 (r=-.26, P=.05) and FVC (r=-.31, P=.02) at enrollment indicating that the higher number of coughs was associated with poor pulmonary function.
Cough Counts and Daily Symptom Diary Data and Visual Analog Scale
Cough counts showed a positive association with limited activities (r=.46, P=.01) and shortness of breath (r=.29, P=.07). Cough counts showed significant associations with symptoms as measured by VAS at nighttime (r=.40, P=.02) and daytime (r=.38, P=.02).
Cough Counts and Asthma Control and Quality of Life
shows correlations between cough counts and asthma control and quality of life during the trial. Cough counts were negatively associated with asthma control and overall quality of life, and the activity and symptom subscales of quality of life.
Cough Counts and Health Care Utilization
There was no association between cough counts and health care use in the 3 months before the trial. However, cough counts showed positive association with health care use during the 7-day trial (r=.61, P<.001) indicating that higher cough counts were associated with the higher use of health care services, particularly with the number of days of hospitalization (r=.72, P<.001) and office visits (r=.72, P<.001). Cough counts were also positively associated with the number of missed school days (r=.70, P<.001).
Greater number of coughs was correlated with poor asthma control and quality of life and higher use of health care services 3 months later ().
|Quality of life total||-.47||.004|
|Emotional function subscale||-.44||.007|
|Health care utilization||.55||.02|
presents the extent to which cough counts predicted asthma control, quality of life, and health care utilization after adjusting for age, gender, race, and family income. Coughs significantly predicted asthma control, and the regression model explained 42% of the variance in asthma control. Cough counts also significantly predicted the quality of life total score (β=-.55, P=.001) and each of the subscales including activity, symptoms, and emotional function 3 months later. Explained variance in quality of life was 38% for the quality of life total score, and 28%, 35%, and 41% for activity, symptoms, and emotional subscales, respectively. Health care utilization during the 3 months following the 7-day trial was significantly predicted by cough counts (β=.74, P=.004), explaining 76% of the variance. Particularly, coughs were associated with ED visits (r=.47. P=.004), asthma specialist visits (r=.45, P=.005), and office visits due to worsening asthma (r=.39, P=.02) that occurred in the 3-month period after the trial.
|Predictors||Asthma control||Quality of life||Health care utilization|
|Age in years||.05||.73||.11||.47||.11||.53||.07||.65||.16||.29||.19||.42|
Sensitivity and Specificity
The capacity of the device to distinguish participants with asthma from those without asthma was assessed using ROC curve analysis. The ROC curve for the presence of asthma diagnosis had an AUC of 0.71 (95% CI 0.58-0.84). The AUC was significantly different from chance (χ21=9.7, P=.002). The optimal cutoff value of the device was 0.56, with 51.3% sensitivity and 72.7% specificity (). The cutoff value discriminating teens with asthma from those without asthma was translated into 0.83 coughs/hour or 19.92 coughs/day.
Overview of the Study Findings
summarizes findings pertaining to the validity of the device. Most of the expected relationships between ADAM data and conventional measures of asthma were substantiated by our findings.
|Types of validity||Expected relationships||Statistical method||Findings|
|Cough counts and FeNO and lung function||Positive association with FeNO; negative association with lung function||Correlation||No significant correlations with FeNO; cough counts were negatively associated with FEV1 and FVC|
|Cough counts and symptom diary data and VAS||Positive association||Correlation||Associated with limited activities and approached significance for shortness of breath and number of rescue medications use in the past 24 hours|
|Cough counts and asthma control||Negative association||Correlation||Cough counts were negatively associated with asthma control|
|Cough counts and quality of life||Negative association||Correlation||Approached significance with quality of life, activity, and symptom subscales|
|Cough counts and health care utilization||Positive association||Correlation||No association between cough counts and health care use before the 7-day trial; however, cough counts showed positive association with health care use during the 7-day trial|
|Cough counts and asthma control and quality of life 3 months later||Cough counts predicting asthma control and quality of life||Multiple regression||Coughs predicted asthma control 3 months later explaining 42% of the variance in asthma control. Coughs predicted the quality of life total score and each of subscales 3 months later, explaining variance in quality of life, which ranged from 28% to 41%|
|Cough counts and health care utilization 3 months later||Cough counts predicting health care utilization||Multiple regression||Coughs predicted health care utilization 3 months later explaining 76% of the variance in health care utilization|
|Area under the curve||ROC curve analysis||0.71 (95% CI 0.58-0.84)|
|Cutoff point||ROC curve analysis||0.56 (0.83 coughs/hour or 19.92 coughs/day)|
|Sensitivity||Discrimination of positive asthma diagnosis by a cutoff||ROC curve analysis||51.3% sensitivity|
|Specificity||Discrimination of negative asthma diagnosis by a cutoff||ROC curve analysis||72.7% specificity|
This study examined the validity of ADAM, an investigational device that automatically monitors coughs continuously in adolescents with asthma. To our knowledge, ADAM is the first fully automated portable device facilitating continuous monitoring of the frequency of coughs that involves real-time processing, analysis, recording, and displaying of symptoms. Previously, we reported technical details  and the acceptance of the device by teen users [ ]. In these earlier reports, we demonstrated the feasibility of developing an algorithm for coughs, but not for wheezes due to the wide intrapersonal and interpersonal variability of the acoustic signature of wheezing. Therefore, ADAM was evaluated solely as a cough-monitoring device at this time. Coughs are widely recognized as a key symptom of asthma [ , ] and the most common symptom of uncontrolled asthma in children and adolescents [ , - ]. Asthma patients report coughs as the most troublesome symptom in their lives and as a symptom of greater importance [ ]. Given the importance of coughing in asthma, ADAM can be a clinically useful monitoring tool not only for the symptom itself, but also for symptom burdens on individuals. In this paper, we examined the validity of the device as an asthma-monitoring tool and its capacity to discriminate asthma cases from controls.
Overall, we found positive temporal correlations, albeit low to moderate, between device data (ie, cough counts) and conventional measures of asthma symptoms and symptom control. Similarly, previous studies reported modest correlations between objective cough rates and subjective measures specific to cough (VAS for cough and cough scores) in individuals with asthma [, ]. In our study, the generic self-report measures of asthma symptoms made it difficult to assess the degree of agreement between the device’s cough counts and the amount of coughs perceived by individuals. Nonetheless, when conceptualizing coughs as an indicator of asthma condition, use of generic measures of asthma control to establish the device’s validity as an asthma-monitoring tool can be justified. The demonstrated relationships between cough counts and concomitantly assessed activity limitation and other symptoms as well as overall asthma control provide support for the validity of the device as a tool for overall asthma monitoring rather than simply for cough. As in another study [ ], the relationship between cough counts and quality of life suggest that experiencing coughing can take a toll on quality of life in teens with asthma. Moreover, the positive relationships between cough counts and health care utilization and school absenteeism during the trial could be further evidence that cough counts as measured by ADAM is a compelling indicator of asthma morbidity.
Unlike earlier reports of no relationship between spirometry data and coughs measured by an objective method [, ] or self-report [ ], we found that the cough counts by ADAM were associated with poor pulmonary function (as indicated by FEV1 and FVC), suggesting that the number of coughs are indicative of airway obstruction. However, consistent with an earlier study [ ], we were unable to establish the relationship between coughing and airway inflammation measured by FeNO. This may be because most (>50%) of the asthma group were asymptomatic (ie, no indication of active airway inflammation) before and during our trial period. Replication of the trial using a large number of patients with active symptoms is needed to determine the nature of relationship between coughs and airway inflammation.
This study demonstrated strong evidence for predictive validity of the device. After adjusting for sociodemographic factors, cough counts were predictive of asthma morbidity and quality of life at 3 months after the 7-day trial. The cough count predicted poor asthma control and quality of life at 3-month post-trial, accounting for 42% and 38% of the variance, respectively. Health care services used during 3 months after the trial were predicted by the cough count by our device and explained as much as 76% of the variance in acute health care utilization 3 months later. In an earlier study , worsening coughing was found to be predictive of severe asthma after 9 years in adult patients. Our findings support not only ADAM’s predictive validity, but also coughs as an important harbinger for upcoming deterioration of teens’ asthma morbidity that could undermine their overall well-being and impose serious burdens on the health care system. As such, the findings underscore the importance and need for a monitoring device such as ADAM that can increase patients’ awareness of the patterns of cough for early detection and has the potential for preventing serious and costly future consequences of asthma morbidity.
The device’s capacity to discriminate correctly the asthma group from those without asthma was assessed to determine sensitivity and specificity. We found poor diagnostic sensitivity of ADAM; that is, the chances of correctly identifying those with asthma were only 51% using 20 coughs/day as a cutoff. The poor diagnostic sensitivity of the device may have been due to the asymptomatic state of more than 50% of the asthma group in which the average cough counts were below the cutoff making it difficult for the device to appropriately identify asthma cases. In the future, studies maximizing differences between subsamples in symptom presentation will be essential for adequately assessing the sensitivity of the device. By contrast, ADAM demonstrated relatively better specificity such that the device correctly classified those who did not have asthma for 73% of cases. Using the device in an environment replete with everyday noises may result in a high number of cough counts exceeding the cutoff, even in those without asthma; thus, incorrectly excluding them from the nonasthma group. We observed occasional false events (2 coughs/hour) depending on environmental noises (70% sensitivity of the cough algorithm) , which is strikingly similar to 2.5 events/hour by the Leicester Cough Monitor [ ]. Given the poor capacity to correctly classify patients with asthma, ADAM is not suitable for determining a clinical diagnosis of asthma, but is intended solely for monitoring cough in those with a confirmed asthma diagnosis. Like any monitoring device, false alarms are still an issue because these can cause unnecessary concerns or undermine users’ confidence in the monitoring tool. Therefore, further optimizing the accuracy of the monitoring device is warranted by refining the algorithm and adopting noninvasive techniques to minimize the influence of environmental noises (eg, direct application of an adhesive microphone on the chest wall).
Several limitations to this study’s design warrant caution. Because the device stored only the number of coughs without sound recording, validating the accuracy of cough counts through the manual confirmation of corresponding cough events was not done. Except for pulmonary function tests and FeNO, we primarily used self-report measures that were not specific to coughs to establish the validity of the device. The generic nature of the measure may have contributed to some of the nonsignificant correlations between the measures and device data. Although self-report measures are inherently subject to recollection bias, we attempted to address the challenge by strategically collecting daily symptom data electronically. In health care utilization data, we observed more than 90% agreement between self-report and medical record review. Therefore, it appears that recollection bias played little influence on our validity outcomes. Nonetheless, future research is needed to evaluate the validity of the device by simultaneously recording raw cough sounds and by using cough specific measures.
Moreover, distinction between the asthma and nonasthma groups was blurred because most of the asthma group did not present active symptoms during the trial, which may account for the device’s poor sensitivity. Also, use of the small and convenient sample of adolescents limited generalizability of the findings. In addition, the length of the 7-day trial might not be long enough to observe any meaningful changes in asthma symptoms, health care utilization, or biological measures including FEV1 and FeNO. A longer observation period is warranted to assess the extent to which the device adequately captures changes over time in symptom patterns and other measures of asthma status. Lastly, there were a few technical challenges and limitations to the optimal operation of the device, which we discussed in our earlier report .
Comparison With Prior Work
Several approaches have been attempted to develop technologies to monitor symptoms objectively, particularly coughs [, , , - ]. However, these existing approaches are considered unsuitable for ambulatory monitoring of symptoms due to practical challenges, including the laborious and time-intensive nature of processing audio data (so not fully automated and unable to provide real-time feedback to users), inability to monitor continuously beyond 24 hours, and the conspicuous appearance of the systems. For instance, the Leicester Cough Monitor [ , , ] uses a similar sound recognition technical approach to that of ADAM. Unlike ADAM, in the Leicester Cough Monitor, the ambulatory component of the system consists strictly of an audio recorder and audio analysis is performed offline, which can take approximately 1 hour to process a 24-hour audio recording. This differs fundamentally from ADAM in which all of the processing/annotation of audio data are performed instantly on the mobile device and the results (cough counts) are provided as feedback in real time. Other cough detection systems [ ] are not intended for ambulatory use or have yet to be validated in vivo for an extended observation period. In that sense, ADAM is the first ambulatory cough-monitoring device on a consumer mobile system with a capacity to fully automatize continuous real-time processing. This is also the first attempt to evaluate the validity of a cough detection device as a symptom-monitoring tool for several consecutive days in adolescents with asthma. Using a popular mobile device, such as iPod, as a platform for ADAM was well received by adolescents with asthma [ ] and the majority of participants used the device daily for a week or longer period supporting ADAM as a sustainable asthma-monitoring tool for adolescents.
Overall, this study demonstrated the validity of ADAM as a symptom-monitoring device in teens with a confirmed asthma diagnosis. Poorly controlled asthma takes a toll on teens’ overall health and quality of life as well as on the health care system due to an increased economic burden associated with the use of urgent types of health care services. ADAM can potentially mitigate the adverse consequences by helping users detect and treat early symptoms before advancing to a poorly controlled state. The device is useful in increasing understanding about one’s current status of asthma control and in predicting asthma morbidity and quality of life for the near future. Such information can make the users become aware of symptoms and triggers and enable them to take appropriate and timely actions to address symptoms or prevent further deterioration of their symptoms. Objective symptom information from the device would be clinically useful in establishing optimum treatment plans and evaluating treatment effects. ADAM can be particularly useful and effective in monitoring symptoms occurring at night when environmental noises are minimal. Nighttime coughs were more common than wheezing  and worsening of asthma symptoms often occurs during nighttime [ ]. As such, nighttime symptoms are often indicative of poorly controlled asthma [ , ], but patients often do not recognize or tend to neglect to report nighttime symptoms [ - ]. ADAM can be an invaluable tool that monitors nighttime symptoms, which would provide important clinical insight into the degree of symptom control and the response to asthma treatment.
Nonetheless, we caution that the device should not be considered as a diagnostic tool and its application to a broader age range remains to be evaluated. Although we suggest coughing as an important indicator of asthma morbidity, understanding the symptom in the context of other information, such as activity level and medication use, can potentially augment its clinical value and relevance. Originally, ADAM was designed to monitor symptoms and activity levels simultaneously capitalizing on the host system’s (iPod) built-in accelerometer to offer an insight into the nature of the relationships between symptoms and activity levels (eg, exercise-induced asthma). However, continuously recording and processing sounds and activity simultaneously quickly drained the platform battery, posing challenges for long-term uninterrupted sound processing. More study is needed to reassess the feasibility of concurrent operation of these 2 applications with optimized code and application power management. ADAM also allowed users to record the use of medications throughout the day to help them systemically review changes in symptom patterns in relation to medication use. This can motivate users’ adherence to treatment plans (when they noted symptom reduction after medication) or inform users or providers of the need for adjusting current medication or dosage (when no relief from symptom was achieved after medication). A clinical trial is needed to examine the effects of the device’s medication tracking function on users’ medication adherence.
ADAM represents an important undertaking in the field of mobile health (mHealth) that has exploded in the past decade to improve health outcomes and patients’ self-management capacity [- ]. The application of mobile technologies in daily assessment of asthma symptoms can be a particularly attractive option for adolescents. Yet to be determined is the extent to which the appeal of mobile technology can translate into better symptom monitoring, awareness, and self-management behaviors, which ultimately leads to improved asthma outcomes.
Potential Clinical Application of ADAM: Sample Case Scenario
The following sample clinical scenario demonstrates the potential application of ADAM in clinical practice:
JD is a 16-year-old boy with moderate persistent asthma and a history of several emergency visits for asthma each year. His provider had prescribed a daily controller medication for him, but he rarely uses it because he has not perceived any significant benefit. He reports not sleeping well, which causes him to feel poorly rested in the morning. He relies heavily on albuterol with several self-administered doses each day. After a discussion with his provider, JD agreed to use ADAM for a month. At a follow-up visit, the provider and JD reviewed the symptom pattern displayed on the screen of the mobile phone monitor. Data from the device revealed significant coughs, predominantly at night, which JD was unaware. After discussing the results, JD agreed to use the controller medication daily to see if the symptoms improved. After 1 month, JD repeated his symptom monitoring with ADAM and discovered his coughing events were significantly decreased compared to the prior monitoring period. In addition, JD was alerted by ADAM when his symptoms became out of control, which allowed him to take precautions and closely follow his asthma action plan. He agreed to continue to use the controller medication for another 2 months and to use ADAM intermittently to help track his symptoms.
This study was funded by the National Institute of Health/National Institute for Nursing Research (R01NR011169-01A1).
HR, as the principal investigator, was responsible for the overall study and drafted the entire manuscript; MJB analyzed the data and contributed to the “Results” section; MS participated in technology development and provided input in describing technical aspects of the device. MFB led and oversaw the technology development for ADAM and contributed to the critical review of the manuscript. We thank Ms Eileen Fairbanks for having coordinated and managed the study on a daily basis and K Comeau, S Miner, and J Mammen for having contributed to participant recruitment and data collection.
Conflicts of Interest
- Bloom B, Cohen RA, Freeman G. Summary health statistics for US children: National Health Interview Survey, 2010. Vital Health Stat 10 2011 Dec(250):1-80 [FREE Full text] [Medline]
- Akinbami LJ, Moorman JE, Garbe PL, Sondik EJ. Status of childhood asthma in the United States, 1980-2007. Pediatrics 2009 Mar;123 Suppl 3:S131-S145. [CrossRef] [Medline]
- Gibson PG, Ram FS, Powell H. Asthma education. Respir Med 2003 Sep;97(9):1036-1044. [Medline]
- Warsi A, Wang PS, LaValley MP, Avorn J, Solomon DH. Self-management education programs in chronic disease: a systematic review and methodological critique of the literature. Arch Intern Med 2004;164(15):1641-1649. [CrossRef] [Medline]
- Bruzzese J, Stepney C, Fiorino EK, Bornstein L, Wang J, Petkova E, et al. Asthma self-management is sub-optimal in urban Hispanic and African American/black early adolescents with uncontrolled persistent asthma. J Asthma 2012 Feb;49(1):90-97. [CrossRef] [Medline]
- Bruzzese J, Bonner S, Vincent EJ, Sheares BJ, Mellins RB, Levison MJ, et al. Asthma education: the adolescent experience. Patient Educ Couns 2004 Dec;55(3):396-406. [CrossRef] [Medline]
- Davis KJ, Disantostefano R, Peden DB. Is Johnny wheezing? Parent-child agreement in the Childhood Asthma in America survey. Pediatr Allergy Immunol 2011 Feb;22(1 Pt 1):31-35. [CrossRef] [Medline]
- Bhogal S, Zemek R, Ducharme FM. Written action plans for asthma in children. Cochrane Database Syst Rev 2006(3):CD005306. [CrossRef] [Medline]
- National Asthma Education and Prevention Program. Expert Panel Report 3 (EPR-3): Guidelines for the Diagnosis and Management of Asthma-Summary Report 2007. J Allergy Clin Immunol 2007 Nov;120(5 Suppl):S94-138. [CrossRef] [Medline]
- Horak E, Grässl G, Skladal D, Ulmer H. Lung function and symptom perception in children with asthma and their parents. Pediatr Pulmonol 2003 Jan;35(1):23-28. [CrossRef] [Medline]
- Fritz GK, McQuaid EL, Spirito A, Klein RB. Symptom perception in pediatric asthma: relationship to functional morbidity and psychological factors. J Am Acad Child Adolesc Psychiatry 1996 Aug;35(8):1033-1041. [CrossRef] [Medline]
- Cabral AL, Conceição GM, Saldiva PH, Martins MA. Effect of asthma severity on symptom perception in childhood asthma. Braz J Med Biol Res 2002 Mar;35(3):319-327 [FREE Full text] [Medline]
- Baker RR, Mishoe SC, Zaitoun FH, Arant CB, Lucas J, Rupp NT. Poor perception of airway obstruction in children with asthma. J Asthma 2000;37(7):613-624. [Medline]
- Mittal V, Khanna P, Panjabi C, Shah A. Subjective symptom perceptual accuracy in asthmatic children and their parents in India. Ann Allergy Asthma Immunol 2006 Oct;97(4):484-489. [CrossRef] [Medline]
- Rietveld S, Everaerd W. Perceptions of asthma by adolescents at home. Chest 2000 Feb;117(2):434-439. [Medline]
- Halterman JS, Yoos HL, Kitzman H, Anson E, Sidora-Arcoleo K, McMullen A. Symptom reporting in childhood asthma: a comparison of assessment methods. Arch Dis Child 2006 Sep;91(9):766-770 [FREE Full text] [CrossRef] [Medline]
- Fuhlbrigge AL, Guilbert T, Spahn J, Peden D, Davis K. The influence of variation in type and pattern of symptoms on assessment in pediatric asthma. Pediatrics 2006 Aug;118(2):619-625. [CrossRef] [Medline]
- Rhee H, Belyea MJ, Elward KS. Patterns of asthma control perception in adolescents: associations with psychosocial functioning. J Asthma 2008 Sep;45(7):600-606 [FREE Full text] [CrossRef] [Medline]
- Britto MT, Byczkowski TL, Hesse EA, Munafo JK, Vockell AB, Yi MS. Overestimation of impairment-related asthma control by adolescents. J Pediatr 2011 Jun;158(6):1028-1030.e1. [CrossRef] [Medline]
- Arga M, Sahbaz H, Bakirtas A, Turktas I, Demirsoy MS. Does self-monitoring by means of symptom diaries improve asthma control in children? J Asthma 2014 Apr;51(3):299-305. [CrossRef] [Medline]
- Bijl-Hofland ID, Cloosterman SG, Folgering HT, Akkermans RP, van Schayck CP. Relation of the perception of airway obstruction to the severity of asthma. Thorax 1999 Jan;54(1):15-19 [FREE Full text] [Medline]
- Koh YI, Choi IS, Lim H. Airway responsiveness as a direct factor contributing to the dyspnoea perception in asthma. Respir Med 2001 Jun;95(6):464-470 [FREE Full text] [CrossRef] [Medline]
- Veen JC, Smits HH, Ravensberg AJ, Hiemstra PS, Sterk PJ, Bel EH. Impaired perception of dyspnea in patients with severe asthma. Relation to sputum eosinophils. Am J Respir Crit Care Med 1998 Oct;158(4):1134-1141. [CrossRef] [Medline]
- Brouwer AF, Roorda RJ, Brand PL. Home spirometry and asthma severity in children. Eur Respir J 2006 Dec;28(6):1131-1137 [FREE Full text] [CrossRef] [Medline]
- Slieker MG, van der Ent CK. The diagnostic and screening capacities of peak expiratory flow measurements in the assessment of airway obstruction and bronchodilator response in children with asthma. Monaldi Arch Chest Dis 2003;59(2):155-159. [Medline]
- Nazir Z, Razaq S, Mir S, Anwar M, Al MG, Sajad M, et al. Revisiting the accuracy of peak flow meters: a double-blind study using formal methods of agreement. Respir Med 2005 May;99(5):592-595 [FREE Full text] [CrossRef] [Medline]
- Kamps AW, Roorda RJ, Brand PL. Peak flow diaries in childhood asthma are unreliable. Thorax 2001 Mar;56(3):180-182 [FREE Full text] [Medline]
- Redline S, Wright EC, Kattan M, Kercsmar C, Weiss K. Short-term compliance with peak flow monitoring: results from a study of inner city children with asthma. Pediatr Pulmonol 1996 Apr;21(4):203-210. [CrossRef] [Medline]
- Yoos HL, Kitzman H, McMullen A, Henderson C, Sidora K. Symptom monitoring in childhood asthma: a randomized clinical trial comparing peak expiratory flow rate with symptom monitoring. Ann Allergy Asthma Immunol 2002 Mar;88(3):283-291. [CrossRef] [Medline]
- Burkhart PV, Dunbar-Jacob JM, Rohay JM. Accuracy of children's self-reported adherence to treatment. J Nurs Scholarsh 2001;33(1):27-32. [Medline]
- van der Meer V, Rikkers-Mutsaerts E R V ER, Sterk PJ, Thiadens HA, Assendelft WJ, Sont JK. Compliance and reliability of electronic PEF monitoring in adolescents with asthma. Thorax 2006 May;61(5):457-458 [FREE Full text] [CrossRef] [Medline]
- Wensley D, Silverman M. Peak flow monitoring for guided self-management in childhood asthma: a randomized controlled trial. Am J Respir Crit Care Med 2004 Sep 15;170(6):606-612. [CrossRef] [Medline]
- Adams RJ, Boath K, Homan S, Campbell DA, Ruffin RE. A randomized trial of peak-flow and symptom-based action plans in adults with moderate-to-severe asthma. Respirology 2001 Dec;6(4):297-304. [Medline]
- Rhee H, Miner S, Sterling M, Halterman JS, Fairbanks E. The development of an automated device for asthma monitoring for adolescents: methodologic approach and user acceptability. JMIR Mhealth Uhealth 2014;2(2):e27 [FREE Full text] [CrossRef] [Medline]
- Sterling M, Rhee H, Bocko M. Automated cough assessment on a mobile platform. J Med Eng 2014;2014:1-9 [FREE Full text] [CrossRef] [Medline]
- Smith AD, Taylor DR. Is exhaled nitric oxide measurement a useful clinical test in asthma? Curr Opin Allergy Clin Immunol 2005 Feb;5(1):49-56. [Medline]
- Harkins MS, Fiato K, Iwamoto GK. Exhaled nitric oxide predicts asthma exacerbation. J Asthma 2004 Jun;41(4):471-476. [Medline]
- Nordvall SL, Janson C, Kalm-Stephens P, Foucard T, Torén K, Alving K. Exhaled nitric oxide in a population-based study of asthma and allergy in schoolchildren. Allergy 2005 Apr;60(4):469-475. [CrossRef] [Medline]
- Spergel JM, Fogg MI, Bokszczanin-Knosala A. Correlation of exhaled nitric oxide, spirometry and asthma symptoms. J Asthma 2005 Dec;42(10):879-883. [CrossRef] [Medline]
- American Thoracic Society, European European Respiratory Society. ATS/ERS recommendations for standardized procedures for the online and offline measurement of exhaled lower respiratory nitric oxide and nasal nitric oxide, 2005. Am J Respir Crit Care Med 2005 Apr 15;171(8):912-930. [CrossRef] [Medline]
- American Thoracic Society. Standardization of spirometry, 1994 Update. Am J Respir Crit Care Med 1995 Sep;152(3):1107-1136. [CrossRef] [Medline]
- Juniper EF, Guyatt GH, Feeny DH, Ferrie PJ, Griffith LE, Townsend M. Measuring quality of life in children with asthma. Qual Life Res 1996 Feb;5(1):35-46. [Medline]
- Juniper EF, O'Byrne PM, Ferrie PJ, King DR, Roberts JN. Measuring asthma control. Clinic questionnaire or daily diary? Am J Respir Crit Care Med 2000 Oct;162(4 Pt 1):1330-1334. [CrossRef] [Medline]
- Janson C, Chinn S, Jarvis D, Burney P. Determinants of cough in young adults participating in the European Community Respiratory Health Survey. Eur Respir J 2001 Oct;18(4):647-654 [FREE Full text] [Medline]
- Rietveld S, Rijssenbeek-Nouwens LH. Diagnostics of spontaneous cough in childhood asthma: results of continuous tracheal sound recording in the homes of children. Chest 1998 Jan;113(1):50-54. [Medline]
- Wildhaber J, Carroll WD, Brand PL. Global impact of asthma on children and adolescents' daily lives: the room to breathe survey. Pediatr Pulmonol 2012 Apr;47(4):346-357. [CrossRef] [Medline]
- Rhee H, Fairbanks E, Butz A. Symptoms, feelings, activities and medication use in adolescents with uncontrolled asthma: lessons learned from asthma diaries. J Pediatr Nurs 2014;29(1):39-46 [FREE Full text] [CrossRef] [Medline]
- Osman LM, McKenzie L, Cairns J, Friend JA, Godden DJ, Legge JS, et al. Patient weighting of importance of asthma symptoms. Thorax 2001 Feb;56(2):138-142 [FREE Full text] [Medline]
- Marsden PA, Smith JA, Kelsall AA, Owen E, Naylor JR, Webster D, et al. A comparison of objective and subjective measures of cough in asthma. J Allergy Clin Immunol 2008 Nov;122(5):903-907. [CrossRef] [Medline]
- Hsu JY, Stone RA, Logan-Sinclair RB, Worsdell M, Busst CM, Chung KF. Coughing frequency in patients with persistent cough: assessment using a 24 hour ambulatory recorder. Eur Respir J 1994 Jul;7(7):1246-1253 [FREE Full text] [Medline]
- Hoskyns EW, Beardsmore CS, Simpson H. Chronic night cough and asthma severity in children with stable asthma. Eur J Pediatr 1995 Apr;154(4):320-325. [Medline]
- Teeter JG, Bleecker ER. Relationship between airway obstruction and respiratory symptoms in adult asthmatics. Chest 1998 Feb;113(2):272-277. [Medline]
- de Marco R, Marcon A, Jarvis D, Accordini S, Almar E, Bugiani M, European Community Respiratory Health Survey Therapy Group. Prognostic factors of asthma severity: a 9-year international prospective cohort study. J Allergy Clin Immunol 2006 Jun;117(6):1249-1256. [CrossRef] [Medline]
- Birring SS, Fleming T, Matos S, Raj AA, Evans DH, Pavord ID. The Leicester Cough Monitor: preliminary validation of an automated cough detection system in chronic cough. Eur Respir J 2008 May;31(5):1013-1018 [FREE Full text] [CrossRef] [Medline]
- Munyard P, Busst C, Logan-Sinclair R, Bush A. A new device for ambulatory cough recording. Pediatr Pulmonol 1994 Sep;18(3):178-186. [Medline]
- Hamutcu R, Francis J, Karakoc F, Bush A. Objective monitoring of cough in children with cystic fibrosis. Pediatr Pulmonol 2002 Nov;34(5):331-335. [CrossRef] [Medline]
- Smith J, Woodcock A. New developments in the objective assessment of cough. Lung 2008;186 Suppl 1:S48-S54. [CrossRef] [Medline]
- Amoh J, Odame K. Technologies for developing ambulatory cough monitoring devices. Crit Rev Biomed Eng 2013;41(6):457-468. [Medline]
- Matos S, Birring SS, Pavord ID, Evans DH. Detection of cough signals in continuous audio recordings using hidden Markov models. IEEE Trans Biomed Eng 2006 Jun;53(6):1078-1083. [CrossRef] [Medline]
- Matos S, Birring SS, Pavord ID, Evans DH. An automated system for 24-h monitoring of cough frequency: the leicester cough monitor. IEEE Trans Biomed Eng 2007 Aug;54(8):1472-1479. [CrossRef] [Medline]
- Shin S, Hashimoto T, Hatano S. Automatic detection system for cough sounds as a symptom of abnormal health condition. IEEE Trans Inf Technol Biomed 2009 Jul;13(4):486-493. [CrossRef] [Medline]
- Manfreda J, Becklake MR, Sears MR, Chan-Yeung M, Dimich-Ward H, Siersted HC, et al. Prevalence of asthma symptoms among adults aged 20-44 years in Canada. CMAJ 2001 Apr 3;164(7):995-1001 [FREE Full text] [Medline]
- Ballard RD, Saathoff MC, Patel DK, Kelly PL, Martin RJ. Effect of sleep on nocturnal bronchoconstriction and ventilatory patterns in asthmatics. J Appl Physiol (1985) 1989 Jul;67(1):243-249. [Medline]
- Bentur L, Beck R, Shinawi M, Naveh T, Gavriely N. Wheeze monitoring in children for assessment of nocturnal asthma and response to therapy. Eur Respir J 2003 Apr;21(4):621-626 [FREE Full text] [Medline]
- Meijer GG, Postma DS, Wempe JB, Gerritsen J, Knol K, van Aalderen WM. Frequency of nocturnal symptoms in asthmatic children attending a hospital out-patient clinic. Eur Respir J 1995 Dec;8(12):2076-2080 [FREE Full text] [Medline]
- Storms WW, Bodman SF, Nathan RA, Byer P. Nocturnal asthma symptoms may be more prevalent than we think. J Asthma 1994;31(4):313-318. [Medline]
- Baughman RP, Loudon RG. Lung sound analysis for continuous evaluation of airflow obstruction in asthma. Chest 1985 Sep;88(3):364-368. [Medline]
- Raherison C, Abouelfath A, Le Gros V, Taytard A, Molimard M. Underdiagnosis of nocturnal symptoms in asthma in general practice. J Asthma 2006 Apr;43(3):199-202. [CrossRef] [Medline]
- Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res 2013;15(5):e95 [FREE Full text] [CrossRef] [Medline]
- Krishna S, Boren SA, Balas EA. Healthcare via cell phones: a systematic review. Telemed J E Health 2009 Apr;15(3):231-240. [CrossRef] [Medline]
- de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone messaging for facilitating self-management of long-term illnesses. Cochrane Database Syst Rev 2012;12:CD007459. [CrossRef] [Medline]
|ADAM: Automated Device for Asthma Monitoring|
|ATS: American Thoracic Society|
|AUC: area under the curve|
|FeNO: fractional exhaled nitric oxide|
|FEV1: forced expiratory volume in first second of expiration|
|FVC: forced vital capacity|
|ICS: inhaled corticosteroids|
|NHLBI: National Heart, Lung, and Blood Institute|
|PAQLQ: Pediatric Asthma Quality of Life Questionnaire|
|ROC: receiver operating characteristic|
|SABA: short-acting beta agonists|
|VAS: Visual Analog Scale|
Edited by G Eysenbach; submitted 23.07.15; peer-reviewed by N Diviani, P Schulz; comments to author 05.08.15; revised version received 18.08.15; accepted 19.08.15; published 16.10.15
©Hyekyun Rhee, Michael J Belyea, Mark Sterling, Mark F Bocko. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 16.10.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.