Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 06.11.20 in Vol 22, No 11 (2020): November

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/19665, first published May 01, 2020.

This paper is in the following e-collection/theme issue:

    Original Paper

    A Patient Self-Checkup App for COVID-19: Development and Usage Pattern Analysis

    1Armed Forces Medical Command, Seongnam, Republic of Korea

    2Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Republic of Korea

    3Department of Medical Humanities and Social Sciences, Yonsei University College of Medicine, Seoul, Republic of Korea

    4Department of Ophthalmology, Yonsei University College of Medicine, Seoul, Republic of Korea

    5Companoid Labs, Yonsei University, Seoul, Republic of Korea

    6Department of Neurology, Yonsei University College of Medicine, Seoul, Republic of Korea

    7Department of Internal Medicine, The Armed Forces Capitol Hospital, Seongnam, Republic of Korea

    8Department of Otorhinolaryngology, The Armed Forces Capitol Hospital, Seongnam, Republic of Korea

    *these authors contributed equally

    Corresponding Author:

    Yu Rang Park, PhD

    Department of Biomedical Systems Informatics

    Yonsei University College of Medicine

    50-1 Yonsei-ro

    Seodaemun-gu

    Seoul, 03722

    Republic of Korea

    Phone: 82 222282493

    Email: yurangpark@yuhs.ac


    ABSTRACT

    Background: Clear guidelines for a patient with suspected COVID-19 infection are unavailable. Many countries rely on assessments through a national hotline or telecommunications, but this only adds to the burden of an already overwhelmed health care system. In this study, we developed an algorithm and a web application to help patients get screened.

    Objective: This study aims to aid the general public by developing a web-based application that helps patients decide when to seek medical care during a novel disease outbreak.

    Methods: The algorithm was developed via consultations with 6 physicians who directly screened, diagnosed, and/or treated patients with COVID-19. The algorithm mainly focused on when to test a patient in order to allocate limited resources more efficiently. The application was designed to be mobile-friendly and deployed on the web. We collected the application usage pattern data from March 1 to March 27, 2020. We evaluated the association between the usage pattern and the numbers of COVID-19 confirmed, screened, and mortality cases by access location and digital literacy by age group.

    Results: The algorithm used epidemiological factors, presence of fever, and other symptoms. In total, 83,460 users accessed the application 105,508 times. Despite the lack of advertisement, almost half of the users accessed the application from outside of Korea. Even though the digital literacy of the 60+ years age group is half of that of individuals in their 50s, the number of users in both groups was similar for our application.

    Conclusions: We developed an expert-opinion–based algorithm and web-based application for screening patients. This innovation can be helpful in circumstances where information on a novel disease is insufficient and may facilitate efficient medical resource allocation.

    J Med Internet Res 2020;22(11):e19665

    doi:10.2196/19665

    KEYWORDS



    Introduction

    On March 11, 2020, the World Health Organization officially characterized COVID-19, a disease caused by the novel coronavirus SARS-CoV-2, as a pandemic [1]. As general preventive guidelines, hygiene control and social distancing were recommended, but in a situation where one shows signs of infection, there are no clear guidelines for patients on how to respond. Many countries, including South Korea, rely on individual assessments and advice provided by a health care worker. This could be carried out by calling a national hotline or one’s local clinic, or by means of telecommunication that assesses and advises people on what to do in the event of a suspected infection [2]. This often results in overwhelmed call centers with long waiting times and untimely response to emergency patients.

    Viral pandemics such as COVID-19 can place extraordinary and sustained demands on public health systems and essential community service providers [3]. Patient triage is essential in such situations to efficiently allocate resources [4,5]. In addition, patients need to be advised on the adequate care measures to be taken not only for themselves but also for others around them. However, for novel infections like COVID-19, it is difficult for both the general public and less experienced medical staff to accurately make decisions.

    Traditionally, for medical advice to be made public, many trials have to repetitively prove that the advice is sound and without any unforeseen side effects. This is, in many ways, appropriate, considering the consequences of erroneous recommendations. However, in the current situation, patients who stand a chance of survival with professional care may still die due to a lack of clinical resources. Thus, a well-timed, imperfect solution may be more beneficial than belated, flawless advice. In addition, this information can easily be confusing to the general public, and false interpretation may cause harm to users. Providing medical knowledge in a friendly and clear manner through an application can be both satisfactory and safe for users. Providing screening algorithms based on experiences from actively involved physicians can help improve the efficiency of the diagnosis process. This is made possible by both accurately selecting the patients to be tested and alleviating the workload imposed on health care workers.

    In this study, we developed an algorithm and a web application for use by the general public to help them decide whether or not to seek professional care in cases where a COVID-19 infection is suspected.


    Methods

    Algorithm Development

    During the initial phases of the COVID-19 outbreak in Korea, the government and the Korean Center for Disease Control and Prevention (KCDC) demonstrated expansive and extensive efforts to isolate infected people and trace and quarantine their contacts. The KCDC guidelines stated that two groups of patients be tested: suspected cases and patients under investigation [6]. A person exhibiting fever or respiratory symptoms within 14 days of contact with a confirmed COVID-19 case was defined as a suspected case. Patients under investigation were defined as those who traveled overseas or those with an epidemiological link to a domestic COVID-19 cluster with fever or respiratory symptoms. In addition, a person suspected to have COVID-19 according to a physician’s opinion was also classified as a patient under investigation and was subject to COVID-19 testing. Many patients were tested based on their physician’s opinion, and no clear guidelines existed on features to be considered.

    Our algorithm was built with the aim of helping a patient decide when they should consult a doctor for COVID-19 testing. It was developed through the consultation of a group of physicians (n=6) directly involved in the process of screening, diagnosis, and/or treatment of patients with COVID-19. The KCDC guidelines were also reviewed for any information on patient selection for COVID-19 testing [6].

    The physician group comprised 1 pulmonologist and 1 infectious disease specialist directly involved in the care of confirmed cases; 1 pulmonologist and 1 otorhinolaryngologist involved in patient screening at the hospital; and 2 neurologists involved in patient screening at public health centers. Based on experience and a literature review, clinical variables that should be considered when screening for patients with COVID-19 were selected, and the decision process of selecting patients to be tested for COVID-19 was discussed.

    The final algorithm of COVID-19 CheckUp is shown in Figure 1. Upon reviewing the KCDC guidelines [6] and several papers in the literature [7-9], the patients who were required to report to the KCDC and undergo a COVID-19 test were clearly defined. KCDC guidelines state that patients who show any symptoms of upper respiratory infection or documented fever combined with direct contact with an individual with confirmed COVID-19 or have visited a well-known outbreak area should report to the KCDC and take a COVID-19 confirmatory test immediately.

    Figure 1. Algorithm for the COVID-19 CheckUp app. BT: body temperature.
    View this figure

    The first part of the algorithm was designed to meet the KCDC guidelines. The user is asked if there was any contact with a confirmed patient or any other source of infection. If the user answered “yes” to these questions and presented any relevant symptoms or fever, they are instructed to report to the KCDC and take a COVID-19 test. A user who had contact a confirmed COVID-19 case is still classified as highly likely to be needing a test even if the patient presents no symptoms and has a normal body temperature.

    Body temperature was classified into four categories: high fever (higher than 38 °C), fever (higher than 37.5 °C but lower than 38 °C), mild fever (higher than 37.2 °C but lower than 37.5 °C), and normal (lower than 37.2 °C). Body temperature was chosen as the most important variable for the process of screening as it was one of the most objective variables the user could provide; further, the pathogens in patients with high fever were more likely to be viral. In addition, patients using antipyretics, including nonsteroidal anti-inflammatory drugs (NSAIDs), were treated like patients with high fever. This was done to reflect cases where fever and symptoms of upper respiratory infection were masked by the chronic use of antipyretics; additionally, patients who took these medications were usually at a higher risk because of their chronic comorbidities. The detailed classification of temperature into four categories (high fever, fever, mild fever, and normal) was used to consider the numerous reports of patients with COVID-19 presenting with only mild fever [10], and at the same time differentiate patients with high fever, who should be considered more gravely.

    Application

    COVID-19 CheckUp was developed using the Flutter software development kit [11]. It was deployed as a web application [12] for better accessibility and maintenance. The application has been designed to be mobile-friendly and interactive. HTML5 local storage was used to store users’ data, which ensures that usage data are never transferred to a server. This structure ensures that the privacy of users’ data can be protected and at the same time eliminates the need for a back-end architecture. To better accommodate international usage, the application has been translated into 5 languages: Korean, English, French, Spanish, and Vietnamese. Screenshots of the COVID-19 CheckUp app are shown in Figure 2. When using the app, the patient is asked a maximum of 7 simple questions based on our algorithm.

    Figure 2. Screenshots of the COVID-19 CheckUp app.
    View this figure

    Usage Patterns

    To gain insight on application usage, Google Analytics [13] was embedded and usage data were collected from March 1 to March 27, 2020. Google Analytics API [14] was used to further help extract data from Google Analytics. The extracted data include frequency of access and location of the user. The data on COVID-19 cases and deaths were extracted from Johns Hopkins University’s COVID-19 data repository on GitHub [15]. The usage numbers were compared by subcontinent and city. Screened cases for each city were also included for analysis. Each local autonomous body in Korea released the number of COVID-19 tests that were conducted. For Indonesia, because data on the number of assessments were not available, the number of screened cases was defined as the sum of patients classified as “under monitoring,” “under supervision,” and “confirmed.” Moreover, usage by age group was compared to their corresponding digital literacy and COVID-19 fatality by age (sourced from the KCDC). Digital literacy by age was obtained from a report on the digital divide in 2019 released by the Ministry of Science and ICT and the National Information Society Agency in Korea [16].

    All the statistical analyses were performed using R 3.6.0 (The R Foundation).


    Results

    In total, 83,640 users accessed the application 105,508 times from 141 countries. In Eastern Asia, where Korea is located, 43,648 users accessed the application 57,030 times. Table 1 reports the number of application users by subcontinent compared with the number of COVID-19 confirmed cases and deaths. Although the application had not been introduced or advertised outside of Korea, the website received heavy traffic from other subcontinents in the following order: Southeast Asia, North America, and Western Europe (Multimedia Appendix 1).

    Table 1. The number of users, new users, sessions, COVID-19 cases, and deaths by subcontinent.
    View this table

    Table 2 shows the list of the top 10 cities with the highest user ratios (user count per 100,000 population). The majority of users were from Korea (n=43,210, 51.64%); almost half (n=40,907, 48.36%) were from outside of Korea.

    In Korea, the digital literacy of users in their 50s and 60s is 73.60% and 35.70%, respectively. However, application usage was similar between the two groups at 5.5% each. Even though the digital literacy of users in their 60s was half that of users in their 50s, the user count was similar for our application (Table 3).

    Table 2. Top 10 cities with the highest user ratios.
    View this table
    Table 3. Application usage, digital literacy, and mortality, compared by age group, in Korea.
    View this table

    Discussion

    Principal Findings

    This study explains the development and deployment of a globally used COVID-19 symptom checkup application. The main focus of this solution was to provide information about the coronavirus disease, including which symptoms should be taken seriously and under what circumstances COVID-19 should be suspected. Many measures were taken to “flatten the curve,” but there were not enough medical resources to smoothly handle the overwhelming spread of the virus [17]. Inevitably, there was mass confusion and distress among the public, which in turn worsened the shortage of medical resources. Our application aimed to break this cycle with, however imperfect, timely knowledge of the unknown virus. This study describes the development of an application to screen for a novel emerging infectious disease that can be deployed worldwide. The usage pattern of the application was not proportional to digital literacy. The application was quickly adapted for use in the military and the government.

    Medical resources may not always be sufficient, especially in a pandemic situation. Traditionally, consultation through telecommunication, such as a national hotline, was used to triage patients. This method may be more thorough than a simple decision support system. However, considering how the spread of the virus is overwhelming the entire health care industry, using ICT (information and communications technology) to alleviate the workload of health care workers can be beneficial.

    Upon building the algorithm for the application, all the participating physicians easily concluded that there cannot be a decisive method to screen patients for COVID-19 infection. This is mainly because many patients are asymptomatic, with some studies reporting that up to 75% of patients can show no symptoms [18-21]. Hence, the algorithm presents a “no evidence” rather than “testing not needed” result when there is no evidence to indicate COVID-19 infection. A total of 10 levels of risk are presented to the user, ranging from “test necessary” and “very high” to “unknown” and “no evidence.” The fine division of the risk level is to inform the user of the factors that should be considered, and which factors are more important than others, for testing.

    Web-based applications are easily accessible because they can be used with a typical preinstalled web browser, rather than having to install a separate app. Moreover, even though our application was developed for a mobile interface, it can be used on any devices such as desktops that connect to the internet. This wide range of supported devices may be beneficial for environments where a mobile device is unavailable.

    Without any advertising abroad, the fast adoption of our application overseas reflects the need for such a solution amidst the general public. The application was translated into a total of 5 languages, increasing accessibility for global users. Referrer data acquired using the Google Analytics tool showed that 20.95% of user acquisition was from Twitter, which may account for the worldwide usage of the application. Additionally, the application was registered in the World Health Organization Digital Health Atlas [22].

    There are concerns regarding ICT accessibility (eg, too complicated to use or unfamiliar to an older audience). However, our study shows that the user distribution is unexpectedly higher in the older age groups with respect to the digital literacy rate of older adults [16]. Considering that disease fatality increases among elderly patients [23], users’ needs seem to outweigh the discomfort of new technologies. The application is constructed as a series of pages with only one question per page. This simple question-and-answer type interface may be more familiar to the elderly user group. However, additional research is needed to confirm the hypothesis.

    Comparison With Prior Works

    There are other applications that help users to screen themselves. Apple Inc, in association with the Centers for Disease Control and Prevention, released the web-based COVID-19 Screening Tool [24]. In addition to physicians, the app also focused on patients, and it was found to be useful in the United States. Spector and colleagues developed the COVID-19 Symptom Tracker app in the United Kingdom, with a partnership between researchers at King’s College London and the health-related data science company ZOE [25]. Judson et al [26] introduced a self-triage and a self-scheduling tool based on well-designed algorithms. Integrated into the electronic medical record (EMR), it enables physicians to easily use the application and possibly enable automatic data collection of user-provided data. EMR-tethered solutions are effective for local users but not for use outside of the region where the associated EMR system is not available. In Korea, the Self-Check Mobile App has been developed by the Ministry of Health & Welfare to monitor symptoms in people for 14 days who arrived in Korea from abroad [27]. We provide a list of similar applications in Multimedia Appendix 2.

    The application described in this paper, the COVID-19 CheckUp app, is officially in use in military hospitals and throughout the military service line. Use of the app to check symptoms is recommended to all personnel with suspected COVID-19 symptoms. Moreover, the city of Seoul has offered to officially support our project by placing a banner linked to our application on their website. Further discussions with government departments are underway.

    Limitations and Conclusion

    There are limitations to this study. The algorithm was based on KCDC guidelines, review of a few publications, and the expert opinions of 6 physicians. Evidence-based guidelines should be developed using data from patients with COVID-19. Insufficient data were available to develop an evidence-based model at the time, but additional studies developing statistical models using the newly acquired patient data will be followed. Additionally, one of the most important variables in the algorithm is body temperature. The typical household thermometer, especially the noncontact infrared thermometers, may be inaccurate, providing false guidance to users [28-30]. Another limitation of this study is that it does not provide any validation results on whether the algorithm was successful in patient triaging or increased efficiency in resource allocation. Concerns about the users’ privacy led to the development of an application that does not collect or send any information provided by the user. Thus, no data were available for use in validating the efficacy of the model. Further, there is regional bias in the usage pattern since the application was developed and advertised only in Korea. However, many people in other countries have used the app since it supports multiple languages. Moreover, the usage pattern was not analyzed temporally because it was difficult to access all the temporal marks that can affect usage patterns.

    An expert-opinion–based algorithm and application for patient screening and guidance can be beneficial in a circumstance where there is insufficient information on a novel disease and medical resources are limited.

    Acknowledgments

    We would like to thank the Armed Forces Medical Command of South Korea for providing the personnel and computing resources for developing the COVID-19 CheckUp application. This study was supported by the Foundational Technology Development Program (NRF2019M3E5D406468212) from the Ministry of Science and ICT, Republic of Korea.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Geographical distribution of app users.

    DOCX File , 150 KB

    Multimedia Appendix 2

    List of COVID-19–related apps.

    DOCX File , 22 KB

    References

    1. WHO Timeline - COVID-19. World Health Organization. 2020 Apr 27.   URL: https://www.who.int/news-room/detail/27-04-2020-who-timeline---covid-19 [accessed 2020-05-01]
    2. Coronavirus Disease-19 (COVID-19): Public Advice. KCDC. 2020.   URL: [accessed 2020-05-01]
    3. Munster VJ, Koopmans M, van Doremalen N, van Riel D, de Wit E. A Novel Coronavirus Emerging in China - Key Questions for Impact Assessment. N Engl J Med 2020 Feb 20;382(8):692-694. [CrossRef] [Medline]
    4. Sandrock CE. Care of the critically ill and injured during pandemics and disasters: groundbreaking results from the Task Force on Mass Critical Care. Chest 2014 Oct;146(4):881-883. [CrossRef] [Medline]
    5. Emanuel EJ, Persad G, Upshur R, Thome B, Parker M, Glickman A, et al. Fair Allocation of Scarce Medical Resources in the Time of Covid-19. N Engl J Med 2020 May 21;382(21):2049-2055. [CrossRef] [Medline]
    6. Coronavirus Infectiousness-19 Response Guidelines for Local Governments Internet. Central Anti-Disaster Countermeasures Headquarters and Central Accident Control Headquarters. 2020.   URL: [accessed 2020-06-03]
    7. Wang D, Hu B, Hu C, Zhu F, Liu X, Zhang J, et al. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China. JAMA 2020 Mar 17;323(11):1061-1069 [FREE Full text] [CrossRef] [Medline]
    8. Wong JEL, Leo YS, Tan CC. COVID-19 in Singapore-Current Experience: Critical Global Issues That Require Attention and Action. JAMA 2020 Apr 07;323(13):1243-1244. [CrossRef] [Medline]
    9. Wu Z, McGoogan JM. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA 2020 Apr 07;323(13):1239-1242. [CrossRef] [Medline]
    10. Kim ES, Chin BS, Kang CK, Kim NJ, Kang YM, Choi J, Korea National Committee for Clinical Management of COVID-19. Clinical Course and Outcomes of Patients with Severe Acute Respiratory Syndrome Coronavirus 2 Infection: a Preliminary Report of the First 28 Patients from the Korean Cohort Study on COVID-19. J Korean Med Sci 2020 Apr 06;35(13):e142 [FREE Full text] [CrossRef] [Medline]
    11. Flutter - Beautiful native apps in record time. Google, LLC.   URL: https://flutter.dev [accessed 2020-05-11]
    12. COVID-19 CheckUp.   URL: https://ncov.docl.org [accessed 2020-10-21]
    13. Google Analytics for Small Businesses. Google, LLC.   URL: https://analytics.google.com/analytics/web/provision/#/provision [accessed 2020-05-01]
    14. Google Analytics. Google, LLC.   URL: https://developers.google.com/analytics/devguides/reporting/core/v4 [accessed 2020-05-01]
    15. Novel Coronavirus (COVID-19) Cases, provided by JHU CSSE Internet. Johns Hopkins University Center for Systems Science and Engineering.   URL: https://github.com/CSSEGISandData/COVID-19 [accessed 2020-06-08]
    16. National Information Society Agency. 2019 The Report on the Digital Divide. South Korea: Ministry of Science and ICT; 2020:22.
    17. Lee D, Lee J. Testing on the move: South Korea's rapid response to the COVID-19 pandemic. Transportation Research Interdisciplinary Perspectives 2020 May;5:100111. [CrossRef]
    18. Day M. Covid-19: identifying and isolating asymptomatic people helped eliminate virus in Italian village. BMJ 2020 Mar 23;368:m1165. [CrossRef] [Medline]
    19. Wang Y, Liu Y, Liu L, Wang X, Luo N, Li L. Clinical Outcomes in 55 Patients With Severe Acute Respiratory Syndrome Coronavirus 2 Who Were Asymptomatic at Hospital Admission in Shenzhen, China. J Infect Dis 2020 May 11;221(11):1770-1774 [FREE Full text] [CrossRef] [Medline]
    20. Hu Z, Song C, Xu C, Jin G, Chen Y, Xu X, et al. Clinical characteristics of 24 asymptomatic infections with COVID-19 screened among close contacts in Nanjing, China. Sci China Life Sci 2020 May;63(5):706-711 [FREE Full text] [CrossRef] [Medline]
    21. Dong Y, Mo X, Hu Y, Qi X, Jiang F, Jiang Z, et al. Epidemiology of COVID-19 Among Children in China. Pediatrics 2020 Jun 16;145(6):e. [CrossRef] [Medline]
    22. DOCL Outcome Prediction Platform (Doctors on the Cloud). World Health Organization.   URL: https://digitalatlas.who.int/en/covid-19/projects/1274/published [accessed 2020-06-08]
    23. Cases in Korea. KCDC.   URL: http:/​/ncov.​mohw.go.kr/​en/​bdBoardList.​do?brdId=16&brdGubun=161&dataGubun=&ncvContSeq=&contSeq=&board_id=&gubun=
    24. COVID-19 Screening Tool. Apple Inc.   URL: https://covid19.apple.com/screening/ [accessed 2020-10-22]
    25. COVID Symptom Study - Help slow the spread of COVID-19. ZOE.   URL: https://covid.joinzoe.com [accessed 2020-06-08]
    26. Judson TJ, Odisho AY, Neinstein AB, Chao J, Williams A, Miller C, et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc 2020 Jun 01;27(6):860-866 [FREE Full text] [CrossRef] [Medline]
    27. Self-Check Mobile App Instructions. The Ministry of Health and Welfare.   URL: http://ncov.mohw.go.kr/selfcheck/ [accessed 2020-10-22]
    28. Hayward G, Verbakel J, Ismail F, Edwards G, Wang K, Fleming S, et al. Non-contact infrared versus axillary and tympanic thermometers in children attending primary care: a mixed-methods study of accuracy and acceptability. Br J Gen Pract 2020 Mar 23;70(693):e236-e244. [CrossRef]
    29. Niven DJ, Gaudet JE, Laupland KB, Mrklas KJ, Roberts DJ, Stelfox HT. Accuracy of Peripheral Thermometers for Estimating Temperature. Ann Intern Med 2015 Nov 17;163(10):768. [CrossRef]
    30. Chung W, Chen C. Evaluation of performance and uncertainty of infrared tympanic thermometers. Sensors (Basel) 2010 Mar 31;10(4):3073-3089 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    EMR: electronic medical record
    ICT: information and communications technology
    KCDC: Korean Center for Disease Control and Prevention
    NSAID: nonsteroidal anti-inflammatory drug


    Edited by G Eysenbach, R Kukafka; submitted 01.05.20; peer-reviewed by T Judson, Y Yang, L Liu, M Gong; comments to author 25.05.20; revised version received 08.06.20; accepted 19.10.20; published 06.11.20

    ©JoonNyung Heo, MinDong Sung, Sangchul Yoon, Jinkyu Jang, Wonwoo Lee, Deokjae Han, Hyung-Jun Kim, Han-Kyeol Kim, Ji Hyuk Han, Woong Seog, Beomman Ha, Yu Rang Park. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 06.11.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.