Background: Obesity is a highly prevalent condition with important health implications. Face-to-face interventions to treat obesity demand a large number of human resources and time, generating a great burden to individuals and health system. In this context, the internet is an attractive tool for delivering weight loss programs due to anonymity, 24-hour-accessibility, scalability, and reachability associated with Web-based programs.
Objective: We aimed to investigate the effectiveness of Web-based digital health interventions, excluding hybrid interventions and non-Web-based technologies such as text messaging, short message service, in comparison to nontechnology active or inactive (wait list) interventions on weight loss and lifestyle habit changes in individuals with overweight and obesity.
Methods: We searched PubMed or Medline, SciELO, Lilacs, PsychNet, and Web of Science up to July 2018, as well as references of previous reviews for randomized trials that compared Web-based digital health interventions to offline interventions. Anthropometric changes such as weight, body mass index (BMI), waist, and body fat and lifestyle habit changes in adults with overweight and obesity were the outcomes of interest. Random effects meta-analysis and meta-regression were performed for mean differences (MDs) in weight. We rated the risk of bias for each study and the quality of evidence across studies using the Grades of Recommendation, Assessment, Development, and Evaluation approach.
Results: Among the 4071 articles retrieved, 11 were included. Weight (MD −0.77 kg, 95% CI −2.16 to 0.62; 1497 participants; moderate certainty evidence) and BMI (MD −0.12 kg/m2; 95% CI −0.64 to 0.41; 1244 participants; moderate certainty evidence) changes were not different between Web-based and offline interventions. Compared to offline interventions, digital interventions led to a greater short-term (<6 months follow-up) weight loss (MD −2.13 kg, 95% CI −2.71 to −1.55; 393 participants; high certainty evidence), but not in the long-term (MD −0.17 kg, 95% CI −2.10 to 1.76; 1104 participants; moderate certainty evidence). Meta-analysis was not possible for lifestyle habit changes. High risk of attrition bias was identified in 5 studies. For weight and BMI outcomes, the certainty of evidence was moderate mainly due to high heterogeneity, which was mainly attributable to control group differences across studies (R2=79%).
Conclusions: Web-based digital interventions led to greater short-term but not long-term weight loss than offline interventions in overweight and obese adults. Heterogeneity was high across studies, and high attrition rates suggested that engagement is a major issue in Web-based interventions.
Facing the global obesity epidemic is a major public health challenge . The prevalence of obesity has nearly doubled over the last 30 years [ ]. Obesity is associated with an increased risk for type 2 diabetes, hypertension, dyslipidemia, cardiovascular diseases, musculoskeletal disorders, psychological stress, and certain types of cancer. All these morbidities significantly increase mortality and reduce quality of life [ ].
Obesity treatment involves a systemic approach with both individual and environmental strategies . The individual interventions are usually delivered face-to-face, which generate high demands for individuals, due to their prolonged course, and a great burden to the health care system due to the high prevalence of obesity [ ]. Despite such efforts, the effectiveness of obesity interventions on weight loss is only modest, particularly in the long-term [ ].
In this context, Web-based digital technology can be a particularly interesting tool for the treatment of overweight and obesity due to its capacity for reaching a large number of people even in remote areas on a 24-hour per 7-day regimen. Delivering weight loss interventions on the Web allows targeting a larger number of people compared to face-to-face interventions and might be less time consuming and more cost-effective for professionals and patients . Previous reviews have shown a modest superiority of digital interventions in comparison to offline interventions with regards to weight loss [ , ]. However, as these reviews included studies that investigated hybrid interventions both in the intervention (eg, Web-based plus short message service text messages) and control groups (ie, face-to-face plus technology-based interventions), the effect of interventions that use only Web-based delivery is not known.
Our aim was to conduct a systematic review and meta-analysis of randomized controlled trials to investigate the effect of Web-based digital interventions in comparison to real-world interventions on anthropometric measures and changes in dietary and physical activity habits in individuals with overweight and obesity.
For the purpose of this review, PubMed or Medline, SciELO, Lilacs, PsychNet, and Web of Science electronic databases were searched up to July 1, 2018. No language restrictions were applied. We searched both for indexed terms and terms in titles or abstracts that corresponded to the following search pattern in PubMed or Medline: (overweight OR obes*) AND (web OR technology OR internet OR computers OR “social media” OR online).
Studies were eligible if they reported data on randomized controlled trials, which recruited adults (≥18 years) with overweight and obesity (body mass index [BMI] ≥25 kg/m2) into a Web-based digital intervention (accessed by browser or Web-based application, regardless of device) versus offline or in-person (face-to-face) interventions. Studies that did not apply any active interventions (wait list) in the control group were also included. Exclusion criteria comprised studies in which overweight and obesity were not a primary selection criterion or those in which the predefined outcomes were not reported. Additionally, studies that included children, adolescents, or pregnant women were excluded. Trials of hybrid interventions (Web-based digital interventions plus face-to-face interventions or other technology-based interventions, such as mobile short message service text messages or digital interventions plus offline interventions) and those that included digital interventions in the control group were also excluded. Moreover, studies evaluating the prevention of weight regain after a previous intervention and those that did not report the predefined outcomes of interest were not included. Multiple reports from the same study were considered as a single one. We considered changes in anthropometric measures and in dietary and physical activity habits as the outcomes of interest.
Two reviewers (AGC and MNLP) independently carried out the selection of the studies according to the predefined eligibility criteria. Any disagreement between them was evaluated by 2 other authors (AMB and AQA). AMB and AQA independently extracted data from reports based on a predefined data extraction form. Any disagreement between them was evaluated by either MdFHD or ALR. When some information was not clear in the report, authors were contacted by email. Hand search was performed in the references of previously published reviews.
Quality of the Evidence
We used the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) methodology  to assess the quality of evidence retrieved by the systematic review. This consists of evaluating the risk of selection (randomization and allocation), performance (blinding of participants and personnel), detection (blinding of outcome assessment), attrition (incomplete outcome data), and reporting (selective reporting) bias of individual studies. In addition, the GRADE methodology suggests assessment of indirectness, inconsistency, imprecision and publication bias of the evidence overall in order to grade the level of the evidence retrieved.
We used a random effects model to calculate summary mean differences (MDs) and 95% CIs for 1 unit change in weight (kg), BMI (kg/m2), waist (cm), or body fat (%). In cases where both “per protocol” and intention-to-treat results were provided, the latter were used to calculate summary MDs. For dietary and physical activity habits, we found a great diversity in the instruments used to measure changes among the groups. This finding precluded meta-analysis, and we performed only qualitative analysis of these outcomes. We used the Cochrane Review Manager software for these analyses .
A random effects meta-regression model was used to determine whether the type of control group (with and without active intervention) was a source of heterogeneity among studies. We performed sensitivity analyses according to the length of follow-up (<6 and ≥6 months) and the type of control intervention (presence or not of a nondigital intervention in the control group). We used Comprehensive Meta-Analysis software version 3 for this analysis .
The search strategy resulted in 4071 articles. After exclusions, as shown in, a total of 11 studies [ - ] that analyzed data from 1525 participants were retrieved. Female sex was predominant in most of the studies. The age of the participants varied from 18 to 65 years. Most of the studies excluded participants with comorbidities and pregnancy as well as participants who were engaged in other weight loss programs. Unhealthy lifestyle habits were not an inclusion criterion in any of the retrieved studies. Recruitment settings varied among the studies and included community populations, physician-referred patients, company employees, and university students or staff. Other characteristics of the studies retrieved are depicted in .
Interventions were predominantly delivered via internet browsers, except 2 that were delivered by a smartphone app [, ]. Diverse behavioral strategies, such as goal setting, self-monitoring and management, social support, modeling, and feedback were applied in the studies. The control groups received either no intervention (wait list) or usual face-to-face interventions ( ).
Quality of the Evidence
With regard to the risk of bias of individual studies (and ), all of the studies reported a sequence generation randomization process. Allocation was not concealed in 3 studies [ , , ]. As expected in this type of intervention, blinding of participants and personnel was not feasible, whereas blinding of the assessor was not reported in 5 [ , , - ]. Moreover, high follow-up attrition rates were a common finding. Moreover, 7 of the 11 retrieved studies showed ≥20% losses to follow-up, and unbalanced losses (intervention>control group) were present in 5 of the 11 studies [ , , , , ].
The quality of the evidence retrieved by the GRADE methodology was considered moderate for the primary outcomes of this review (weight and BMI change), as shown in the summary of findings table (). Although indirectness, imprecision, and publication bias were not major issues in this body of evidence, heterogeneity (I2=94%; P<.001, for weight loss as the outcome) was high and explained mainly (R2=0.79) by differences in the type of control group as shown by meta-regression analysis.
Absolute weight and BMI changes were reported in 10 [- , - ] and 9 [ - ] studies, respectively. Overall, changes in weight (MD −0.77 kg; 95% CI −2.16 to 0.62; ) and MDs in BMI (MD −0.12 kg/m2; 95% CI −0.64 to 0.41; ) were not significantly different between the digital interventions and the offline interventions. Only 2 studies reported results on waist circumference [ , ]. There was no difference between the intervention and control groups for this outcome (−0.54 cm; 95% CI −5.17 to 4.10), as shown in . Only 1 study reported changes in percent body fat and did not find a significant difference between the intervention and control groups (−1.40%; 95% CI −2.93 to 0.13) [ ].
Lifestyle Habits and Other Outcomes
Among the 11 studies, 8 reported outcomes on dietary or physical activity habit changes [, - , - ]. However, the instruments used to measure qualitative and quantitative dietary and physical activity characteristics were very different across the studies. This precluded us to perform a quantitative review of these outcomes. Most of the studies reported that there was no significant difference between the intervention and control groups, except for dietary habits in 3 of the studies ( ) [ , , ]. Moreover, 5 studies [ , , , , ] reported data on substitutive measures of cardiovascular morbidity—blood pressure, glucose metabolism, or cholesterol. None of them found any difference between the intervention and control groups. None of the 11 studies investigated hard endpoints, such as cardiovascular morbidity and mortality.
There was no difference between the groups in terms of quality of life in the 3 studies that assessed it [- ]. The Web-based intervention was cost-effective in comparison to a 6-month in-person intervention in 1 [ ] of the 3 studies that evaluated cost-effectiveness [ - ].
|Study||Behavioral theory||Behavioral strategy||Technology strategy||Type of control intervention||Behavioral strategy in control group||Tools used in control group|
|Allen et al 2013 ||Social cognitive theory, behavioral self-management, and motivational interviewing counseling||Self-management, mindful empowerment, and feedback||Web-based smartphone app||In person||Coaching and goal setting||N/Aa|
|Chung et al 2014 ||Not specified||Self-monitoring, knowledge, personalized feedback||Image-based electronic portal and caloric calculator||Paper food diary||Self-monitoring||Paper diary|
|Collins et al 2012 ||Social cognitive theory||Self-efficacy, goal setting, and self-monitoring of weight, body measurements, exercise, and diet; outcome expectations (knowledge-based web components); modeling; and social support||Website and telephone contact||Wait list||N/A||N/A|
|Dunn et al 2016 ||Theory of planned behavior, mindfulness, and small steps to change||Web-based lessons of the Eat Smart, Move More, Weigh Less, an evidence-based, 15-week, adult weight management program||Website||Wait list||Theory of planned behavior, mindfulness, and small steps to change||Eat Smart, Move More, Weigh Less, an evidence-based, 15-week, adult weight management program|
|Hurkmans et al 2018 ||Not specified||Knowledge, self-monitoring, help button||Mobile app||In person||Self-monitoring, action planning, relapse prevention||N/A|
|Kraschnewski 2011 ||Positive deviance framework||Modeling, goal setting, knowledge, personalized feedback, self-monitoring||Videos||Wait list||N/A||N/A|
|Krukowski et al 2011 ||Not specified||Knowledge, self-monitoring, stimulus control, problem solving, goal setting, relapse prevention, and assertiveness training||Chat group, pedometers, and website platform||In person||Knowledge, self-monitoring, stimulus control, problem solving, goal setting, relapse prevention, and assertiveness training||Group sessions and printed information on dietary intake and physical activity|
|McConnon et al 2007 ||Not specified||Counseling, personal feedback||Not specified||Primary-care based printed information||Not available||Printed information|
|Padwal et al 2017 ||Not specified||Knowledge, self-management, self-monitoring, goal setting, stress management||Website||In person||Knowledge, self-management, self-monitoring, goal setting, and stress management||N/A|
|Steinberg et al 2013 ||Not specified||Self-monitoring, knowledge and skills (portion control, restaurant eating, structured exercise, problem solving, stimulus control, and relapse prevention)||Cellular-connected “smart” scale for daily weighing, website, and email||Wait list||N/A||N/A|
|Yardley et al 2014 ||Cognitive behavioral theory||Skills, self-regulation, and feedback||Website lessons, challenges, and email||In person||Not available||N/A|
aN/A: not applicable.
|Study||Dietary caloric intake||Dietary quality||Physical activity|
|Allen et al 2013 ||No difference||N/Aa||No difference|
|Chung et al ||N/A||N/A||N/A|
|Collins et al 2012 ||Lower in the intervention group||No difference||No difference|
|Dunn et al 2016 ||No difference||No difference||No difference|
|Hurkmans et al 2018 ||Lower in control group||No difference||No difference|
|Kraschnewski et al, 2011 ||No difference||No difference||No difference|
|Krukowski et al ||N/A||No difference||No difference|
|McConnon et al ||No difference||No difference||No difference|
|Padwal et al 2017 ||N/A||N/A||N/A|
|Steinberg et al 2013 ||Lower in the intervention group||No difference||No difference|
|Yardley et al 2014 ||N/A||N/A||N/A|
aN/A: not available.
In the subgroup of studies in which there was an active intervention in the control group, there was a significant difference between Web-based interventions and nontechnology interventions regarding weight loss (MD 0.82 kg; 95% CI 0.06 to 1.59;). When the analysis was restricted to the subgroup of studies that did not have any intervention in the control group, the Web-based intervention was superior to control (MD −2.14 kg; 95% CI −2.65 to −1.64; ).
When studies were analyzed according to the length of follow-up, there was greater weight loss (MD −2.13 kg; 95% CI −2.71 to −1.55) in the Web-based intervention group than in the offline intervention group in the subgroup of studies with <6 months of follow-up, whereas there was no difference between the intervention and control groups in the subgroup of studies with ≥6 months of follow-up (MD −0.17 kg; 95% CI −2.10 to 1.76), as shown inand , respectively.
In this meta-analysis, we found that the use of Web-based digital interventions exclusively was not superior to the use of offline interventions in terms of weight or BMI loss in individuals with overweight and obesity except in the short-term. These findings were based on moderate-quality evidence. Changes in dietary and physical habits of overweight and obese individuals were not different between these 2 types of intervention either.
The findings of superiority of the intervention in comparison to the control for short-term but not long-term weight loss suggest that long-term use and adherence to digital interventions are important issues to consider when planning this kind of intervention. Moreover, the superiority of digital intervention in the subgroup of studies that had no specific intervention in the comparison group suggests that this tool might be more valuable to induce weight loss in patients who do not have access to any kind of in-person intervention.
Intervention-induced weight loss was of small clinical significance. This happened even in studies with a short-term follow-up. Low engagement to the interventions delivered by the Web-based tools might explain these modest results and might be a proxy for the low motivation of participants . These modest results also highlight the need to investigate the components and tools of Web-based platforms that lead to the maintenance of users’ motivation, interest, and participation, which play a key role in enhancing adherence to healthy behaviors.
A great diversity of behavioral techniques was found in the intervention groups across studies. Behavioral strategies with multiple components comprised most of the intervention strategies. This makes it difficult to infer which components are more effective in promoting weight loss and change of health habits and precluded us from identifying whether the results were due to differences in the nature of the interface (Web-based vs face-to-face) or in the behavioral strategy. Additionally, the principles of the interventions applied in the control group were not similar to those applied in the intervention group within each study.
High risk of attrition bias was identified in 5 of the 11 studies. Although most of them followed up participants in the short-term (less than 6 months), loss of ≥20% of participants over the follow-up period was common both in the intervention and control groups. This suggests that Web-based interventions probably do not overcome the low adherence to treatments, which is commonly reported in obesity studies. Another issue of concern regarding the quality of the studies was the scarcity of data on hazardous outcomes related to the weight loss. Since appetite disorders as well as muscle and bone mass reduction may be consequences of weight loss, it was desirable that the studies had included these issues in the results. Differences in the type of control group (with and without intervention) explained a major part of the high heterogeneity found in the meta-analysis.
The thorough revision, which included 5 databases with no language restriction, is a major strength of this study. On the other hand, the high heterogeneity and high risk of attrition bias make recommendations of using Web-based interventions for individuals with overweight and obesity based on their effectiveness on weight loss of moderate certainty.
There is moderated certainty in our findings that Web-based digital health interventions are more effective than nontechnology interventions in promoting short-term but not long-term weight loss. Moreover, Web-based interventions do not seem superior to nontechnology ones in terms of changes in dietary and physical activities. The high dropout rates in the retrieved studies contributed to a lowered quality of evidence and suggest that designing interventions that maintain participants’ engagement and motivation over time might be fundamental to the success of digital interventions.
This study was supported with grants from the National Institute of Science and Technology for Health Technology Assessment (IATS) – CNPq/Brazil. AMB was was funded by Programa de Pós Doutorado (Prêmio Capes de Teses em Medicina I-2013) from Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes). ALR was supported by CNPq (research fellowship number 310679/2016–8) and FAPEMIG (PPM-00428-17).
Conflicts of Interest
Multimedia Appendix 1
Individual study characteristics.PDF File (Adobe PDF File), 37KB
Multimedia Appendix 2
Summary of findings table according to the GRADE methodology.PDF File (Adobe PDF File), 48KB
- World Health Organization Fact Sheets. 2017 URL: http://www.who.int/en/news-room/fact-sheets/detail/obesity-and-overweight [accessed 2018-08-14] [WebCite Cache]
- GBD 2015 Obesity Collaborators, Afshin A, Forouzanfar MH, Reitsma MB, Sur P, Estep K, et al. Health Effects of Overweight and Obesity in 195 Countries over 25 Years. N Engl J Med 2017 Dec 06;377(1):13-27 [FREE Full text] [CrossRef] [Medline]
- Bray GA, Frühbeck G, Ryan DH, Wilding JPH. Management of obesity. Lancet 2016 May 07;387(10031):1947-1956. [CrossRef] [Medline]
- Wadden TA, Butryn ML, Hong PS, Tsai AG. Behavioral treatment of obesity in patients encountered in primary care settings: a systematic review. JAMA 2014 Nov 05;312(17):1779-1791 [FREE Full text] [CrossRef] [Medline]
- Douketis JD, Macie C, Thabane L, Williamson DF. Systematic review of long-term weight loss studies in obese adults: clinical significance and applicability to clinical practice. Int J Obes (Lond) 2005 Oct;29(10):1153-1167. [CrossRef] [Medline]
- Dávalos ME, French MT, Burdick AE, Simmons SC. Economic evaluation of telemedicine: review of the literature and research guidelines for benefit-cost analysis. Telemed J E Health 2009 Dec;15(10):933-948. [CrossRef] [Medline]
- Arem H, Irwin M. A review of web-based weight loss interventions in adults. Obes Rev 2011 May;12(5):e236-e243 [FREE Full text] [CrossRef] [Medline]
- Hutchesson MJ, Rollo ME, Krukowski R, Ells L, Harvey J, Morgan PJ, et al. eHealth interventions for the prevention and treatment of overweight and obesity in adults: a systematic review with meta-analysis. Obes Rev 2015 May;16(5):376-392. [CrossRef] [Medline]
- Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011 Apr;64(4):401-406. [CrossRef] [Medline]
- Review Manager (RevMan) computer program, 5 ed. Copenhagen: The Nordic Cochrane Centre: The Cochrane Collaboration; 2014.
- Borenstein M, Hedges L, Higgins J, Rothstein H. Comprehensive Meta-Analysis Computer program. 3 ed. Englewood, NJ: Biostat; 2006.
- Allen JK, Stephens J, Dennison Himmelfarb CR, Stewart KJ, Hauck S. Randomized controlled pilot study testing use of smartphone technology for obesity treatment. J Obes 2013;2013:151597 [FREE Full text] [CrossRef] [Medline]
- Chung LMY, Law QPS, Fong SSM, Chung JWY. Teledietetics improves weight reduction by modifying eating behavior: a randomized controlled trial. Telemed J E Health 2014 Jan;20(1):55-62 [FREE Full text] [CrossRef] [Medline]
- Collins CE, Morgan PJ, Jones P, Fletcher K, Martin J, Aguiar EJ, et al. A 12-week commercial web-based weight-loss program for overweight and obese adults: randomized controlled trial comparing basic versus enhanced features. J Med Internet Res 2012 Apr 25;14(2):e57 [FREE Full text] [CrossRef] [Medline]
- Dunn C, Olabode-Dada O, Whetstone L, Thomas C, Aggarwal S, Nordby K, et al. Using synchronous distance education to deliver a weight loss intervention: A randomized trial. Obesity (Silver Spring) 2016 Jan;24(1):44-50 [FREE Full text] [CrossRef] [Medline]
- Hurkmans E, Matthys C, Bogaerts A, Scheys L, Devloo K, Seghers J. Face-to-Face Versus Mobile Versus Blended Weight Loss Program: Randomized Clinical Trial. JMIR Mhealth Uhealth 2018 Jan 11;6(1):e14 [FREE Full text] [CrossRef] [Medline]
- Kraschnewski JL, Stuckey HL, Rovniak LS, Lehman EB, Reddy M, Poger JM, et al. Efficacy of a weight-loss website based on positive deviance. A randomized trial. Am J Prev Med 2011 Dec;41(6):610-614. [CrossRef] [Medline]
- Krukowski RA, Tilford JM, Harvey-Berino J, West DS. Comparing behavioral weight loss modalities: incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity (Silver Spring) 2011 Aug;19(8):1629-1635 [FREE Full text] [CrossRef] [Medline]
- McConnon Á, Kirk SF, Cockroft JE, Harvey EL, Greenwood DC, Thomas JD, et al. The Internet for weight control in an obese sample: results of a randomised controlled trial. BMC Health Serv Res 2007 Dec 19;7:206 [FREE Full text] [CrossRef] [Medline]
- Padwal RS, Klarenbach S, Sharma AM, Fradette M, Jelinski SE, Edwards A, et al. The evaluating self-management and educational support in severely obese patients awaiting multidisciplinary bariatric care (EVOLUTION) trial: principal results. BMC Med 2017 Mar 02;15(1):46 [FREE Full text] [CrossRef] [Medline]
- Steinberg DM, Tate DF, Bennett GG, Ennett S, Samuel-Hodge C, Ward DS. The efficacy of a daily self-weighing weight loss intervention using smart scales and e-mail. Obesity (Silver Spring) 2013 Sep;21(9):1789-1797 [FREE Full text] [CrossRef] [Medline]
- Yardley L, Ware LJ, Smith ER, Williams S, Bradbury KJ, Arden-Close EJ, et al. Randomised controlled feasibility trial of a web-based weight management intervention with nurse support for obese patients in primary care. Int J Behav Nutr Phys Act 2014 May 21;11:67 [FREE Full text] [CrossRef] [Medline]
- Pappa GL, Cunha TO, Bicalho PV, Ribeiro A, Couto Silva AP, Meira W, et al. Factors Associated With Weight Change in Online Weight Management Communities: A Case Study in the LoseIt Reddit Community. J Med Internet Res 2017 Dec 16;19(1):e17 [FREE Full text] [CrossRef] [Medline]
|BMI: body mass index|
|GRADE: Grades of Recommendation, Assessment, Development and Evaluation|
|MD: mean difference|
Edited by G Eysenbach; submitted 08.12.17; peer-reviewed by O Uthman, IN Gomez; comments to author 18.03.18; revised version received 14.08.18; accepted 30.08.18; published 08.01.19Copyright
©Alline M Beleigoli, Andre Q Andrade, Alexandre G Cançado, Matheus NL Paulo, Maria De Fátima H Diniz, Antonio L Ribeiro. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 08.01.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.