Published on 14.09.17 in Vol 19, No 9 (2017): September
Reliability of an e-PRO Tool of EORTC QLQ-C30 for Measurement of Health-Related Quality of Life in Patients With Breast Cancer: Prospective Randomized Trial
Background: Breast cancer represents the most common malignant disease in women worldwide. As currently systematic palliative treatment only has a limited effect on survival rates, the concept of health-related quality of life (HRQoL) is gaining more and more importance in the therapy setting of metastatic breast cancer. One of the major patient-reported outcomes (PROs) for measuring HRQoL in patients with breast cancer is provided by the European Organization for Research and Treatment of Cancer (EORTC). Currently, paper-based surveys still predominate, as only a few reliable and validated electronic-based questionnaires are available. Facing the possibilities associated with evolving digitalization in medicine, validation of electronic versions of well-established PRO is essential in order to contribute to comprehensive and holistic oncological care and to ensure high quality in cancer research.
Objective: The aim of this study was to analyze the reliability of a tablet-based measuring application for EORTC QLQ-C30 in German language in patients with adjuvant and (curative) metastatic breast cancer.
Methods: Paper- and tablet-based questionnaires were completed by a total of 106 female patients with adjuvant and metastatic breast cancer recruited as part of the e-PROCOM study. All patients were required to complete the electronic- (e-PRO) and paper-based versions of the HRQoL EORTC QLQ-C30 questionnaire. A frequency analysis was performed to determine descriptive sociodemographic characteristics. Both dimensions of reliability (parallel forms reliability [Wilcoxon test] and test of internal consistency [Spearman rho and agreement rates for single items, Pearson correlation and Kendall tau for each scale]) were analyzed.
Results: High correlations were shown for both dimensions of reliability (parallel forms reliability and internal consistency) in the patient’s response behavior between paper- and electronic-based questionnaires. Regarding the test of parallel forms reliability, no significant differences were found in 27 of 30 single items and in 14 of 15 scales, whereas a statistically significant correlation in the test of consistency was found in all 30 single items and all 15 scales.
Conclusions: The evaluated e-PRO version of the EORTC QLQ-C30 is reliable for patients with both adjuvant and metastatic breast cancer, showing a high correlation in almost all questions (and in many scales). Thus, we conclude that the validated paper-based PRO assessment and the e-PRO tool are equally valid. However, the reliability should also be analyzed in other prospective trials to ensure that usability is reliable in all patient groups.
Trial Registration: ClinicalTrials.gov NCT03132506; https://clinicaltrials.gov/ct2/show/NCT03132506 (Archived by WebCite at http://www.webcitation.org/6tRcgQuou).
J Med Internet Res 2017;19(9):e322
Epidemiological Relevance of Breast Cancer
Breast cancer represents the most common malignant disease in women worldwide, with more than 71,000 new cases diagnosed every year in Germany . In spite of improvement in progression-free survival (PFS) through promising targeted therapy options for hormone receptor-positive breast cancer, metastatic breast cancer remains an incurable disease [ - ]. In the year 2012, 17,853 women died of breast cancer in Germany alone [ ]. Although early-stage breast cancer is often associated with high survival rates, the prognosis of metastatic breast cancer is significantly poorer, and therefore, the aim of treatment is mostly palliative because of minor probability of curation in patients with metastatic breast cancer [ ]. Depending on the phenotype, median overall survival (OAS) after diagnosis of metastatic breast cancer is 2-3 years [ ], ranging from 13.3 months for triple-negative [ ] and 34.4 months for HER2-positive subtype of breast cancer [ ].
Health-Related Quality of Life in Metastatic Breast Cancer
As currently systematic palliative treatment had only limited effect on survival rates, the concept of health-related quality of life (HRQoL) is gaining more and more importance in the therapy setting of metastatic breast cancer. Especially against the background of emerging side effects accompanying multiple oncological treatment lines, treatment should primarily aim at restoration and conservation of patients’ HRQoL before prolonging the survival of patients . Additionally, the diagnosis of an incurable disease represents an enormous emotional burden resulting in psychosocial distress that might impair the patient’s well-being [ , , ]. This is also taken into account by the recent German S3-guidelines for diagnosis, treatment, and aftercare of patients with breast cancer, recommending regular assessment of HRQoL during treatment [ ]. HRQoL is defined as an individual’s perception of their position in life in the context of the culture and value systems in which they live and in relation to their goals, expectations, standards, and concerns by World Health Organization [ ]. HRQoL is divided into domains (1) health, (2) subjective feelings, (3) leisure time activities, (4) social relationships, (5) general activities, and (6) life satisfaction [ ]. For many years, HRQoL in patients with breast cancer has been investigated. Especially in patients with metastatic disease, the measurement of HRQoL is important, as the primary goal of therapy is to afford them a high quality of life during their remaining lifespan [ - ].
Health-Related Quality of Life and Patient-Reported Outcomes
The most important obstacle has been the absence of widely accepted, standardized methods for carrying out such assessments, as much of the data suggest that clinicians miss or underestimate a large proportion of the symptomatic adverse events experienced by patients [, ]. Furthermore, the assessment of adverse events and HRQoL by health care professionals is inconsistent when compared with the opinion of other professionals [ - ]. In this context, an independent report by the patient herself through patient-reported outcomes (PROs) could be more reliable and feasible. The US Food and Drug Administration (FDA) defines a PRO as any report of the status of a patient’s health condition that comes directly from the patient (or in some cases, a caregiver or surrogate), without interpretation of the patient’s response by a clinician or anyone else [ ]. PROs comprise various aspects of the subjectively perceived state of health from patient’s point of view, such as HRQoL, satisfaction with care, and drug adherence [ - ]. In this closely related areas, validated PROs are already the accepted gold standard for data collection, being used in clinical trials, as well as regulatory drug approvals (approximately 25% of US drug labels now include PRO-derived data) [ - ]. This aspect is also designated for optimization within the National Cancer Plan to enable information exchange between in-patient and outpatient treatment [ ]. Concerning feasibility of patient’s self-reported state of health in the oncological setting, a previous study demonstrated that most patients are willing and able to self-report their experiences with treatment [ ]. Moreover, the Eastern Cooperative Oncology Group could not find any association between the patient’s performance or functional status and compliance rates [ , ]. A study indicated that integrating HRQoL reports in daily clinical routine might represent an effective and time-saving option to improve medical care, as communication between patient and physician can be facilitated without extended interviews with the patient [ ]. However, although regular assessment of HRQoL is generally recommended, routine evaluation is not yet provided in clinical practice [ , ]. One of the major PROs for measurement HRQoL in patients with breast cancer is provided by the European Organization for Research and Treatment of Cancer (EORTC). The EORTC QLQ-C30 as a modular approach is available in more than 100 languages and is used to assess HRQoL in patients with various cancers within the scope of clinical trials, as well as in daily routine [ ].
Electronic Monitoring of Patient-Related Outcome
With the expansion of digital tools, assessment of PRO by using an electronic equipment (e-PRO), such as tablet computers, is becoming a promising and economically viable approach, as real-time HRQoL monitoring allows early detection of patients at risk, ongoing improvement of oncological treatment, and ensuring the patient’s safety . Current data show that monthly compliance referring to frequency and completion of questionnaires with home Web reporting was high in breast cancer, warranting strategies to enhance compliance with routine care settings [ ]. Additionally, some studies have suggested high feasibility of an electronic patient self-report platform in oncological patients, with mean compliance rates ranging from 75% to 85%, high patient satisfaction, and good usability of systems even among the non-Web avid and elderly patients [ , ]. Velikova et al [ ] demonstrated that routine electronic HRQoL assessment in patients with breast cancer could positively influence physician-patient communication, potentially improving emotional functioning and HRQoL. Additionally, there is evidence that the completion of the questionnaire itself may improve the patient’s well-being, regardless of whether the results are fed back to physicians [ ]. However, knowledge regarding patient acceptance, feasibility, and barriers remains limited, especially in relation to health status, socioeconomic aspects, and the influence of other variables on patient’s response behavior [ - ]. Currently, paper-based surveys of PRO still predominate, as there are only a few reliable and validated e-PRO questionnaires. The paper-based versions are frequently assigned into a tablet-based format without verification of reliability. As the aspects that influence the patient’s willingness to use e-PRO and their response behavior by using e-PRO remain unclear, this strategy can endanger significance of e-PRO surveys [ ]. For instance, the EORTC QLQ-C30 questionnaire has been used worldwide [ ], but only reliable paper-based versions of it, although e-PRO has become much more prevalent (and “user-friendly”) [ ]. Facing the possibilities that are coming along with the evolving digitalization in medicine, the validation of electronic versions of well-established PRO is essential in order to contribute to a comprehensive and holistic oncological care and to ensure high quality in cancer research.
Aims and Objectives
The aim of this study was to analyze the reliability of a tablet-based e–PRO-measuring application for EORTC QLQ-C30 in German in patients with adjuvant (curative) and metastatic breast cancer compared with the established paper-based version. It should be analyzed if the response behavior of patients with breast cancer is influenced by the type of answering the questionnaire (answering by using paper and pencil or tablet computer) in a statistically significant way. We wanted to know whether there are differences in response behavior between the validated paper-based PRO version of EORTC QLQ-C30 and a new e-PRO version. The other aim was to examine the feasibility of using an e-PRO version of the EORTC QLQ-C30 for the future tablet-based measurement of HRQoL in patients with metastatic and adjuvant breast cancer in clinical practice. To achieve the aims, the patients were asked to fill out both paper- and tablet-based EORTC QLQ-C30 questionnaires.
Sample and Study Design
From July 2015 to May 2016, paper- and tablet-based PRO questionnaires were completed by a total 106 female patients with adjuvant and metastatic breast cancer treated consecutively at the Department of Women’s Health in Tuebingen, Germany, and the National Cancer Centre in Heidelberg, Germany. The patients were recruited as part of the e-patient-reported outcomes and compliance analysis (PROCOM) study. The aim of e-PROCOM was to evaluate the general patient acceptance and practicability of a Web-based application for a PRO questionnaire for patients with adjuvant or metastatic breast cancer. The patients were asked to participate to compare the response behavior in paper- and Web-based questionnaires for analyzing the reliability of the e-PRO versions of the EORTC QLQ-C30 questionnaires (version 3) . Inclusion criteria of e-PROCOM were female gender, full legal age (18 years and older), adjuvant or metastatic breast cancer diagnosis, sufficient language skills in German, and signed declaration on consent forms. Exclusion criterion was participation in other studies to minimize the burden of questionnaires. The patients were asked to complete the questionnaire during an outpatient visit to the hospital under the supervision of an attending physician. The study was designed as a double-centered (Tuebingen and Heidelberg), two-armed, prospective randomized trial. All patients were required to fill out the electronic- (e-PRO) and paper-based HRQoL questionnaire of EORTC QLQ-C30. Patients in arm A were assigned tablet computer, followed by paper questionnaire in the same session. Patients in arm B filled out the paper-based version, followed by the tablet-based questionnaire. The randomization procedure is based on the permuted-block randomization, which strives to generate equally large groups of treatment [ , ]. The postexposure acceptance for using the e-PRO tool was high (92%), as the patients were asked whether they could potentially imagine using tablet-based tools before using e-PRO [ ]. Patients were informed about the aims of the study and asked for their consent ex ante. The study was approved by the Ethics Committee at the University of Tuebingen (project number 089/2015B02).
The data collection was performed in 5 parts. The first part focused on the patientsʼ socio-economic variables. The second part contained the EORTC QLQ-C30, consisting of 30 questions in 5 subscales, various symptom scales, and individual items related to the patientsʼ health status on a multidimensional level. In addition, 28 of 30 questions are designed with a 4-point Likert scale and 2 questions with a 7-point Likert scale. Mean values were calculated in accordance with the official EORTC guidelines, which require a separate score to be calculated for each scale. The scores range from 0 to 100 [, ]. The third part of the assessment also targeted HRQoL by administering FACT-B questionnaire, consisting of 37 questions with responses in 5 dimensions on a 5-point Likert scale [ , ]. The publication of FACT-B’s e-PRO reliability analysis is under preparation. The patients in the fourth part of the assessment were asked about preexisting technical skills, their willingness to use e-PRO, and potential barriers in relation to their health status [ ], and the fifth part concerned the patients’ evaluation of the e-PRO tool (publication in preparation). The patients filled out the second and third parts of the assessment both paper- and tablet-based, whereas they answered only paper-based questionnaire in the other parts. In this study, we report the results of the second part of the assessment (reliability analyses of e-PRO tool of EORTC QLQ-C30).
Specifics of the PRO Tool
For e-PRO measurement, we used the PiiA (“Patient-informiert-interaktiv-Arzt”) Web-based application, which presents the relevant questions to be completed on a tablet computer. The PiiA-portal is a Web-based solution for capturing PROs, which was self-developed by the working group. Patients receive anonymous user credentials and are asked to complete FACT-B and QLQ-C30 questionnaires.shows the user interface of the first set of questions of the German EORTC QLQ-C30. The tool is constructed similar for all 28 questions with a 4-point Likert scale. shows the user interface of the 7-point Likert scale questions. After completing the questionnaires, patients log out and the pseudo-anonymized data will be backed up on a local storage device and securely locked.
A frequency analysis was first performed using IBM SPSS Statistics (version 24) to determine the descriptive sociodemographic characteristics of the patients. After that, we analyzed both dimensions of reliability (parallel forms reliability and test of internal consistency) and examined the disparity of responses and the rate of consistency between the paper-based PRO and e-PRO answers. Both dimensions of reliability were calculated for the 30 single items and 15 scales, resulting from the single items in accordance with the EORTC guidelines . According to the Shapiro-Wilks test, the paired samples were not normally distributed, and therefore, we used the Wilcoxon test to identify possible statistically significant differences in the test of parallel forms reliability. Additionally, a Bland-Altman plot was created for the scale “overall state of health,” which represents the HRQoL ( ). Earlier, the mean values of the paper-based PRO and the e-PRO measurements were calculated with the official EORTC guidelines [ ], which require a separate score calculated for each scale, with scores ranging between 0 and 100. The consistency analyses were performed by calculation of correlation analyses (Spearman rho and agreement rates) for every EORTC question together with inter-item correlation (Pearson correlation) and rank correlation (Kendall tau) for each scale. With Pearson correlation, the internal consistency of a scale can be measured; it describes the extent to which the tasks or questions of a scale are interrelated. While Spearman rho test examines the internal consistency of the individual questions (specifically, the reliability of the e-PRO individual questions against the paper-based questions), Pearson correlation and Kendall tau tests are used to determine the scales calculated according to EORTC guidelines. Previously, we performed chi-square and Shapiro-Wilks tests between patients with metastatic and adjuvant breast cancer to identify possible statistically significant differences in relation to HRQoL. In all analyses, P values <.05 (2-tailed) were considered indicative of statistically significant differences (α=.05). As the analysis behaves as an explorative study, all reported P values can be received as purely descriptive. As we did not find any significant differences in relation to the response behavior between arms A and B in a pretest, we assessed both the arms together to compare the paper-based and e-PRO questionnaires of the patients. Bland-Altman plot was produced by using XLSTAT 2017. Missing values (which arose when patients did not answer individual questions) were ignored in the statistical calculation.
Overall, 106 female patients with breast cancer were recruited, who completed both paper- and tablet-based EORTC QLQ-C30 questionnaires. Originally, n=153 patients were assessed for eligibility, of which 47 were excluded during recruiting, allocation, and data analyses as shown in the Consolidated Standards of Reporting Trials (CONSORT) flow diagram (). A total of 53 patients were assigned tablet computer, followed by paper in the same session (arm A), whereas the same number of patients filled out the paper-based version, followed by the tablet-based questionnaire (arm B). Patients who had not completed more than half of the EORTC questions either paper- or tablet-based were excluded (1 patient in arm A and 2 patients in arm B). We did not find significant differences between the two arms in the response behavior likewise in sociodemographic status and in therapy setting, wherefore both arms were appreciated together. Previously, both arms were compared in all single items. Furthermore, 10 patients (arm A) and 16 patients (arm B), respectively, produced missing data in some questions (more often in the tablet-based questionnaire).
and show the sociodemographic characteristics of the study group, with 72% patients in adjuvant therapy and 28% in metastatic situation. We did not find significant intragroup differences between patients with metastatic and adjuvant breast cancer either in e-PRO or in paper-based PRO. Although the adjuvant and metastatic patients differ by focalizing age and HRQoL as the metastatic patients were older and reported a poorer HRQoL, we found no differences between the e-PRO response behavior of both groups. There were no differences in reliability of all single items and scales between metastatic and adjuvant patients because of which the whole collective was appreciated together. The mean age of the whole collective amounted to 51.0 years, and nearly one-third of the patients showed a higher level of education (high school diploma).
|Sociodemographic characteristics||Descriptive analyses (N=106)|
|Mean (SD)||51.0 (11.31)|
|Median (range, minimum-maximum)||52 (54, 30-84)|
|Level of education (1=lowest; 5=highest)|
|Interquartile range (25%-Quartil; 75%-Quartil)||2.0 (3.0; 5.0)|
|Sociodemographic characteristics||n (%)||95% CI|
|Kind of education|
|No qualification||1 (.9)||0-6|
|Main or secondary school leaving certificate||43 (40.6)||32-50|
|Advanced technical certificate||19 (17.9)||10-26|
|High school diploma (“Abitur”)||33 (31.1)||22-40|
|Not specified||10 (9.4)||2-15|
|Adjuvant treatment||76 (71.7)||61-83|
Parallel Forms Reliability
The e-PRO tool seems to be reliable in the dimension of parallel forms reliability, as only few significant differences could be found.shows the results of the Wilcoxon test of the 30 single items in the EORTC QLQ-C30. In only 3 items (in relation to tiredness and pain and need to rest), there were weak statistically significant differences between paper-based PRO and e-PRO. Tiredness was ranked a bit higher in the e-PRO questionnaire with noticeable differences by focalizing median’s characteristics (MDPaper-based PRO=2.0 vs MDe-PRO=3.0) similar to need to rest, whereas the patients confirmed more pain in the paper-based questionnaire. In 27 of the questions in the EORC QLQ-C30, there were no statistically significant differences between the patient’s answers in the paper-based questionnaire and e-PRO.
No statistically significant differences could be found in the Wilcoxon test for scoring values in the function and symptom scales of the EORTC QLQ-C30, except for the score “tiredness,” where the difference was significantly low (). The patient’s response behavior between paper-based PRO and e-PRO was similar on the 5 functional scales, the 9 symptom scales, and in the overall state of health, as the identified differences in the single items had balanced each other in relation to the scoring values’ calculation.
shows a Bland-Altman plot of the scale “overall of health” to compare agreement of measurements in a graphical way by plotting the difference between paper-based PRO and e-PRO against their mean. It is apparent that the deviations from the mean value of the difference are almost exclusively within the confidence interval, so the response behavior between paper-based PRO and e-PRO is not significantly different in the collective.
|Scale no.||Item no.||Short description of item||Paper-based PROa||Electronic-based PRO||P value|
|Mean (SD)||Median |
|Mean (SD)||Median |
|I||1||Strenuous activities||2.40 (1.13)||2.0 (2.0)||2.33 (1.11)||2.0 (2.0)||.18 |
|I||2||Long walk||2.19 (1.10)||2.0 (2.0)||2.23 (1.13)||2.0 (2.0)||.79|
|I||3||Short walk||1.56 (.85)||1.0 (1.0)||1.58 (.86)||1.0 (1.0)||>.99|
|I||4||Stay in bed or chair||1.79 (.87)||2.0 (1.0)||1.75 (.79)||2.0 (1.0)||.29|
|I||5||Self-care||1.18 (.55)||1.0 (.0)||1.14 (.51)||1.0 (0.0)||>.99|
|II||6||Limited in work||2.22 (1.05)||2.0 (2.0)||2.28 (.99)||2.0 (1.0)||.19|
|II||7||Hobbies or limited leisure activities||2.30 (1.10)||2.0 (2.0)||2.38 (1.04)||2.0 (1.0)||.18|
|III||21||Tense||2.06 (.85)||2.0 (2.0)||2.06 (.80)||2.0 (2.0)||.82|
|III||22||Worried||2.40 (.90)||2.0 (1.0)||2.49 (.81)||3.0 (1.0)||.31|
|III||23||Irritated||2.01 (.78)||2.0 (2.0)||2.02 (.79)||2.0 (2.0)||.71|
|III||24||Depressed||2.12 (.90)||2.0 (2.0)||2.18 (.91)||2.0 (2.0)||.85|
|IV||20||Concentration||1.86 (.97)||2.0 (1.0)||1.95 (1.06)||2.0 (2.0)||.06|
|IV||25||Memory difficulties||1.93 (.90)||2.0 (2.0)||1.96 (.91)||2.0 (2.0)||.53|
|V||26||Family life||2.21 (.96)||2.0 (2.0)||2.36 (1.01)||2.0 (1.0)||.09|
|V||27||Social life||2.39 (1.02)||2.0 (1.0)||2.41 (1.05)||2.0 (1.0)||.64|
|VI||10||Need to rest||2.33 (.93)||2.0 (1.0)||2.48 (.88)||2.0 (1.0)||.005b|
|VI||12||Felt weak||2.26 (.99)||2.0 (1.0)||2.20 (1.02)||2.0 (2.0)||.59|
|VI||18||Felt tired||2.48 (.93)||2.0 (1.0)||2.55 (.89)||3.0 (1.0)||.03b|
|VII||14||Nausea||1.54 (.78)||1.0 (1.0)||1.55 (.80)||1.0 (1.0)||.95|
|VII||15||Vomiting||1.09 (.45)||1.0 (.0)||1.13 (.54)||1.0 (.0)||>.99|
|VIII||9||Had pain||1.94 (.93)||2.0 (2.0)||1.92 (.85)||2.0 (2.0)||.02b|
|VIII||19||Pain interfered||1.95 (.99)||2.0 (2.0)||1.91 (.93)||2.0 (1.5)||.19|
|IX||8||Shortness of breath||1.92 (.87)||2.0 (1.0)||1.84 (.87)||2.0 (1.0)||.27|
|X||11||Sleep disturbance||2.15 (1.01)||2.0 (2.0)||2.23 (1.05)||2.0 (2.0)||.13|
|XI||13||Lack of appetite||1.56 (.88)||1.0 (1.0)||1.54 (.87)||1.0 (1.0)||.89|
|XII||16||Constipation||1.52 (.78)||1.0 (1.0)||1.47 (.70)||1.0 (2.0)||.95|
|XIII||17||Diarrhea||1.43 (.76)||1.0 (1.0)||1.40 (.72)||1.0 (1.0)||.15|
|XIV||28||Financial impact of disease||1.97 (1.03)||2.0 (2.0)||1.93 (1.03)||2.0 (2.0)||.28|
|XV||29||Physical condition||4.63 (1.43)||5.0 (2.0)||4.63 (1.37)||5.0 (2.0)||.95|
|XV||30||General QoLc||4.67 (1.49)||5.0 (2.13)||4.72 (1.59)||5.0 (2.0)||.29|
aPRO: Patient reported outcomes.
bStatistically weak significant difference.
cQoL: quality of life.
|Scale||Number of questions (items) QLQ‑C30||Paper-based PROa||Electronic-based PRO||P value|
|Mean (SD)||Median |
|Mean (SD)||Median |
|Physical resiliencea (I)||5 (1-5)||73.14 (24.83)||80.0 (33.3)||73.89 (23.65)||80.0 (33.3)||.08|
|Resilience at work and during leisure time activitiesa (II)||2 (6-7)||57.92 (34.61)||66.67 (50)||55.77 (32.41)||66.67 (50)||.59|
|Emotional resiliencea (III)||4 (21-24)||61.53 (23.95)||66.67 (41.67)||60.88 (23.22)||58.33 (33.34)||.53|
|Cognitive resiliencea (IV)||2 (20-25)||70.18 (26.00)||66.67 (50)||68.67 (27.67)||66.67 (50)||.12|
|Social resiliencea (V)||2 (26,27)||56.93 (30.46)||66.67 (50)||54.23 (32.58)||66.67 (50)||.12|
|Tiredness (VI)||3 (10,12,18)||45.21 (28.41)||33.33 (36.11)||45.10 (28.08)||33.33 (44.44)||.05b|
|Nausea or vomiting (VII)||2 (14,15)||10.42 (18.13)||.0 (1.47)||11.52 (19.80)||.0 (3.19)||.28|
|Pain (VIII)||2 (9,19)||31.25 (30.46)||33.33 (50.0)||29.34 (26.65)||33.33 (50.0)||.35|
|Shortness of breath (IX)||1 (8)||30.58 (29.06)||33.33 (33.33)||28.00 (29.10)||33.33 (33.33)||.70|
|Sleep disturbance (X)||1 (11)||38.45 (33.64)||33.33 (66.67)||41.06 (34.84)||33.33 (66.67)||.71|
|Lack of appetite (XI)||1 (13)||18.65 (29.18)||.0 (33.33)||18.14 (29.04)||.0 (33.3)||.72|
|Constipation (XII)||1 (16)||17.31 (25.87)||.0 (33.33)||15.69 (23.37)||.0 (33.33)||.47|
|Diarrhea (XIII)||1 (17)||14.22 (25.41)||.0 (33.33)||13.27 (23.96)||.0 (33.33)||.19|
|Financial impact of disease (XIV)||1 (28)||32.19 (34.33)||33.33 (66.67)||31.05 (34.40)||33.33 (66.67)||.20|
|Overall state of healtha (XV)||2 (29,30)||60.78 (23.75)||66.67 (41.67)||61.30 (23.82)||66.67 (33.33)||.52|
aItems are scaled from worst to best, with high scores representing a good QoL profile.
bStatistically weak significant difference.
Test of Internal Consistency
shows the Spearman rho correlation values and agreement rates, which were obtained for every question of the EORTC QLQ-C30 questionnaire. In all 30 questions, a high correlation (>.7) was found between paper-based PRO and e-PRO. In 23 questions, the correlation levels was >.85, and in 1 question, we found a maximal correlation of 1.0. In all 30 correlated questions, agreement rates fluctuated between 66.6% and 100%.
shows the results of internal consistency testing for the function and the symptom scales of the EORTC between paper-based PRO and e-PRO. There were high correlations in the response behavior (>.7) in all 5 functional scales, all 9 symptom scales, and the overall state of health. Statistically high significant correlations (>.9) between paper-based PRO and e-PRO was found in all 5 functional scales by focalizing interitem correlation (Pearson correlation). The rank correlation in all functional scales was also high, as Kendall tau coefficient ranged between .79 and .92. In the overall state of health, the correlation was .88 (Pearson correlation) and .77 (Kendall tau). In the symptom scales, the consistency rates were between .80 (shortness of breath) and .96 (lack of appetite) in relation to interitem correlation, together with rates between .71 (shortness of breath) and .93 (diarrhea) with regard to rank correlation.
|Scale no.||Item no.||Short description of items||Spearman rho||P value of Spearman rhob||Agreement (%)|
|I||4||Stay in bed or chair||.848||<.001||84.1|
|II||6||Limited in work||.863||<.001||76.8|
|II||7||Hobbies or limited leisure activities||.867||<.001||81.8|
|VI||10||Need to rest||.860||<.001||78.2|
|IX||8||Shortness of breath||.792||<.001||77.3|
|XI||13||Lack of appetite||.947||<.001||92.4|
|XIV||28||Financial impact of disease||.922||<.001||85.8|
aQoL: quality of life.
bstatistically high significant correlations.
|Scale||Pearson correlation (95% CI)||P valuea||Kendall tau||P valuea|
|Physical resilience||.979 (.966-.987)||<.001||.918||<.001|
|Resilience at work and during leisure time activities||.900 (.851-.933)||<.001||.795||<.001|
|Emotional resilience||.941 (.906-.963)||<.001||.848||<.001|
|Cognitive resilience||.914 (.865-.945)||<.001||.866||<.001|
|Social resilience||.921 (.874-.950)||<.001||.819||<.001|
|Nausea or vomiting||.956 (.928-.973)||<.001||.873||<.001|
|Shortness of breath||.798 (.712-.860)||<.001||.710||<.001|
|Sleep disturbance||.921 (.874-.951)||<.001||.810||<.001|
|Lack of appetite||.959 (.933-.974)||<.001||.916||<.001|
|Financial impact of disease||.919 (.872-.949)||<.001||.863||<.001|
|Overall state of health||.878 (.823-.916)||<.001||.769||<.001|
aStatistically high significant correlations.
In both dimensions of reliability (parallel forms reliability and internal consistency), we found high correlations with only few differences in the patient’s response behavior between paper-based PRO and e-PRO in the EORTC QLQ-C30 questionnaire. In the test of parallel forms reliability, we found statistically significant differences in only 3 of 30 questions. By focalizing the function scales and the symptom scales, there was only one statistically significant difference between the patient’s answers in both procedures. In the dimension of consistency, there were high correlation and agreement rates in all items and scales. Due to only few differences and high correlations in almost all single items and scales, the PiiA tool’s e-PRO version of EORTC QLQ-C30 seems to be reliable for HRQoL measurement in patients with metastatic and adjuvant breast cancer. Due to the results, we cannot expect that the future use of the PiiA tool in the same patient group will show significant differences between paper-based PRO and e-PRO version of QLQ-C30, or that patient’ response behavior will be significantly influenced by the survey tool after transition to electronic-based patient surveys. Therefore, the tool is suitable for ascertaining HRQoL in patients with metastatic or adjuvant breast cancer.
Comparison With Prior Work
Although e-PRO applications are on the rise, paper-based surveys of PRO still predominate clinical research, as there is a lack of reliable electronically validated questionnaires. One of the most used questionnaires for the measurement of HRQoL, especially in patients with breast cancer, is the EORTC QLQ-C30, with reliable paper-based format in many languages but without reliable electronic-based version in German. Electronic-based utilization of EORTC QLQ-C30 and other PRO without verification of reliability could endanger significance of e-PRO surveys, wherefore corresponding analysis in relation to differences and correlations between the paper-based vision of EORTC QLQ-C30 and newly developed online tools is of great importance. It can be assumed that several aspects (ie, sociodemographics, technical skills, health condition, and maybe design specifics of the e-PRO tool) influence the patient’s willingness to use e-PRO and their response behavior, which underlines the necessity of reliability analyses [, ].
However, there were some limitations in the study design and the methodological implementation, which could possibly reduce data’s validity. In 3 questions of the test of parallel forms reliability, we found several missing values that could be because of the length of the survey. The patients were surveyed while they were receiving chemotherapy intervention, and they were not permitted to take the questionnaire home to complete it. Obviously, the length of the questionnaire had an effect on the patients’ concentration, as missing values were found especially in the second response run of the EORTC questionnaire. Possibly, the burden of therapy was potentially affecting the ability of some patients to fill both paper- and tablet-based questionnaires during an outpatient visit. Missing values were found particularly in the questions “need to rest” and “felt tired,” which were potentiated unfavorable to the dimension “tiredness” in the dimension of parallel forms reliability. In addition, the tumor stage, extent of metastasis, and the administered therapy were beyond the scope of study. Furthermore, psycho-oncological information was not collected, although psycho-oncological distress is a commonly associated burden that could potentially influence the willingness to use e-PRO and therefore e-PRO’s reliability [, ]. It was not possible to determine whether the state of health was lower and the psychological distress was higher in those patients who could not be motivated to participate in the study compared with those who could be included, wherefore a selection bias was in place. Therefore, it is not possible to assess conclusively whether the e-PRO version is reliable for all patients with metastatic and adjuvant breast cancer or only for those with substantiated willingness. It must also be ensured that the proven reliability for the mentioned patient group applies only to the electronic-based version of the EORTC QLQ-C30 questionnaire but not to the PiiA tool per se.
Strengths of the Study
The EORTC QLQ-C30 questionnaire has been used worldwide , but only with reliable paper-based versions, whereas e-PRO has become much more prevalent (and “user-friendly”) [ ]. It is one of the strengths of this study that a new tool for e-PRO measurement was developed and analyzed regarding reliability in patients with breast cancer, while other studies often assign paper-based versions into a tablet-based format without verification of reliability. Reliability was ascertained in a multidimensional way, as parallel forms reliability (Wilcoxon test) and internal consistency (by focalizing Spearman rho, agreement rates, Pearson correlation and Kendall tau) were calculated. Our data show that patients with both the adjuvant and metastatic breast cancer are able to use e-PRO, as the PiiA tool was reliable in both patient groups. However, willingness to use e-PRO in patients with metastatic and adjuvant breast cancer is dependent on technical exposition [ ]. The results of the study can improve the quality of e-PRO measurements, as they seem to be generalizable, and the PiiA application of EORTC QLQ-C30 can be used for reliable e-based measurement of HRQoL in other studies and clinical routine. The tool is reliable in female patients with breast cancer, as hurdles for e-PRO could be found especially in metastatic patients [ ].
Electronic-based PRO is constantly being adopted in clinical research and clinical routine, which underlines the necessity of reliable questionnaires. The evaluated PiiA’s version of the EORTC QLQ-C30 is reliable for patients with breast cancer in adjuvant or metastatic setting because high correlation was found in almost all questions (and in many scales). Thus, we conclude equality between the validated p-PRO assessment and the used e-PRO tool. However, the reliability in other prospective trials should also be analyzed to ensure the reliable usability in all patient groups.
Conflicts of Interest
PG has received honoraria from Novartis and financial support for symposia from Novartis, Roche, and PharmaMar. The other authors declare that they have no conflict of interest.
Multimedia Appendix 1
CONSORT-EHEALTH checklist (v1.6.1).PDF File (Adobe PDF File), 1MB
- Krebsdaten. 2016. Bericht zum krebsgeschehen in Deutschland 2016 URL: http://www.krebsdaten.de/Krebs/DE/Content/Publikationen/Krebsgeschehen/Krebsgeschehen_download.pdf?__blob=publicationFile [accessed 2017-08-28] [WebCite Cache]
- Verma S, Miles D, Gianni L, Krop IE, Welslau M, Baselga J, et al. Trastuzumab emtansine for HER2-positive advanced breast cancer. N Engl J Med 2012 Nov 08;367(19):1783-1791 [FREE Full text] [CrossRef] [Medline]
- Baselga J, Cortés J, Kim S, Im S, Hegg R, Im Y, et al. Pertuzumab plus trastuzumab plus docetaxel for metastatic breast cancer. N Engl J Med 2012 Jan 12;366(2):109-119. [CrossRef] [Medline]
- Baselga J, Campone M, Piccart M, Burris HA, Rugo HS, Sahmoud T, et al. Everolimus in postmenopausal hormone-receptor-positive advanced breast cancer. N Engl J Med 2012 Feb 09;366(6):520-529. [CrossRef] [Medline]
- Hortobagyi GN, Stemmer SM, Burris HA, Yap Y, Sonke GS, Paluch-Shimon S, et al. Ribociclib as first-line therapy for HR-positive, advanced breast cancer. N Engl J Med 2016 Dec 03;375(18):1738-1748. [CrossRef] [Medline]
- Finn RS, Crown JP, Lang I, Boer K, Bondarenko IM, Kulyk SO, et al. The cyclin-dependent kinase 4/6 inhibitor palbociclib in combination with letrozole versus letrozole alone as first-line treatment of oestrogen receptor-positive, HER2-negative, advanced breast cancer (PALOMA-1/TRIO-18): a randomised phase 2 study. Lancet Oncol 2015 Jan;16(1):25-35. [CrossRef] [Medline]
- Finn RS, Martin M, Rugo HS, Jones S, Im S, Gelmon K, et al. Palbociclib and letrozole in advanced breast cancer. N Engl J Med 2016 Dec 17;375(20):1925-1936. [CrossRef] [Medline]
- Dufresne A, Pivot X, Tournigand C, Facchini T, Altweegg T, Chaigneau L, et al. Impact of chemotherapy beyond the first line in patients with metastatic breast cancer. Breast Cancer Res Treat 2008 Jan;107(2):275-279. [CrossRef] [Medline]
- Kucharczyk MJ, Parpia S, Walker-Dilks C, Banfield L, Swaminath A. Ablative therapies in metastatic breast cancer: a systematic review. Breast Cancer Res Treat 2017 Jul;164(1):13-25. [CrossRef] [Medline]
- Kassam F, Enright K, Dent R, Dranitsaris G, Myers J, Flynn C, et al. Survival outcomes for patients with metastatic triple-negative breast cancer: implications for clinical practice and trial design. Clin Breast Cancer 2009 Feb;9(1):29-33. [CrossRef] [Medline]
- Savci-Heijink CD, Halfwerk H, Hooijer GK, Horlings HM, Wesseling J, van de Vijver MJ. Retrospective analysis of metastatic behaviour of breast cancer subtypes. Breast Cancer Res Treat 2015 Apr;150(3):547-557 [FREE Full text] [CrossRef] [Medline]
- Wintner LM, Giesinger JM, Zabernigg A, Sztankay M, Meraner V, Pall G, et al. Quality of life during chemotherapy in lung cancer patients: results across different treatment lines. Br J Cancer 2013 Oct 29;109(9):2301-2308 [FREE Full text] [CrossRef] [Medline]
- Trautmann F, Hentschel L, Hornemann B, Rentsch A, Baumann M, Ehninger G, et al. Electronic real-time assessment of patient-reported outcomes in routine care-first findings and experiences from the implementation in a comprehensive cancer center. Support Care Cancer 2016 Jul;24(7):3047-3056. [CrossRef] [Medline]
- Goodwin PJ, Ennis M, Bordeleau LJ, Pritchard KI, Trudeau ME, Koo J, et al. Health-related quality of life and psychosocial status in breast cancer prognosis: analysis of multiple variables. J Clin Oncol 2004 Oct 15;22(20):4184-4192. [CrossRef] [Medline]
- Awmf. 2012. Interdisziplinaere S3-leitlinie fuer die diagnostik, therapie und nachsorge des mammakarzinoms URL: http://www.awmf.org/uploads/tx_szleitlinien/032-045OL_k_S3__Brustkrebs_Mammakarzinom_Diagnostik_Therapie_Nachsorge_2012-07.pdf [accessed 2017-08-28] [WebCite Cache]
- WHO. WHOQOL: measuring quality of life URL: http://www.who.int/healthinfo/survey/whoqol-qualityoflife/en/ [accessed 2017-08-28] [WebCite Cache]
- Ritsner M, Modai I, Endicott J, Rivkin O, Nechamkin Y, Barak P, et al. Differences in quality of life domains and psychopathologic and psychosocial factors in psychiatric patients. J Clin Psychiatry 2000 Nov;61(11):880-9; quiz 890. [Medline]
- Cortés J, Baselga J, Im Y, Im S, Pivot X, Ross G, et al. Health-related quality-of-life assessment in CLEOPATRA, a phase III study combining pertuzumab with trastuzumab and docetaxel in metastatic breast cancer. Ann Oncol 2013 Oct;24(10):2630-2635. [CrossRef] [Medline]
- Korpela J, Mali P, Kaljonen A, Salminen E. Quality of life of patients with metastatic breast cancer treated with epirubicin and docetaxel. IJCM 2011;02(03):346-351. [CrossRef]
- Karamouzis MV, Ioannidis G, Rigatos G. Quality of life in metastatic breast cancer patients under chemotherapy or supportive care: a single-institution comparative study. Eur J Cancer Care (Engl) 2007 Sep;16(5):433-438. [CrossRef] [Medline]
- Raphael J, Verma S. Overall survival (OS) endpoint: an incomplete evaluation of metastatic breast cancer (MBC) treatment outcome. Breast Cancer Res Treat 2015 Apr;150(3):473-478. [CrossRef] [Medline]
- Stefanovic S, Wallwiener M, Karic U, Domschke C, Katic L, Taran F, et al. Patient-reported outcomes (PRO) focused on adverse events (PRO-AEs) in adjuvant and metastatic breast cancer: clinical and translational implications. Support Care Cancer 2017 Feb;25(2):549-558. [CrossRef] [Medline]
- Abernethy AP, Herndon JE, Wheeler JL, Day JM, Hood L, Patwardhan M, et al. Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: pilot study of an e/Tablet data-collection system in academic oncology. J Pain Symptom Manage 2009 Jun;37(6):1027-1038. [CrossRef] [Medline]
- Pakhomov SV, Jacobsen SJ, Chute CG, Roger VL. Agreement between patient-reported symptoms and their documentation in the medical record. Am J Manag Care 2008 Aug;14(8):530-539 [FREE Full text] [Medline]
- Wuerstlein R, Kirkovits T, Drewes C, Schiltz D, Bauerfeind I, Haidinger R, et al. Abstract P2-10-05: eHealth in modern breast cancer treatment: new possibilities in communication between patients, doctors and nursing staff:. Cancer Res 2015 Apr 30;75(9 Supplement):Abstract nr P2-10-05. [CrossRef]
- FDA (Food and Drug Administration). FDA. 2009. Guidance for industry: patient-reported outcome measures; use in medical product development to support labeling claims, Stand URL: https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf [accessed 2017-08-28] [WebCite Cache]
- Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-reported outcomes (PROs): putting the patient perspective in patient-centered outcomes research. Med Care 2013 Aug;51(8 Suppl 3):S73-S79 [FREE Full text] [CrossRef] [Medline]
- Bitton A, Onega T, Tosteson AN, Haas JS. Toward a better understanding of patient-reported outcomes in clinical practice. Am J Manag Care 2014 Apr;20(4):281-283 [FREE Full text] [Medline]
- Valderas JM, Alonso J. Patient reported outcome measures: a model-based classification system for research and clinical practice. Qual Life Res 2008 Nov;17(9):1125-1135. [CrossRef] [Medline]
- Szende A, Leidy NK, Revicki D. Health-related quality of life and other patient-reported outcomes in the European centralized drug regulatory process: a review of guidance documents and performed authorizations of medicinal products 1995 to 2003. Value Health 2005;8(5):534-548 [FREE Full text] [CrossRef] [Medline]
- Basch E. Patient-reported outcomes in drug safety evaluation. Ann Oncol 2009 Dec;20(12):1905-1906 [FREE Full text] [CrossRef] [Medline]
- Basch E, Jia X, Heller G, Barz A, Sit L, Fruscione M, et al. Adverse symptom event reporting by patients vs clinicians: relationships with clinical outcomes. J Natl Cancer Inst 2009 Dec 2;101(23):1624-1632 [FREE Full text] [CrossRef] [Medline]
- Basch E, Goldfarb S. Electronic patient-reported outcomes for collecting sensitive information from patients. J Support Oncol 2009;7(3):98-99. [Medline]
- Basch E. New frontiers in patient-reported outcomes: adverse event reporting, comparative effectiveness, and quality assessment. Annu Rev Med 2014;65:307-317.
- Willke RJ, Burke LB, Erickson P. Measuring treatment impact: a review of patient-reported outcomes and other efficacy endpoints in approved product labels. Control Clin Trials 2004 Dec;25(6):535-552. [CrossRef] [Medline]
- Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004 Feb 15;22(4):714-724 [FREE Full text] [CrossRef] [Medline]
- Wallwiener M, Simoes E, Sokolov AN, Brucker SY, Fasching PA, Graf J. Health-related quality of life in metastatic and adjuvant breast cancer patients. Geburtshilfe Frauenheilkd 2016 Oct;76(10):1065-1073 [FREE Full text] [CrossRef] [Medline]
- Aaronson NK, Ahmedzai S, Bergman B, Bullinger M, Cull A, Duez NJ, et al. The European organization for research and treatment of cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst 1993 Mar 3;85(5):365-376. [Medline]
- Basch E, Iasonos A, Barz A, Culkin A, Kris MG, Artz D, et al. Long-term toxicity monitoring via electronic patient-reported outcomes in patients receiving chemotherapy. J Clin Oncol 2007 Dec 01;25(34):5374-5380. [CrossRef] [Medline]
- Judson TJ, Bennett AV, Rogak LJ, Sit L, Barz A, Kris MG, et al. Feasibility of long-term patient self-reporting of toxicities from home via the internet during routine chemotherapy. J Clin Oncol 2013 Jul 10;31(20):2580-2585 [FREE Full text] [CrossRef] [Medline]
- Graf J, Simoes E, Wißlicen K, Rava L, Walter CB, Hartkopf A, et al. Willingness of patients with breast cancer in the adjuvant and metastatic setting to use electronic surveys (ePRO) depends on sociodemographic factors, health-related quality of life, disease status and computer skills. Geburtshilfe Frauenheilkd 2016 May;76(5):535-541 [FREE Full text] [CrossRef] [Medline]
- Hartkopf AD, Graf J, Simoes E, Keilmann L, Sickenberger N, Gass P, et al. Electronic-based patient-reported outcomes: willingness, needs, and barriers in adjuvant and metastatic breast cancer patients. JMIR Cancer 2017 Aug 07;3(2):e11 [FREE Full text] [CrossRef] [Medline]
- Barentsz MW, Wessels H, van Diest PJ, Pijnappel RM, Haaring C, van der Pol CC, et al. Tablet, web-based, or paper questionnaires for measuring anxiety in patients suspected of breast cancer: patients' preferences and quality of collected data. J Med Internet Res 2014;16(10):e239 [FREE Full text] [CrossRef] [Medline]
- Steele GC, Gill A, Khan AI, Hans PK, Kuluski K, Cott C. The electronic patient reported outcome tool: testing usability and feasibility of a mobile app and portal to support care for patients with complex chronic disease and disability in primary care settings. JMIR Mhealth Uhealth 2016 Jun 02;4(2):e58 [FREE Full text] [CrossRef] [Medline]
- Ayana BA, Negash S, Yusuf L, Tigeneh W, Haile D. Reliability and validity of amharic version of EORTC QLQ-C 30 questionnaire among gynecological cancer patients in Ethiopia. PLoS One 2016;11(6):e0157359 [FREE Full text] [CrossRef] [Medline]
- Schwartzberg L. Electronic patient-reported outcomes: the time is ripe for integration into patient care and clinical research. Am Soc Clin Oncol Educ Book 2016;35:e89-e96 [FREE Full text] [CrossRef] [Medline]
- Lachin JM. Statistical properties of randomization in clinical trials. Control Clin Trials 1988 Dec;9(4):289-311. [Medline]
- Schulz KF. Randomized controlled trials. Clin Obstet Gynecol 1998 Jun;41(2):245-256. [Medline]
- EORTC. 2001. EORTC QLQ C-30 scoring manual URL: http://www.eortc.be/qol/files/scmanualqlq-c30.pdf [accessed 2017-09-06] [WebCite Cache]
- Brady MJ, Cella DF, Mo F, Bonomi AE, Tulsky DS, Lloyd SR, et al. Reliability and validity of the functional assessment of cancer therapy-breast quality-of-life instrument. J Clin Oncol 1997 Mar;15(3):974-986. [Medline]
- Kemmler G, Holzner B, Kopp M, Dünser M, Margreiter R, Greil R, et al. Comparison of two quality-of-life instruments for cancer patients: the functional assessment of cancer therapy-general and the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-C30. J Clin Oncol 1999 Sep;17(9):2932-2940. [CrossRef] [Medline]
- Schaeffeler N, Pfeiffer K, Ringwald J, Brucker SY, Wallwiener M, Zipfel S, et al. Assessing the need for psychooncological support: screening instruments in combination with patients' subjective evaluation may define psychooncological pathways. Psychooncology 2015 Dec;24(12):1784-1791. [CrossRef] [Medline]
|CONSORT: Consolidated Standards of Reporting Trials|
|EORTC QLQ‑C30: European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30 item|
|e-PRO: electronic-based patient-reported outcome|
|e-PROCOM: Patient-Reported Outcomes and Compliance Analysis|
|FACT-G: Functional Assessment of Cancer Therapy-General|
|HRQoL: health-related quality of life|
|PRO: patient-reported outcome|
|QoL: quality of life|
|SD: standard deviation|
|SPSS: Statistical Package for the Social Sciences version|
Edited by G Eysenbach; submitted 28.06.17; peer-reviewed by A Birukov, N Schaeffeler; comments to author 13.07.17; revised version received 08.08.17; accepted 14.08.17; published 14.09.17
©Markus Wallwiener, Lina Matthies, Elisabeth Simoes, Lucia Keilmann, Andreas D Hartkopf, Alexander N Sokolov, Christina B Walter, Nina Sickenberger, Stephanie Wallwiener, Manuel Feisst, Paul Gass, Peter A Fasching, Michael P Lux, Diethelm Wallwiener, Florin-Andrei Taran, Joachim Rom, Andreas Schneeweiss, Joachim Graf, Sara Y Brucker. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 14.09.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.