Published on in Vol 20, No 12 (2018): December

How Returning Aggregate Research Results Impacts Interest in Research Engagement and Planned Actions Relevant to Health Care Decision Making: Cohort Study

How Returning Aggregate Research Results Impacts Interest in Research Engagement and Planned Actions Relevant to Health Care Decision Making: Cohort Study

How Returning Aggregate Research Results Impacts Interest in Research Engagement and Planned Actions Relevant to Health Care Decision Making: Cohort Study

Original Paper

1Division of Adolescent/Young Adult Medicine, Boston Children's Hospital, Boston, MA, United States

2Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, United States

3Department of Pediatrics, Harvard Medical School, Boston, MA, United States

Corresponding Author:

Elissa R Weitzman, ScD, MSc

Division of Adolescent/Young Adult Medicine

Boston Children's Hospital

300 Longwood Avenue

BCH 3187

Boston, MA, 02115

United States

Phone: 1 617 355 3538

Fax:1 617 730 0699


Background: Collection of patient-reported outcomes measures (PROs) may augment clinical data and inform health research, improving care, yet approaches to sustaining interest among patient cohorts in research participation are needed. One approach may involve returning aggregate research results (ARRs), which may help patients contextualize personal experiences, prompt conversations with providers or family, and encourage information seeking. This model has been demonstrated for Web-based patient-centered registries. Studies with clinical cohorts may further elucidate the model, its impacts on interest in research participation and planned actions, and potential for participants to experience this as helpful or harmful—gap areas.

Objective: We sought to investigate the impacts of returning ARRs comprising summaries of PROs and clinical metrics to parents of children with rheumatic disease, assessing interest in future research participation among parents who viewed ARRs and plans for acting on returned information. Further, we sought to investigate reactions to viewing ARRs and how these reactions impacted planned actions.

Methods: Clinical and PRO data were obtained about children in a national clinical disease registry, summarized, and processed into annotated infographics, comprising ARRs for children’s parents. Parents who viewed ARRs (n=111) were surveyed about the information’s perceived value and their reactions. Reaction patterns were summarized using principal components analysis (PCA), and associations among reaction patterns and interest in research participation and planned actions were estimated using multivariate logistic regression.

Results: Parental endorsement of the value of ARRs for understanding their child’s condition and making care decisions was high (across 10 topics for which ARRs were shared, 42.2%-77.3% of the parents reported information was “very valuable”). Most (58/111, 52.3%) parents reported being more interested in participating in research after viewing ARRs, with the remainder reporting that their interest levels were unchanged. Reactions to viewing ARRs reflected experiencing validation/affirmation and information burden based on PCA. Reactions were not associated with child demographic or clinical characteristics and PROs, except that parents from households with less education reported greater information burden than those from more educated households (P=.007). In adjusted models, parents with higher validation/affirmation scores had increased odds of reporting heightened interest in research participation (adjusted odds ratio [AOR] 1.97, 95% CI 1.18-3.30), while higher information burden scores were associated with decreased odds of planned discussions with their child (AOR 0.59, 95% CI 0.36-0.95) and increased odds of planned discussions with providers (AOR 1.75, 95% CI 1.02-3.00).

Conclusions: Returning ARRs may foster a “virtuous cycle” of research engagement, especially where ARRs are experienced favorably and affect plans to share and discuss ARRs in support of a child’s chronic disease care and treatment. Reactions to ARRs vary with education level, underscoring the need for attention to equity for this model.

J Med Internet Res 2018;20(12):e10647



Growing evidence supports the importance of engaging patients in research to share information about their disease and treatment experiences and health-related quality of life [1,2]. Where shared data flow into health care systems and clinical epidemiologic studies, better symptom management, improved treatments, and better outcomes result [3]. In support of engaging patients in sharing high-quality data, considerable investment has been made in developing standardized patient-reported outcomes measures (PROs) that characterize aspects of patients’ physical, mental, and social health [3-6]. PROs enable scientific rigor and help capture the patient voice, consistent with the paradigm shifting efforts to advance patient-centered outcomes research [7-10]. Yet, open questions remain about how to motivate ongoing patient engagement in research, and this question is central to ambitious efforts to activate vast cohorts in donating PROs and clinical data [11].

One possible approach is to operationalize a process whereby patients donate health data that are subsequently processed and returned in aggregate. Here, the return of aggregate research results (ARRs) about the cohort is hypothesized to motivate a virtuous cycle of data donation that can help grow the evidence base to advance more acceptable, effective therapies [12]. Viewing ARRs may be motivating for research participation if the returned information helps patients appraise personal experiences of disease and treatment [13] and informs conversations with health care providers and family members, factors that are relevant to health care decision making [14]. These actions, which reflect an engaged and activated patient and socially embedded nature of health care decision making, are central to models of chronic illness care [15-17]. These factors are also consistent with survey reports about what motivates sharing of personal health information [18-20] and are reflected in the appeal of Web-based patient-centered health information repositories [17]. Additional empirical testing of this “closed-loop” model is essential for ascertaining whether it fosters interest in research participation among clinical cohorts and to better understand its potential for being experienced as helpful or harmful—gap areas that are central to ensuring equipoise.

Returning ARRs may be informative and reassuring for some research participants, providing a normalizing context around experiences; ARRs may also be overwhelming and disquieting for others, including if ARRs show evidence of problems experienced by others with the same condition. Other factors, including the level of education and health literacy, might also affect acceptability. Concern about the balance of benefit or harm experienced when viewing ARRs may be especially acute for conditions that are rare, have treatments that rest on an immature evidence base or incompletely ameliorate symptoms or health-related quality of life, and pose risks for side effects [21]. Arguably, motivating ongoing research engagement for such conditions is especially important because gaps in knowledge might be filled, driving improvements in therapies at the system level and decision making at the patient or family level.

Pediatric-onset rheumatic disease (RD) is a trenchant case for examining these issues. Among youth, RD is rare but rising in incidence [22], with affected children facing significant hurdles regarding health-related quality of life due to the chronic relapsing nature of RD, unpredictable disease course, and difficult treatments [1-3]. Lack of a mature evidence base guiding RD care makes maintaining patient engagement in research especially important for gathering information about disease and treatment experiences to improve care and outcomes. For example, in a prior study focused on youth with RD, we found that treatment-related problems for standard RD therapies were common and contributed to poor health-related quality of life even after controlling for patient clinical characteristics and ameliorative effects of these treatments on symptoms, such as pain [23].

This study aims to investigate impacts of returning ARRs on interest in participating in future research within a larger project focused on investigating pediatric-onset RD. We engaged parents of children with RD in donating PROs about their child’s health and treatment experiences and then returned cohort-level summaries of clinical measures and PROs to parents, testing whether viewing these ARRs increased parents’ interest in future research participation and their intentions to discuss ARRs with others or seek further information. Such discussions might encourage shared decision making, consistent with theory [6,24]. We hypothesized that the receipt of ARRs would be highly motivating for future research engagement and that reactions to and planned actions arising from viewing ARRs would vary with participants’ experiences of benefit and burden.

Study Design and Setting

We obtained data from The Learning Cohort (TLC) study [23], which surveyed consented parents of youth with a pediatric-onset RD enrolled in the Childhood Arthritis and Rheumatology Research Alliance (CARRA) Registry [25] during subspecialty care visits. Surveys included PROs to capture aspects of disease or treatment experience and well-being [23]. PRO survey measures were programmed in the Research Electronic Data Capture system [25], from whence they could flow into the registry using a modular, ontology-based, federated informatics infrastructure constructed from open-source software; this infrastructure provides research investigators full ownership and access to their contributed data while supporting permissioned and robust data-sharing across federated sites [26]. In total, 4 CARRA Registry sites participated in this study, with Institutional Review Board approval from each. All participants provided informed consent. Details of the TLC study are published elsewhere [23].

Toward the close of the study period, the research team aggregated PRO data from the full TLC cohort (N=202 dyads) and clinical metrics from the CARRA Registry. These results were summarized by the research team into a curated, annotated set of ARRs, reflecting areas of leading concern to parents of registry-involved youth based on an initial survey of their information needs conducted when they enrolled [23]. ARRs comprised 68 unique slides (including titles and section headers) delivered on a tablet computer at a routine visit and were delivered as static infographics (visuals, figures, and charts). Topics covered study methods, clinical and treatment characteristics of children in the cohort, including medications used, patterns of health-related quality of life, experiences of pain and morning stiffness, and treatment problems (Figure 1). All materials were pretested, including the ARR slides, and the survey that was to be administered to parents to elicit reactions to returned data. The process was iterative to address all concerns. Pretests for accessibility were conducted with 5 parent volunteers and a representative of a family-based disease advocacy group. Pretests for accuracy and safety were conducted with 6 pediatric rheumatologists and 1 pediatric emergency room physician/ informatician.

From August 2015 to February 2016, in lieu of collecting additional PROs during clinic visits, an approximately 50% convenience sample of participating parents viewed ARRs (119 were approached, 115 consented; 96.6% consent rate), after which they completed a survey about their reactions to these materials; 111 parents provided complete data on their reactions to ARRs.


Demographic, Clinical and Health Characteristics

Parents reported their child’s age; sex, race/ethnicity; diagnosis (juvenile idiopathic arthritis, systemic lupus erythematosus, or mixed connective tissue disease); overall health status; health-related quality of life [27]; pain interference [28]; morning stiffness; experience of serious side effects from a medication; methotrexate intolerance [29]; and highest education attained in the family. Disease duration was obtained from the CARRA Registry. Time in cohort was calculated as the number of days from the enrollment date (initial PRO collection date) to the final PRO collection date. Sample mode and mean imputation were used for 9 participants with incomplete data on demographic or clinical characteristics.

Perceived Value of and Reactions to the Return of Aggregate Research Results

Novel measures were developed and used to assess parents’ reactions to the return of ARRs. Perceived value of the return of ARRs was ascertained with the question, “Overall, how valuable might this summary information be when understanding and making decisions about your child’s condition and care?” asked for each of 10 topics that were shared in ARRs. Responses were given using a 3-point Likert scale (very valuable, somewhat valuable, and not valuable) and reported as frequencies. Sample size for each item ranged from 106 to 111 due to participant nonresponse on select items. Parents’ reactions to seeing ARRs were determined by the extent to which they agreed or disagreed with the following 6 statements, each rated on a 5-point Likert scale (strongly agree, agree, undecided, disagree, strongly disagree): “Seeing summary information about the experiences of other study participants was comforting because it made me feel like my experiences are shared and validated as real;” “Overall, seeing summary information about other patients’ experiences help me understand my child’s experience;” “Reviewing this type of information is within my comfort zone;” “Overall, this type of information raises more questions than it answers;” “This type of information requires more knowledge or expertise than I have to understand it;” and “I prefer to let my rheumatologist digest this type of information.” The mean of nonmissing items within the scale was used to impute missing response for 5 participants missing 1 or 2 (of 6) items.

Engagement Outcomes

Parents were asked to report their interest in participating in research studies after seeing ARRs; options included more interested, less interested, and not any more or less interested. As one respondent endorsed “less interested,” this response was combined with “not any more or less interested” to create a dichotomous variable. Parents were asked to report their planned actions after reviewing the summary information provided in ARRs by selecting all that applied from the following list: (1) discuss contents of the slide-deck with my child’s health care provider; (2) discuss contents of the slide-deck with my child; (3) explore different medications; (4) look up information about something I saw in this slide-deck; and (5) do something else.

Statistical Analyses

All analyses were conducted using SAS 9.4 software (SAS Institute, Inc, Cary, North Carolina). Summary statistics were generated to describe sample characteristics; differences in demographic and clinical characteristics between parents who did (n=111) or did not (n=91) receive ARRs were evaluated using t tests, Kruskal-Wallis, or chi-square (χ²) tests, as appropriate. Principal components analysis (PCA) was conducted to investigate the commonalities between ARR reactions and generate summary variables based on individual reaction measures. All factors with eigenvalues >1 were retained consistent with the standard practice [30,31], leading to a 2-factor solution, and an orthogonal rotation was applied to generate 2 uncorrelated scales (hereafter referred to as “validation/affirmation” and “information burden”).

Figure 1. Example content from aggregate research results returned to parents.
View this figure

Validation/affirmation reflects reactions to ARRs that reflect experiences of recognizing the legitimacy of personal experiences, greater insight into their child’s condition, and level of comfort with ARRs. Information burden reflects reactions to ARRs that reflect experiences akin to being over one’s head or uncomfortable with information in ARRs owing to perceived lack of personal expertise, preferences for their provider to digest the information and uncertainty. Pearson correlations, t tests, and analysis of variance (as appropriate) were used to examine bivariate relationships between each of the 2 ARR reaction constructs and demographic and clinical characteristics. Subsequently, multivariate logistic regression was used to model engagement outcomes as predicted by the 2 ARR reaction constructs; both unadjusted (controlling only for both factors simultaneously) as well as models adjusting for child age, race/ethnicity, and parent education were used.

Sample Characteristics

Children in this cohort were 12.0 years of age on average (SD 3.6) and predominantly female (n=161, 79.7%), diagnosed with juvenile idiopathic arthritis (n=187, 92.6%), white/non-Hispanic individuals (n=152, 75.2%), and had parents with any college education (n=144, 71.3%); the average disease duration was 7.7 (SD 3.5) years. No differences were observed in demographic or clinical characteristics between those who did or did not receive ARRs (Table 1).

Perceived Value of and Reactions to the Return of Aggregate Research Results

The proportion of parents finding the ARR topics to be “very valuable” ranged from 42.2% (for experiences of morning stiffness) to 77.3% (for medication problems; Figure 2).

PCA identified a 2-factor solution with high loadings for all 6 ARR reaction items (Figure 3).

The validation/affirmation construct was defined by 3 items reflecting having one’s experience validated (factor loading=0.873), improved understanding of their child’s condition (factor loading=0.865), and feeling that the ARR materials are within their comfort zone (factor loading=0.564). The information burden construct was defined by reports of requiring more knowledge to understand ARRs (factor loading=0.856), preferring a physician to “digest” ARRs (factor loading=0.717), and viewing ARRs raising more questions than were answered (factor loading=0.682). Parents from households with less education reported greater information burden than those from more educated households (P=.007; Table 2); factors were not significantly associated with other demographic or clinical characteristics.

Engagement Outcomes

The majority of the parents (58/111, 52.3%) reported that after seeing ARR, they were more interested in participating in research. Higher validation/affirmation scores were associated with nearly a 2-fold increase in the odds of reporting more interest in research participation (adjusted odds ratio [AOR]: 1.97, 95% CI 1.18-3.30; Table 3). Nearly one-third (35/111, 31.5%) of the parents reported that they would discuss the contents of ARRs with their child, nearly one-fifth (20/111, 18.0%) reported they would look up information, 15.3% (17/111) said that they would discuss ARR contents with their child’s health care provider, and approximately one-tenth (11/111, 9.9%) reported they would explore different medications. Parents who reported greater information burden were less likely to report wanting to discuss the contents of ARRs with their child (AOR 0.59, 95% CI 0.36-0.95) but were more likely to report wanting to discuss the contents of ARRs with their child’s health care provider (AOR 1.75, 95%CI 1.02-3.00).

Table 1. The parent-reported sample characteristics by receipt of aggregate research results (ARRs).
CharacteristicTotal (N=202)Received ARRs (n=111)Did not receive ARRs (n=91)P value

Child age (years), mean (SD)12.0 (3.6)11.7 (3.6)12.3 (3.7).29

Child sex, n (%).87

Female161 (79.7)88 (79.3)73 (80.2)a

Male41 (20.3)23 (20.7)18 (19.8)

Child race or ethnicity, n (%).63

White, non-Hispanic152 (75.2)85 (76.6)67 (73.6)

Non-white or Hispanic50 (24.8)26 (23.4)24 (26.4)

Highest level of parent education, n (%).79

High school graduate or less58 (28.7)31 (27.9)27 (29.7)

Any college144 (71.3)80 (72.1)64 (70.3)
Clinical characteristics

Diagnosis, n (%).50

Juvenile idiopathic arthritis187 (92.6)104 (93.7)83 (91.2)

Systematic lupus erythematosus or mixed connective tissues disease15 (7.4)7 (6.3)8 (8.8)

Disease duration (in years), mean (SD)7.7 (3.5)7.7 (3.5)7.7 (3.6).99

Methotrexate use and intolerance, n (%).65

No methotrexate use103 (51.0)59 (53.2)44 (48.4)

Use with no intolerance60 (29.7)30 (27.0)30 (33.0)

Methotrexate intolerance39 (19.3)22 (19.8)17 (18.7)

Overall healthb, mean (SD)8.1 (2.0)8.2 (1.8)8.0 (2.3).45

Typical morning stiffness in the past 2 weeks (minutes)c.30

>1536 (17.8)17 (15.3)19 (20.9)

≤15166 (82.2)94 (84.7)72 (79.1)

Lifetime serious medication side effects.64

One or more52 (25.7)30 (27.0)22 (24.2)

None150 (74.3)81 (73.0)69 (75.8)

Pediatric Quality of Life Inventory 4.0 scores, mean (SD)

Total score75.7 (18.3)74.9 (18.4)76.7 (18.4).50

Psychosocial score76.1 (18.0)75.2 (18.2)77.1 (17.8).46

Physical score75.1 (22.6)74.4 (22.3)75.9 (23.1).65

Patient-Reported Outcomes Measurement Information System Pain Interferencee, mean (SD)50.5 (10.9)51.2 (11.0)49.7 (10.9).35

Time in cohort (days), mean (SD)95.8 (100.9)105.1 (100.7)84.3 (100.4).14

aNot applicable.

bParents’ rating of their child’s overall health from 1 to 10, where higher scores indicate better health.

cParents’ report of the number of minutes of morning stiffness their child experiences on a typical day over the past 2 weeks.

dThe possible range of scores is from 0 to 100, with higher score indicating better quality of life.

eRaw pain interference scores were transformed into a “t score” for each participant. The t score rescales the raw score into a standardized score with a mean of 50, SD of 10, and the possible range of 38-78, with higher score indicating more pain interference.

Figure 2. Parents’ ratings of the value of each patient-reported topic presented in aggregate research results for understanding and making decisions regarding their child’s condition and care.
View this figure
Figure 3. Percentages and factor loadings of response items regarding reactions to the return of aggregate research results.
View this figure
Table 2. Reactions to aggregate research results (ARRs) by demographic and clinical characteristics (n=111).
CharacteristicFactor 1: Validation/affirmationaP valueFactor 2: Information burdenbP value
Demographic characteristics

Child age (years), correlation coefficient0.

Child sex, mean (SD)

Female–0.05 (1.02).290.07 (1.02).15

Male0.20 (0.92)c–0.27 (0.89)

Child race or ethnicity, mean (SD)

White, non-Hispanic–0.01 (1.05).92–0.06 (0.96).24

Non-white or Hispanic0.02 (0.81)0.20 (1.11)

Highest level of parent education, mean (SD)

High school graduate or less0.07 (0.89).670.41 (1.10).007

Any college–0.03 (1.04)–0.16 (0.92)
Clinical characteristics

Diagnosis, mean (SD)

Juvenile idiopathic arthritis–0.02 (1.02).520.01 (0.99).69

Systematic lupus erythematosus or mixed connective tissues disease0.24 (0.70)–0.15 (1.15)

Disease duration (years), correlation coefficient0.

Methotrexate use and intolerance, mean (SD)

No methotrexate use–0.01 (0.96).270.07 (0.97).54

Use with no intolerance0.20 (0.82)–0.17 (1.01)

Methotrexate intolerance–0.25 (1.28)0.04 (1.08)

Overall healthd, correlation coefficient−0.07.48−0.03.73

Typical morning stiffness in the past 2 weeks (minutes)e, mean (SD)

>150.11 (1.05).610.05 (1.02).83

≤15–0.02 (1.00)–0.01 (1.00)

Lifetime serious medication side effects, mean (SD)

One or more0.27 (0.95).09–0.25 (0.78).10

None–0.10 (1.01)0.09 (1.06)

Pediatric Quality of Life Inventory 4.0scoresf, correlation coefficient


Psychosocial Score–0.11.25–0.10.29

Physical Score–0.05.61–0.11.26

Patient-Reported Outcomes Measurement Information System Pain Interferenceg, correlation coefficient0.06.54–0.02.82

Time in Cohort (days), correlation coefficient–0.02.800.08.43

aReflects the extent to which parents feel their experience is validated or affirmed when viewing ARRs; higher scores indicate greater agreement.

bReflects the extent to which parents experienced information burden when viewing; higher scores indicate greater agreement.

cNot applicable.

dParent’s rating of their child’s overall health from 1 to 10, where higher scores indicate better health.

eParent’s report of the number of minutes of morning stiffness their child experiences on a typical day over the past 2 weeks.

fThe possible range of scores is from 0 to 100, with higher score indicating better quality of life.

gRaw pain interference scores were transformed into a “t score” for each participant. The t score rescales the raw score into a standardized score with a mean of 50, SD of 10, and the possible range of 38-78. A higher score indicates more pain interference.

Table 3. Associations between reactions to aggregate research results and engagement outcomes. 
OutcomeaOutcome prevalence, n (%)Unadjusted models, OR (95% CI)Adjusted models, OR (95% CI)
More interest in participating in research

Validation/affirmation58 (52.3)1.97 (1.21-3.18)1.97 (1.18-3.30)

Information burden1.33 (0.89-1.98)1.36 (0.89-2.09)
Planned actions

Discuss with child

Validation/affirmation35 (31.5)1.22 (0.79-1.87)1.18 (0.75-1.86)

Information burden0.69 (0.45-1.07)0.59 (0.36-0.95)

Look up information

Validation/affirmation20 (18.0)0.93 (0.58-1.47)0.87 (0.54-1.41)

Information burden0.68 (0.39-1.17)0.64 (0.36-1.15)

Discuss with providers

Validation/affirmation17 (15.3)0.69 (0.43-1.11)0.69 (0.42-1.13)

Information burden1.63 (0.98-2.71)1.75 (1.02-3.00)

Explore different medications

Validation/affirmation11 (9.9)1.03 (0.54-1.98)1.06 (0.53-2.10)

Information burden1.15 (0.63-2.11)1.31 (0.69-2.52)

aFrequency and unadjusted prevalence of engagement outcomes among those who received aggregate research results (n=111). Planned action prevalence did not sum to 100% as participants could endorse multiple actions. “Unadjusted” models controlled for validation/affirmation and information burden scales only. Adjusted models controlled for the child’s age, race/ethnicity, and highest education attained in the family in addition to both validation/affirmation and information burden scales.

This study provides evidence that returning ARRs to study participants who have donated data is highly motivating for ongoing research participation. By investigating effects of sharing ARRs on interest in research participation and planned actions stemming from this experience, findings extend the understanding of benefits of patient engagement in health care research [32]. More than half of the parents in the cohort exposed to ARRs reported increased interest in research participation as a function of receiving ARRs—good news for participatory research models, including those predicated on engaging cohorts in donating data. Moreover, parents endorsed plans for proactive engagement in their child’s treatment and outcomes after receiving ARRs, with specific plans varying as a function of validation/affirmation and information burden. Overall, reactions to ARRs aligned with hopes for fostering motivation for research participation and activation in the health care process [33,34].

As hypothesized, the experience of viewing ARRs was not “one size fits all”—participants’ responses to the model reflected both experiences of validation/affirmation, wherein parents gained value from contextualizing disease or treatment experiences through viewing cohort-level data, and information burden, wherein aggregate results may be cognitively overwhelming. Yet, as greater information burden was associated with plans to discuss results with a provider, even parent participants who may have felt overwhelmed by ARRs might benefit if they are stimulated to talk with health care providers to understand findings and discuss any implications for their child. Such activities could foster improved patient-provider partnership and shared decision making. Notably, experiencing greater information burden was evident among participants who reported lower levels of educational attainment. As such, achieving goals of optimizing and equalizing health benefits and reducing the potential for disparities stemming from this model may require additional support around interpreting and processing ARRs [34]. Nationally, low levels of health literacy and numeracy indicate that large segments of the US population may face barriers to understanding health data, underscoring the importance of attending to these issues [35,36].

This investigation was undertaken with a cohort situated in a well-defined national multisite disease registry, whose members’ diagnoses were clinically confirmed—significant strengths. Still, several limitations merit mention. First, findings reflect the experiences of engaged parents of registry participants who viewed a specific set of ARRs and may not generalize to other populations that may differ in conditions, concerns, and experiences (including the history of benefit or harm from research and care); the ARR contents are also specific to the population. Second, this study leveraged the visit structure of a registry-engaged clinical cohort, layering data collection and return onto the natural visit cycle of this cohort. Additional research using a randomized trial design would inform comparisons of engagement outcomes for participants who did and did not receive ARRs over time. Third, participants shared structured PROs, some validated, others novel; items describing motivation to participate in research and experiences of receiving ARRs are not validated but capture important patient-centered dimensions of research experience. Fourth, parent proxy reports of child clinical characteristics and PROs and parent reactions to the return of ARRs may differ from those of the child [9,37], and results from this study should not be construed as reflecting child (ie, patient) reactions to the return of ARRs or reactions of another parent or guardian. Relatedly, excepting the measure of the highest parent education attained, demographic characteristics describe the child not parents. Congruence between child and parent experiences of this model and further investigation of effects of parent demographics on outcomes may further inform this work and merit future study. Lastly, self-reported data are subject to recall and social desirability biases; however, the use of structured and validated measures and electronic data collection help protect against known validity threats.

In sum, viewing ARRs increased motivation for research participation among a majority of study participants and shows promise for driving greater patient activation and engagement. Results of this study are encouraging in light of national plans for engaging volunteers in donating personally generated data, including PROs, to drive precision medicine and comparative effectiveness research [7,11,38,39]. To the extent that these efforts utilize a closed loop approach in which ARRs are returned to a learning and sharing cohort, they may thrive. Protecting against the potential for unintentionally worsening health disparities is vital as results show that participants from households with less parent education were more likely to experience information burden from viewing ARRs. Should this lead to the differential engagement or attrition of less educated participants, biases in study findings could be introduced and the ultimate fairness and beneficence of the model undermined.


This study was funded by grants from the National Library of Medicine award #R01LM011185; the National Institute of Arthritis and Musculoskeletal and Skin Diseases award #U19AR069522 and #R21AR070944; and the Maternal and Child Health Bureau (Title V, Social Security Act), Health Resources and Services Administration award #T71MC00009. The funding bodies were not involved in the design of the study or collection, analysis, and interpretation of data, nor in writing, review or approval of the manuscript.

Conflicts of Interest

None declared.

  1. Manary M, Boulding W, Staelin R, Glickman S. The patient experience and health outcomes. N Engl J Med 2013 Jan 17;368(3):201-203. [CrossRef] [Medline]
  2. Washington AE, Lipstein SH. The Patient-Centered Outcomes Research Institute--promoting better information, decisions, and health. N Engl J Med 2011 Oct 13;365(15):e31. [CrossRef] [Medline]
  3. Basch E. Patient-Reported Outcomes - Harnessing Patients' Voices to Improve Clinical Care. N Engl J Med 2017 Jan 12;376(2):105-108. [CrossRef] [Medline]
  4. Concannon T, Fuster M, Saunders T, Patel K, Wong J, Leslie L, et al. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med 2014 Dec;29(12):1692-1701 [FREE Full text] [CrossRef] [Medline]
  5. Asch D, Rosin R. Engineering Social Incentives for Health. N Engl J Med 2016 Dec 29;375(26):2511-2513. [CrossRef] [Medline]
  6. Lavallee DC, Chenok KE, Love RM, Petersen C, Holve E, Segal CD, et al. Incorporating Patient-Reported Outcomes Into Health Care To Engage Patients And Enhance Care. Health Aff (Millwood) 2016 Apr;35(4):575-582. [CrossRef] [Medline]
  7. Ahmed S, Berzon RA, Revicki DA, Lenderking WR, Moinpour CM, Basch E, International Society for Quality of Life Research. The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy. Med Care 2012 Dec;50(12):1060-1070. [CrossRef] [Medline]
  8. Levine DR, Mandrell BN, Sykes A, Pritchard M, Gibson D, Symons HJ, et al. Patients' and Parents' Needs, Attitudes, and Perceptions About Early Palliative Care Integration in Pediatric Oncology. JAMA Oncol 2017 Sep 01;3(9):1214-1220 [FREE Full text] [CrossRef] [Medline]
  9. Brudvik C, Moutte S, Baste V, Morken T. A comparison of pain assessment by physicians, parents and children in an outpatient setting. Emerg Med J 2017 Mar;34(3):138-144 [FREE Full text] [CrossRef] [Medline]
  10. Selby JV, Forsythe L, Sox HC. Stakeholder-Driven Comparative Effectiveness Research: An Update From PCORI. JAMA 2015 Dec 01;314(21):2235-2236. [CrossRef] [Medline]
  11. Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med 2015 Feb 26;372(9):793-795 [FREE Full text] [CrossRef] [Medline]
  12. Califf R, Robb M, Bindman A, Briggs J, Collins F, Conway P, et al. Transforming Evidence Generation to Support Health and Health Care Decisions. N Engl J Med 2016 Dec 15;375(24):2395-2400. [CrossRef] [Medline]
  13. Oerlemans S, Arts L, Horevoorts N, van de Poll-Franse LV. “Am I normal?” The Wishes of Patients With Lymphoma to Compare Their Patient-Reported Outcomes With Those of Their Peers. J Med Internet Res 2017 Aug 15;19(8):e288 [FREE Full text] [CrossRef] [Medline]
  14. Dwamena F, Holmes-Rovner M, Gaulden C, Jorgenson S, Sadigh G, Sikorskii A, et al. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Database Syst Rev 2012 Dec 12;12:CD003267. [CrossRef] [Medline]
  15. Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Millwood) 2001;20(6):64-78. [CrossRef] [Medline]
  16. Gee P, Greenwood D, Paterniti D, Ward D, Miller L. The eHealth Enhanced Chronic Care Model: a theory derivation approach. J Med Internet Res 2015 Apr 01;17(4):e86 [FREE Full text] [CrossRef] [Medline]
  17. Bradley M, Braverman J, Harrington M, Wicks P. Patients' motivations and interest in research: characteristics of volunteers for patient-led projects on PatientsLikeMe. Res Involv Engagem 2016;2:33 [FREE Full text] [CrossRef] [Medline]
  18. Sanderson S, Linderman M, Suckiel S, Diaz G, Zinberg R, Ferryman K, et al. Motivations, concerns and preferences of personal genome sequencing research participants: Baseline findings from the HealthSeq project. Eur J Hum Genet 2016 Jan;24(1):14-20 [FREE Full text] [CrossRef] [Medline]
  19. Weitzman ER, Kelemen S, Kaci L, Mandl KD. Willingness to share personal health record data for care improvement and public health: a survey of experienced personal health record users. BMC Med Inform Decis Mak 2012 May 22;12:39 [FREE Full text] [CrossRef] [Medline]
  20. Weitzman E, Kaci L, Mandl K. Sharing medical data for health research: the early personal health record experience. In: J Med Internet Res. Sharing medical data for health research: The early personal health record experience. J Med Internet Res Internet Journal of Medical Internet Research; May 25, 2010:e14.
  21. Politi MC, Lewis CL, Frosch DL. Supporting shared decisions when clinical evidence is low. Med Care Res Rev 2013 Feb;70(1 Suppl):113S-128S. [CrossRef] [Medline]
  22. Helmick C, Felson D, Lawrence R, Gabriel S, Hirsch R, Kwoh C, National Arthritis Data Workgroup. Estimates of the prevalence of arthritis and other rheumatic conditions in the United States. Part I. Arthritis Rheum 2008 Jan;58(1):15-25 [FREE Full text] [CrossRef] [Medline]
  23. Weitzman E, Wisk L, Salimian P, Magane K, Dedeoglu F, Hersh A, et al. Adding patient-reported outcomes to a multisite registry to quantify quality of life and experiences of disease and treatment for youth with juvenile idiopathic arthritis. J Patient Rep Outcomes 2018;2:1-11 [FREE Full text] [CrossRef] [Medline]
  24. Wagner E. Chronic disease management: what will it take to improve care for chronic illness? Eff Clin Pract 1998;1(1):2-4. [Medline]
  25. Ringold S, Beukelman T, Nigrovic P, Kimura Y, CARRA Registry Site Principal Investigators. Race, ethnicity, and disease outcomes in juvenile idiopathic arthritis: a cross-sectional analysis of the Childhood Arthritis and Rheumatology Research Alliance (CARRA) Registry. J Rheumatol 2013 Jun;40(6):936-942. [CrossRef] [Medline]
  26. Natter M, Quan J, Ortiz D, Bousvaros A, Ilowite N, Inman C, et al. An i2b2-based, generalizable, open source, self-scaling chronic disease registry. J Am Med Inform Assoc 2013 Jan 01;20(1):172-179 [FREE Full text] [CrossRef] [Medline]
  27. Varni J, Seid M, Rode C. The PedsQL: measurement model for the pediatric quality of life inventory. Med Care 1999 Feb;37(2):126-139. [Medline]
  28. Amtmann D, Cook K, Jensen M, Chen W, Choi S, Revicki D, et al. Development of a PROMIS item bank to measure pain interference. Pain 2010 Jul;150(1):173-182 [FREE Full text] [CrossRef] [Medline]
  29. Bulatović M, Heijstek M, Verkaaik M, van Dijkhuizen EHP, Armbrust W, Hoppenreijs E, et al. High prevalence of methotrexate intolerance in juvenile idiopathic arthritis: development and validation of a methotrexate intolerance severity score. Arthritis Rheum 2011 Jul;63(7):2007-2013 [FREE Full text] [CrossRef] [Medline]
  30. Jolliffe IT. Principal Component Analysis, Second Edition. New York, USA: Springer-Verlag New York; 2002.
  31. Dunteman G. Principal components analysis. Beverly Hills: Sage Publications; 1989:65-69.
  32. Domecq J, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res 2014 Feb 26;14:89 [FREE Full text] [CrossRef] [Medline]
  33. Hibbard J, Greene J, Overton V. Patients with lower activation associated with higher costs; delivery systems should know their patients' 'scores'. Health Aff (Millwood) 2013 Feb;32(2):216-222. [CrossRef] [Medline]
  34. Hibbard J. Patient Activation and Health Literacy: What's the Difference? How Do Each Contribute to Health Outcomes. Stud Health Technol Inform 2017;240:251-262. [Medline]
  35. Keim-Malpass J, Letzkus L, Kennedy C. Parent/caregiver health literacy among children with special health care needs: a systematic review of the literature. BMC Pediatr 2015 Aug 05;15:92 [FREE Full text] [CrossRef] [Medline]
  36. Friis K, Lasgaard M, Osborne R, Maindal H. Gaps in understanding health and engagement with healthcare providers across common long-term conditions: a population survey of health literacy in 29,473 Danish citizens. BMJ Open 2016 Jan 14;6(1):e009627 [FREE Full text] [CrossRef] [Medline]
  37. Rescorla L, Ginzburg S, Achenbach T, Ivanova M, Almqvist F, Begovac I, et al. Cross-informant agreement between parent-reported and adolescent self-reported problems in 25 societies. J Clin Child Adolesc Psychol 2013;42(2):262-273. [CrossRef] [Medline]
  38. Devine E, Alfonso-Cristancho R, Devlin A, Edwards T, Farrokhi E, Kessler L, CERTAIN Collaborative. A model for incorporating patient and stakeholder voices in a learning health care network: Washington State's Comparative Effectiveness Research Translation Network. J Clin Epidemiol 2013 Aug;66(8 Suppl):S122-S129 [FREE Full text] [CrossRef] [Medline]
  39. Beskow LM, Burke W, Fullerton SM, Sharp RR. Offering aggregate results to participants in genomic research: opportunities and challenges. Genet Med 2012 Apr;14(4):490-496 [FREE Full text] [CrossRef] [Medline]

AOR: adjusted odds ratio
ARR: aggregate research result
CARRA: Childhood Arthritis and Rheumatology Research Alliance
PCA: principal components analysis
PRO: patient-reported outcome measure
RD: rheumatic disease
TLC: The Learning Cohort

Edited by G Eysenbach; submitted 02.04.18; peer-reviewed by S Rush, A Hoffman; comments to author 24.07.18; revised version received 15.08.18; accepted 10.09.18; published 21.12.18


©Elissa R Weitzman, Kara M Magane, Lauren E Wisk. Originally published in the Journal of Medical Internet Research (, 21.12.2018.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.