This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Social media, mobile and wearable technology, and connected devices have significantly expanded the opportunities for conducting biomedical research online. Electronic consent to collecting such data, however, poses new challenges when contrasted to traditional consent processes. It reduces the participant-researcher dialogue but provides an opportunity for the consent deliberation process to move from solitary to social settings. In this research, we propose that social annotations, embedded in the consent form, can help prospective participants deliberate on the research and the organization behind it in ways that traditional consent forms cannot. Furthermore, we examine the role of the comments’ valence on prospective participants’ beliefs and behavior.
This study focuses specifically on the influence of annotations’ valence on participants’ perceptions and behaviors surrounding online consent for biomedical research. We hope to shed light on how social annotation can be incorporated into digitally mediated consent forms responsibly and effectively.
In this controlled between-subjects experiment, participants were presented with an online consent form for a personal genomics study that contained social annotations embedded in its margins. Individuals were randomly assigned to view the consent form with positive-, negative-, or mixed-valence comments beside the text of the consent form. We compared participants’ perceptions of being informed and having understood the material, their trust in the organization seeking the consent, and their actual consent across conditions.
We find that comment valence has a marginally significant main effect on participants’ perception of being informed (
This work explores the effects of adding a computer-mediated social dimension, which inherently contains human emotions and opinions, to the consent deliberation process. We proposed that augmenting the consent deliberation process to incorporate multiple voices can enable individuals to capitalize on the knowledge of others, which brings to light questions, problems, and concerns they may not have considered on their own. We found that consent forms containing positive valence annotations are likely to lead participants to feel less informed and simultaneously more trusting of the organization seeking consent. In certain cases where participants spent little time considering the content of the consent form, participants exposed to positive valence annotations were even more likely to consent to the study. We suggest that these findings represent important considerations for the design of future electronic informed consent mechanisms.
Social media, mobile and wearable technology, and connected devices have significantly expanded the opportunities for conducting research online. Already recognized as a rich resource for psychological and social research [
Electronic consent poses new challenges when contrasted to traditional consent processes. Whereas individuals were formerly able to engage with a professional in additional face-to-face dialogue, potential online research participants have fewer opportunities to ask questions and express their concerns in real time. Furthermore, the use of certain presentation techniques and design interventions may influence an individual’s decision to participate [
While electronic consent can reduce the participant-researcher dialogue, the online environment allows the consent deliberation process to move from solitary to social settings. A computer-supported social environment could enable individuals deliberating on their consent decision to connect with each other, share information, formulate and evaluate different perspectives, and ultimately understand the risks and benefits of the research beyond the scope of one-on-one dialogue with a research staff member.
In a previous study [
Following our first study, a number of questions remained concerning the extent to which annotations containing bias or emotional valence may influence users’ deliberative processes and consent decisions, and the necessity of “policing” such information contributed by anonymous users in a high-risk context. User-generated content contains human emotion and bias by its very nature and can influence others: “…affect appears to influence what we notice, what we learn, what we remember, and ultimately the kinds of judgments and decisions we make” (p. 273) [
Traditionally, medical genetic testing targeted individual loci and was performed for specific medical contexts (eg, when investigating a suspected genetic condition). A medical expert mediated the consent process for testing and returning results. A precipitous decline in the costs of genome-scale testing, however, has led to widespread access of personal genomic data. Several companies currently offer genome-scale testing services directly to consumers. Direct-to-consumer genetic testing (DTCGT) is a relatively new and developing online service that enables individuals to acquire genetic information without the mandatory involvement of a health care provider by sending a saliva sample to a DTCGT company at the cost of a few hundred dollars. DTCGT users are often asked to share their genetic and family history information with biomedical researchers who partner with the DTCGT provider. Genetic results, including traits, ancestry, and in some cases, health information, are reported using interactive online apps [
The decision to consent to participate in biomedical research is generally mediated by two main factors: participants’ comprehension of the details of the study and their trust in the research organization [
Prior research on the design of consent forms has not yielded consistent results. Early studies on the design of consent forms focused on text readability [
Social annotations consist of three elements: the resource (ie, the text in question), the users, and the metadata created by the users. In a paper on the collective dynamics of social annotation, Catutto et al [
Cross and Sproull [
Access to socially constructed information can impact the decisions an individual makes in areas ranging from consumer products [
When user-contributed information is generated and added voluntarily to digitally mediated documents, they are not usually policed by a centralized authority [
Any potential for false information can have significant impacts on prospective participants. An individual’s ability to respond appropriately to a situation requires the ability to correctly interpret and react to incoming information, particularly in compliance-gaining settings [
Social annotations communicate both information and emotion: as a form of human communication they inherently carry information about the contributor’s emotional state or judgment about the content [
Prior research on the influence of user-generated comment valence has largely been done in the context of consumer reviews. Chen and Xie [
Studies on text with affective dimensions suggest that positive and negative sentiment could lead to greater cognitive involvement in terms of attention as well as better memory of the text [
Messages evoking or communicating particular sentiments result in different forms of engagement with the message. Berger [
Beyond the effective and appropriate communication of information, previous research shows that trust plays a crucial role in the decision to disclose sensitive information online [
Drawing from the literature above, our research model is depicted in
Deliberating whether to participate in medical research can be a complex process, though individuals’ decision-making abilities are limited [
H1a. Participants exposed to negative- and positive-valence annotations will feel more informed about their decision to consent or not than participants exposed to mixed-valence comments.
H1b. Participants exposed to negative- and positive-valence annotations will feel that they understand the content of the consent form better than participants exposed to mixed-valence comments.
Dinev and Hart [
H2a. The effect of exposure to social annotation on the extent to which participants feel informed will be stronger for individuals with lower privacy concern when exposed to negative valence comments than when exposed to mixed- or positive-valence comments.
H2b. The effect of exposure to social annotation on the extent to which participants feel they understand the content of the consent form will be stronger for individuals with lower privacy concern when exposed to negative-valence comments than mixed- or positive-valence comments.
We propose that annotation valence also plays a role in how individuals assess the trustworthiness of the organization seeking consent. Prior research has examined the role of technology-mediated social influence in protecting users in trust-related situations such as security and privacy threats [
H3a. Participants exposed to negative-valence annotations will trust the organization less than participants exposed to either mixed- or positive-valence comments.
Prior research has shown that individuals with high and low privacy concern form trust in online contexts differently from each other [
H3b. The effect of exposure to social annotation on the extent to which participants trust the organization will be stronger for individuals with lower privacy concern when exposed to negative-valence comments than mixed- or positive-valence comments.
Research model depicting dependent, independent, interaction terms, and study hypotheses.
We conducted a between-subjects experimental study to explore the effects of message valence in online social annotations on users’ beliefs and behavior surrounding consent.
A website was developed specifically for this experiment. A link to the study was made available on Amazon Mechanical Turk, and participants were paid US $5.00 for completing the questionnaires. Participation in the study was limited to English speakers with a record of at least 100 prior tasks at an approval rate exceeding 99%. Since DTCGT is marketed to the general population, we chose to recruit users via Amazon Mechanical Turk. The population of Amazon Mechanical Turk is diverse and reflective of the general population, making it a viable venue for data collection [
Participants were asked to take part in a study seeking to understand how users engage and learn from personal genomic information. They were first asked to answer several questions about their Internet usage (ie, privacy questionnaire) and to complete a tutorial on genomics. They were then asked to review the consent form for an
In order to maintain ecological validity, participants were led to believe that the additional genome mapping study was a real study in which they could participate. Participants were told that if they consented, they would be linked to an external page where they would be asked to provide their email address, phone number, and basic health information and would be contacted by an administrator of the genomics study to coordinate further (
Consent question used in study.
A privacy questionnaire and personal genomics tutorial preceded the consent form. Because the majority of the risks and issues with digitally mediated research center on data privacy, particularly in the context of genomics research, we used a measure of pre-existing privacy concern to assess an individual’s existing attitude towards online privacy-related issues. We used a validated 16-item measure for privacy concern developed by Buchanan et al [
Buchanan et al’s [
Question # | Question content |
1 | In general, how concerned are you about your privacy while using the Internet? |
2 | Are you concerned about online organizations not being who they claim they are? |
3 | Are you concerned that you are asked too much personal information when you register or make online purchases? |
4 | Are you concerned about online identity theft? |
5 | Are you concerned about people online not being who they say they are? |
6 | Are you concerned that information about you could be found on an old computer? |
7 | Are you concerned who might access your medical records electronically? |
8 | Are you concerned about people you do not know obtaining personal information about you from your online activities? |
9 | Are you concerned that if you use your credit card to buy something on the Internet your card number will be obtained/intercepted by someone else? |
10 | Are you concerned that if you use your credit card to buy something on the Internet your card will be mischarged? |
11 | Are you concerned that that an email you send may be read by someone else besides the person you sent it to? |
12 | Are you concerned that an email you send someone may be printed out in a place where others could see it? |
13 | Are you concerned that a computer virus could send out emails in your name? |
14 | Are you concerned about emails you receive not being from whom they say they are? |
15 | Are you concerned that an email containing a seemingly legitimate Internet address may be fraudulent? |
The personal genomics tutorial comprised learning materials on the human genome and personal genomics developed by the Personal Genetics Education Project [
Sample genomic report presented to users in the training portion of this study.
Following the genomics tutorial, participants were presented with the consent form for an additional, optional study in which their genomes would be mapped and their family health history and trait information would be collected online. The study was framed as a voluntary contribution to research (rather than a commercial service in exchange for payment), but those who chose to participate would receive their results in a free, online report. The content of the consent form was based on Office for Human Research Protections guidelines [
The experimental consent form included comment boxes with social annotations in the margins of the screen (
The three experimental conditions included one iteration of the consent form in which the onscreen annotations contained all of the positive-valence comments, one iteration that contained only the negative-valence comments, and a final iteration that contained mixed-valence comments: positive and negative valence comments were alternated equally in the text, beginning with a positive-valence comment. To compare across these conditions, we placed comments at the same point in the text, referencing the same passages and topics in the text of the consent form.
Prior research on the effects of message valence has largely compared positive- to negative- valence messages to each other, or messages containing some valence with neutral messages. Participants’ feedback in early stages of the study indicated that comments in this context are rarely neutral: personal genomics is an important topic that evokes emotionally charged responses. To preserve ecological validity, we therefore chose to examine the effects of mixed-valence annotations rather than neutral annotations or annotations whose overall effect was neutral.
Annotations in each condition also displayed an indicator showing how many other (hypothetical) study participants “liked” the comments. The number of “likes” for each comment was determined by the researchers and ranged from 0-46 likes on a comment. The same number of likes were displayed for each comment, in each condition (ie, both the positive and negative valence instances of a comment in each of the three conditions had the same number of likes).
Participants in this study had the ability to interact with the annotations and likes embedded in the consent form (unlike in our first study where the comments were entirely static). We wanted to provide the participants the opportunity to engage with the annotations more directly and in ways that you might find elsewhere online. In our study, we used the SideComments application programming interface to implement functionality that allowed participants to respond to or “like” existing comments or to create their own highlights and textual annotations. They could also click on a comment to open or close it or could hover over an in-text highlight to open the associated comment. Stylized profile photos were used to improve the ecological validity of the annotations: websites that incorporate social annotations frequently implement some mechanism for signaling to participants that the comments came from multiple authors.
To ensure that the added level of interactivity did not present a confound in our study of message content, we devised and tested an iteration of the interface in which the comments were non-interactive. The comments were identical in message and placement to the annotations in the interactive mixed-valence condition. We recruited 137 participants and presented them with the same study as participants in the interactive conditions, and Student’s
Comparison of measures between an interactive, mixed-valence condition, and a non-interactive, mixed-valence condition.
Interactive, mixed-valence condition | Non-interactive, mixed-valence condition | |||
Decision was informed | 4.5 (0.69) | 4.46 (0.65) | .74 | |
Understood all the material | 4.19 (0.93) | 4.25 (0.76) | .72 | |
Trust the organization seeking my consent | 3.82 (0.82) | 3.66 (0.94) | .26 | |
Consent, n (%) | 20 (43%) | 65 (49%) | .61 | |
No consent, n | 26 | 67 |
Screenshot of consent form with highlighted text and social annotations.
Following their decision to consent to the personal genomic study described in the consent form, users were presented with questions about their deliberative process and perceptions of the consent form (see
Questions used to evaluate each hypothesis.
Hypothesis | Question |
H1a, H2a | I feel that my decision (to consent or not) was an informed decision. |
H1b, H2b | I feel that I understood the material presented and I have no additional questions. |
H3a, H3b | Based on what I have seen and read in this consent form, I feel like I can trust the HCIPGP to use and protect my data in the ways outlined in the consent form. |
Prior research has shown that demographic variables can influence how informed participants feel [
Analysis of variance with covariates was used to identify main effects of condition and interaction effects where applicable, while controlling for demographic variables and participants’ pre-existing attitude towards information privacy. Post-hoc Tukey tests were performed to further examine the results pairwise. The interactivity measures (ie, number of times participants opened, liked, or hover over comments, and how many comments they wrote) were found to contain positive skew (ie, a larger number of participants interacted relatively little with the interactive features of the consent form). To correct for this skew and produce a relatively symmetrical distribution of actions, we transformed the counts for each interactive measure by using its square root in the analysis [
A total of 152 participants took part in this study: 56 participants were assigned to the negative valence condition, 46 participants to the mixed valence condition, and 47 participants to the positive valence condition. The average age of participants was 34.25 years (SD 10.78), and 72 (48.3%) participants were female. One participant had some high-school education, 12 participants had high school diplomas, 58 participants had some college education, 59 participants had bachelor degrees, 14 participants had master’s degrees, 3 participants had doctoral degrees, and 2 participants declined to state their education.
Participants spent 3.88 minutes on average (SD 3.14 min) studying the genomics tutorial, and 3.96 minutes on average (SD 2.21 min) studying Jamie’s sample genomics test results. Only 3 (out of 152) answered fewer than 3 out of 6 genome tutorial questions, or fewer than 2 out of 3 of the genome report questions, incorrectly. These individuals were removed from the dataset, leaving 149 viable participants.
Correlation analysis was used to test whether the domain comprehension scores from the entire population impacted the extent to which they felt their decision was informed (ie, informed consent). Within the subset of viable participants, the correlation analysis between participants’ comprehension scores and perceptual variables failed to reach significance. The domain comprehension score was therefore not controlled for going forward.
Participants had a mean rating of 2.93 (between 1 and 5, SD 0.87) on our measure of privacy concern.
In the condition with the negative-valence comments, participants spent an average of 7.57 minutes (SD 8.56 min) studying the consent form before deciding whether to consent. In the mixed condition, participants spent 8.18 minutes (SD 7.14 min), and in the positive condition participants spent 5.82 min (SD 4.20 min) prior to deciding whether to consent. An analysis of variance testing the distribution of time across conditions shows that condition does not a have a significant main effect on time: the amount of time spent studying the consent form did not differ significantly between social annotations’ valence. We did observe, however, a significant effect of gender on time: female participants took significantly longer to read the consent form (mean 488.72, SD 488.04) than male participants (mean 357.38, SD 300.65;
Overall, participants who consented spent significantly less time studying the consent form than participants who did not consent (mean 5.62 min, SD 7.36 min and mean 8.39, SD 5.86 min, respectively;
The number of times participants liked, opened, or added comments to the consent form did not differ significantly across conditions (see
Our main findings are presented in
Results from the comparison between the negative-, mixed-, and positive-valence conditions.
Negative valence comments | Mixed valence comments | Positive valence comments | |||
Decision was informed | 4.45 (0.63) | 4.5 (0.69) | 4.17 (0.94) | .07 | |
Understood all the material | 3.98 (1.05) | 4.19 (0.92) | 4.28 (0.69) | ns | |
Trust the organization seeking my consent | 3.59 (1.14) | 3.82 (0.82) | 4.02 (0.90) | .08 | |
Liked comments | 1.43 (2.62) | 1.80 (2.52) | 1.53 (2.67) | ns | |
Commented | 1.62 (3.04) | 1.61 (2.27) | 1.19 (1.65) | ns | |
Opened comment | 5.46 (7.30) | 7.54 (9.11) | 5.72 (5.87) | ns | |
Hovered over in-text highlight | 2.88 (7.61) | 4.36 (6.50) | 1.56 (2.98) | .08 | |
Time (s) | 454.12 (513.87) | 461.89 (392.68) | 341.00 (234.09) | .012 | |
Consent, n (%) | 27 (48.21%) | 20 (43.48%) | 27 (57.44%) | ns | |
No consent, n | 29 | 26 | 20 |
The rate of consent did not differ significantly across conditions: 48% (27/56) of participants consented in the negative valence condition, 43% (20/46) consented in the mixed-valence condition, and 57% (27/47) consented in the positive condition. There was, however, a significant interaction between condition and the amount of time participants spent studying the consent form on the consent rate (
Proportion of participants who consented in each condition depending on whether they spent more or less than the median amount of time studying the consent form.
The experimental intervention had a marginally significant main effect on participants’ beliefs (
Our results indicate that condition does not have a main effect on participants’ belief that they understood the content of the consent form, and this effect does not differ according to participants’ prior privacy preserving attitudes and behavior. We therefore reject hypotheses H2a and H2b.
Condition had a marginal main effect on the extent to which participants reported trusting the organization (
Although we observed a significant, negative main effect of privacy concern on participants’ trust in the organization (
We also observed a marginally significant effect of age (older participants tended to trust the organization less than younger participants: B=-0.01,
Impact of the interaction of condition and number of likes on the extent to which participants reported trusting the organization.
In this study, we found that the valence communicated in social annotations, which are embedded in an interactive informed consent form, can influence individuals’ perceptions and beliefs about consent. In particular, we show that consent forms containing positive valence annotations are likely to lead participants to feel less informed and simultaneously more trusting of the organization seeking consent. In certain cases where participants spent little time considering the content of the consent form, participants exposed to positive valence annotations were even more likely to consent to the study.
While our findings that participants in the mixed-valence condition felt more informed than participants in the positive-valence condition may seem surprising in the context of previous studies comparing positive- and negative-valence messages, we argue that it contributes to our understanding of social influence in contexts where sentiment is effectively mixed. Prior research shows that individuals tend to focus on the negative elements of the consent process as a result of the information provider’s desire to warn others about threats, and the information seeker’s desire to acquire more information about a potential problem [
Our results show that participants’ trust in the organization also differs across condition: participants in the negative valence condition were significantly less trusting than participants in the positive valence condition. This finding is supported by previous research showing that negative messages tend to be more persuasive in general [
Notably, even when valence was extreme (as in the positive and negative manipulations), there was no significant impact on the ultimate metric of consent rates. This seems to indicate that implementing social comments on consent processes may risk little in terms of actual consent rates, while giving participants an increased sense of autonomy by helping them feel more informed. This is generally consistent with the results of our previous study [
This study has demonstrated that social annotation interventions can have an impact in a biomedical informed consent decision-making context. In contrast to the spaces where social annotation studies have traditionally been conducted (eg, consumer products, online search platforms, and security feature adoption), human subjects research requires decisions that are intensely personal and can have substantial ramifications for the individual as well as their families. Our research demonstrates that
Social influence in online environments and its effect on users in social recommender systems has been the topic of substantial research in recent years [
Our results also contribute to the literature on valence in social annotation. The existing research on mixed-valence social annotations is sparse: authors focus on comparing positive to negative valence comments [
This study represents a new and expanded understanding of the multidimensionality of social annotation in a high-risk decision-making context. Our previous study showed that the inclusion of social annotation does not merely improve or worsen the user’s experience (as put forth in existing studies); rather, it changes how participants reflect on their ability to make informed decisions for themselves in complex ways. Here we extend that line of research to provide a unique and nuanced perspective on how inherent qualities of user-generated content, namely emotional valence, can influence and engage individuals. This is particularly salient in the context of informed consent because the focus of deliberation is not among members for the purpose of consensus agreement, but within the individual [
While this study demonstrates how exposure to computer-supported social annotations impacts individuals’ perceptions in the context of informed consent, it has a number of limitations. Though we believe that the demonstrated increase in the perception of being informed suggests that social annotations can benefit prospective participants, the experiment was structured to study the effects of exposure to annotations on participants’
Furthermore, we look at the impact of a narrow range of emotional valence that is operationalized in their extremes; that is to say that it is unlikely that the user will be confronted with only positive, only negative, or perfectly mixed-valence comments. It is more likely that they would be confronted with some complex mix of the two that leans toward an overall positive or negative effect. Furthermore, we prioritized using ecologically representative comments in our study rather than controlling for the strength of sentiment contained in each comment individually. Additional research is needed to control for and understand the impact of less extreme and less consistent examples of emotional valence [
The small sample sizes used in this study may also have obscured findings related to participants’ perceptions, given that the manipulation of sentiment was relatively subtle. We believe that the results we have presented here are compelling for an exploratory study such as this one, but future research should consider larger sample sizes when investigating related questions.
Finally, a number of important questions remain for further investigation that will help us determine whether social annotation interventions are appropriate in this context. Evaluating the effect of creating and actively engaging with social annotation on user behavior requires us to understand how to solicit meaningful content from participants, what motivates individuals to contribute content, what privacy issues are associated with contributing and accessing health-related information, and how (or whether) to “police” information contributed by anonymous others in a form with such a significant impact: additional research is needed to understand whether moderating user-contributed information to create the desired effect is ethical and effective. Knowing that we may be able to improve certain aspects of the process of deliberating consent by incorporating novel and non-traditional sources of information, however, obligates us as a community to explore social annotation interventions further.
Electronic consent has become increasingly popular in Internet research in general and biomedical research in particular. The work presented here explores the effects of adding a computer-supported social dimension, which inherently contains human emotions and opinions, to the consent deliberation process. In our first study we found that exposure to social annotations results in participants’ feeling that their decision was more informed, but simultaneously less confident in their understanding of the genomics material presented in the consent form as well as less trusting of the organization soliciting the consent. Based on these findings, we proposed that augmenting the consent deliberation process with multiple voices can enable individuals to capitalize on the knowledge of others, which brings to light questions, problems, and concerns they may not have considered on their own. In this study, we examined the influence of human emotion contained in these voices on participants’ perceptions and beliefs about consent. We found that consent forms containing positive valence annotations are likely to lead participants to feel less informed and simultaneously more trusting of the organization seeking consent. In certain cases where participants spent little time considering the content of the consent form, participants exposed to positive valence annotations were even more likely to consent to the study. We suggest that these findings represent important considerations for the designers of such systems. We also call for future research that may extend the research on socially enabled online consent forms to examine the role of novel user-generated sources of information, and may develop new measures and indicators for evaluating social informed consent.
direct-to-consumer genetic testing
This work was partially funded by Grants IIS-1017693 and IIS-1422706 from the National Science Foundation (NSF), Division of Information and Intelligent Systems (IIS).
None declared.