This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Laypeople frequently discuss medical research findings on Web-based platforms, but little is known about whether they grasp the tentativeness that is inherent in these findings. Potential influential factors involved in understanding medical tentativeness have hardly been assessed to date.
The research presented here aimed to examine the effects of personality factors and of other users’ previous contributions in a Web-based forum on laypeople’s understanding of the tentativeness of medical research findings, using the example of research on deep brain stimulation.
We presented 70 university students with an online news article that reported findings on applying deep brain stimulation as a novel therapeutic method for depression, which participants were unfamiliar with. In a randomized controlled experiment, we manipulated the forum such that the article was either accompanied by user comments that addressed the issue of tentativeness, by comments that did not address this issue, or the article was accompanied by no comments at all. Participants were instructed to write their own individual user comments. Their scientific literacy, epistemological beliefs, and academic self-efficacy were measured. The outcomes measured were perceived tentativeness and tentativeness addressed in the participants’ own comments.
More sophisticated epistemological beliefs enhanced the perception of tentativeness (standardized β=.26,
Students’ understanding of the tentativeness of research findings on deep brain stimulation in an online forum is influenced by a number of character traits and by the previous comments that were contributed to the forum by other users. There is potential for targeted modification of traits such as scientific literacy, epistemological beliefs, and academic self-efficacy to foster critical thinking in laypeople who take part in online discussions of medical research findings.
The resources made available to medical research have seen a steady increase over the past century [
Like all scientific research, that in the medical domain is affected by the inherently uncertain, temporary, and revisionary nature of scientific findings [
When confronted with medical research findings in the media, it is important for non-professionals to detect and to understand this tentativeness [
Laypeople’s understanding of tentativeness has gained relevance with the rise of Web 2.0. With the term laypeople, we refer to all Internet users who are not medical experts but nevertheless are interested in reading and understanding online newspaper texts about medical topics. So with laypeople, we mean casual readers of science journalistic texts who do not necessarily have to be patients or individuals with the same medical condition as described in the text. Readers of science journalistic articles often have an academic background and are therefore not representative of the average population. The role of laypeople has changed, since they are no longer mere recipients of information from journalists. Nowadays, anybody with Internet access may receive medical information and personal opinions from other non-professionals [
Since evidence suggests that about 85% of those people who read online news articles also read associated user comments [
Factors related to scientific understanding and scientific knowledge are known to have an impact on people’s ability to critically deal with scientific information. Accordingly, we assume that such factors may also play a role when it comes to understanding the tentativeness of medical research findings presented in the media. One such factor is scientific literacy [
Another factor that has an impact on how critically people handle scientific information and that, as a consequence, is supposed to influence whether they grasp the tentativeness of medical research findings is a person’s beliefs about the nature of knowledge, which is referred to in the literature as epistemological beliefs [
Finally, academic self-efficacy, meaning a belief in one’s own competence to be able to work effectively in an academic context [
Previously, scientific literacy, epistemological beliefs, and academic self-efficacy have all been associated with the ability to apply critical thinking in traditional psychological experiments [
In addition to these three personality factors, we were also interested in the impact of a potentially relevant situational aspect because in Web-based discussion forums people are not at all limited to the mere critical reception of information. Interactive functions of Web 2.0 also allow active contributions, and so Web-based forums are typically characterized by a vigorous exchange of opinion, personal support, and guidance [
In a laboratory study, we presented students, who were laypeople with respect to the neurosurgical procedure of deep brain stimulation (DBS), with an online medical news article on that procedure. DBS involves the implantation of remote-controlled electrodes into the brain and has been used in experimental studies to alleviate symptoms of depression. Although its effectiveness is promising, it remains as yet unproven [
We examined the relationships between scientific literacy, epistemological beliefs, academic self-efficacy, and people’s understanding of the tentativeness of the findings reported in the article, as well as potential effects of topics from preceding users’ comments on that same outcome. We examined participants’ perceived tentativeness of the case study findings reported in the online medical news article (measured by a questionnaire) and the extent to which they took tentativeness into consideration in their own comments in the forum (addressed tentativeness).
We posited the following hypotheses:
H1: Greater scientific literacy will enhance perceived tentativeness (H1a) and addressed tentativeness (H1b).
H2: More sophisticated epistemological beliefs in the medical domain will enhance perceived tentativeness (H2a) and addressed tentativeness (H2b).
H3: Greater academic self-efficacy in the science domain will enhance perceived tentativeness (H3a) and addressed tentativeness (H3b).
H4: Addressed tentativeness will depend on the degree to which comments by other users also address this issue.
Relationships between perceived tentativeness and addressed tentativeness were also explored.
In a randomized controlled experiment, participants were randomly allocated to one of three experimental conditions. In all three conditions, they were asked to write a comment in response to an online newspaper-style text, elaborating on their views as extensively as possible. Conditions differed in terms of the “user” comments purportedly made previously, which participants could see in the simulated online forum. In control condition A, no comments were present. In condition B, previous comments were present but did not address the tentativeness of the findings reported in the article. In condition C, previous comments addressed the issue of tentativeness. Scientific literacy, epistemological beliefs, and academic self-efficacy were the trait variables measured. The outcome variables identified and measured were perceived tentativeness and actively addressed tentativeness in participants’ own comments.
We recruited 70 participants aged 18-35 years from a university-wide pool of volunteers who confirmed upon study entry that they had no prior knowledge of DBS; 19 participants were male (27.1%) and 49 were female (70.0%). Two participants chose not to disclose their sex. Data were complete for all other measured variables. All participants were university students and all except one had obtained general qualification for university entrance. None of these participants was excluded from the study. Twenty-four participants were randomly assigned to condition A, 23 to condition B, and 23 participants to condition C. All gave full written informed consent and received 6€ for participating in a session that lasted approximately 45 minutes.
All participants received the same online newspaper-style article reporting a case study of a female patient with depression. The article was constructed in part on the basis of actual newspaper articles; the case study itself was fictional. It described a case in which a patient, following unsuccessful treatment with psychotropic drugs and psychotherapy, experienced substantial improvements in her quality of life after the application of DBS. The article was fabricated for the study in the style typical of mainstream media. It pointed out that the use of DBS in the patient was only experimental but did not explicitly stress the tentativeness of the content in the report.
Subsequent to reading the newspaper article, participants in the two comments conditions saw five fictitious comments by previous “users.” The “user” comments were designed to be similar in both groups in terms of word count (total of 276 and 306 words across comments for conditions B and C respectively). In addition, the comments were identical with regard to the (relatively neutral) attitude toward DBS that was expressed. The comments in both groups differed only in that for condition B the comments focused on the content of the text without addressing the tentativeness of the findings (eg, by relating the case study to the user’s own experiences with depression). For condition C, the five comments explicitly discussed the issue of tentativeness (eg, by pointing out that long-term effects of DBS were not identifiable on the basis of the case study, that the success described in the study may be due to chance or placebo effects, or that alternative explanations for the patient’s improved condition might exist).
We performed a test of the experimental material after completion of the main study to confirm that our manipulation of the level of tentativeness conveyed by the comments had been successful. For this purpose, 24 university students aged 19-30 years (mean age 22.4 years [SD 3.0]; 83% female) were presented with all 10 comments in random order and for each rated their agreement with the statement, “The comment communicates that the findings are preliminary and should be interpreted with care” on 7-point Likert scales. The five comments in condition B received an average rating of 1.53 (SD 0.39); those of condition C received an average rating of 5.35 (SD 0.63). As expected, the sum scores across the five comments for condition B (mean 7.67, SD 1.95) were significantly lower than for condition C (mean 26.75, SD 3.17),
A total of four questionnaires were administered. Participants’ level of scientific literacy was measured by the Nature of Science Assessment (NoS), which has been used previously to measure scientific literacy in a student sample [
Epistemological beliefs specific to the medical domain were assessed using the 24-item Connotative Aspects of Epistemological Beliefs (CAEB) Questionnaire [
The level of participants’ academic self-efficacy in the science domain was measured using a 4-item scale [
Items measuring academic self-efficacy.
Number | Item |
1 | “I am usually able to understand scientific content.” |
2 | “If I put enough effort into it, I succeed in gaining a good overview of the natural sciences.” |
3 | “If I have questions related to the field of science, I am usually able to help myself.” |
4 | “Without help, I am not able to deal with scientific topics at all.” (reversed item) |
Participants’ perceived tentativeness of the findings presented in the journalistic article was measured by a 6-item scale (see
Items measuring perceived tentativeness.
Number | Item |
1 | “The findings of the study are not very definite.” |
2 | “On the basis of this study, our understanding of DBS in depression is not complete yet.” |
3 | “The study is conclusive.” (reversed item) |
4 | “The findings are reliable.” (reversed item) |
5 | “The study offers a solid basis on which to decide on the future use or non-use of DBS in depression.” (reversed item) |
6 | “The findings of the study should be seen as preliminary only.” |
Addressed tentativeness in participants’ comments was coded independently by 2 raters who were naïve to the research questions and blind to the experimental conditions. Prior to viewing the comments, a list of aspects of tentativeness relating to the newspaper article was set up and used to rate the extent to which tentativeness was addressed by the participants in their comments (score 0-6; resulting from one point for each of the 6 following aspects of tentativeness that were addressed by the participants: “uncertain long-term effects,” “single case study,” “lack of control condition,” “potential for placebo effect,” “inability to draw conclusions on all patients with depression on basis of study,” “need for further studies”). Before rating the participants’ comments, the 2 raters first became familiarized with the scoring procedure using five training comments. Following completion of the coding of all 70 comments for addressed tentativeness by the 2 raters and subsequent calculation of interrater reliability (see below), the average of the scores assigned by the raters was calculated for each comment and was used for the purpose of analyses.
Demographic information was obtained using a standard self-report questionnaire with items on age, sex, and the main subject of study.
The study was performed on laptops in the laboratory. Initially, participants read the newspaper article presented on the screen in their own time. On the next screen, the “user” comments on the article were presented for the two comments conditions (B and C) in a simulated forum (condition A did not receive any “user” comments), and participants in all three conditions were asked to write their own comments in a space provided on that same page. Instructions were non-specific, with the request to simply comment on the article in any way. No time limits or limits to a word count were imposed. Subsequently, the tentativeness questionnaire, the NoS scale, the CAEB scale, and the academic self-efficacy questionnaire, as well as demographic information were completed.
Scales were initially assessed for internal consistency, and interrater reliability for addressed tentativeness scores was calculated. Bivariate Pearson correlations explored associations among all predictor and outcome variables. In order to investigate the relationship of the trait variables with perceived and addressed tentativeness scores, a linear regression model was calculated for each of the two outcome measures. Scientific literacy (NoS), epistemological beliefs (CAEB), and academic self-efficacy scores were all entered concurrently into the models in order to evaluate the independence of associations. Each of the two models controlled for experimental condition. The model on addressed tentativeness additionally controlled for word count on the basis that participants who wrote longer comments had a relatively greater chance of obtaining high scores on that measure compared with participants who wrote shorter comments. Finally, an analysis of covariance (ANCOVA) with adjustment for word count and for all three trait variables compared addressed tentativeness scores among the three experimental conditions. All statistical analyses were two-tailed.
The study had ethical approval from the Institutional Ethics Committee (approval reference: LEK 2014/001).
Internal consistencies for the CAEB (Cronbach alpha=.88), academic self-efficacy (alpha=.86), and perceived tentativeness (alpha=.74) scales were found to be good. Agreement between the 2 raters on addressed tentativeness was very good (intraclass correlation coefficient=.90). The average of the addressed tentativeness score was negatively skewed, with 24 participants (34.4%) receiving a score of 0, meaning that these individuals did not address the issue of tentativeness in their comments at all. Nonetheless, we decided not to transform the variable, given that the transformation of count data such as these may lead to a bias in results when used in parametric analyses [
Sample characteristics.
Characteristics | Values and scores | |
Age (years), mean (SD) | 24.50 (3.95) | |
Female sex, n (%) | 49 (70.0) | |
|
||
|
The humanities | 23 (32.9) |
|
Natural sciences | 10 (14.3) |
|
Pedagogics | 6 (8.6) |
|
Economics | 6 (8.6) |
|
Law | 6 (8.6) |
|
Other | 19 (27.1) |
Scientific literacy (NoS) (possible range 0-7), mean (SD) | 2.36 (1.46) | |
Epistemological beliefs (CAEB) (possible range 0-144), mean (SD) | 75.41 (16.38) | |
Academic self-efficacy (possible range 0-16), mean (SD) | 11.09 (2.88) | |
Perceived tentativeness (possible range 0-36) | 24.07 (5.57) | |
Addressed tentativeness (average of 2 raters; possible range 0-6), median (IQR) | 1.0 (0.0-2.0) | |
Comment word count (range 38-383), mean (SD) | 165.0 (73.2) |
Two-tailed Pearson correlation analyses (see
Pearson correlations among measured variables.
|
|
|||||
Age | Scientific literacy | Epistemological beliefs | Academic self-efficacy | Addressed tentativeness | Perceived tentativeness | |
Scientific literacy | .09 (.474) |
|
|
|
|
|
Epistemological beliefs | .10 (.433) | .07 (.583) |
|
|
|
|
Academic self-efficacy | .03 (.794) | .12 (.335) | -.09 (.480) |
|
|
|
Addressed tentativeness | .18 (.134) | .24 (.017)a | -.03 (.798) | .28 (.019)a |
|
|
Perceived tentativeness | .25 (.035)a | -.03 (.828) | .25 (.037)a | .14 (.406) | .39 (.001)a |
|
Word count | .05 (.698) | -.05 (.668) | -.01 (.932) | .01 (.962) | .22 (.063) | .10 (.416) |
a
In order to further investigate the findings from the univariate analyses presented in
Models of perceived tentativeness and addressed tentativeness on scientific literacy, epistemological beliefs, and academic self-efficacya.
|
Perceived tentativeness | Addressed tentativeness | ||
Standardized |
|
Standardized |
|
|
Scientific literacy | -.06 (.46) | .603 | .25 (.09)b | .025 |
Epistemological beliefs | .26 (.04)b | .034 | -.05 (.01) | .631 |
Academic self-efficacy | .14 (.24) | .238 | .31 (.05)b | .007 |
aFindings from two linear regression models (for perceived and addressed tentativeness, respectively) with all predictor variables entered in a single step. Both models controlled for experimental condition; the analysis of addressed tentativeness additionally controlled for word count. Total
bValues are significant.
In order to evaluate the effect of experimental condition on addressed tentativeness, an ANCOVA with adjustment for the word count of the participants’ own comments as well as for their scientific literacy, epistemological beliefs, and academic self-efficacy assessed mean addressed tentativeness in the three experimental conditions. As expected in H4, there was an overall effect of condition on addressed tentativeness,
This study investigated the roles that science-related interindividual character differences and themes discussed in other people’s online forum contributions play in the critical evaluation of an online medical news article by students who were laypeople with respect to the topic of the article. To our knowledge, this is the first study to assess the extent to which users discuss medical tentativeness in their comments on a medical topic in a simulated online forum.
Results of the measurements carried out in the study showed that participants with greater scientific literacy and those with higher academic self-efficacy actively addressed the issue of tentativeness to a relatively greater extent than participants who had lower scores in these dimensions. Importantly, the findings for scientific literacy and academic self-efficacy were independent of one another. Epistemological beliefs were not a predictor of addressed tentativeness. However, in line with our expectation, we found evidence showing that participants who believed medical knowledge to be relatively more complex (ie, had more sophisticated epistemological beliefs) perceived the tentativeness in the article’s research findings to a higher degree. Neither scientific literacy nor academic self-efficacy was related to perceived tentativeness (it should be noted here that in a previous study [
The finding that people with more sophisticated epistemological beliefs demonstrate a greater ability to detect tentativeness in medical research is consistent with the literature on epistemological beliefs and critical thinking in science in general [
The association of higher scientific literacy and higher academic self-efficacy with greater addressed tentativeness is consistent with previous investigations that had identified relationships of these factors with critical thinking ability [
We had also expected that participants who were exposed to comments already appearing in the forum that addressed the issue of tentativeness would be influenced in the content of their contributions. A difference was indeed found between the experimental conditions insofar as participants in the condition receiving comments that mentioned tentativeness (condition C) scored higher on addressed tentativeness compared to the participants in the control condition (condition A). With this finding, we have extended previous evidence that had identified salience of tentativeness in a journalistic article as predictive of perceived tentativeness [
In the past, online forum posts on medical topics have mainly been used for thematic language analyses [
However, the somewhat artificial setting of the forum that included researcher-generated comments and text-based experimental manipulation in a single session is a potential shortcoming of this study. Our findings are further limited by the fact that analyses were based on a sample of university students. Future laboratory studies should consider using non-specific samples in order to determine whether the effects identified here extend to the general, including non-student, population. Around one third of the student sample presented here failed to address tentativeness at all in their comments, despite being equipped with at least basic scientific education in secondary school that should provide them with the capacity to deal with scientific material and with the aptitude to apply critical thinking. Accordingly, laypeople in the general population would be expected to address tentativeness even less than was reported here.
We further used only one topic in the current study (DBS), and our findings may not generalize to other scientific topics, including medical research topics. It may be the case that the effects seen in this study in fact would be different if participants were to deal with tentative research findings in other domains. Future studies should therefore aim to extend the present results by evaluating other scientific topics. Nonetheless, by focusing on DBS, we have highlighted a field of medical research that may warrant particular care in the communication of research findings to the general population. Researchers working on DBS and who interact with science journalists and university media outlets may be advised to consider the online dissemination of their findings. Online forums in particular may represent one useful way to influence the perception of DBS research and to increase knowledge in the general population of this often life-saving procedure.
In addition to the manipulation of other possible situational factors that could influence people’s behavior in online forums, future studies should make use of real-life forums to investigate health-related thinking and behavior in patients and laypeople [
Finally, future research should also take a variety of control variables into account that might have an impact on how laypeople deal with tentative scientific information. Such control variables might include people’s personal interest in medical issues in general or in the particular medical topic at hand; whether they have been diagnosed with a relevant disease (here, depression), or in the case of more established therapeutic treatments, their prior knowledge of the medical procedure.
The processes involved in laypeople’s active contribution to online medical forums are highly complex and dynamic and are therefore difficult to investigate systematically. The study reported here has made an advance by applying a simulated online forum in a controlled laboratory setting. This allowed for the manipulation of situational aspects as well as the precise measurement of trait factors that may influence laypeople’s behavior in such a forum. We have shown that the ability to understand the tentativeness of DBS research in an online forum was not at all universal. Influential factors included people’s scientific literacy, epistemological beliefs, and academic self-efficacy. Their understanding of tentativeness was additionally and independently also affected by other users’ comments already appearing in the simulated forum.
We have made an observation that calls for awareness in future investigations, particularly in those involving real-life online forums. Users appear to be only as “good” at applying critical thinking as the existing system of the forum itself. In order to recognize the tentativeness of medical research findings, readers of online medical news articles do not only depend on article authors to refer to the fact that research findings may be uncertain, temporary, controversial, or inconsistent [
Targeting the specific trait characteristics that were identified in this study as being influential on the ability to understand the tentativeness of medical research, and which are modifiable to greater or lesser degrees [
analysis of covariance
Connotative Aspects of Epistemological Beliefs
deep brain stimulation
Nature of Science
The research reported here was funded by a grant from the German Federal Ministry of Education and Research awarded to Joachim Kimmerle and Ulrike Cress (Grant No. 01GP1306B).
None declared.