This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Web-based decision aids are known to have an effect on knowledge, attitude, and behavior; important components of informed decision making. We know what decision aids achieve in randomized controlled trials (RCTs), but we still know very little about how they are used and how this relates to the informed decision making outcome measures.
To examine men's use of an online decision aid for prostate cancer screening using website transaction log files (web-logs), and to examine associations between usage and components of informed decision making.
We conducted an observational web-log analysis of users of an online decision aid, Prosdex. Men between 50 and 75 years of age were recruited for an associated RCT from 26 general practices across South Wales, United Kingdom. Men allocated to one arm of the RCT were included in the current study. Time and usage data were derived from website log files. Components of informed decision making were measured by an online questionnaire.
Available for analysis were 82 web-logs. Overall, there was large variation in the use of Prosdex. The mean total time spent on the site was 20 minutes. The mean number of pages accessed was 32 (SD 21) out of a possible 60 pages. Significant associations were found between increased usage and increased knowledge (Spearman rank correlation [ρ] = 0.69,
Increased usage of Prosdex leads to more informed decision making, the key aim of the UK Prostate Cancer Risk Management Programme. However, developers realistically have roughly 20 minutes to provide useful information that will support informed decision making when the patient uses a web-based interface. Future decision aids need to be developed with this limitation in mind. We recommend that web-log analysis should be an integral part of online decision aid development and analysis.
ISRCTN48473735; http://www.controlled-trials.com/ISRCTN48473735 (Archived by WebCite at http://www.webcitation.org/5pqeF89tS)
Enabling patients to make informed decisions is the new benchmark for high quality care as recently highlighted by the NHS (National Health Service) Constitution document in which the patient’s right to choice and information were emphasized [
Decision aids are designed to facilitate informed decision making, which is characterized by improved knowledge and attitudes that are congruent with subsequent behaviors [
Decision aids are therefore powerful tools that have the potential to influence patients’ health care decisions and use of services. Given their prominence in health care and policy making, clinicians and developers are realizing the need for a greater understanding of how decision aids are developed and used. If decision aids have the potential to influence patients’ decisions, then we need to know how we can optimize their design, particularly for diverse groups of users. We also need to know if usage of decision aids is sufficiently high to justify the expense involved in developing increasingly sophisticated decision support tools for patients.
Until recently, the formats of decision aids (eg, leaflets and videos) did not allow researchers to analyze patterns of use, particularly if the decision aid was used by the patient when alone. The migration of decision aids to the Internet, however, has offered the potential of analyzing use of decision aids in greater detail by using website transaction log files (web-logs). Within the context of website interface design, web-log analysis has long been realized as an important tool for examining website usability and usage [
We believe, however, that far greater research opportunities are offered by analyzing patients’ usage of decision aids using web-logs. First, we can examine associations between usage and key outcomes that decision aids are known to affect, including the components of informed decision making. In the current study, we hypothesized that increased usage of the decision aid would be associated with informed decision making. Secondly, we can explore the usage data to inform the future design of web-based decision aids. Based on the field-testing of Prosdex in its development stage [
Prosdex [
We conducted an observational study of users of a web-based PSA test decision aid (Prosdex
The study took place in the context of a randomized controlled trial of Prosdex [
Participants were asked to access and view an online study version of Prosdex without supervision in their own home or in a setting of their choice. After viewing the website, participants were automatically redirected to an online questionnaire that included questions to assess informed decision making (described below) and asked participants to select from predefined categories their age, ethnicity, marital status, and educational level.
A screenshot of the Prosdex main contents list
A screenshot of a page on Prosdex showing the decision stacker (right) and the navigation bar (left)
A screenshot of the decision summary in use
Prosdex is divided into three key modules: “The PSA Test,” “It’s your Choice,” and “Prostate Cancer” (
The key outcomes derived from the web-logs were:
Total time spent on site
Total number of main content pages accessed on site
Time spent viewing each module (“The PSA Test,”’ “It’s your Choice,” “Prostate Cancer”)
Time spent viewing each section (eg, “What is the PSA test?” and “PSA test results”
Time spent viewing each page (eg, “Why is the PSA test done?”)
Number of pages accessed in each module and section and actual pages accessed
Total number of interactive elements used (ie, videos and animations)
Whether the participant tracked their decision using the “Decision Stacker”
Whether the participant used the integrated print function
The 3 components of informed decision making assessed were: (1) knowledge of PSA testing and prostate cancer-related issues; (2) attitudes toward testing; (3) behavior, for which a proxy measure was the declared intention to have the PSA test.
Prior to the web-logs being collected, the Prosdex website was coded so that each page/file request had a unique code. By doing this, we were able to determine from the web-logs the pages that had been viewed and which multimedia elements had been accessed by each participant.
The web-log files were generated by web-server software that tracked users’ interaction with the server
The component outcomes of informed decision making were measured in an online questionnaire that participants completed immediately after viewing Prosdex
The study received ethical approval from South East Wales Research Ethics Committee, 06/WSE04/13 (REC reference number). All participants gave informed consent before taking part in the trial after the nature and possible consequences of the study were explained.
Frequency distributions were made to assess participants’ use of the site (eg, total number of pages selected and total time spent on each page). Data were examined for outliers. To assess whether there were distinct subgroups of users characterized by the number of pages accessed, we examined the frequency distribution and the Q-Q (Quantile - Quantile) plot.
Correlations between the number of pages accessed and informed decision making outcomes (knowledge, attitude, intention) were assessed at the individual level using Spearman’s rho. For comparisons across subgroups,
There were 129 participants (all men) allocated to the Prosdex intervention group in the associated RCT [
For 8 of the 82 men who successfully viewed Prosdex, we found an excessive amount of time spent viewing particular pages, indicating that these men had likely left the computer on for long periods of time with no interaction (in one case this was 11 days). Viewing times for these participants distorted the mean amount of time spent on each page, section, module, as well as the site as a whole, and were therefore excluded from descriptive analyses relating to time.
Of the 82 men whose web-logs indicated that they had successfully viewed Prosdex, 73 completed the online questionnaire and provided complete outcome data. Data for these 73 men were included in correlation analyses. The 17 men who attempted to access the Prosdex site but were unsuccessful (therefore providing no web-log data) had, however, completed the online questionnaire and provided complete outcome data. These men were compared with the 73 men who successfully accessed the site and who had complete outcome data.
Participants’ characteristics (n = 82)
Characteristic | Percent (Number) | |
|
||
50-59 | 61 (50) | |
60-69 | 22 (18) | |
70 or over | 6 (5) | |
Unknown | 11 (9) | |
|
||
No formal qualifications | 11 (9) | |
School leaver age 16 (formal qualification) | 20 (16) | |
School leaver age 18 (formal qualification) | 8 (7) | |
Clerical or commercial qualification | 6 (5) | |
Graduate or post-graduate qualification | 44 (36) | |
Unknown | 11 (9) | |
|
||
Married or cohabiting | 81 (66) | |
Single or never married | 2 (2) | |
Divorced or separated | 6 (5) | |
Unknown | 11 (9) | |
|
||
White | 88 (72) | |
Mixed race | 1 (1) | |
Unknown | 11 (9) |
A summary of participants’ use of Prosdex: mean time (seconds); mean number of pages viewed; percentage of available pages viewed and percentage of videos/animations viewed among men in the low, intermediate, and high access groups
Time | Access | |||||||
Module/Section | Mean (SD)a Time in |
Range | Total Number of Pages in Module/ |
Mean Number |
Percent of Available Pages Viewed Among Men in Each Group | |||
Low Access |
Intermediate Access (n = 18) | High Access |
||||||
PROSDEX site | 1191 (914) | 75-3672 | 60 | 32 (21) | 19 | 60 | 94 | |
|
412 (329) | 0-1385 | 19 | 11 (6) | 37 | 58 | 91 | |
Introduction | 8 (19) | 0-99 | 1 | 0.5 (0.5) | 24 | 28 | 37 | |
What is the PSA test? | 91 (66) | 0-310 | 3 | 2 (1) | 67 | 85 | 94 | |
PSA test results | 132 (129) | 0-772 | 5 | 3 (2) | 45 | 68 | 92 | |
PSA controversies | 103 (118) | 0-420 | 6 | 3 (2) | 28 | 44 | 93 | |
What is screening? | 76 (87) | 0-327 | 4 | 2 (2) | 22 | 56 | 97 | |
|
218 (263) | 0-996 | 19 | 9 (8) | 5 | 51 | 97 | |
Introduction | 13 (15) | 0-77 | 1 | 1 (0.5) | 54 | 83 | 93 | |
Making the best decisions | 18 (49) | 0 -394 | 2 | 1 (1) | 4 | 58 | 94 | |
Knowing enough to choose | 65 (85) | 0-302 | 5 | 2 (2) | 1 | 70 | 99 | |
Shared decision |
49 (70) | 0-285 | 4 | 2 (2) | 3 | 42 | 98 | |
Choices about the PSA test | 44 (70) | 0-341 | 4 | 2 (2) | 2 | 32 | 99 | |
Screening in other countries | 29 (42) | 0-212 | 3 | 1 (1) | 5 | 39 | 94 | |
|
428 (449) | 0-1824 | 22 | 12 (9) | 15 | 70 | 94 | |
Introduction | 15 (14) | 0-63 | 1 | 1 (0.5) | 43 | 100 | 96 | |
The prostate | 70 (68) | 0-231 | 3 | 2 (1) | 20 | 70 | 95 | |
Understanding the risks | 60 (82) | 0-503 | 4 | 2 (2) | 12 | 76 | 100 | |
Detecting prostate cancer | 114 (169) | 0-733 | 5 | 3 (2) | 17 | 63 | 91 | |
Treating prostate cancer | 115 (180) | 0-883 | 5 | 2 (2) | 10 | 68 | 90 | |
Early treatment | 50 (59) | 0-228 | 4 | 2 (2) | 12 | 65 | 94 |
a based on 74 participants (8 outliers excluded)
b based on 82 participants (8 outliers did not sufficiently distort usage data)
A summary of participants’ use of video clips and animations on Prosdex among men in the low, intermediate, and high access group
Module/Section | Number of Videos |
Percent of Available Videos (Animations) Viewed Among Men in Each Group | ||
Low access (n = 37) | Intermediate Access (n = 18) | High Access (n = 27) | ||
PROSDEX site | 25 (8) | 0.5 (2) | 8 (16) | 8 (21) |
The PSA Test | 9 (2) | 1 (5) | 3 (14) | 2 (30) |
It’s Your Choice | 1 (2) | 0 (0) | 0 (6) | 4 (13) |
Prostate Cancer | 14 (4) | 0 (1) | 12 (22) | 12 (21) |
Participants (n = 74, outliers excluded) spent a mean of 20 (SD 15) minutes on Prosdex. The shortest time spent on Prosdex was 1 minute and the longest time was 61 minutes. Participants spent a mean of 7 (SD 5) minutes on the “The PSA Test” module, a mean of 4 (SD 4) minutes on the “It’s your Choice” module, and a mean of 7 (SD 7) minutes on the “Prostate Cancer” module. The longest time spent on each module was 23 minutes, 17 minutes, and 30 minutes respectively. The relatively large standard deviations obtained highlight the large variability in the time that men spent on the modules.
The participants (n = 82) viewed a mean number of 32 (SD 21) out of a possible 60 main content pages on the Prosdex site with only seven men (8.5%) viewing all 60. A mean of 11 (SD 6) pages out of 19 were viewed from the “PSA Test” module. Eight men (10%) viewed all 19 pages while four men (5%) did not view any pages from this module. A mean number of nine (SD 8) pages out of 19 was viewed from the “It’s your Choice” module, with 17 men (20%) viewing all 19 pages, and 16 men (20%) not accessing any pages. A mean number of 12 (SD 9) pages out of 22 was viewed from the “Prostate Cancer” module. All 22 pages were viewed by 19 men (23%) while 13 men (16%) did not view any pages from this module. Overall, men spent longer and viewed more pages in the “PSA Test” and “Prostate Cancer” modules than they did in the “It’s Your Choice” module.
The interactive features of the site included videos and animations. The mean number of videos viewed was 1 out of 25, and the mean number of animations viewed was 1 out of 8. Of the 82 men, 64 (78%) did not view any video clips, and 44 (54%) did not view any animations. Of the 82 men, 37 (45%) used the “decision stacker,” designed to facilitate involvement in the decision making process. The majority of these men only used the stacker once, and therefore, usage was minimal. Only 3 men out of 82 (4%) used the integrated print functionality, printing only one item of information each.
By examining the frequency distribution and Q-Q (quantile-quantile) plot of the number of pages accessed, we identified a bimodal distribution. The frequency distribution of number of pages accessed suggested two modes, at ≤ 40% and ≥ 80% of the pages, with a relative dearth of intermediate values. Therefore, we defined three groups in terms of the number of pages participants accessed: low access was defined as 0 to 40% (ie, 0 to 24 pages), intermediate access as 41 to 79% (ie, 25 to 47 pages), and high access as 80 to 100% (ie, 48 to 60 pages).
Correlations between usage (measured by number of pages accessed) and the measures of informed decision making were assessed (Spearman rank correlation [ρ], two-tailed). Focusing on number of pages as an indicator of usage allowed for the inclusion of participants with outliers on time data. There was a significant positive correlation between the total number of pages viewed and the overall knowledge score (ρ = 0.69,
A negative correlation was found between the total number of pages viewed and attitude to screening (ρ = -0.52,
Significant differences were demonstrated in knowledge scores, attitudes towards the test, and intention to have the test between men who accessed less than 40% of the website and men who accessed 80 to 100%. Specifically,
On comparing participants with successful and unsuccessful access to Prosdex
This web-log analysis of men using an online decision aid demonstrates a strong correlation between increased usage and increased knowledge, a less favorable attitude to the PSA test, and a congruent reduction in intention to take the test. We have found a significant dose-response relationship whereby informed decision making increases with increased usage of the website.
We found that men who used Prosdex spent a mean time of 20 minutes viewing the website before quitting. Therefore, while increased access is preferable due to the significant dose-response relationship, developers realistically have roughly 20 minutes in which to support an informed decision online. However, it is possible that users would spend even less time viewing such a site outside of the research context, so this time frame may be further limited. As predicted, we also found that men who used Prosdex did not comprise a homogeneous group, and access was not characterized along a continuum. Instead, the bimodal distribution of the data highlighted two distinct groups of users, characterized as low access and high access. The key findings of this study have important consequences for the future design of decision aids, as discussed at the end of the paper.
This is the first study to analyze the web-logs of an online decision aid collected during an online study. We not only successfully identified patterns of usage, but also demonstrated correlations with outcome measures obtained from an associated randomized controlled trial. Specifically, we were able to identify associations between actual usage of a decision aid and components of informed decision making. If decision aids are designed to facilitate informed decision making, it is important that we understand the type of usage that leads to this.
Using a novel method of analyzing website log files, we have identified research limitations that require improvement. First, web-logs were unavailable for 47 of the men; 30 men did not attempt to access the site. This is a relatively high nonparticipation rate that could have an impact on the overall findings. There was evidence of software problems as 17 of the 47 participants were unsuccessful in their attempt to log in to the site. Future web-based research should ensure that software support is made available to users to minimize participant dropout associated with software downloads. The second weakness related to interpretation of the time data and the assumptions made from the web-logs. Although a long time spent on a particular page could indicate that the user took time to read it, it might also mean that the user left the computer and returned later. Given that the web-logs were collected in the home context, where it is likely that there are more interruptions, it is possible that the time data may be overestimated. Therefore, we cannot be sure that time spent on the site meant that the user was viewing the page for that time.
We recommend that interpretations of web-log data should be based primarily on the number of pages viewed rather than time spent. These two measures are highly correlated, but the former is more stable and robust. The correlations between usage and the outcome measures must also be interpreted with caution. While the dose-response correlations suggest that increased usage leads to more informed decision making, it might be the case, for example, that men with more doubts and less positive attitudes toward the test prior to viewing Prosdex were more likely to spend longer viewing the site.
Molenaar’s [
However, we should note that the differences in time spent might also be a function of the population sampled. First, there is some evidence to suggest that women are generally more motivated to be involved in decisions about their health [
The study has expanded on existing research [
There is evidence that Prosdex promotes informed decision making in men, and we highlight factors that should inform the future design of decision aids. First, for the population using Prosdex
We caution against the simple response of developing shorter decision aids, with possibly two versions for high and low users. A more valid response, in our opinion, would be to move away from the traditional linear design of decision aids toward designing tools with a stratum approach (
A diagram representing the stratum approach to developing decision aids
The top layer would present users with the key messages and allow them to access the most important and relevant information with little navigation. This layer would be aimed at the low access group. It would ensure they receive the information they need to make an informed decision in a short time period. For those who wish to dig deeper, the more complex features (eg, interactive elements) and detailed information would be accessible. By adopting this stratum approach, developers could address the needs of different types of users and deliver the relevant and most important information within the relatively limited timeframe. We believe that developers who conduct further research and decision aid development along these lines will be able to support informed decision making for the greatest number of people. Additionally, the “golden” 20-minute time limit, found in the current study, could provide a useful heuristic for other developers. However, we recommend that web-log analysis be an integral part of the development process for online decision aids as well as a tool for posthoc analysis, so that developers can establish the “critical window” that is relevant to their own situation.
The study was funded by Cancer Research, UK. The researchers are entirely independent from the funders (Grant number: C6475/A7490). Cardiff University agreed to act as Sponsor for the above project, as required by the Research Governance Framework for Health and Social Care (Sponsorship reference: SPON304-06). The sponsor acted as employer of members of the research team. The sponsor and funder were not involved in the review and final approval of the manuscript. We are grateful to all the men who participated in this study, and to the general practices involved.
None declared
National Health Service
Prostate Cancer Risk Management Programme
prostate specific antigen
Quantile – Quantile plot
randomized controlled trial
website transaction log file