This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Routine measurement of Patient Reported Outcomes (PROs) linked with clinical data across the patient pathway is increasingly important for informing future care planning. The innovative electronic Patient-reported Outcomes from Cancer Survivors (ePOCS) system was developed to integrate PROs, collected online at specified post-diagnostic time-points, with clinical and treatment data in cancer registries.
This study tested the technical and clinical feasibility of ePOCS by running the system with a sample of potentially curable breast, colorectal, and prostate cancer patients in their first 15 months post diagnosis.
Patients completed questionnaires comprising multiple Patient Reported Outcome Measures (PROMs) via ePOCS within 6 months (T1), and at 9 (T2) and 15 (T3) months, post diagnosis. Feasibility outcomes included system informatics performance, patient recruitment, retention, representativeness and questionnaire completion (response rate), patient feedback, and administration burden involved in running the system.
ePOCS ran efficiently with few technical problems. Patient participation was 55.21% (636/1152) overall, although varied by approach mode, and was considerably higher among patients approached face-to-face (61.4%, 490/798) than by telephone (48.8%, 21/43) or letter (41.0%, 125/305). Older and less affluent patients were less likely to join (both
The informatics underlying the ePOCS system demonstrated successful proof-of-concept – the system successfully linked PROs with registry data for 100% of the patients. The majority of patients were keen to engage. Participation rates are likely to improve as the Internet becomes more universally adopted. ePOCS can help overcome the challenges of routinely collecting PROs and linking with clinical data, which is integral for treatment and supportive care planning and for targeting service provision.
In recent decades, the number of people living with and beyond cancer has increased substantially [
Cancer registries and increasingly electronic health records (EHR) [
Online surveys are an obvious way forward (inexpensive and widely used), but the challenges to their use in health care include ensuring the involvement of all patients, obtaining meaningful patient consent, combining PROs with medical information, and maintaining data security. At the Eindhoven cancer registry in the Netherlands, an online system has been successfully established, complementing a mailed paper questionnaire alternative, to collect and link PROs data to registry data from patients identified via the registry post registration: the Patient Reported Outcomes Following Initial treatment and Long-term Evaluation of Survivorship (PROFILES) system [
We have designed and built a potentially UK-scalable system for administering Patient Reported Outcome Measures (PROMs) online at specified post-diagnostic time-points to patients identified and consented in the clinical setting, for linking and storing the collected PROs data with patients’ clinical data in the regional registry, and for semiautomating the associated patient monitoring and correspondence. This is the first such system developed in the United Kingdom and is known as electronic Patient-reported Outcomes from Cancer Survivors (ePOCS). A comprehensive description of the design and development of ePOCS has been published open-access and includes a graphical representation of the system components and data flows as well as details concerning data linkage [
This study aimed to test the technical and clinical feasibility of the novel ePOCS system by running it in two UK National Health Service (NHS) settings over 2 years. Feasibility outcomes included system informatics performance, patient recruitment, retention, representativeness and questionnaire completion (response rate), patient feedback, and administration burden involved in running the system.
Screenshot of the website homepage of the ePOCS system.
Screenshot of the login page of the ePOCS system.
Screenshot of an ePOCS system questionnaire item (item 7 from the 21-item Social Difficulties Inventory).
Screenshot of an ePOCS system questionnaire item (item 3 from the 47-item Quality of Life in Adult Cancer Survivors scale).
Screenshot of part of the ePOCS system Tracker, used to generate and send required daily patient correspondence (due invitations, reminders, etc, appear in red).
Following ethical approval from the NHS Leeds (East) Research Ethics Committee (ref. 10/H1306/65), a prospective, repeated-measures feasibility study was run in the Yorkshire Cancer Network (YCN) Cancer Centre and one YCN Cancer Unit, in the United Kingdom. The comprehensive protocol for the feasibility study has been published open-access [
Adult patients were eligible if diagnosed with potentially curable breast, colorectal, or prostate cancer within the last 6 months and were English literate. The target was to approach all eligible patients during the recruitment period (November 2010 to September 2011).
Eligible patients were identified during discussions in routine multidisciplinary meetings and/or through consultation of medical notes by NHS research nurses and/or oncology clinicians, who then initially approached patients about study participation. Wherever feasible, patients were approached and informed about the study in-person, typically during a routine hospital appointment. Where this was not possible (eg, patient missed their appointment), patients were sent a letter about the study signed by their consultant, or were sometimes telephoned if the patient knew the recruiting research nurse/clinician. Recruiting research nurses and clinicians completed a paper form for each approached patient, on which they recorded, among other things, the mode and location of approach. Participants provided written informed consent, and their consent status was recorded in the EHR [
When joining the study, patients were asked to provide an email address that was used for all study correspondence (eg, invitations to complete questionnaires, reminders). For patients who did not provide an email address, all study correspondence was mailed. Participants were asked to complete questionnaires comprising multiple PROMs within 6 months of diagnosis (T1), and at 9 months (T2) and 15 months (T3) post diagnosis. At T2 and T3, participants had up to 6 weeks to complete the questionnaire. The PROMs chosen for each questionnaire were typical of those likely to be used in future applications of the system and covered various psychosocial and quality-of-life issues ([
A pen-and-paper feedback questionnaire devised by the researchers and a prepaid addressed envelope were mailed to all retained participants post T3. The feedback questionnaire contained a mix of 28 closed and open questions chiefly covering ease of use of various aspects of the system at the different time-points, perceived positive aspects of the system, and suggestions for system improvement. The full feedback questionnaire is reproduced in
Throughout the study, the ePOCS research team diligently maintained a “patient contact” log of all patient-initiated inquiries to the team. For each inquiry (ie, each instance of contact), among other things, the date, mode of communication (eg, email, telephone, letter), and a detailed reason for the contact were recorded on a database. The ePOCS system Tracker automatically generates all the administrative actions due each day (ie, all the patient invitations, reminders, thank yous, etc, that need sending on that date). In order to test that the Tracker was correctly generating all the necessary correspondence, the research team also manually worked out all the administrative actions due each day for 6 months at the study start and when the first participants reached T2 and T3.
The Illness Perception Questionnaire-Revised (IPQ-R) [
The EuroQol-5D, version 2 (EQ-5Dv2) [
The Medical Outcomes Study 36-Item Short-Form Health Survey, version 2 (SF-36v2) [
The Social Difficulties Inventory (SDI-21) [
The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire (QLQ) (EORTC-QLQ) [
The Quality of Life in Adult Cancer Survivors (QLACS) scale [
Psychometric information about the measures and further associated references are provided in the published study protocol [
Technical success and reliability were evaluated by (1) calculating the proportion of patients with successful linkage of ePOCS PROs data and registry data, (2) comparing, and subsequently exploring any discrepancy between, the manually worked out administrative actions due each day (eg, the invitations, reminders required) with those generated automatically by the system Tracker, and (3) examining the number and type of information technology (IT)-related patient inquiries recorded in the “patient contact” log.
Recruitment (ie, consent rate [CR]) was assessed by calculating the proportion of eligible patients recruited relative to all eligible patients approached. Potential differences in CR by mode of approach (face-to-face, letter, telephone), and location (Cancer Centre, Cancer Unit) were also explored. The representativeness of recruited patients was assessed by examining differences in sociodemographic and clinical characteristics between eligible consenting patients and eligible approached patients who did not join the study. The types and frequency of reasons for nonparticipation, recorded by the consenting research nurses and clinicians, were also analyzed.
Retention was assessed by calculating the proportion of consented patients still in study relative to all consented patients, and the representativeness of retained patients was assessed by examining differences in sociodemographic and clinical characteristics between patients who remained in study and patients who withdrew from the study.
Questionnaire completion, or the response rate (RR), was assessed at all 3 time-points in two ways: RR1 is the number of fully and partially completed questionnaires / all eligible patients approached minus those who have died, and RR2 is the number of fully and partially completed questionnaires / all eligible consented patients minus those who have died.
We defined a fully completed questionnaire as one in which all the items have been answered (ie, responded to, as patients could choose to answer that they “prefer not to answer”), and a partially completed questionnaire as one in which less than all of the items have been answered (ie, one or more of the items had no response).
Associations between patient characteristics and questionnaire completion were explored.
For each PROM at each time-point, the proportion of missing data, median completion time, and psychometric reliability were also assessed. Missing data were calculated as the number of “prefer not to answer” item responses within the total PROM dataset (number of items in PROM multiplied by the number of patients who
Participant opinion regarding ePOCS was evaluated from the post-T3 feedback questionnaire. Closed questions were analyzed using proportions. Free-text comments were read by the ePOCS research team (HJ, LA, PW) and following discussion key themes were agreed. The text was imported into QSR NVivo 9 with the main coding undertaken by HJ. Coding consistency, coding saturation, and consensus discussions were undertaken by PW and HJ. Quotes were grouped and examples chosen to best represent the majority opinion for each theme.
Administrative burden was assessed by examining (1) the successful functioning of the Tracker system in automatically generating the administrative actions due each day (eg, the invitations, reminders required) (see feasibility outcome 1), as this minimizes workload, (2) proportion of patients providing an email address for study correspondence, as this too reduces workload compared to printing and mailing study correspondence, and (3) the dates, types, and frequency of patient-initiated inquiries recorded in the "patient contact" log.
Quantitative data were analyzed using IBM-SPSS Statistics-19. Group differences were examined using chi-square tests,
The ePOCS system successfully linked PROs data with clinical registry data for 100% of patients (N=636). Two key problems were identified in the day-to-day running of ePOCS from the comparison of the manually worked out administrative actions due each day (eg, the number of invitations, reminders required and for which particular patients) with those generated automatically by the system Tracker. Some programming updates to the EHR that were not notified in advance to the ePOCS team affected registry data transfers, resulting in void or inaccurate actions generated in the Tracker database. These were resolved quickly. The second problem concerned date of definitive diagnosis. The time-points for questionnaire completion were determined from patients’ date of diagnosis at the time of consent, which was entered into QTool (the questionnaire administration component of the ePOCS system) to guide the timing of questionnaire administration for each patient. However, within the hospital EHR, patient diagnoses can change following diagnostic tests, and when this happened, a new diagnosis date was transferred to the Tracker (the patient monitoring and correspondence component of the ePOCS system), which was different from the original diagnosis date entered into QTool, thus causing QTool-Tracker asynchrony. This resulted in 8 (0.7%) missed invitations of 1227 due. Additional system programming prevented the “original” diagnosis date taken at the time of consent and used in QTool from being overwritten in the Tracker, thus resolving the problem. The majority of IT-related inquiries from participants using the system (n=86) concerned issues with logging on, and notably, confusion between similar-looking (eg, zero/letter o) and case-sensitive letters/numbers in usernames/passwords.
Of 1152 eligible patients approached, 636 consented to participate (55.21%). Patient recruitment is detailed in
Participants (mean 61.3, SD 11.09 years) were significantly younger than declining patients (mean 66.0, SD 12.05 years;
Clinical and sociodemographic characteristics of participants.
Characteristic | Cancer Centre |
Cancer Unit |
Total |
||
|
|||||
|
|
||||
|
Breast | 228 (44.7) | 69 (54.8) | 297 (46.7) | |
Colorectal | 170 (33.3) | 22 (17.5) | 192 (30.2) | ||
Prostate | 112 (22.0) | 35 (27.8) | 147 (23.1) | ||
|
|
||||
|
Men, median age 66 years (range 23-92) | 223 (43.7) | 51 (40.5) | 274 (43.1) | |
Women, median age 58 years (range 24-88) | 287 (56.3) | 75 (59.5) | 362 (56.9) | ||
|
|
||||
|
20% most deprived | 98 (19.2) | 21 (16.8) | 119 (18.7) | |
20-40% most deprived | 98 (19.2) | 25 (20.0) | 123 (19.4) | ||
20% middle deprived | 79 (15.5) | 17 (13.6) | 96 (15.1) | ||
20-40% least deprived | 131 (25.7) | 38 (30.4) | 169 (26.6) | ||
20% least deprived | 104 (20.4) | 24 (19.2) | 128 (20.2) | ||
|
|
||||
|
Yes | 408 (80.0) | 120 (95.2) | 528 (83.0) | |
No | 102 (20.0) | 6 (4.8) | 108 (17.0) | ||
|
|||||
|
|
||||
|
White British | 409 (97.6) | 114 (95.0) | 523 (97.0) | |
White other | 5 (1.2) | 5 (4.2) | 10 (1.9) | ||
British minority ethnic group | 5 (1.2) | 1 (0.8) | 6 (1.1) | ||
|
|
||||
|
Single | 22 (5.2) | 7 (5.8) | 29 (5.4) | |
Married/Co-habiting/Civil partnership | 316 (75.2) | 95 (79.2) | 411 (76.1) | ||
Widowed | 43 (10.2) | 9 (7.5) | 52 (9.6) | ||
Separated/Divorced | 26 (6.2) | 7 (5.8) | 33 (6.1) | ||
Other | 13 (3.1) | 2 (1.7) | 15 (2.8) | ||
|
|
||||
|
No formal qualifications | 109 (26.7) | 15 (12.9) | 124 (23.6) | |
School qualifications | 103 (25.2) | 34 (29.3) | 137 (26.1) | ||
University degree/s | 82 (20.0) | 26 (22.4) | 108 (20.6) | ||
Vocational qualification/s | 52 (12.7) | 13 (11.2) | 65 (12.4) | ||
Other | 63 (15.4) | 28 (24.1) | 91 (17.3) | ||
|
|
||||
|
Full-time employment | 141 (33.6) | 35 (29.2) | 176 (32.6) | |
Part-time employment | 60 (14.3) | 19 (15.8) | 79 (14.6) | ||
Homemaker | 15 (3.6) | 3 (2.5) | 18 (3.3) | ||
Retired | 187 (44.5) | 54 (45.0) | 241 (44.6) | ||
Other | 17 (4.0) | 9 (7.5) | 26 (4.8) |
aValue in parentheses is the number of missing values.
Flow chart of study recruitment.
Almost all participants (95.1%, 605/636) were still enrolled in the study at T3, with 12 deaths and 19 withdrawals accounting for attrition. Participants who withdrew were older (mean 69.7, SD 9.34 years) than those who stayed in study (mean 60.8, SD 10.97 years;
At T1, 85.1% of invited participants fully or partially completed the questionnaire, and at T2 and T3, this value was 70.0% and 66.3% respectively (see
Age (
Of the 344 participants who fully completed the questionnaire at all three time-points, 82 required no reminders (3 missing values) (24.0%, 82/341). Ninety-two required one or more reminders at all three time-points (27.0%, 92/341), and the remaining 167 (49.0%, 167/341) received one or more reminders at one or two time-points. Among participants who fully completed all three questionnaires, age (
The rates of missing data (ie, in fully completed PROMs, so patients choosing “I would prefer not to answer this question”), completion times, and internal reliability for each PROM at each time-point are shown in
Feedback questionnaires were sent to 599 of the 605 T3 participants (2 died and 4 withdrew during the T3 window). Feedback was returned by 71.6% of participants (429/599) with most returns from those who had completed all questionnaires (69.9%, 300/429). Most participants reported that they found it
Most participants responded positively when asked what they liked about the electronic system (item 13) (69.0%, 296/429): “Easy and relaxed, able to complete at your own time, in your own environment” [colorectal cancer patient, male, 62 years old], “It was convenient and easy to use with the option of reviewing answers (given) when required. I liked the option to be able to leave the system but come back to complete later” [prostate cancer patient, male, 65 years old], and “It was easy to use (once I had logged on with help from my husband). I am not very computer literate but could easily use the system” [breast cancer patient, female, 47 years old].
In many cases participants indicated that they preferred ePOCS to a paper system: “It is interactive. I liked receiving an email telling me it was time to complete the questionnaire. I appreciated receiving an email reminding me to complete the questionnaire when I had not done so. I liked the ‘paperless’ system” [colorectal cancer patient, female, 46 years old], and “Very easy to access. Less trouble than using pen and paper and having to post the result” [colorectal cancer patient, male, 73 years old].
However, 27.0% (116/429) of participants did not provide a response to item 13, and a small number of participants made indifferent or negative comments (4.0%:17/429): “I did not have any likes/dislikes about the system, it was like any other questionnaire” [prostate cancer patient, male, 79 years old] and “Nothing at all. I hate computers and prefer a written system like this” [breast cancer patient, female, 56 years old].
For participants who indicated that they would have preferred a paper system (item 18), the reasons mainly concerned lack of computer knowledge, not having to rely on others for help, and finding it easier to get an overview of a whole questionnaire: “No experience of computers and related points” [colorectal cancer patient, female, 88 years old], “I would prefer paper because my daughter has a busy life and can’t always help me and I couldn’t do it myself” [breast cancer patient, female, 51 years old], and “Easier to preview questions and review answers” [colorectal cancer patient, male, 41 years old].
About a third of participants (33.8%, 145/429) commented on how ePOCS might be changed (item 14), with most improvement comments (75.9%, 110/145) concerning the number, type, repetition, and layout of the questions: “Rather too many questions and some feeling of overlap” [breast cancer patient, female, 70 years old].
Although not asked to, some participants volunteered reasons for their participation in the study, and altruism and a sense of belonging to a community were commonly cited: “If it helps in any way to achieve better treatment and after care, I’m all for it” [colorectal cancer patient, male, 76 years old], and “I liked answering the questions because I felt it gave me more of an understanding of my condition and I didn’t feel like it was just me with these symptoms” [breast cancer patient, female, 40 years old].
The Tracker database was easy and quick to use with all required daily correspondence (invitations, reminders, thank-yous) automatically generated and populated with the appropriate patient’s details, ready for sending (after health status was verified). As most participants provided an email address (83.0%, 528/636), sending the required numerous reminders (
Questionnaire completion, reminders sent, and response rates at all time-points (number of items in questionnaire [number varies dependent upon diagnostic group] T1=80, T2=102-108, T3=148-154).
|
Time 1 (T1)a | Time 2 (T2)b | Time 3 (T3)c | ||||
|
|||||||
|
Died | 0 | 5 | 12 | |||
Actively withdrew | 0 | 9 | 19 | ||||
Technical error | 0 | 5 | 3 | ||||
|
636 | 617 | 602 | ||||
|
|
520 | 417 | 394 | |||
|
|
||||||
|
0 | 238 (45.8%) | 209 (50.1%) | 208 (52.8%) | |||
1 | 168 (32.3%) | 119 (28.5%) | 95 (24.1%) | ||||
2 | 80 (15.4%) | 55 (13.2%) | 61 (15.5%) | ||||
3 | 31 (6.0%) | 32 (7.7%) | 28 (7.1%) | ||||
Missing | 3 (0.6%) | 2 (0.5%) | 2 (0.5%) | ||||
|
|
21 | 25 | 20 | |||
|
|
||||||
|
0 | 0 (0%) | 0 (0%) | 0 (0%) | |||
1 | 1 (4.8%) | 7 (28.0%) | 10 (50.0%) | ||||
2 | 6 (28.6%) | 8 (32.0%) | 9 (45.0%) | ||||
3 | 14 (66.7%) | 9 (36.0%) | 1 (5.0%) | ||||
Missing | 0 (0%) | 1 (4.0%) | 0 (0%) | ||||
|
|
95 | 175 | 188 | |||
|
|
||||||
|
0 | 0 (0%) | 0 (0%) | 0 (0%) | |||
1 | 3 (3.2%) | 10 (5.7%) | 5 (2.7%) | ||||
2 | 5 (5.3%) | 5 (2.9%) | 4 (2.1%) | ||||
3 | 87 (91.6%) | 160 (91.4%) | 177 (94.1%) | ||||
Missing | 0 (0%) | 0 (0%) | 2 (1.1%) | ||||
|
|
||||||
|
|
RR1e | 541/1152 (47.0%) | 442/1147 (38.5%) | 414/1140 (36.3%) | ||
|
|
RR2f | 541/636 (85.1%) | 442/631 (70.0%) | 414/624 (66.3%) |
aT1 window – between date of consent and 6 months post diagnosis.
bT2 window – a 6-week window for completion with the midpoint at 9 months post diagnosis.
cT3 window – a 6-week window for completion with the midpoint at 15 months post diagnosis.
dReminders were not sent to those who contacted the ePOCS team and actively withdrew after the invitation/reminder was sent.
eRR1=number of fully and partially completed questionnaires/all eligible patients approached minus those who died.
fRR2=number of fully and partially completed questionnaires/all eligible consented patients minus those who died.
Time to complete, missing data, and psychometric reliability for standard validated ePOCS PROMs (in addition to the standard validated PROMs shown here, participants also completed other questions, eg, about sociodemographic information, employment, and the financial costs of cancer).
PROM (n items) |
|
Fully completed, |
Completion timea, min:sec | Missing datab | Internal reliability, Cronbach αc | ||
Median | Range | Total % | N scales α≥.70 | α range | |||
IPQ-R (66) | T1 | 531 | 12:46 | 04:44–410:32 | 0.80 | 7/7 (100%) | .78–.90 |
EQ-5Dv2 (6) | T1 | 526 | 01:54 | 00:39–29:05 | 0.29 | n/a | n/a |
|
T2 | 426 | 01:30 | 00:31–54:20 | 0.31 | n/a | n/a |
|
T3 | 402 | 01:24 | 00:29–63:42 | 0.46 | n/a | n/a |
SF-36v2 (36) | T2 | 432 | 08:31 | 02:59–39:14 | 0.39 | 8/8 (100%) | .83–.95 |
|
T3 | 400 | 07:44 | 02:35–262:37 | 0.35 | 8/8 (100%) | .85–.95 |
SDI-21 (21) | T2 | 423 | 03:47 | 01:17–29:13 | 3.15 | 4/4 (100%) | .72–.89 |
EORTC-QLQ-BR23 (23) | T2 | 196 | 03:56 | 01:24–32:57 | 2.13 | 4/5 (80%) | .69–.92 |
|
T3 | 183 | 03:36 | 01:14–40:04 | 2.68 | 4/5 (80%) | .64–.92 |
EORTC-QLQ-CR29 (29) | T2 | 117 | 06:35 | 03:07–712:33 | 1.38 | 2/5 (40%) | .45–.90 |
|
T3 | 104 | 05:45 | 02:36–26:12 | 1.39 | 3/5 (60%) | .69–.83 |
EORTC-QLQ-PR25 (25) | T2 | 117 | 04:27 | 02:10–28:41 | 0.68 | 3/5 (60%) | .41–.82 |
|
T3 | 111 | 04:13 | 02:01–44:41 | 1.23 | 2/5 (40%) | .43–.80 |
QLACS (47) | T3 | 407 | 09:56 | 03:36–288:52 | 2.25 | 12/12 (100%) | .75–.94 |
aCompletion time descriptive statistics are based on participants who started and completed a PROM on the same calendar day.
bMissing data (ie, patients’ choosing to respond “I would prefer not to answer this question”) per PROM is based on the number of patients who fully completed that PROM.
cSpearman-Brown reliability coefficient for 2-item subscales.
Reasons for participant inquiries to the ePOCS team over the study period.
This study has tested and demonstrated the technical and clinical feasibility of an innovative electronic system for collecting PROs online and linking them with clinical registry data. In general, the results showed that the system informatics performed successfully, demonstrated encouraging rates of patient recruitment, retention, and questionnaire completion, revealed predominately positive feedback from patients, and showed a low administration burden involved in running the system. However, patients who joined and stayed in the study were not wholly representative of all invited/recruited patients.
The informatics underlying the ePOCS system demonstrated successful proof-of-concept. The system successfully linked PROs with registry data for 100% of patients. The set-up work involved in establishing the linkage capacity was undertaken over several weeks during system building by a member of the registry IT team (but took approximately 2 working days compressed). When the system is running, as in the current study, an output of linked data can be instantaneously “pulled off” the system. Importantly, the labor involved in the initial linkage work is not impacted by the number of patients in the system and would therefore remain modest even if use of the system were scaled up considerably. The system also worked efficiently with relatively few day-to-day running problems. Speedy resolution of problems was possible due to the close working relationship developed between all parties: the ePOCS research team, the design teams of the QTool and Tracker components of the ePOCS system, and the EHR and registry IT teams. The modest number of IT-related inquiries from patients mainly concerned confusion with some letters/numbers in usernames and passwords and was easily resolved; importantly, this can be avoided when running the system in the future by more careful design of the composition of usernames and passwords.
Over half the patients invited joined up to use the ePOCS system. The participation rate of 55.21% (636/1152) is encouraging when compared with response rates for mailed cancer patient surveys (31%-64%) [
Patients who declined to join the feasibility study were older and less affluent (and patients who withdrew from the study were also older) and most commonly cited IT reasons for nonparticipation. Older age and socioeconomic deprivation are characteristics commonly associated with study nonparticipation [
Almost all invited participants fully or partially completed all the questionnaires (89.3%), despite the relatively large number of items at each time-point (ie, between 80 and 154). Deprivation was negatively associated with questionnaire completion, and IT issues may also have influenced this, although the low number of feedback questionnaires returned from non/partial-responders makes this impossible to determine. Receiving reminders led to improved completion of questionnaires, although the impact diminished somewhat at T2 and T3. Among participants who fully completed all three questionnaires, patients who did not provide an email address were more likely to need a reminder. Participants who receive invitations in the mail have to make a special effort to go to a computer, boot up, and find the ePOCS website prior to logging on to complete questionnaires, whereas participants who receive an email invitation are already online and can simply use the weblink in the email to log on and complete questionnaires. The time taken to complete individual PROMs varied considerably (eg, 6-item EQ-5Dv2 at T1 ranged from 00:39 to 29:05 minutes:seconds), suggesting that some participants broke off partway through to complete other activities. As has been observed elsewhere (eg, [
Participant feedback was generally positive and endorsed the ePOCS approach. Over 90% of participants found it
The ePOCS system was not administratively onerous to run. The Tracker successfully automatically generated all the administrative actions due each day (eg, the invitations, reminders required). Checking the Tracker daily and sending all the due correspondence, which was mostly via email, took a member of the research team between just 15 and 30 minutes. In the ePOCS study, 2 members of the research team shared this task (to allow for sickness/holiday absence). The number of participant inquiries received over the course of the study averaged a modest 11 per month, even with 636 patients in study, indicating that the time required to provide support to patients using the system is not burdensome. This is similar to the experience of the PROFILES system in which only around 2% of patients contact the PROFILES helpdesk [
ePOCS provides an infrastructure to routinely collect and link PROs to clinical and cancer registration data. Preparatory ePOCS work indicated patients’ disinclination to consent when critical treatment decisions are being considered (not tested in this study) or close to the time of diagnosis [
IT is playing an increasing role in the delivery of high-quality cancer care [
To our knowledge, this is the first study to test and report on the feasibility of a system like ePOCS. Strengths of the study include multisite recruitment, the large number of patients invited to participate (over 1000), and the examination of patients’ use of the system on multiple, longitudinal occasions. We also consider an important strength to be the focus on technical and administrative feasibility which, alongside patient consent and response rates, are of key importance for those seeking to use and run a system like ePOCS.
The principal limitation of the study is that we were unable to test the system in a more authentic context. Unavoidably, in order to obtain patients’ informed consent, the current study was presented to patients as being about the ePOCS system. However, future studies that use the ePOCS system will be presented to patients with emphasis on the PROs to be collected and analyzed, and the ePOCS system will be mentioned only secondarily as the data collection tool. It will be important to examine patient recruitment, retention, and response rates in future PROs studies that simply use the ePOCS system rather than aim to test it. The second notable limitation concerns the minimal feedback obtained from patients who declined participation and who consented but did not complete questionnaires. In health information technology (HIT) research generally there is a lack of studies exploring the perspectives and experiences of patients who choose not to engage with HIT and/or who withdraw participation. We are currently planning a study with such patients, aimed at understanding and overcoming modifiable barriers to patients’ acceptance and use of HIT.
Routine collection of PROs is integral for planning patient-centred, compassionate, and personalized health care. This study has shown that the ePOCS system performs well, is accepted by the majority of patients, and is an efficient means to collect and collate PROs data at scale. Although IT usage is not currently universal, every year more patients will become Internet users. Until then, and for those who choose not to engage with e-systems, conventional alternatives will also have to be offered. This should not hold back plans for introducing systems such as ePOCS, as the majority of patients are keen to engage and provide information they believe will help future cancer patients.
Copy of the feedback questionnaire used to obtain patient opinion on the ePOCS system.
consent rate
Electronic Health Record
electronic Patient-reported Outcomes from Cancer Survivors (system)
health information technology
Index of Multiple Deprivation
information technology
National Health Service
Patient Reported Outcomes Following Initial treatment and Long-term Evaluation of Survivorship (system)
patient reported outcomes
patient reported outcome measures
response rate
Yorkshire Cancer Network
This study was funded by Macmillan Cancer Support, with research nurses supported by the West Yorkshire Comprehensive Local Research Network, and sponsored by the University of Leeds. We thank the patients who participated in this study, the ePOCS research nurses, the study steering group, Johnathan Joffe for clinical support, and the Psychosocial Oncology and Clinical Practice Research Group.
None declared.