This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
In health care, the use of physical parameters to monitor physical disease progress is common. In mental health care, the periodic measurement of a client’s functioning during treatment, or routine outcome monitoring, has recently become important. Online delivery of questionnaires has the potential to reduce clinicians’ resistance to the implementation of routine outcome monitoring. Online delivery enables clinicians to receive results on a questionnaire in a graphic directly after data entry. This gives them insight into the progress of a client at a single glance.
To explore clinicians’ perspectives on a routine outcome monitoring procedure where questionnaires and feedback on scores were delivered online. Questionnaires could also be filled out on paper and then entered into the online system by a research assistant.
In 2009 we sent an online survey, consisting of five yes-or-no questions and six open-ended questions, to all clinicians in the 14 mental health care organizations working with the routine outcome monitoring system in the Netherlands. Of the 172 clinicians contacted, 80 (47%) opened the link and 70 of these 80 (88%) clinicians completed the survey.
Clinicians seldom used the graphical feedback from the Web-based system, which indicates that direct feedback on scores did not enhance the implementation of routine outcome monitoring. Integration into the electronic patient record and more training on interpretation and implementation of feedback in daily practice were seen as the primary points for further improvement. It was mainly the availability of a research assistant that made the routine outcome monitoring procedure feasible.
Without a research assistant and training in the interpretation of outcomes, software programs alone cannot ensure effective implementation of monitoring activities in everyday practice.
Regular monitoring of clients with chronic physical diseases using specific outcome measures—for example glucose levels in diabetes or blood pressure in patients with heart failure—is common in health care [
Online delivery of questionnaires may facilitate routine outcome monitoring by (1) reducing the logistic burden, (2) being able to provide feedback directly to clinicians, which is an extension of the paper-and-pencil method, and (3) reducing the amount of missing data, because one may not proceed without answering all questions [
To stimulate the use of routine outcome monitoring in old-age psychiatry in the Netherlands, we introduced the Mental Health Care Monitor Older Adults (MEMO) system. This Web-based system was designed to deliver the benefits mentioned above and contains Web-based versions of questionnaires, automatically calculated sum scores, graphics presenting the symptom course over time, and reminders to clinicians of when follow-up assessments are needed. Depending on their own individual preferences, clinicians could also fill out the questionnaires on paper and a research assistant would then enter the data into the Web-based system. The present study explored the acceptability of using a Web-based routine outcome monitoring system in old-age psychiatry from the perspective of the clinician.
MEMO was started in 2008 for a period of 5 years. During information meetings, managers received extensive information about the procedure of MEMO. After these meetings, 14 of the 41 mental health care organizations throughout the Netherlands applied to participate in MEMO. All clients referred by their general practitioner to these 14 organizations and who progressed to treatment after intake at an outpatient clinic for old-age psychiatry were monitored. The only exclusion criterion was a primary diagnosis of a cognitive disorder. Data were collected at fixed intervals and included diagnostic variables, measures of mental and social functioning, treatment characteristics, and client satisfaction (see
Before the introduction of MEMO, outcomes were not measured in such a structured and uniform way in old-age psychiatry in the Netherlands. To ensure high-quality data collection, funding was provided to organizations to employ a research assistant. We also provided annual training sessions for the use of the Health of the Nation Outcome Scales for older adults (HoNOS 65+) and the Web-based system (see below).
Measurement overview of Mental Health Care Monitor Older Adults (MEMO).
Measure | Rater | Time point | |||
Intake | 4 months | 8 months | 12 months/end |
||
Demographic variables | MHPa | Yes | No | No | No |
DSM-IV-TRb classification | MHP | Yes | No | No | No |
HoNOS 65+c (mental and social functioning) | MHP | Yes | Yes | Yes | Yes |
GDS-15d (severity of depressive symptoms)e | Client | Yes | Yes | Yes | Yes |
Type of treatmentf | MHP | No | Yes | Yes | Yes |
Life events | MHP | No | No | No | Yes |
MHCCTg (client satisfaction) | Client | No | No | No | Yes |
a Mental health care professional.
b Diagnosis according to the
c Health of the Nation Outcome Scales for older adults [
d Geriatric Depression Scale 15-item version [
e To be filled out only by clients with a primary diagnosis of a mood disorder.
f Categories are based on the Dutch reimbursement categorization: psychological therapy, supportive counseling, activating techniques, psychopharmaceutical treatment, relation or system therapy, and electroconvulsive therapy.
g Mental Health Care Client Thermometer [
As clients neither were subjected to interventions nor had to obey behavioral rules for MEMO, the medical ethics committee Medisch-ethische Toetsingscommissie instellingen Geestelijke Gezondheidszorg decided that the Dutch Medical Research Involving Human Subjects Act was not applicable and the study did not require ethics approval. Nevertheless, all clients were informed about the scientific purposes of MEMO, and clients who did not agree to the use of their anonymous data were excluded.
NetQuestionnaires Netherlands (Utrecht, the Netherlands;
To safeguard the data collected, we observed the Data Protection Act and ensured confidentiality through the use of an HTTPS connection, which secured the communication on the Internet, as well as a login procedure and a separate environment for each organization, which prevented the exchange of client data between organizations [
After a client was added to the system, based on the diagnosis, the questionnaires that had to be filled out at baseline were shown and could be filled out immediately online. During follow-up, the system tracked each client and signaled every 4 months, up to 12 months, by email alerts to both clinicians and research assistants when new questionnaires had to be completed. Immediately after data entry, the system generated a graphic visualizing the progress of the client and thereby giving feedback to the clinician. For example, the feedback in
For clinicians who preferred to use paper-and-pencil forms, feedback was delayed until the research assistant had entered the data into the system. The participating organizations asked clients to fill out their questionnaires on paper.
Example of the graphical feedback generated by the Web-based system to show clinicians the progress of their client during treatment. Clarification of Dutch language terms:
To evaluate the MEMO procedure, we emailed an online questionnaire to all 172 clinicians who were authorized to use the Web-based system after 1 year of data collection. They were asked to help improve the procedure of MEMO by filling out the questionnaire, which would take 10 minutes at most. Responses would not be communicated to their organization or manager. To avoid duplicate entries the software did not allow clinicians to open the questionnaire more than once. In addition to general information (organization, sex, age, profession, and work experience), the questionnaire consisted of 5 items that could be answered with either yes or no: (1) Do you think the MEMO procedure is feasible? (2) Do you use the graphical feedback from the system? If yes, the following questions were displayed: (3) Do you think the graphical feedback is relevant to your work? (4) Based on the graphical feedback, did you change the treatment plan for a specific client? (5) Do you think this type of feedback should be continued? Each question was accompanied by an open-ended question that gave them an opportunity to explain their answer. Next, clinicians were asked to grade the graphical feedback (on a scale of 1 to 10). The final question for all clinicians was “Do you have any suggestions for data collection and giving feedback in mental health care for older adults?” The questions were spread over 4 pages: (1) general information, (2) feasibility questions, (3) questions on the graphical feedback, (4) suggestions for data collection. Clinicians were able to review and change their answers in the course of completing the questionnaire. Responses were automatically captured in a database. To improve the response rate, (1) contrary to the yes-or-no questions, the open-ended questions were not mandatory to prevent premature discontinuation, (2) clinicians received a reminder after 1 month, and (3) a gift voucher of €25 was raffled off among the respondents in each organization.
Qualitative data analysis was performed on the answers to the open-ended questions. The contents of the answers were labeled (by MV). These labels were data driven. When coding was complete, the frequencies of the codes were analyzed.
Of the 172 clinicians, 80 (47%) opened the link to the evaluation questionnaire; 70 of these 80 (88%) clinicians completed the questionnaire (n = 41, 59% female; mean age 47.5, range 27–64 years; mean work experience 22, range 1–44 years). Participating clinicians were nurses (n = 38, 54%), psychologists (n = 24, 34%), psychiatrists or geriatricians (n = 5, 7%), and social workers (n = 3, 4%). Characteristics of nonresponders are unknown.
Although 54 of the 70 (77%) clinicians considered the procedures of MEMO to be feasible, only 11 of 70 (16%) actually used the graphical feedback (
Clinicians’ answers to the yes-or-no questions on the feasibility of Mental Health Care Monitor Older Adults (MEMO) and the use of feedback from the Web-based system.
Question | n | Positive answer (yes) | ||
n | % | |||
1 | Procedure MEMO feasible | 70 | 54 | 77% |
2 | Use of graphical feedback | 70 | 11 | 16% |
3 | Graphical feedback relevant | 11 | 10 | 91% |
4 | Change of treatment plan | 11 | 4 | 36% |
5 | Continuation of graphical feedback | 11 | 11 | 100% |
Of 70 clinicians, 21 (30%) gave further suggestions, and 8 of these 21 (38%) suggested that the Web-based system could be improved by incorporating the system within the electronic patient record, through which alerts and graphics are seen directly in the record. The additional login procedure besides the electronic patient record was thought to be the main barrier to the use of the Web-based system. Instead, they filled out the questionnaires on paper. Of the 21 clinicians, 6 (29%) suggested that data collection be extended to other instruments beyond the HoNOS 65+ to obtain more detailed information. Of the 21 clinicians, 5 (24%) suggested that more attention should be paid to the implementation of routine outcome monitoring in everyday practice. Remarkably, only two comments about increased workload were given.
Although the majority of clinicians considered the procedure of MEMO to be feasible, only a few actually used the graphical feedback from the Web-based system. The most common suggestion given to enhance the use of the feedback was to incorporate the Web-based system into the electronic patient record, so that the scores and graphics of their clients would be more easily accessible. This means that previous findings that software programs support direct feedback and monitor activities in clinical practice [
As we did not integrate our Web-based system into the electronic patient record, it is likely that the availability of a research assistant made our routine outcome monitoring procedures feasible. This is in line with the findings of a previous study on younger adults, which showed that routine outcome monitoring is highly feasible when supported by trained assistants [
First, the overall response rate (70/172, 41%) is rather low. Nevertheless, most nonresponders (92/172, 54%) did not open the link and therefore were not aware of the content of the survey. As 88% (70/80) of the clinicians who opened the link also filled out the questionnaire, it seems unlikely that the response rate is related to the content of the questionnaire. Second, as the explanatory items were not mandatory, some clinicians may have too easily skipped some of these questions. Therefore, reasons for not using the feedback or suggestions to improve the procedure of MEMO may have been missed. Third, open-ended questions were labeled by one person. Since the questions were structured and the clinicians’ answers were brief, it is likely that having a second person coding would have led to the same results. Fourth, organizations voluntarily applied for participation in MEMO, which could imply that participating clinicians were more in favor of using a Web-based system than were clinicians in organizations that did not participate. However, this seems unlikely to have been the case, because it was the managers who made the decision to participate in MEMO and not the clinicians themselves.
The availability of a research assistant seems to enhance the acceptability of routine outcome monitoring to clinicians. Integration of routine outcome monitoring software into the electronic patient record and more training on how to implement direct feedback in daily practice would make the Web-based system easier to use for clinicians.
Health of the Nation Outcome Scales for older adults
Mental Health Care Monitor Older Adults
This study was financed by the Netherlands Ministry of Health, Welfare and Sport. We are grateful for this financial assistance in enabling MEMO.
MV drafted the manuscript and helped to design the study. ROV helped to draft the manuscript and supervised the study. AMP designed and supervised the study and the present manuscript.
None declared.