This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Social media has become mainstream and a growing number of people use it to share health care-related experiences, for example on health care rating sites. These users’ experiences and ratings on social media seem to be associated with quality of care. Therefore, information shared by citizens on social media could be of additional value for supervising the quality and safety of health care services by regulatory bodies, thereby stimulating participation by consumers.
The objective of the study was to identify the added value of social media for two types of supervision by the Dutch Healthcare Inspectorate (DHI), which is the regulatory body charged with supervising the quality and safety of health care services in the Netherlands. These were (1) supervision in response to incidents reported by individuals, and (2) risk-based supervision.
We performed an exploratory study in cooperation with the DHI and searched different social media sources such as Twitter, Facebook, and healthcare rating sites to find additional information for these incidents and topics, from five different sectors. Supervision experts determined the added value for each individual result found, making use of pre-developed scales.
Searches in social media resulted in relevant information for six of 40 incidents studied and provided relevant additional information in 72 of 116 cases in risk-based supervision of long-term elderly care.
The results showed that social media could be used to include the patient’s perspective in supervision. However, it appeared that the rating site ZorgkaartNederland was the only source that provided information that was of additional value for the DHI, while other sources such as forums and social networks like Twitter and Facebook did not result in additional information. This information could be of importance for health care inspectorates, particularly for its enforcement by risk-based supervision in care of the elderly. Further research is needed to determine the added value for other health care sectors.
Social media has become a mainstream online tool that enables any individual to connect, create, consume, and control content, independently of time and place [
Not surprisingly, the scientific community has become increasingly interested in social media and health care rating sites, and many studies have been conducted on this topic. It is clear that health care rating sites may affect choices that people make about their health care [
Although correlations are not always strong and several questions remain unanswered, social media seems to be a potential source of information about individuals’ perspectives of the quality and safety of care. If so, it could be useful for organizations such as health care inspectorates that want to make use of these individuals’ perspectives to supervise the quality and safety of care.
An example of such an organization is the Dutch Healthcare Inspectorate (DHI), an agency under the Ministry of Health, Welfare and Sport. It is the official regulatory body charged with supervising the quality and safety of health care services, prevention activities, and medical products in the Netherlands. The DHI has organized its supervision in several ways to ensure compliance with (professional) standards and guidelines and to ensure patient safety. The two most important ways are (1) incident-based supervision, and (2) analyses of various types of risk information, also known as risk-based supervision [
To the best of our knowledge, the use of social media in supervising the quality and safety of health care services by health care inspectorates has not been studied intensively. Consequently, we performed an exploratory study, which consisted of two parts. The first part was aimed at identifying added value of social media, for supervision based on incidents reported to the health care inspectorate. We searched social media sources to find additional information about incidents for which insufficient information was available. Health care inspectors determined the potential value of the information found. The second part of this study was focused on the added value of social media for risk-based supervision after assessment of specific risks of health care providers. We searched social media to find additional information to identify whether it could function as an indicator for high-risk providers. Experts from the health care inspectorate determined the potential additional value of the information found.
First, we identified usable, efficient, and reliable social media sources or tools to search social media. Social media sources consisted of individual original sources such as Twitter or Facebook, and tools consisted of standard search engines such as Google or specific social media monitoring tools (SMMTs). SMMTs are tools that allow users to search several social media sources at the same time. Social media sources and tools to search social media were selected by going through three operational steps. First, two researchers with past experience in performing online searches in academic and grey literature, determined the following inclusion criteria for use in this study: (1) related to the Dutch language, (2) ability to search for messages posted up to at least 2 years back, since the DHI takes into account reports about incidents that took place within the past 2 years, (3) accessible, since private social networks are inaccessible, and (4) supporting Boolean search. The latter is a technique that allows the creation of specific search strings making use of operators like “AND”, “NOT”, and “OR” [
Second, we searched for search tools on the Internet and asked experts to share their best options. We also screened an existing online list of SMMTs for relevant tools [
Third and last, test searches were performed with all social media sources and/or tools that corresponded to the criteria. Two researchers individually performed three test searches for randomly selected health care providers. They developed a search strategy to find any information about this health care provider that was available online, such as experiences or ratings. Relevant features and/or findings were added to a table and results and experiences were discussed together. This step not only allowed the researchers to get acquainted with the selected tools, but also to identify any issues regarding reliability, specificity, or usability. For example, if a tool was hosted on a website that was extremely slow and if it was clear that this problem could not be solved, the researchers could decide to exclude the tool. For practical reasons, we aimed at selecting three to five of the most suitable social media sources and/or tools for use in this study. For the second search, we selected social media sources and/or tools based on the results of the first search and removed sources or tools that had not resulted in any new information from the list.
We started by creating searches for 20 reported incidents, which were randomly selected from the DHI database. Random selection was needed to keep a wide focus and to assure that the analysis could apply to different sectors such as elderly care, hospital care, and non-hospital care. Two researchers (TB and LV) individually studied the summaries of the first five reported incidents and created search queries for each incident. Summaries were found suitable for this purpose since they contained all information needed for this study, including a description of the incident, date, name of the person that reported the incident, health care provider, specialty, and organization. For each search strategy, the goal was to find additional information (within social media sources) related to the reporting individual, the disease or treatment, the health care provider, and/or the organization that could be of value for the DHI in determining the relevance of the reported incident. We then tailored the search queries to each social media source or tool, since every source or tool had different options such as rules and use of search operators. To reduce inter-researcher variability, the researchers discussed and improved the search queries and results for the first five searches, until they both agreed on the search query.
Next, one of the two researchers (TB) created search queries for the 15 incidents remaining and performed the searches. Any issues were resolved by consulting another researcher (LV). Furthermore, the second researcher screened search queries, in order to make sure that the basics of each query were identical and operators (eg, AND, NOT) were used in the right way. Print-screens were made for all searches. In the case of many hits, only print-screens of the first two pages with results were saved.
Finally, all results related to the original reported incident were summarized in a text file and shared with DHI inspectors. Per sector, one inspector determined the additional value (eg, information leads to specific actions or may influence a decision) of the information found. They were asked to choose between “0: No additional value of the information found” or “ 1: Information is relevant and contains additional information”. Furthermore, an explanation of each answer was retrieved.
We also performed searches for incidents after purposive sampling: four reported incidents in one of the five major health care sectors to retain a wide focus (hospital care, primary care, mental care, long-term care, and home care). We asked the DHI to select incidents for which it was unclear whether action by the health care inspectorate was necessary. The search queries were developed and results were summarized using the same procedure as the first search. Since the first round taught us that two options for categorization did not provide enough differentiation, we added two categories. Inspectors could determine the relevance of all information found by selecting one of the following options: Relevant, information leads to immediate action (3), Relevant, information leads to further investigations (2), Relevant, information found leads to a signal in the file of the organization (1), or No additional value (0). A description of each option is provided in
First, we asked DHI experts to select high-risk themes for which additional information was preferred. Second, one researcher (TB) developed a search query for each high-risk theme, which was peer-reviewed by a second researcher (LV). After discussion and improvement, a DHI expert working on the selected high-risk themes peer-reviewed the queries, which resulted in the final search strategies. Third, we performed searches for the high-risk themes and presented results in a spreadsheet. In case a result concerned a rating, the ones without a textual description or ratings ≥7 (on a scale from 1 “extremely bad” to 10 “excellent”) were excluded. The latter was done since we reasoned that the useful information for inspectorates would be found in ratings <7. Subsequently, both researchers independently filtered results based on relevance by reading individual results and comparing it to the search query. All results for which no relation to the search could be determined were excluded. For example, if the search included “long-term elderly care”, results about providers working in other sectors such as care delivery by a local GP were excluded. Fourth and last, one expert per theme determined the relevance of each result following the same procedure and the same options as described above and in
Since this study used anonymous data from the public domain and without patient involvement, no ethical approval was needed in the Netherlands. More specifically, all data we used from the rating site ZorgkaartNederland.nl are publicly available. Furthermore, we obtained permission from the DHI to perform the study and acquired the data needed for this study.
We identified 11 possible tools that could be used to find additional information for incident-based supervision. Based on each tool’s features, we selected four tools to perform our searches: Coosto, Google, Addictomatic, and ZorgkaartNederland. Further information about each tool’s features and the selection of tools is provided in
Added value of information for incident-based supervision.
|
Incident-based supervision, Round 1 (n=20) | Incident-based supervision, Round 2 (n=20) |
Immediate action required | 0 | 0 |
Information leads to further research | 0 | 1 |
Information leads to signal | 3 | 2 |
Information found but no added value | 3 | 0 |
No information found | 14 | 17 |
Regarding the searches for risk-based supervision, the DHI selected the high-risk sector long-term elderly care, combined with four specific themes that form major safety risks: hygiene, professional expertise, medication safety, and restriction of freedom. Based on results from the searches for incident-based supervision, Coosto was selected as the preferred tool to perform all four searches. ZorgkaartNederland was selected as the preferred source since the first part taught us that it was the only source that provided us with relevant results.
Added value of information for risk-based supervision.
|
|
Indicator 1: Long-term elderly care and hygiene | Indicator 2: Long-term elderly care and expertise | Indicator 3: Long-term elderly care and medication (safety) | Indicator 4: Long-term elderly care and restriction of freedom |
|
|
n (%) | |||
Hits (n) |
|
79 (100%) | 117 (100%) | 49 (100%) | 11 (100%) |
Results remaining after exclusion of ratings 7 or higher |
|
38/79 (48%) | 90/117 (77%) | 34/49 (69%) | 7/11 (64%) |
Results remaining for assessment by health care inspector after exclusion ratings not related to search query |
|
25/79 (32%) | 68/117 (58%) | 21/49 (43%) | 2/11 (18%) |
|
|||||
|
Immediate action required | 0/25 (0%) | 0/68 (0%) | 0/21 (0%) | 0/2 (0%) |
|
Information leads to further research | 9/25 (36%) | 1/68 (1%) | 1/21 (5%) | 0/2 (0%) |
|
Information leads to signal | 13/25 (52%) | 31/68 (46%) | 15/21 (71%) | 2/2 (100%) |
|
No added value of the information found | 3/25 (12%) | 36/68 (53%) | 5/21 (24%) | 0/2 (0%) |
In this study, we showed that a Dutch health care rating site can be used to identify additional information for supervising quality and safety, especially in long-term elderly care. These findings indicate that social media may enable supervisory bodies to include the patients’ perspective in a more efficient way. Regarding incident-based supervision, social media provided relevant additional information in six of 40 incidents, and for risk-based supervision, social media provided relevant additional information in 72 of 116 cases. Additional information led to a signal or to further research. Although these numbers are promising for supervisory bodies looking for an efficient way of collecting information from the patients’ perspective on health care providers, several things need to be discussed.
In keeping with our approach, we aimed at including all Dutch social media sources. It appeared that the rating site ZorgkaartNederland was the only source that provided information that was of additional value for the DHI. It may seem surprising that only one social media source, among the plethora of other sources such as forums and social networks like Twitter and Facebook, resulted in additional information. Apparently, Dutch people are not likely to share their experiences with health care in combination with the name of their health care provider in tweets or public Facebook posts. Regarding ZorgkaartNederland, it should be considered that it is the only Dutch website with a list of all officially registered health care providers and organizations, which aims to collect as many ratings as possible. Furthermore, ZorgkaartNederland is a non-commercial initiative supported by the Dutch Federation for Patients and Consumers. As a result, people willing to share a rating or their experiences are likely to use ZorgkaartNederland. There are hardly any serious competitors. Therefore, ZorgkaartNederland could be, at least at the present time, a valuable source of information for the DHI. It would be worthwhile to explore the possibility of creating a direct link between relevant social media sources such as ZorgkaartNederland and the DHI (eg, via an open API), allowing the DHI to make a selection of ratings available for use in daily practice. In fact, the DHI is already implementing a system in which information from ZorgkaartNederland is imported into a risk database for its daily supervision.
In this study, we performed searches for the two most used instruments by health care inspectorates: supervision in response to reported incidents and risk analysis. The results show that the searches for risk-based supervision resulted in more relevant information than searches for reported incidents. This can be explained by the search goals for every search. Regarding searches for supervision based on reported incidents, searches were created using the information from the reported incident only, and we aimed to find information that could indicate similar events or structural problems regarding one provider or organization only. Regarding the searches for supervision led by risk analysis, we created searches aimed at finding relevant information about a specific theme in a group of health care providers or organizations within an entire sector. The added value of the information varied strongly: restriction of freedom (100%), hygiene (88%), medication safety (76%), and expertise (47%). This might be explained by the extent in which lay people are able to judge health care situations (eg, it is easy to see that toilets are dirty, but it is harder to determine whether you were given the correct dosage of medication). Searching for additional information for an entire theme with one search (risk analysis) is less time consuming than creating a unique search for every individual case (reported incidents).
In the present study, we performed searches for risks in four sectors in part one and one sector in part two. Therefore, future projects should investigate the generalizability to other health care sectors. This seems to be particularly relevant since the use of rating sites may be different for different demographical groups [
The use of social media for supervision of health care has some limitations, such as the lack of sufficient information, which has been recognized before [
We conclude that social media could be used to include the patient’s perspective in supervision. This information could be of importance for health care inspectorates, particularly for its enforcement by risk-based supervision of elderly care. Further research is needed to determine the added value for other health care sectors.
Scoring options to indicate the value of the information found.
Features of the potential search tools.
Dutch Healthcare Inspectorate
social media monitoring tool
We gratefully acknowledge all DHI representatives who supported us during the various steps of this research project. Their advice and expertise helped us to carry out the two case studies efficiently.
This study was performed in the context of the academic collaborative center on supervision and was funded by ZonMW, the Netherlands Organization for Health Research and Development.
None declared.