To investigate the opinions about the quality factors of hospitals in South Korea, particularly the quality of pediatric hospitals, we collected user-generated content from online communities hosted by Korean Web portals. Preliminary examination showed that online communities specializing in pediatrics served as places for parents to actively discuss (through questions and answers) the quality of pediatric hospitals, sharing their thoughts, ideas, and experiences. Considering that Naver [20] and Daum [21] operate the dominant local online communities for parents and caregivers, we decided to analyze the content of these two portals.

Our aim was to collect user-generated content that could contain users’ experiences with and opinions about pediatric hospitals from online health communities. The structure of the content can be divided into three levels: threads, messages, and sentences. Threads often contain several messages, whereas messages are short and often comprise only a few sentences or sentence fragments. Figure 1 is a snapshot of the hospital recommendation content of online communities, where (2) is the thread title, (4) indicates the details of the thread, and (5) contains the messages of the thread.

Table 1 shows statistics on the content from the selected parents/caregiver online communities from Naver and Daum between April 2007 and May 2013. We obtained 32,422 threads and 173,748 messages. On an average, there were 5.59 pediatric hospital-related discussions per thread among the selected regions’ community members. More specifically, 18.45% out of the total messages contain clue expressions for hospital quality factors.

It is worth noting that over the sample period, there were increases in both Internet usage rate in South Korea and the content in online health communities on the two portal sites. As of 2013, the number of Internet users in South Korea was 40,080,000, (ie, 82.09% of the population) [22]. About 50% of Internet users accessed social media for personal use, and more than 70% of users have used an online community between 2010 and 2012. In the case of user-related content in online health communities, the number of threads has rapidly increased since its launch in 2003 but the number seems to have stabilized since 2010. In Figure 2, we present both these trends; our proposed method will properly address such changes in volume (the number of threads for 2013 was extrapolated using data available up to May 2013).

To efficiently manage social media data in terms of hospital service quality measurement, we defined social media-based hospital service quality factors by referring to previous quality models. However, it is also worth noting that there exist several important research studies [23-29] on service quality models for hospitals. In terms of measuring patient satisfaction, the SERVQUAL model [23], which consists of five dimensions: (1) responsiveness, (2) reliability, (3) assurance, (4) tangibles, and (5) empathy, is the most widely used tool [24]. The SERVQUAL model helps obtain customer ratings of perception and expectation on an ordinal scale. Due to its innate incompleteness, several approaches employ a modified SERVQUAL approach. For example, [25] added two quality dimensions, “caring” and “patient outcomes”, to SERVQUAL, and [26] added “core medical outcomes” and “professionalism/skill/competence”.

In terms of consumer perspectives, five quality factors for health care providers can be defined: (1) warmth, caring, and concern, (2) medical staff, (3) technology and equipment, (4) specialization and scope of services available, and (5) outcome [27].

For use in the hospital accreditation process, the Joint Commission [28] defined nine quality dimensions for hospitals: (1) efficacy, (2) appropriateness, (3) efficiency, (4) respect and caring, (5) safety, (6) continuity, (7) effectiveness, (8) timeliness, and (9) availability. The Joint Commission model is more comprehensive because it encompasses SERVQUAL and the five quality factors for health care providers.

In an effort to develop a valid and reliable instrument for hospital management’s strategic and operational decision-making, the Key Quality Characteristics Assessment for Hospitals model was developed [29], which consists of eight factors: (1) respect and caring, (2) effectiveness and continuity, (3) appropriateness, (4) information, (5) efficiency, (6) meals, (7) first impressions, and (8) staff diversity. The model used inputs from both health care providers and consumers. However, the existing service quality measurement models are not appropriate for interpreting social media data in terms of a hospital’s service quality because users write their experiences without predefined formats or principles.

To this end, we manually listed all the keywords that appeared more than five times in our corpus. Then we selected only terms related to services quality, for example terms regarding the status, impression, and treatments of hospitals. We finally resolved the model for social media–based hospital service quality factors to consist of six quality factors (ie, “service”, “professionalism”, “process”, “environment”, “impression”, and “popularity”), as shown in Figure 3. Each service quality factor has an average of 1000 keywords as features that can be used for analyzing the content of user-written messages. The six service quality factors have their own subcategories and subordinating items, as listed in Table 2. Each quality factor has more than one subcategory, numbered from f1–f29 and e1–e10, according to their functional or emotional characteristic, respectively.

Before analyzing in depth the results of Steps 5 and 6 in various ways, we performed manual evaluations on the extraction and classification results for things such as hospital names, quality factors, and recommendation types. For the evaluation of hospital name extraction, we performed human tagging of a random sample of 9450 messages, which amounted to 5.4% of the total messages. In addition, for the evaluation of quality factor detection and recommendation type classification, we selected the five most frequently occurring hospitals/clinics from the six regions and manually checked their subordinating quality factors and recommendation type to check

if their extractions/classifications were correct. The evaluation results are summarized in Table 3.

We achieved a recall (the proportion of hospital names in the original text that were extracted correctly) of 73% and precision (the proportion of extracted hospital names that were correct) of 84%, giving F1 = 2 × (precision) × (recall)/(precision + recall) = 77.7%. The measure F1 is the harmonic mean of precision and recall. Although accuracy measures of 80%-90% have often been achieved in named entity extraction in English, there are very few cases of such high accuracy in informal texts written in Korean due to various types of acronyms, misspellings, and frequently incorrect spacing between words. These factors are currently performance bottlenecks together with the fact that Korean is an agglutinative language.

Quality factor detection is a kind of keyword matching that checks if a given word exists in one of the categories of the six quality factors. Its performance is slightly over 90% due to errors in word spacing and homonym problems of the Korean language. The currently achieved F1 score is considerably satisfactory. However, the detection performance could be enhanced further by considering compound nouns and more sophisticated processing of erroneously written sentences for clearer word spacing.

Our recommendation classification, a type of sentiment analysis, is a challenging task in that it should track polarity transitions within a message. In spite of handling polarity transitions within a message by referring to [34], the verified classification performance is 77.8% on average. This indicates that sentiment analysis in Korean still has room for further improvement. Nevertheless, our quality factor detection and sentiment analysis provides the option to facilitate the in-depth analysis of messages by region.

By applying the text mining techniques introduced earlier to online health community messages, we obtained a collective intelligence driven by our social media–based hospital service quality factors model. As a first step to investigate the potential of this knowledge, we analyze the occurrence patterns of preferred quality factors among different big cities in South Korea.

Previously, labor-intensive surveys have been used for hospital service quality evaluation. In contrast with such costly manual surveys, we can easily access multidimensional analysis results from the occurrence ratios of keywords included in one of the six quality factor categories. The occurrence ratios of each quality factor signify caregivers’ preferences when they select pediatric hospitals/clinics.

Panel (a) of Figure 5 gives a high-level view of the six key quality factor distributions for the six largest cities (ie, Seoul, Daegu, Busan, Daejeon, Incheon, and Gwangju) in South Korea. The plots in panel (a) of Figure 5 indicate the occupying ratios of each quality factor for each region. Although there are small variations among the different six quality factors, two distinctive factors (popularity and professionalism) occur most frequently.

Such trends are maintained every month, as shown in panel (b) of Figure 5, where the raw frequencies of quality factors are averaged over all regions. In contrast, when we observe regional data, the mean relative frequency ratios of their constituting factors differ somewhat by region and season. Panels (c) and (d) of Figure 5 show the mean relative frequency ratios per month during 2006-2013 for two selected cities, Daejeon and Gwangju, respectively. The numbers in panels (b)-(d) of Figure 5 are given as percentages of occurrence.

We assume that the analysis results driven by social media–based hospital service quality factors form a kind of collective intelligence that represents caregiver beliefs regarding the quality of local hospitals and can be effectively used to discover hospital recommendations within online health communities. Our analysis provides insight into regional and seasonal trends in quality factors that can be obtained from observing pertinent keywords/terms.

With the growing popularity of Web 2.0, a number of caregivers who have children are participating in reviewing, recommending, and exchanging opinions about pediatric hospitals. In this regard, a thorough understanding of the qualitative measures of a hospital could be important for selecting an appropriate pediatric hospital/clinic. Therefore, our contributions are threefold: (1) we define six types of quality factors feasible for social media–based hospital service quality analysis, (2) we propose a text mining approach to extract several hospital quality factors and to classify the messages into recommendation types (recommended or not recommended), and (3) we provide an in-depth analysis that can be used for many shareholders including caregivers, hospital staff, and government officials of health care policy.

Despite the value of the new analysis generated by this work, this study has several limitations in terms of data coverage. Social media, like other tools that assess public opinion, lends itself to selection bias, and it is difficult to assess whether this study population represents a fair sample of the general population [36]. In addition, owing to the limited information (accounting for regional tendencies), more in-depth analysis as to why different fluctuation patterns occur is required. However, it goes beyond the scope of this study, as the reasons are highly correlated with demographic characteristics.

Problems such as the inherent anonymity of users and the inability to authenticate content can be slightly mitigated because, in contrast to Twitter, messages from online health communities contain more detailed information originating from real personal experiences at hospitals. As the coverage of recommendation of pediatric hospitals/clinics in social media is not without its pitfalls, the findings highlight the potential for social media to perpetuate and perhaps even promote sources of information of unknown quality.

This study is based on our firm belief that social media can be used to gauge how patients and caregivers respond to a particular hospital. However, it has been previously found that social media also has risks. Broader scope and easily accessible platforms can lead to the wide dissemination of misinformation. We may overestimate the effect of our quality factor metrics because of the misinterpretation or over-simplification of findings from other socio-scientific studies. Nevertheless, our experimental analysis shows the great potential for social media–based key quality factor analysis for hospitals.

A natural next step would be to validate the results of our analysis by comparing hospital service quality as measured through the Internet user-related data with traditional quality measures as well as patient satisfaction surveys. Previously in the literature, Greaves et al [5] did sentiment analysis on online free-text comments about hospitals and compared the results with the paper-based national inpatient survey results for all 161 acute hospitals with adult services in the United Kingdom. They found that the online comments are moderately associated with the survey and thus online comments can be used to assess patients’ opinion about hospital performance. While they compared hospital ranks, they did not look into the detailed categories of hospital service quality factors that we attempted in this study. In fact, to the best of our knowledge, a patient survey on hospital service quality factors has never been conducted at the national level. Currently in Korea, there exist several government-run websites that provide information on hospitals, for example the Ministry of Health and Welfare [37], Statistical Information Service of Health and Welfare [38], and Health Insurance Review and Assessment Service [31]. However, none of these accreditation bodies provide hospital evaluation information at the patient's satisfaction level, especially in terms of the six service quality factors used in this study. We searched and found academic studies detailing sentiment analysis of social network service mentioning about hospitals in Korea utilizing the SERVEQUAL model [39,40]. However, Song [39] analyzed social media comments about three hospitals only for 3 months. Although Kang and Song [40] compared their sentiment analysis results with survey data and verified that social media results corresponded with survey results, their analysis was confined to the top five hospitals in Korea. In fact, there have been few studies verifying social media content about hospital quality to date, mainly due to the lack of hospital quality surveys conducted at the national level. If a survey of hospital service quality will be conducted at the national level, it would be important and interesting to validate how well our social media-based analysis reflects the actual quality of hospitals.

Although we leave such validation work to be conducted in future studies, real-time information extracted from online sites via our proposed methodology can be still helpful to government authorities as well as information-seeking patients at both local and national levels. Recently, information from social media has been analyzed and used at the national level in the United Kingdom. In particular, the UK government recommends the use of social media and sentiment analysis for rapidly measuring hospital performance and poor care [41] and also runs the website National Health Service (NHS) Choices [42], where people can rate the quality of care received by their family practice.

This work and future studies in this field are critical to inform both research and health care communities about the current status of hospital service quality and to highlight the need for careful, evidence-based analysis of user-preferable quality factors and recommendations based on them. To further explain the possible causes of hospital selection patterns, we plan to analyze the results in conjunction with demographic data.