We collected questions posted by health consumers from 1st January to 10th August, 2014, with the tags “hypertension (高血压)” or “blood pressure (血压)” under the Q&A (有问必答) section on a Chinese health website with more than 35 million registered users [27] and imported the data into a MySQL (MySQL Community Edition, Oracle) database. The resulting database included 98,032 messages, from which 2000 messages were randomly selected as the sample for analysis.

In this study, “question” is defined as a request that a health consumer has posted on the website on a certain subject to elicit answers from physicians, which was identified based on meaning, not form. We focused on questions related to hypertension (高血压), which was sometimes expressed as “high blood pressure (高血压),” or simply as “high pressure (高压).” Therefore, we manually discarded messages that did not match the definition and that were irrelevant to hypertension but which contained similar words such as “high pressure oxygen (高压氧),” “hyperbaric cabin (高压舱),” “high voltage (高压电),” “pressure cooker (高压锅),” and so on. A new message was randomly selected from the database when an irrelevant message was discarded from the sample, so as to keep the sample size at 2000.

The website provides a template for users to generate questions, which includes three fields: (1) describe your health status (病情描述), (2) treatments or tests in the past (曾经的治疗或检查情况), and (3) what kinds of help do you want (想得到怎样的帮助). This template might lead to confusion in customers regarding how to post their questions. To deal with this case, we developed a rule: if the phrase “what kinds of help do you want (想得到怎样的帮助)” was found in the message, then we would take the sentence after the phrase as the “question.” Otherwise, we would take the whole message as the “question.” By doing this, we collected 2000 questions with an average length of 48 words.

A topic-based classification schema was developed based on TGCQ [14] and the Layered Model of Context for Consumer Health Information Searching (LMCC) [15], and some categories were divided into more specific sub-categories to code the specific information needs. We produced the annotations in 4 rounds (Figure 1). In round 1, one annotator (specialized in medical informatics) annotated all the 2000 sample questions, following the classification schema. Some categories were added to accommodate questions that did not fall into any existing specialty during the process. For purposes of consistency improvement among coders, and the usability of the classification, a list of annotation rules was developed and some question patterns were enumerated for even the smallest category. As a result, the preliminary classification schema for consumer health questions included 101 topic categories and 32 annotation rules.

In round 2, four other annotators (two specialized in medicine and two specialized in informatics) independently annotated 200 questions randomly selected from the sample, using the classification schema. The authors compared the consistency of the five coding results (including the one in the first round) and categorized the 200 questions into three groups: (1) all annotators agreed (n=73), (2) only one disagreed (n=63), and (3) more than one disagreed (n=64). Then we focused on the last group. We addressed ambiguous elements by further specifying annotation rules and improving the descriptions of the question patterns.

In round 3, the revised classification was distributed to the five annotators who independently annotated another 300 questions randomly selected from the remaining sample of 1800 messages. This step was done to measure the interrater reliability of the classification schema as well as to further modify it.

In the last round, each of three annotators independently annotated 500 from the remaining 1500 messages. So each of the 2000 sample questions were annotated by at least two annotators. The authors compared the coding results and the disparities were discussed to achieve an agreement. The codes agreed upon during this step were regarded as the final schema. The number of questions in each category was calculated, and categories in which no questions were filled were deleted (such as physical characteristics of drugs, pharmacodynamics, and mechanism of drug action).

The 2000 questions annotated by the above steps were used to train and test the classifiers for the primary level topics, including Diagnosis, Treatment, Condition Management, Healthy Lifestyle, and Health Provider Choice.

The interrater reliability of the classification schema was evaluated by the kappa statistic, which could correct agreement that occurred by chance. Kappa=(P_o-P_e)/(1-P_e), where P_o is the observed agreement and P_e is the agreement expected by chance [40]. When the number of categories was large, as in this study, P_e would be close to zero, and the kappa value would be close to P_o [14]. Thus, we directly used P_o as the kappa value. The bigger the kappa value, the better the agreement. We assume that when the user asked more than one question, it was acceptable to answer any one of them. Therefore, a liberal reliability criterion was used: a match was recorded if either the main or minor topics assigned by one annotator matched the other’s assignments.

The performance of automatic classification methods was evaluated by precision (p), recall (r) and F₁ score, all of which are commonly used as evaluation metrics for text categorization, and we report the average of each metric. Precision is the number of correctly classified cases divided by the total number of cases classified for the category; recall is the number of correctly classified cases divided by the total number of cases of that class; and F₁ score is the harmonic mean of precision and recall, calculated as F₁ =(2×p×r) / (p + r).

The final classification schema was a four-hierarchical-level of specificity, consisting of 48 quaternary categories (see Multimedia Appendix 1) and 35 annotation rules. The first level included seven areas, namely diagnosis, treatment, condition management, epidemiology, healthy lifestyle, health provider choice, and other. A branching structure of secondary, tertiary, and quaternary levels describes more specific topics of the questions than its upper level. One or more closely related question patterns were listed for each quaternary category. Table 1 shows examples of consumer health questions in Chinese with their pattern and annotated tags on the topics of diagnosis and treatment, respectively.

Table1. An example of consumer health questions in Chinese with their pattern and annotated tags.

This study found that although health consumers would ask numerous health questions about themselves or their families, the general topics of the questions were limited to a small number and each category of the topics had its particular question patterns. The 2000 Chinese consumer health questions were annotated with 2000 major codes and 607 minor codes. The distribution of the sample questions on the primary level category is shown in Table 2. 26.35% of the questions were annotated with more than one topic, which demonstrated that health consumers tend to ask more than one question at a time and the question messages usually belong to multiple topic categories [11] (Multimedia Appendix 1). These findings indicated that the various consumer health questions could be represented by limited topics and keywords, and the task to classify those topics was a multi-label problem [37].

The kappa statistic for the five annotators was 0.63 in the quaternary level of the classification, indicating “substantial” reliability, better than in several similar studies, such as assigning topics to general clinical questions (kappa=0.53) [14]. When only the primary and secondary levels were considered, the kappa value increased to 0.75. When only the seven broad areas in the primary level were considered, the kappa value was 0.82, slightly better than automatically classifying question types for consumer health questions in English conducted by Roberts et al [20].

The Φ-score of each feature was calculated for each binary classifier. We found that their distribution between different types of features differed greatly. The performance of classifiers using features with Φ ≥ avg Φ was not worse than that of those classifiers using all the features in the corresponding types, and some of them were even higher than the latter. Taking the topic of Lifestyle as an example, the average and standard deviation of Φ in each feature type are shown in the third and fourth columns in Table 3. The avg Φ of bag-of-words was 0.0016 with a standard deviation of 0.0067, while the values of keywords with maximum TF were 0.0008 and 0.0009, respectively. The average F₁ score of the classifier was 74.08% when using all the 6154 features in part-of-speech, while the performance increased to 78.84% when just taking the 1490 features with Φ ≥ avg Φ (Figure 2). Similar cases can be seen in feature types of noun rear chunks, interrogative + noun or verb head chunks, verb rear chunks + interrogative, keywords with maximum TF, TF-IDF, and so on. The observations indicated that some of the features in each type either do nothing to the classifiers or have some side effects on them. Rejecting these features could not only save the computing resources so as to increase the efficiency, but also improve the performance of the classifiers.

Therefore, the features with Φ ≥ avg Φ in every feature type were selected as input features for machine-learning, in order to keep all the useful features in different types and to improve the performance of the classifiers. Thus, each classifier received a different feature set, and the number of features within them are showed in the third column in Table 4. For example, words such as “drinking (饮 酒),” “eat a meal (吃 饭),” “breakfast (早餐),” “stay up late (熬夜),” “weight (体重),” “daily life (平时),” “nurse one’s health (调理),” and so on were the effective bag-of-words features for the classifier for Healthy Lifestyle but not effective for the classifier for Diagnosis. On the contrary, words such as “diagnose (诊断),” “judge (判断),” “indicate (提示),” “physical examination (查体),” “cardiac murmur (杂音),” “head rush (脑充血),” “dazed (昏沉沉),” and so on were the effective bag-of-words features for the classifier for Diagnosis but not effective for the classifier for Healthy Lifestyle.

The results were obtained from SVMs in the kernlab package because it performed the best among all the classification algorithms available in the R project. The research findings showed that the feature spaces were reduced from 65.40% to 85.13% by dropping features under the threshold (Table 4). Furthermore, the performance of each classifier also improved significantly (Figure 3). For example, the average F₁ score of the classifier for Diagnosis dramatically increased from 79.84% to 98.55%, while the feature space reduced from 15349 to 5311 (reduced 65.40%). The feature space of the classifier for Treatment reduced greatly (Down from 15349 to 4216, reduced 72.53%), although the improvement of performance was not so obvious (F1 score increased from 75.82% to 76.02%). The results of 10-fold cross validation on each binary classifier affirmed the hypothesis we proposed in the Feature Selection sections.

The performance on the classification of most topics of consumer health questions in Chinese was high. The evaluation metrics (average precision, recall, and F₁ score) of Diagnosis, Condition Management, Healthy Lifestyle, and Health Provider Choice were above 95%, while, the metrics of Treatment and Epidemiology were 76.02% and 71.77%, which were relatively weaker than the others. The standard deviation indicated that the performance of each binary classifier was relatively robust. As of 6-category classification results, we achieved an average precision of 91.41%, recall of 89.62%, and F₁ score of 90.24%.

A classification schema of consumer health questions was built in this study and 2000 hypertension-related consumer health questions in Chinese were manually annotated based on this schema. The research findings demonstrated that health consumers were mainly concerned about what was wrong with their health (or the health of someone they cared about), why it was wrong, how to treat it (including choosing which provider to treat), whether the drugs they used had adverse effects or would do harm in some conditions (eg, pregnancy, breast feeding), whether they could recover from the illness, and what they could do to improve their health in everyday life (mainly diet suggestions).

We explored a machine-learning method to automatically classify these Chinese consumer health questions into one of the six primary level topics, with a novel scoring metric to select the most effective features from the abundant feature types we had explored. The results proved that selecting the features with Φ ≥ avg Φ in each feature type as input features for machine- learning not only increased the efficiency, but also improved the performance of the classifiers successfully. From the 6-category classification results, we achieved an average precision of 91.41%, recall of 89.62%, and F₁ score of 90.24%.

One of the limitations of this work is that the sample questions we used to build the classification schema and to train the automatic classifiers were from only one Chinese health website and defined to be hypertension or blood pressure related. Therefore, the applicability of the classification schema and the validity of the automatic classifiers for the vast majority of questions from other websites and other diseases remain to be tested. Another limitation of this work is that some types of features, such as keywords and bag-of-words, might be correlated. However, our feature selection algorithm did not take the impact of correlation into consideration. We only reached moderate performances on the automatic classifiers for the general topics of Treatment and Epidemiology, whereas the reasons for this remain to be explored in the future.

One of the specialties of this research was that Chinese consumer health questions were chosen as the research object. We built a classification schema of consumer health questions which consisted of 48 quaternary categories and 35 annotation rules, and we annotated 2000 questions in Chinese that were randomly selected from nearly 100 thousand messages about hypertension. Then, by using these annotated questions as the corpus, we explored a machine-learning method to automatically classify Chinese consumer health questions into six general topics to facilitate users’ information needs analysis and answer extraction. We explored an abundant number of feature types and adopted a novel method to select all the effective features with Φ ≥ avg Φ. The results proved that our classification approach was relatively more efficient and effective as compared with similar studies.