Autism spectrum disorder (ASD) is a neurodevelopmental condition characterized by difficulties in social communication and interaction, as well as restricted and repetitive behaviors. According to the Report on the Development of Autism Education and Rehabilitation Industry in China III, more than 10 million people in China are affected by autism-related disorders. Approximately 20% are children, and an estimated 200,000 new cases are reported each year. As the etiology remains unclear and no curative pharmacological treatment is available, ASD management relies primarily on long-term rehabilitation and psychological intervention. Families, therefore, face sustained caregiving responsibilities and decision-making pressure [1-3].

Under conditions of prolonged care and high uncertainty, many parents turn to the internet to obtain health information, emotional support, and practical guidance. Prior studies have shown that, in the management of chronic and developmental disorders, access to autism-related resources and care pathways is often uneven [4], and online channels can help compensate for limited access to offline health care resources [5-7]. Online health communities (OHCs) have thus become important platforms for families affected by ASD to seek information and peer support [8,9]. Their 24-hour accessibility and anonymity lower participation barriers, making support more accessible for families with time constraints or those living in remote areas [10-13].

However, the openness of online environments also introduces challenges related to information quality and commercial influence. Inaccurate or promotion-oriented content may interfere with family decision-making [14-17]. Two main types of OHCs can be distinguished. One consists of open, one-to-many peer forums. The other includes structured, one-to-one physician-patient consultation platforms. The former emphasizes experience sharing and emotional support, whereas the latter focuses on professional diagnostic and treatment consultation. Yet systematic comparisons across different platform structures within the same disease context remain limited.

To address this gap, this study examines 2 representative autism-related platforms in China—an open forum platform (eg, Baidu Tieba) and the physician-patient consultation platforms (eg, Chunyu Doctor and Haodf). Using a unified analytical framework, we aim to compare topic structures, poster identity distributions, and help-seeking pathways across platform types.

Previous studies have recognized OHCs as important spaces for information exchange, peer support, and decision-making in the management of chronic and developmental conditions, including ASD. However, most research has focused on single platforms, particularly Western social media sites such as Reddit, Facebook, and Twitter [18,19]. Methodologically, studies have largely relied on manual coding or unsupervised topic modeling. In addition, the commercial posters on an open forum platform and their influence on community structure remain underexamined. In addition, user identity in an open forum platform is an important topic [20,21]. Quantitative analysis of identity stratification and participation hierarchies is also limited.

In recent years, large language models (LLMs) have demonstrated strong performance in biomedical and public health text tasks, including topic classification, summarization, and information extraction [22-25]. OHC texts are highly heterogeneous and context-dependent. Under standardized prompts and in-context examples, LLMs can generate candidate labels and improve annotation efficiency and consistency [26,27]. However, the systematic application of LLMs to cross-platform OHC analysis, particularly with agreement-based evaluation of model scale effects and stability, remains limited.

Specifically, existing research remains limited in 3 key respects. First, although LLMs have increasingly been used for text annotation, most existing frameworks still rely on the direct deployment of a prespecified model, with less attention paid to systematic multimodel benchmarking under standardized conditions. Second, research on autism-related OHCs has focused more on topic identification than on comparing poster identity structures across platform types, particularly the visibility and concentration of commercial actors in an open forum platform. Third, although different online health platforms may support different forms of information seeking, fewer studies have examined cross-platform differences in topic distribution and help-seeking pathways under a unified annotation scheme.

To address these gaps, this study conducts a cross-platform comparison of 2 representative autism-related OHCs in China under a unified annotation framework. Specifically, we examine (1) the effectiveness of the annotation framework and the impact of model scale on annotation performance; (2) the distribution of poster identity structures within an open forum platform, with particular attention to the structural positions of commercial actors; and (3) differences in topic distribution and help-seeking pathways across platform types. By systematically analyzing model performance, identity structures, and topic distributions, this study seeks to provide a transparent, reproducible, and cross-platform analytical framework for applying LLMs in OHC research and to inform platform governance and information support strategies.

This study has both methodological and practical significance. From a practical perspective, differences in participation structures and information pathways across OHCs may shape how caregivers access information, evaluate resources, and make decisions; if such differences are not adequately recognized and addressed, the risk of information asymmetry may be further amplified. From a methodological perspective, by systematically examining model scale effects, identity stratification, and differentiated help-seeking pathways, this study provides an empirical basis for understanding patterns of information demand in China’s digital health landscape. The proposed LLM-assisted annotation framework enables transparent, reproducible, and cross-platform analysis of large-scale online community data, thereby offering a methodological foundation for optimizing platform governance, regulating commercial content, and developing more targeted information support strategies.

The remainder of this study is organized as follows. The Methods section describes the data sources, data processing, LLM-assisted annotation framework, and ethical considerations. The Results section presents the evaluation of model scale effects and the cross-platform comparative analysis results. The Discussion section interprets these findings in light of relevant theories and further outlines their practical implications.

As illustrated in Figure 1, this study developed, validated, and applied an LLM-assisted annotation framework to conduct a cross-platform comparative analysis of autism-related OHCs.

The methodological content comprised 4 major components: data sources, data processing, the LLM-assisted annotation framework, and ethical considerations.

Through this structured framework, the study enabled scalable, large-scale analysis of heterogeneous community text while ensuring cross-platform comparability, annotation consistency, and methodological transparency.

This study drew data from 3 representative Chinese OHCs: Baidu Tieba [28], Chunyu Doctor [29], and Haodf [30].

Baidu Tieba is a large topic-based online forum platform in China that follows an open, one-to-many interaction model. Users create themed subforums (“bars”) to initiate posts and engage in discussions. The “Autism” subforum is publicly accessible and includes caregivers, individuals with autism, and related practitioners, forming a typical peer-support communication space.

Chunyu Doctor and Haodf are established internet-based physician-patient consultation platforms in China. They operate under a structured, one-to-one interaction model. Patients or caregivers can consult licensed physicians online to obtain diagnostic advice, intervention guidance, and follow-up communication. These platforms function within regulated service and content governance frameworks.

Two datasets were assembled as the basis for model validation and substantive analysis: (1) an open forum dataset, derived from the Autism subforum on the Baidu Tieba (hereafter “Baidu Tieba”) open dataset (2017-2019) released by the second “Open Data for Shared Wisdom” National College Open Data Innovation Competition. The dataset included 8097 original posts and 3516 unique posters, representing discussions within an open forum interaction environment (Figure 2); and (2) the physician-patient consultation dataset was collected via Scrapy [31] from publicly available, deidentified consultation records on the Chunyu Doctor (Figure 3) and Haodf (Figure 4) platforms (2017-2019). English translations of the Chinese text shown in Figures 2-4 are provided in Multimedia Appendix 1. The dataset included 3640 physician-patient consultations involving 168 physicians, reflecting help-seeking behavior within structured physician-patient consultation settings.

Before annotation and analysis, all data were preprocessed to ensure cross-platform data quality and structural consistency. We removed noninformative or abnormal entries (eg, system error strings such as “Http Message object has no attribute getheaders” and empty posts) as well as exact or near-duplicate content. In addition, posts identified as “Unclear Expression” were treated as quality-control exclusion records rather than as a substantive topic category for final analysis. For the open forum dataset, we reconstructed the one-to-one correspondence among thread topics, post content, poster IDs, and poster names to standardize analytical units and ensure traceability of poster-level variables. For the physician-patient consultation platform, core patient consultation content was extracted as the structured unit of analysis to enhance cross-platform comparability. The physician-patient dataset included 7687 consultation-demand records derived from 3640 physician-patient consultations involving 168 physicians. After data cleaning and structural reconstruction, the final dataset included 7535 posts from Baidu Tieba and 7687 consultation-demand records from Chunyu Doctor and Haodf.

To enable systematic cross-platform analysis and comparability, we constructed a unified annotation scheme encompassing 3 dimensions: poster identity, discussion topics, and poster participation level.

In this study, interannotator agreement refers to the degree of consistency between the 2 human coders on the labeled subset. Human-LLM agreement refers to the degree of consistency between LLM-generated labels and the human reference labels on the same subset and was used to assess whether model outputs could approximate human annotation reliability. Both types of agreement were evaluated using accuracy, F₁-score, and Cohen κ.

Accuracy was computed as the proportion of correctly classified instances:

where TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives, respectively.

Cohen κ coefficient was used to measure agreement beyond chance:

where P_o denotes the observed agreement, and P_e denotes the expected agreement by chance. According to commonly accepted methodological guidelines, κ values between 0.61 and 0.80 are generally interpreted as indicating substantial agreement, whereas values above 0.80 indicate near-perfect agreement.

Precision and recall were defined as:

The F₁-score was calculated as the harmonic mean of precision and recall:

In Baidu Tieba, poster identities were identified and classified using a coding manual (Multimedia Appendix 2). A double-blind manual annotation procedure was conducted to categorize posters into 4 groups: patients, family members of patients (caregivers), commercial rehabilitation practitioners (commercial posters), and others. Two coders independently assigned labels without access to each other’s judgments. After interannotator agreement was evaluated, discrepancies were resolved through discussion to obtain consensus labels for characterizing the composition of different participant groups.

The thematic classification framework was developed based on the Baidu Tieba dataset. Using inductive open coding, the research team established a unified 2-level taxonomy (primary and secondary topics). Primary topics refer to the broad communicative function of a post, whereas secondary topics refer to more specific thematic categories. Because content on a physician-patient consultation platform is inherently help-seeking in nature, all consultation-demand records were classified within the seeking help primary-topic framework, and only the secondary-topic categories were applied. The secondary help-seeking themes were derived from the open coding results. Without altering the overall structure, the taxonomy was adapted for application to the Chunyu Doctor and Haodf consultation dataset, thereby ensuring cross-platform consistency and comparability. Detailed classification rules are provided in the coding manual.

In addition, to capture participation intensity and role differentiation within Baidu Tieba, we constructed a participation-level variable based on platform-displayed poster rank information. According to predefined thresholds (Table 1), posters were classified into 3 mutually exclusive categories: low-activity posters (fans), moderately active posters (members), and highly active core contributors (leaders). This stratification was used to examine differences in topic engagement and help-seeking behaviors across participation levels.

In terms of model selection, rather than prespecifying a single model for annotation, this study adopted the DeepSeek series of LLMs as the base models. The DeepSeek models demonstrate strong Chinese language understanding and support scalable parameter configurations, making them suitable for domain adaptation and multiconfiguration benchmarking. Under a unified prompting strategy and annotation framework, we conducted benchmark evaluations across models with different parameter sizes to compare their annotation performance under varying conditions. We used standardized Chinese prompt templates with a fixed structure, including role, task, label set, operational definitions, decision and disambiguation rules, auxiliary use of poster identity information, output constraints and format, and input fields (Textbox 1). After this development stage, the prompt templates were fixed within each task and applied consistently across all evaluated models. Performance differences primarily reflect model-related variation under a shared prompting condition rather than iterative prompt engineering during benchmarking. The specific models evaluated are listed in Table 2. All models performed the annotation task in inference mode with a temperature of 0.1. The complete prompts are provided in Multimedia Appendix 3, and the code and related settings are available in a GitHub repository [32] referenced in the Data Availability statement.

The framework consisted of 3 stages. In the first stage, a human reference dataset was constructed, and interannotator agreement was assessed. Approximately 20% of the data from both platforms were sampled and independently annotated by 2 researchers, who subsequently resolved discrepancies through discussion. The selection of a 20% sampling proportion was informed by prior methodological studies in LLM-assisted qualitative content analysis, which suggest that this proportion provides a balance between reliability assessment and annotation efficiency [27].

In the second stage, model benchmarking and selection were conducted. Multiple model scales were evaluated on the human reference subset using accuracy, F₁-score, and Cohen κ to assess human-LLM agreement. Each model was run 3 times, and stability was assessed using the SD. Stability was quantified as the SD of accuracy, F₁-score, and Cohen κ across 3 independent runs; lower SD indicates higher stability. Based on agreement performance, stability across repeated runs, and practical deployment considerations, we compared the relative trade-offs among model scales. For full-dataset annotation in this study, we selected the model with the strongest overall agreement and stability in order to minimize error propagation in the final analytical dataset.

To further characterize model errors, we reviewed records in the sampled subset that were misclassified in at least 2 of the 3 repeated runs and defined them as recurrent errors. On this basis, we focused on the recurrent errors exhibited by the final model selected for full-dataset annotation and further examined the distribution of error types accounting for more than 10% across different platform types and classification tasks.

In the third stage, full-dataset annotation was performed. After verifying that the human-LLM agreement reached an acceptable level, the selected model was applied to annotate the entire dataset. The resulting annotations served as the basis for subsequent statistical analysis and cross-platform comparisons.

The data analyzed in this study included only publicly accessible and anonymized content from online platforms, specifically discussion threads, replies, and question-answer records from Baidu Tieba, Chunyu Doctor, and Haodf. Importantly, no personally identifying information was collected beyond poster aliases, ensuring that the privacy of individuals remained fully protected throughout the study.

To prioritize poster privacy, the research team took deliberate steps to avoid publishing or sharing any raw poster posts or consultation texts in their original form. Instead, the findings presented in this study are derived solely from aggregated and anonymized data. This approach guarantees that no individual poster can be identified from the results, further safeguarding privacy.

Since the research relied exclusively on publicly available and deidentified data, a formal ethics review was deemed unnecessary. However, the research team remained committed to responsible data practices and ensured that all analyses were conducted in compliance with the platforms’ terms of service and applicable data use policies.

After preprocessing and full-scale annotation, a total of 7687 consultation-demand records were included from the physician-patient consultation platforms. In contrast to the open forum platform, discussions on these platforms were more concentrated on clinically oriented concerns.

As shown in Table 8, the most prevalent category was “autism intervention” (2864/7687, 37.26%), followed by “autism resource recommendation (evaluation)” (1358/7687, 17.67%), “autism diagnosis” (1304/7687, 16.96%), and “autism examination” (935/7687, 12.16%). Together, these 4 topics accounted for 84.05% (6461/7687) of all consultation-demand records.

Topics related to “autism symptom” accounted for 6.10% (469/7687), while “other help-seeking” represented 5.28% (406/7687). “Etiology and trigger-related inquiry” comprised 3.53% (271/7687), and “cost and financial burden inquiry” accounted for 1.04% (80/7687).

The physician-patient consultation platforms demonstrated a clinically focused topic distribution, with the majority of consultations concentrated in intervention, diagnosis, resource evaluation, and examination-related categories.

Existing research on autism-related OHCs has primarily focused on Western platforms, particularly Reddit and other social media sites, emphasizing topic analysis and identity construction. In China, related studies have largely concentrated on open forum platforms such as Baidu Tieba. For example, Fong et al [18] analyzed 740,042 Reddit posts spanning the prepandemic to postpandemic periods and used BERTopic to automatically identify core discussion topics, finding that diagnosis management, intervention choices, and daily coping consistently remained high-frequency themes. Similarly, Deng et al [37] applied Latent Dirichlet allocation topic modeling to 12,667 Baidu Tieba posts and likewise identified diagnosis and intervention as central discussion themes. These findings are highly consistent with the results of this study, suggesting that across both English- and Chinese-language platforms, ASD online communities consistently center their discussions on diagnostic decision-making and intervention-related resources.

However, most existing studies focus primarily on topic structure identification, while systematic quantitative analysis of poster identity composition, participation hierarchy, and cross-platform differences remains limited. Skafle et al [38] qualitatively examined how social media shapes identity formation among individuals with autism, but they did not provide structured measurements of the proportional distribution of different identity groups or their content patterns within communities. Likewise, topic modeling approaches based on Latent Dirichlet allocation or BERTopic generally lack the capacity to identify poster roles or underlying interest affiliations.

Building on this foundation, this study extends the analytical scope by systematically identifying and quantifying the identity structure of posters on Baidu Tieba, including patients, caregivers, and commercially affiliated institutions. It further examines differences in topic distribution and participation intensity across identity groups. The results indicate that in the open forum platform, the proportion of commercial posters increases at higher activity levels, and their contributions predominantly consist of advertisements and resource promotion. This extends prior autism-related OHC research from topic identification alone to the joint analysis of topic distribution, identity structure, and participation hierarchy.

In addition, under a unified secondary topic and identity coding framework, this study conducts a cross-platform comparison between the open forum platform and the physician-patient consultation platform. The findings demonstrate that the physician-patient consultation platform is more concentrated in patient and caregiver identities and is more oriented toward professional intervention and management guidance, whereas the open forum platform exhibits greater identity diversity and content stratification. This design makes it possible to compare how different platform types shape help-seeking pathways and information demand patterns within a common analytical framework.

Methodologically, compared with prior studies that have used LLMs for annotation [39-43], this study offers several improvements. First, rather than deploying a single prespecified model, the framework begins with systematic multimodel benchmarking. Using the DeepSeek series as the base models, we evaluated 7B, 14B, 32B, and 671B configurations under the same fixed prompting strategy. This design makes performance differences mainly attributable to model-related variation under shared prompting conditions, while also allowing model scale effects to be observed directly. In particular, the results show that although performance improved with increasing model size, gains in accuracy and agreement became clearly limited beyond mid-sized models, suggesting diminishing marginal returns from further scaling under the current task design. Such explicit assessment of scale effects and cost-performance trade-offs is relatively uncommon in conventional annotation frameworks. Second, the framework is accompanied by full methodological documentation as well as publicly available code and supporting materials, making it easier for other researchers to adopt, test, and extend it in other domains. In this sense, the framework serves not only for topic annotation itself, but also as a transparent and reusable methodological basis for model selection and large-scale cross-platform analysis.

This study conducted a cross-platform comparative analysis of autism online support within the Chinese cultural and health care context, examining both an open forum platform and the physician-patient consultation platform.

Methodologically, we developed and validated an LLM-assisted annotation framework. The framework enabled large-scale automated text analysis while maintaining annotation consistency and semantic stability, as evaluated by accuracy, F₁-score, and Cohen κ. This approach substantially reduced manual coding burden and demonstrated strong stability and scalability, providing a transferable methodological pathway for OHC research.

Using this framework and drawing on relevant theoretical perspectives, we analyzed the identity structure of posters in the open forum platform, with particular attention to the structural position of commercial posters. Commercial posters accounted for a notable proportion and were more prevalent at higher activity levels. Across platform types, both differences and similarities were observed in topic distribution and expressed needs. These patterns reflect sustained demand for resources under conditions of health uncertainty, as well as users’ adaptive information-seeking strategies based on perceived trust and accessibility of different channels.

Several limitations should be noted. First, the data were collected between 2017 and 2019, before the COVID-19 pandemic. OHCs and social media platforms may have evolved substantially since 2019 in terms of platform governance, commercialization patterns, digital health service use, and users’ online help-seeking behaviors. These changes may affect both participation structures and topic distributions. The COVID-19 pandemic may also have influenced users’ help-seeking behaviors, content, and modalities. Therefore, the findings of this study reflect the baseline characteristics of Chinese-language autism-related online communities before the COVID-19 pandemic, and other contexts warrant further investigation. Second, the linguistic training background and potential biases of the DeepSeek model series may have influenced annotation results. Third, full-dataset topic annotation was based on automated model predictions without exhaustive human correction. Therefore, the final annotated dataset likely retains a certain degree of classification error, which may reduce the precision of fine-grained topic classification and cross-platform comparisons.

Future research could incorporate data from multiple time periods to conduct longitudinal comparisons and validate findings using different types of LLMs or domain-specific fine-tuned models to further enhance the accuracy, robustness, and external applicability.

This study developed and validated an LLM-assisted annotation framework to systematically analyze topic structures and participation patterns across 2 types of autism OHCs in China. When metrics were arithmetically averaged across all annotation tasks, the best-performing model achieved near-human agreement (accuracy=79.18%, SD 0.20%; κ=0.736, SD 0.003; F₁-score=0.727, SD 0.006) and showed stable performance across repeated runs.

Both platform types demonstrated sustained high demand for resource evaluation, indicating strong information-seeking needs under conditions of health uncertainty. The open forum platform was caregiver-dominated, with a growing proportion of commercial posters at higher activity levels. In contrast, the physician-patient consultation platform showed a more concentrated, clinically oriented topic distribution.

More broadly, these findings should be understood as reflecting the prepandemic stage of Chinese digital health communication, providing a useful baseline for future longitudinal comparison rather than a direct account of present-day platform dynamics.