This study used publicly available, deidentified Reddit posts and was determined to be exempt from full IRB review at Boğaziçi University (exemption/application number 2025-13).

As this study involved the secondary analysis of publicly available data, the IRB waived the requirement for informed consent.

All posts were analyzed in aggregate, and no usernames or other identifying information was retained. The data were stored on secure computers. To prevent search-based reidentification, we paraphrased all illustrative post excerpts in the main manuscript and Multimedia Appendix 1 while preserving the linguistic features that each excerpt was selected to illustrate. Local Interpretable Model-Agnostic Explanations (LIME) attributions and all quantitative analyses were generated using the original, unparaphrased text.

No compensation was provided because all data were obtained from publicly available sources.

No images of individual users were included.

We randomly selected 30 posts from each of 3 control subreddits (r/Showerthoughts, r/DeepThoughts, and r/ImposterSyndrome), with all identifying metadata removed. Both annotators independently coded these 90 blinded posts.

For the r/Narcissism subreddit, we assessed both intrarater reliability and interrater agreement. Each annotator independently completed 30 blinded annotations: 15 reannotations of posts they had previously labeled to assess intrarater reliability and 15 annotations of posts originally labeled by the other annotator to assess interrater agreement.

Labels L7 (expresses markers) and L8 (claims traits) were not assigned solely based on users’ self-reports. Instead, 2 experienced psychiatrists (with 23 and 20 years of clinical practice) evaluated each post for clinically meaningful markers consistent with the DSM-5 diagnostic criteria, including grandiosity, need for admiration, lack of empathy, entitlement, and interpersonal exploitation. This clinical validation ensured that our positive class (n=185) comprised posts containing psychiatrist-validated NPD markers across all categories, distinguishing our approach from studies that rely solely on self-reported data.

Using multiple, fine-grained labels allowed us to capture different aspects of the problem. Specifically, the labeling scheme makes it possible to:

We developed a custom software application [43] to support multilabel annotation. Figure 1 shows the annotation interface.

We designed 3 experimental setups (A, B, and C). The experimental design for setup A is presented in Table 2.

For clarity, we define posts exhibiting linguistic markers of pathological grandiosity consistent with the DSM-5 diagnostic criteria for NPD as NPD-trait-positive and posts without such markers as trait-negative control posts (see the “Limitations” section for a discussion of construct interpretation). Depending on the experimental setup, the specific label assignments to the NPD-trait-positive and trait-negative classes varied.

We aimed to predict whether a post involves an individual (not necessarily the author) exhibiting NPD traits.

We replicated the experiments from setup A with a revised problem definition (Table 4), resulting in a different label-to-class mapping. Whereas setup A aimed to predict whether the author of a post (1) displayed NPD traits (L7), (2) claimed to exhibit NPD traits (L8), or (3) had a clinical diagnosis of NPD (L9), setup B aimed to predict whether the post involved an individual with NPD traits, regardless of whether that individual was the author or someone else mentioned in the post. Accordingly, we converted the annotations of posts from the r/Narcissism subreddit into binary class labels by including the additional labels L2, L4, and L6 in the positive class. The label-to-class mapping used in setup B is presented in Table 5.

We compared 2 feature-engineering pipelines. The frequency and sentiment analysis (FSA) approach combined classical text features (TF-IDF n-grams and sentiment scores generated using TextBlob), extracted with scikit-learn and the Natural Language Toolkit. The second pipeline generated dense embeddings from transformer models, including all-MiniLM-L6-v2 via sentence-transformers v2.2.2 and OpenAI’s text-embedding-3-large. For each pipeline, we trained 4 classifiers: LR as an interpretable baseline, SVM for high-dimensional margin-based separation, RF for capturing nonlinear interactions, and XGB for enhanced predictive performance. These models were selected to represent both linear and ensemble approaches.

All models were evaluated using an 80/20 stratified train-test split (random_state=42), with hyperparameters optimized through 5-fold cross-validation within the 80% training set. We report F₁-score (the harmonic mean of precision and recall), ROC-AUC, and accuracy on the held-out 20% test set. Binary predictions were generated using the default 0.5 decision threshold, and the test set was not used to optimize the operating point. For the multiclass experiment (C1), we report weighted F₁-score (scikit-learn average=“weighted”), in which each class’s F₁-score is weighted according to its support. To ensure full reproducibility, all stochastic steps, including data splitting, SMOTE-Tomek balancing, feature selection, and model training, used fixed random seeds. Environment details, exact code snippets, and package versions are provided in Multimedia Appendix 2.

Given the relatively small size of our dataset, we selected 5-fold cross-validation as a balance between reliable performance estimation and reasonable computational cost. Using 10-fold cross-validation would have reduced each validation fold to only 10% of the training data, potentially increasing variance in performance estimates. Moreover, our preliminary experiments showed negligible improvements in mean F₁-score compared with 5-fold cross-validation. The remaining 20% of the data was held out entirely (not used during cross-validation) and evaluated only once after completion of all model tuning to provide an unbiased assessment on the test set.

We adhered to the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [51], documenting each stage of our workflow, from data sourcing and outcome definition to feature engineering, hyperparameter tuning, internal validation, and performance reporting. This ensured transparency and reproducibility (see Multimedia Appendix 3 for the completed checklist).

Echoing De Choudhury et al’s [4] depression prediction work [4] and Kosinski et al’s [1] trait forecasting findings [1], our results indicate that online language contains strong signals of NPD-trait markers. Embedding-based models reliably distinguished psychiatrist-labeled NPD-trait posts from control posts across diverse experimental configurations (discussed below and in the “Limitations” section). The interpretability analysis of the FSA + LR baseline further showed that the classifier’s feature profile varied systematically across experimental designs.

After keyword removal, the FSA + LR model relied on features with plausible clinical relevance rather than explicit diagnostic labels. Reflective terms (eg, “aware”) carried positive coefficients for the NPD-positive class, consistent with self-referential reflective language in r/Narcissism (eg, a poster acknowledging awareness of these patterns in themselves). In the A5 no-keyword model, “aware” and “hurt” appeared among the positive predictors, consistent with reflective and distress-related language in the support community. As r/Narcissism is a self-selected community of users engaging with NPD-related content, these within-community word-frequency differences reflect discourse patterns in the support community itself. Whether they represent NPD-specific distress or broader emotional-distress vocabulary is addressed in the section “What Do These Automated Classifiers Actually Learn?”.

To assess the practical reliability of these models, we evaluated them under a range of conditions, from clean comparisons to more challenging, noisy scenarios.

We observed consistently high performance across metrics such as F₁-score and accuracy, particularly in experiments A1-A3 and B1-B3. These experiments focused on detecting authors exhibiting NPD traits or posts involving individuals with NPD-consistent trait markers. The availability of contextual cues likely contributed to the superior performance observed in these experiments compared with the remaining experiments.

The performance decreases observed in experiments A4, A5, B4, and B5 were expected and can be attributed to the inclusion of noisy, negatively labeled data from the r/Narcissism subreddit. Many of these negatively labeled posts were related to pathological grandiosity markers or discussed individuals exhibiting NPD traits. As posts from both classes shared terminology related to narcissism, distinguishing between the classes became more challenging.

Experiment A5 still achieved relatively strong performance because author-specific contextual cues helped the model distinguish posts written by individuals exhibiting NPD traits. However, experiment B5 was more negatively affected because it lacked direct cues from the author. The model instead had to interpret indirect references, third-party mentions, and more complex contextual information.

This performance pattern also raises a construct validity concern specific to setup B. As the positive class includes 6 label types (L2, L4, L6, and L7-L9; see the “Methods” section, setup B), the training signal combines linguistically distinct populations, ranging from self-aggrandizing or trait-disclosing language to distress-oriented accounts of others’ behaviors. A classifier trained on such heterogeneous signals may learn a composite discourse pattern rather than specific psychological trait markers. The lower F₁-score in B5 (0.744) compared with A5 (0.892) is consistent with this interpretation: when both classes share r/Narcissism-related vocabulary and the positive class contains heterogeneous psychological perspectives, classification becomes more challenging. Therefore, a positive classification in setup B indicates that a person exhibiting NPD traits is involved in the post (as the author or as the subject of discussion), rather than indicating that the author themselves exhibits specific trait markers.

Multiclass classification in experiment C1 aimed to differentiate among 3 related categories and therefore involved greater complexity than the binary classification tasks. As expected, C1 performed below the binary experiments. Nonetheless, it achieved a reasonable weighted F₁-score of 0.685 (bert-base-uncased) for the 3-way classification task (irrelevant vs NPD-related vs an individual exhibiting NPD traits).

A crucial follow-up question is what these automated classifiers actually learn: do they simply identify keywords, or do they capture broader linguistic patterns?

We used identical hyperparameter ranges across models, limiting variation attributable to model tuning and allowing a more direct comparison of feature representations. Embedding models consistently outperformed the FSA approach, likely because of their richer representational capacity and ability to capture contextual linguistic patterns beyond surface-level features (see Table 10).

They demonstrated strong performance, with F₁-scores ranging from 0.744 to 1.0. However, further validation across diverse datasets is required to confirm these findings. All embedding models performed similarly; the 2 OpenAI models showed the best overall performance, with mean F₁-scores within 0.01 of each other, and text-embedding-ada-002 performing marginally better. It ranked first or tied for first in 9 of 14 experiments, whereas text-embedding-3-large ranked first or tied for first in 6 of 14 experiments. These counts overlap because the 2 models tied for first place in several experiments. The mean F₁-scores across experiments were 0.907 (SD 0.101) for text-embedding-ada-002 and 0.897 (SD 0.106) for text-embedding-3-large. The complete results are provided in Sheet S2 in Multimedia Appendix 6.

We attribute the perfect precision or recall scores (1.0) achieved in a few experiments to evaluation on a small test set. Additionally, the clear separation of linguistic features between r/Narcissism and r/ImposterSyndrome enabled the models to achieve perfect F₁-scores (1.0).

To assess whether the models captured linguistic patterns beyond explicit diagnostic labels, we retrained them after removing keyword families at 2 levels of aggressiveness using 2 complementary analyses. The first involved removing 4 keyword families (narcissism, npd, disorder, and personality) and evaluating the models using the same grid-search pipeline as in the main analysis across all feature models. The embedding models remained largely unaffected, with a mean decrease of only 0.37%, compared with 3.25% for the FSA models (Multimedia Appendix 6).

The second analysis involved more aggressive removal of 6 keyword families by additionally excluding the keyword families “self-absorbed” and “diagnose,” which are broader and not exclusively specific to NPD. This analysis was conducted separately using the FSA + LR interpretability pipeline. The FSA + LR pipeline showed a mean relative F₁-score decrease of 3.6% across 13 configurations, with individual experiments ranging from a 15.0% decrease to an 8.4% increase. Two configurations showed slight improvements after keyword removal (B5: +8.4%; C3: +1.8%). These values represent the differences between the keyword-removed and original-text versions of the FSA + LR pipeline, not the changes in the best-model F₁-scores reported in Table 10. The embedding models again showed minimal changes, with mean relative F₁-score decreases of 1.9% for all-MiniLM-L6-v2 and 0.8% for text-embedding-3-large (per-experiment results are provided in Section S8 in Multimedia Appendix 1). The FSA and embedding results are not directly comparable because the FSA decrease is based on a single classifier, whereas the embedding decreases are averaged across 3 classifiers (LR, SVM, and RF). Therefore, this comparison should be considered descriptive rather than a like-for-like evaluation.

No experiment showed a substantial relative F₁-score decline after keyword removal, indicating that the models retained predictive signal beyond explicit diagnostic labels. Per-experiment significance tests were underpowered because of the small test-set sizes (n=50-204); therefore, we based this robustness claim on the consistent pattern observed across experiments rather than on any individual test (see the “Statistical Power for Per-Experiment Comparisons” section). LR coefficient analysis of the keyword-excluded models showed that, in the absence of diagnostic terms, the highest-ranked features shifted toward broader emotional, cognitive, and relational cues (see Sections S4-S6 in Multimedia Appendix 1). However, LIME-based local explanations were less informative than expected because of a structural sparsity artifact in the TF-IDF features (see Section S7 in Multimedia Appendix 1).

Coefficient analysis of the FSA + LR pipeline provides additional evidence that the model captures signal beyond diagnostic keywords. In the no-keyword A2 model, the top positive coefficients included mental, behavior, important, issues, and society—terms reflecting general psychological and social vocabulary rather than diagnostic labels (although A2’s near-perfect classification may partly reflect topical separability between r/Narcissism and r/ImposterSyndrome). In the within-subreddit A5 model, where both classes shared community vocabulary, the top coefficients after keyword removal included think, low, second, aware, and look, suggesting reliance on common cognitive and observational terms. Across the deep-tier experiments, keyword removal shifted the composition of the top 20 coefficients from 5%-40% diagnostic terms to 0%, with broader emotional, cognitive, relational, and behavioral cues filling the vacated ranks. This shift is partly mechanical because removing keywords from a fixed top-20 list necessarily promotes the next-highest-ranked features. Thus, the analysis shows what the model relies on when keywords are unavailable, rather than demonstrating that nonkeyword features were always the primary drivers. A residual surrogate term (covert, shorthand for covert narcissist) remained highly ranked in the no-keyword models. A third ablation tier that additionally removed this term along with 2 other surrogate terms (tendencies and traits) produced negligible further changes in F₁-score (see the “Statistical Power for Per-Experiment Comparisons” section). The keyword removal list did not exhaust NPD community jargon (eg, supply, love bombing, and gray rock). However, these terms did not appear among the highest-ranked coefficients in any deep-tier experiment (see Sections S4.4, S5.3, and S6.5 in Multimedia Appendix 1), indicating that they did not anchor the FSA + LR interpretability analysis, although other unlisted community terms may have contributed residual signal (see Section S9 in Multimedia Appendix 1).

These coefficient profiles reveal a design-dependent pattern in what the classifier relies on. In the cross-subreddit experiments (A1 and A2), where 8 of the top 20 positive coefficients in the original model were diagnostic keywords, keyword removal eliminated these terms and shifted the profile toward a mix of behavioral, cognitive, and relational cues (eg, mental, behavior, and abused in A2; told, self, exhausting, and therapy in A1). In A5, where both classes shared r/Narcissism vocabulary, NPD-related terms largely canceled out as a discriminative signal: even before keyword removal, only 1 of the top 20 positive coefficients was a diagnostic keyword. The features that remained discriminative in A5 were terms associated with distress, self-awareness, and interpersonal difficulty (eg, afraid, hurt, and aware). These represent the residual signal once shared NPD vocabulary is neutralized. This design-dependent variation indicates that the classifier adapts its feature reliance to the available vocabulary contrast rather than depending on a fixed set of keywords. Whether the residual signal in A5 reflects NPD-specific distress patterns or broader emotional distress within the r/Narcissism community remains an empirical question that the current design cannot resolve. A related sampling caveat applies to "aware" specifically, which may reflect self-recognition among users who already identify with NPD-consistent traits rather than NPD-trait expression more generally (see the “Feature Profile and Construct Interpretation” section). Coefficient and LIME analyses apply only to the FSA + LR baseline.

While these results demonstrate the potential of these models, they should be interpreted with caution given the limited dataset size. Further validation across broader online environments and real-world settings is needed.

When interpreting the results, several limitations should be considered. Our study was based on English-language data, although the approach could also be applied to other languages using embedding models. We used Reddit posts containing at least 100 characters as the data source. Although we expect the models to generalize to other platforms, different media may exhibit distinct linguistic dynamics that should be taken into account.

We identified 185 positively annotated posts, and this research can be extended as the subreddit grows and additional posts are annotated by mental health professionals. Experiments A1, A3, A4, and A5 used 185 positively labeled and 185 negatively labeled posts. Using an 80/20 train-test split resulted in a test set of 74 posts (37 per class). Although it is mathematically possible to achieve perfect scores in some experiments, the perfect precision and recall scores observed may be attributable to the limited test-set size, as correctly classifying a small number of cases can yield perfect performance metrics. This limitation is particularly relevant for experiment A2, in which the control group (r/ImposterSyndrome) contained only 63 posts, resulting in 13 test samples for this class (total test n=50: 37 positive and 13 negative). Therefore, the perfect precision and recall scores (1.0) achieved in this experiment should be interpreted with caution. To account for the statistical uncertainty associated with the small control sample, we calculated a 95% Wilson score CI of 0.772-1.000 for the 13-sample control class.

While the results demonstrate the feasibility of detecting NPD traits using machine learning, we recognize the inherent limitations of a small dataset, including the potential for overfitting. To address this, we balanced the dataset using undersampling (A5 and B5) and SMOTETomek (A2, B2, and setup C) and applied robust validation techniques, including 5-fold cross-validation. The consistent performance across embedding models is encouraging, but further research using larger and more diverse datasets is needed to confirm the generalizability of these findings. This study lays the groundwork for future research exploring scalable approaches for mental health applications in online forums. In the imbalanced experiments, the SMOTETomek step was applied to the feature representation used in each experiment, including transformer embeddings. SMOTE generates minority-class instances by interpolating between neighboring samples, which works well for sparse bag-of-words features but is more difficult to interpret in a dense contextual embedding space, where an interpolated point does not generally correspond to a linguistically coherent representation. The same concern applies to Tomek link removal. Therefore, this theoretical limitation also applies to these experiments.

To assess the practical impact of this limitation, we reran the 30 binary embedding pipelines (A1-A5 and B1-B5 × 3 classifiers) under 3 conditions: baseline SMOTETomek, no resampling, and class_weight=“balanced.” The mean absolute ΔF₁-score between the baseline and no-resampling conditions was 0.006, with a maximum of 0.047. ROC-AUC was similarly stable (maximum |Δ|=0.038). The mean signed ΔF₁-score was +0.000, and the direction of change was mixed across the largest differences, indicating that SMOTETomek neither systematically improved nor degraded performance at this scale. A 2×2 factorial analysis of the 6 A2/B2 cells, where SMOTE was actually applied, isolated the interpolation effect at 0.000 in 5 of the 6 cells, and the remaining cell differed by no more than 0.01 F₁-score. Applying the same protocol to the 9 binary setup C cells (C2-C4 × 3 classifiers, where SMOTETomek was also applied) yielded a mean absolute ΔF₁-score of 0.018, with a maximum of 0.071 and mixed directionality (4 of the 9 cells showed negative changes). ROC-AUC again remained stable (maximum |Δ|=0.024). For the multiclass C1 cells, scored using weighted F₁-score and weighted one-versus-rest ROC-AUC as in Table 10, the ΔF₁-scores were −0.009 (LR), +0.075 (SVM), and +0.074 (RF), whereas ROC-AUC again remained stable (maximum |Δ|=0.021). Thus, although the theoretical concern remains valid, its practical impact on the results presented in Table 10 appears to be limited for this corpus and these embedding models.

Our data collection ended in August 2019, before significant changes in online mental health discourse. The COVID-19 pandemic substantially altered patterns of online self-expression, help-seeking behavior, and mental health discussions across social media platforms. In addition, the emergence of new platforms (eg, TikTok) and changes in Reddit community dynamics may have influenced how individuals discuss narcissism and personality disorders online. These temporal factors may affect the model’s generalizability to current discourse, and validation using post-2019 data is needed before deployment in contemporary settings.

Our best-performing models rely on the OpenAI embedding APIs (text-embedding-ada-002 and text-embedding-3-large), which are proprietary and closed source. This dependency poses challenges for long-term scientific reproducibility because these models may be updated or deprecated by the vendor without notice. By contrast, open-weight alternatives such as BERT or LLaMA offer greater stability for replication studies. However, several open-source embedding models (eg, all-MiniLM and nomic-embed-text) achieved comparable performance in our experiments, providing reproducible alternatives for future implementations.

We conducted the interpretability analysis using the FSA + LR pipeline, which provides direct access to feature weights and coefficients. Detailed feature-level inference focused on the within-subreddit A5 experiment (FSA + LR baseline F₁-score=0.667), in which both the positive and control posts originated from r/Narcissism. This design minimizes subreddit-level stylistic and topical confounding (see the “Subreddit Style Confounds” section), making A5 the most appropriate setting for attributing learned features to NPD-related discourse rather than community membership. Cross-experiment robustness is reported across all 13 binary experiments in Section S3 in Multimedia Appendix 1. A2 and A1 are presented as additional examples (see Sections S4 and S5 in Multimedia Appendix 1), but feature-level construct claims are limited to A5 (see Section S6 in Multimedia Appendix 1). The embedding models that achieved the highest F₁-scores in this study (up to 0.892 for A5) remain opaque; therefore, feature-level interpretability claims apply only to this lower-performing baseline and may not generalize to the embedding models.

LIME provided local, instance-level explanations, but contextual words (eg, school, working, wife, and friends) occupied 53%-79% of the top 5 explanation slots across experiments. This predominance reflects both a TF-IDF sparsity artifact and genuine reliance on subreddit-specific life topics. The coefficient tables show that relational and contextual terms (eg, wife, friends, society, and men) carry substantial positive coefficients for the NPD-positive class. Thus, classification performance appears to reflect a combination of psychologically relevant cues and community-level topical patterns, whose relative contributions cannot be disentangled using the current methodology. Coefficient analysis serves as the primary source of interpretability evidence, whereas LIME provides supplementary local illustrations.

The 9-category feature-coding scheme was applied by a single coder without assessment of interrater reliability. Therefore, the category distributions presented in Multimedia Appendix 1 should be considered approximate. Section S7 in Multimedia Appendix 1 documents the boundary-case ambiguities and potential bias introduced by the default coding decisions.

We based this research on Reddit posts annotated by psychiatrists (NBA and OZ). Although these annotations provided a reliable foundation for model development, definitive clinical assessment would require formal diagnosis according to DSM-5 criteria followed by analysis of the patients’ online content. In addition, our annotation criteria focused on DSM-5 NPD diagnostic features, which primarily emphasize markers of grandiosity.

In setup B, the positive class comprised 6 label types representing both posts whose authors exhibited NPD traits (L7, L8, and L9) and posts referring to individuals with NPD traits without indicating that the author exhibited those traits (L2, L4, and L6). See Table 5 for the complete label composition and the “Task Complexity and Contextual Information” section for the implications for construct validity. Within r/Narcissism, L4 posts typically describe experiences of interpersonal harm (eg, references to a relative with narcissistic traits). By contrast, setup A provides a more specific classifier for identifying authors who themselves exhibit NPD traits.

The keyword-removal robustness analysis included 13 binary experiments with test-set sizes ranging from 50 to 204 posts. At these sample sizes, McNemar tests comparing the original-text and keyword-removed models on identical test sets yielded only 1-13 discordant pairs per comparison, which is too few in most cases for the test to reliably detect realistic effect sizes. Accordingly, the bootstrap 95% CIs for the F₁-score differences were wide and frequently included 0. Therefore, a nonsignificant result reflects insufficient statistical power rather than equivalence and should not be interpreted as either confirming or refuting model robustness.

A1 was the only exception. With 13 discordant pairs—the largest number across all experiments—it was the only comparison with sufficient power to detect an effect, and keyword removal resulted in a statistically significant decrease in F₁-score (McNemar P=.02). This outcome is expected for a cross-subreddit, keyword-dependent design (see Section S5.3 in Multimedia Appendix 1 for the coefficient profile). However, it does not alter the overall pattern: the robustness argument depends on whether any experiment showed a complete performance collapse after keyword removal, and none did. Thus, the robustness claim is based on the consistent pattern of point estimates across experiments, several of which showed little or no performance degradation, rather than on the results of any individual per-experiment test (see the “What Do These Automated Classifiers Actually Learn?” section).

Usernames and other author identifiers were not recorded (see the “Privacy and Confidentiality” section). Consequently, the train-test split was performed at the post level rather than the user level, and user-disjoint partitioning was not possible. As a result, posts from the same author may have appeared in both the training and test sets. This could allow the model to learn author-specific writing styles rather than generalizable NPD-trait markers, potentially inflating performance on the held-out test set, particularly for prolific contributors. As author identifiers were not retained, we could neither quantify this effect nor re-partition the data on a user-disjoint basis. We therefore acknowledge this as a transparency limitation of the reported performance metrics.

Our validation of the control subreddits (r/Showerthoughts, r/DeepThoughts, and r/ImposterSyndrome) confirmed minimal evidence of NPD traits, consistent with the expected prevalence in the general population. However, future research could strengthen this validation by annotating larger samples from these control communities. Incorporating additional data sources may also improve the robustness of comparative analyses.

Our control subreddits (r/DeepThoughts and r/Showerthoughts) represent neutral reflective discourse, whereas r/ImposterSyndrome represents self-focused but nongrandiose narratives. Future research should evaluate model performance on communities related to other personality disorders (eg, r/BPD) to assess specificity. However, such analyses would require new data collection, appropriate IRB approval, and disorder-specific annotation schemes and were therefore beyond the scope of this study.

The test-set prevalence (see the “Class Imbalance and Resampling Strategy” section) was substantially higher than the estimated population prevalence of 1%-6%. Consequently, the reported precision and F₁-scores are likely to overestimate performance in real-world deployment. Adjusting precision for prevalence using Bayes theorem, P = (π·TPR)/(π·TPR + [1 − π]·FPR), where TPR is true-positive rate, shows the expected effect. For A5 (text-embedding-ada-002 + LR in Table 10; TPR=0.892, FPR=0.108), the adjusted precision is approximately 0.08 at π=0.01, 0.20 at π=0.03, and 0.35 at π=0.06. However, these point estimates are based on a very small number of false positives (4 of 37 in A5) and should therefore be interpreted with caution because of their wide uncertainty.

Propagating the 95% Wilson CI for A5’s FPR (0.043-0.247) through the Bayes formula at π=0.03 yields an adjusted precision of approximately 0.10-0.40. These values should be interpreted as illustrative orders of magnitude rather than as lower bounds for deployment precision. The adjustment also assumes that the test-set TPR and FPR remain unchanged in deployment, a strong assumption given that the negative class in real-world settings would differ from the control subreddits used in this study. By contrast, ROC-AUC and recall are invariant to prevalence. Any operational deployment would therefore require prevalence-aware calibration, such as Platt scaling, threshold tuning, or precision-recall analyses across a defined range of prevalence values, rather than relying on the operating point observed in the test set.

We acknowledge that r/Narcissism contains heterogeneous content, including posts from self-identified individuals with narcissistic traits, help seekers, individuals questioning whether they exhibit such traits, and people discussing others with narcissistic traits in their lives. As a result, the model learns from multiple linguistic contexts related to narcissism rather than from a purely clinical sample of individuals with confirmed diagnoses. However, the use of psychiatrist-validated labels (L7 and L8) helps ensure that the model captures clinically relevant linguistic patterns despite this source heterogeneity.

The classifier’s feature profile is design dependent (see the “What Do These Automated Classifiers Actually Learn?” section). The cross-subreddit experiments relied substantially on diagnostic keywords in the original model, whereas the within-subreddit A5 experiment discriminated primarily on distress, self-awareness, and relational cues once shared NPD vocabulary was neutralized. This variation cautions against adopting a single construct interpretation across all 13 experiments. Sections S5.3, S6.5, and S9 in Multimedia Appendix 1 present detailed evidence of these design-dependent feature profiles. For the scope of the keyword removal list and the potential influence of unlisted community jargon, see the “What Do These Automated Classifiers Actually Learn?” section and Section S9 in Multimedia Appendix 1.

We acknowledge that stylistic differences between subreddits may influence classification when control posts originate from communities such as r/Showerthoughts or r/DeepThoughts, which have writing conventions that differ from those of r/Narcissism. Experiments A5 and B5 use posts exclusively from r/Narcissism for both the positive and negative classes, thereby eliminating cross-subreddit stylistic variation—the same design feature that motivates A5 as the primary setting for the feature-level interpretability analysis (see Section S6 in Multimedia Appendix 1). A5 (n=370) achieved an accuracy of 0.892 and an F₁-score of 0.892, substantially exceeding chance performance even after removing subreddit-level stylistic variation. A content-versus-style feature ablation analysis (cross-validated and reproducible using the published code package) showed that, within A5, content features (word TF-IDF semantics and sentiment) achieved 80.6% accuracy compared with 68.2% for style features (orthographic and punctuation statistics plus part-of-speech distributions), a 12.4-percentage-point advantage for content (SD 5.9 across folds). As the resampled cross-validation folds were not independent, we report this difference without a significance test. Although style features alone performed above the 50% chance baseline, content features provided substantially better discrimination. In the combined model, RF feature importance was 97% content and 3% style. However, because content and style features overlap, this marginal importance likely understates the standalone contribution of style, which is more directly reflected by the ablation analysis.

The advantage of content features is not attributable to cross-subreddit confounding. The difference in accuracy between content and style features was similar in the cross-subreddit experiments (mean 11.8 percentage points across A1-A3) and the within-subreddit A5 experiment (12.4 percentage points).

In the cross-subreddit experiments, the content signal included both established diagnostic vocabulary and community-specific terms. In the within-subreddit setting, both content and style signals may reflect community-level discourse norms in addition to trait-relevant variation. Future research could use adversarial debiasing and validation across diverse mental health communities (eg, r/BPD, r/depression, and r/anxiety) to further disentangle trait-relevant markers from community-level discourse norms.