This paper is in the following e-collection/theme issue:

Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (993) Development and Evaluation of Research Methods, Instruments and Tools (732) Secondary Use of Clinical Data for Research and Surveillance (469) Diagnostic Tools in Mental Health (322) mHealth for Screening (294) Research Instruments, Questionnaires, and Tools (733)

Published on in Vol 27 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/70451, first published .
The Impact of Individual Factors on Careless Responding Across Different Mental Disorder Screenings: Cross-Sectional Study

The Impact of Individual Factors on Careless Responding Across Different Mental Disorder Screenings: Cross-Sectional Study

The Impact of Individual Factors on Careless Responding Across Different Mental Disorder Screenings: Cross-Sectional Study

Authors of this article:

Huawei Kuang1 Author Orcid Image ;   Lichao Zhu2 Author Orcid Image ;   Haonan Yin3 Author Orcid Image ;   Zihe Zhang1 Author Orcid Image ;   Biao Jing1 Author Orcid Image ;   Junwei Kuang4 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    1School of Energy and Constructional Engineering, Shandong Huayu University of Technology, Dezhou, China

    2Tencent (China), Shenzhen, China

    3Paul Merage School of Business, University of California, Irvine, CA, United States

    4Department of Industrial Engineering, School of Business Administration, South China University of Technology, No. 381 Wushan Road, Guangzhou, China

    Corresponding Author:

    Junwei Kuang, PhD


    Background: Online questionnaires are widely used for large-scale screening. However, careless responding (CR) from participants can compromise the reliability of screening outcomes. Prior studies have focused on the effects of individual and environmental factors on CR, but the effect of questionnaire type remains underexplored.

    Objective: This study investigates the individual factors influencing CR in online mental health screening and assesses how the effect of these factors varies across different psychological questionnaires.

    Methods: This study analyzed data from 24,367 participants across 4 questionnaires (PHQ-9 [Patient Health Questionnaire-9], PSS [Perceived Stress Scale], ISI [Insomnia Severity Index], and GAD-7 [Generalized Anxiety Disorder-7 Scale]). CR was defined as the proportion of items completed in less than 2 seconds per item. We used a multiple linear regression model to examine the effect of individual factors (sex, age, education, smoking, and drinking) on CR across 4 questionnaires. In addition, response times were visualized to identify patterns between careless and careful responders.

    Results: Females demonstrate lower levels of CR than males when completing the PHQ-9 (β=−.172, 95% CI −0.104 to −0.089; P<.001), PSS (β=−.234, 95% CI −0.162 to −0.14; P<.001), ISI (β=−.207, 95% CI −0.13 to −0.114; P<.001), and GAD-7 (β=−.177, 95% CI −0.108 to −0.093; P<.001). Older participants demonstrated lower levels of CR on the PHQ-9 (β=−.036, 95% CI −0.007 to −0.003; P<.001), ISI (β=−.036, 95% CI −0.007 to −0.003; P<.001), and GAD-7 (β=−.053, 95% CI −0.009 to −0.005; P<.001), but their age was unrelated to CR on the PSS. Interestingly, compared with participants with an associate-level education, those with a high education (bachelor’s, master’s, or doctoral degree) demonstrated higher levels of CR, especially those with a master’s degree (PHQ-9: β=.098, 95% CI 0.136 to 0.188; P<.001 and GAD-7: β=.091, 95% CI 0.125 to 0.178; P<.001). Smokers exhibited varied patterns, with current smokers demonstrating lower levels of CR on the PHQ-9 (β=−.022, 95% CI −0.064 to −0.016; P=.001) and GAD-7 (β=−.014, 95% CI −0.051 to −0.002; P=.03), whereas occasional smokers demonstrated higher levels of CR on the PSS (β=.019, 95% CI 0.010 to 0.050; P=.003) than nonsmokers. Drinkers demonstrated lower levels of CR than nondrinkers, with the strongest effect among occasional drinkers on the PHQ-9 (β=−.163, 95% CI −0.103 to −0.087; P<.001). Analysis of response times revealed that participants tended to spend less time on PHQ-9 and GAD-7 surveys, and CR on PSS and ISI surveys was characterized by skipping questions.

    Conclusions: The effect of individual factors on CR varies across questionnaire types. These findings offer valuable insights for questionnaire designers and administrators, highlighting the need for targeted intervention.

    J Med Internet Res 2025;27:e70451

    doi:10.2196/70451

    Keywords

    careless response; digital health; online health screening; mental health; questionnaire 


    Background

    Mental illnesses, such as depression and anxiety, are associated with symptoms including low mood, difficulty concentrating, and a diminished interest in life, profoundly affecting both individuals’ well-being and societal stability [1,2]. Psychological issues have become increasingly prevalent worldwide. The COVID-19 pandemic has exacerbated this trend, leading to a sharp rise in mental health cases [3]. According to the World Health Organization (WHO), nearly 1 billion people globally endure from mental disorders, with cases of anxiety and depression increasing by 25% following the pandemic [4]. However, due to mental illnesses being stigmatized in contexts, particularly depression and anxiety, individuals often hesitate to disclose their conditions to health care providers [5]. This reluctance results in a significant number of unreported and untreated cases, ultimately exacerbating the overall disease burden [6,7]. Consequently, implementing effective mental health screening is essential for early detection and prevention [8,9].

    Mental health screening questionnaires are standardized tools designed to assess an individual’s mental health status through a series of structured questions. Traditionally, these questionnaires were distributed by administrators to participants for completion, after which they were collected and analyzed. However, this method is often hindered by inefficiencies in distribution, data collection, and analysis. The emergence of internet technology has revolutionized this process by introducing online psychological screening questionnaires, which offer a more efficient and scalable alternative. Online questionnaires can improve the recognition of mental illness and increase patients’ opportunities to receive digital mental health interventions [10]. Compared to paper questionnaires, they are more easily accepted by respondents in modern mental health screening [11]. Despite their advantages, a significant limitation of online questionnaires is the increased risk of careless responding (CR), where participants may fail to read or fully engage with the items [12-14]. This can lead to data that inaccurately reflects their true mental health status [15], thus undermining the validity of the results and subsequent psychological interventions. Consequently, identifying the factors that influence CR and improving response validity has become a critical priority.

    Related Works

    Existing research has explored both external and internal factors influencing careless responses in online surveys. External factors include extrinsic motivation, social contact, and environmental distractions. For instance, Gadiraju et al [16] found that participants recruited through crowdsourcing platforms frequently use strategies to minimize time and effort in exchange for compensation. As survey duration increases, participants’ motivation and attention tend to decline, significantly elevating the risk of CR. In addition, Johnson [17] argued that the lack of direct interaction between respondents and administrators, as well as physical distance in online surveys, may reduce participants’ sense of responsibility, leading to more careless responses. Environmental distractions can further exacerbate CR [18].

    Internal factors, such as participants’ intrinsic motivation, interest, and personality traits, also play a significant role in CR. For example, participants with low survey interest are more likely to exhibit careless responses in lengthy surveys compared to shorter ones [19]. As survey duration increases, participants’ motivation diminishes, further elevating the probability of careless responses [15]. In addition, Maniaci and Rogge [20] found that self-reported agreeableness, conscientiousness, and openness to experience were negatively correlated with CR. Bowling et al [21] reported that acquaintance-reported agreeableness, conscientiousness, emotional stability, and extraversion were negatively correlated with CR. Similarly, Meade et al [13] found that self-reported agreeableness, friendliness, and emotional stability were negatively correlated with CR in a sample of undergraduate students. McKay [22] further revealed that benevolent personality traits (eg, self-esteem) were negatively correlated with CR, while malevolent traits (eg, interpersonal coldness and sensation-seeking) were positively correlated.

    Research Gaps

    While existing research has extensively explored external and internal factors influencing CR in online surveys, these studies have primarily focused on nonpsychological screening contexts. Unlike general questionnaires, psychological screening instruments often include intrusive or sensitive items (eg, “Have you had suicidal thoughts in the past two weeks?”), which may trigger defensive avoidance or emotional arousal in respondents. As a result, the factors contributing to CR in psychological screening questionnaires may differ significantly from those in general online surveys, highlighting the need for further investigation into this unique context.

    Moreover, existing research has produced inconsistent findings regarding the same factors influencing CR, potentially due to the heterogeneous effect of questionnaire type. Most studies analyze data from a single questionnaire type, whereas mental health screening questionnaires encompass diverse forms, such as the Patient Health Questionnaire-9 (PHQ-9) [23], Generalized Anxiety Disorder-7 Scale (GAD-7) [24], Insomnia Severity Index (ISI) [25], and Perceived Stress Scale (PSS) [26]. Thus, the same factors may yield different conclusions across different psychological screening instruments. However, few studies have examined response behaviors across multiple psychological screening questionnaires within the same participant group.

    Objective

    This study aims to explore the factors influencing CR in online mental health screening questionnaires and assess how the effect of these factors varies across different types of psychological screening instruments. The findings can improve the validity of psychological screening outcomes, ultimately advancing mental health services.


    Study Design and Participants

    This cross-sectional study is based on observational data collected through an online mental health screening initiative conducted by Xinxiang Medical University during the COVID-19 outbreak (February 27-March 17, 2021) [27], which aimed to assess psychological distress among students to inform campus mental health support services. Eligible participants were currently enrolled undergraduate or graduate students at Xinxiang Medical University who voluntarily participated in the screening and provided informed consent for future research purposes.

    Data Collection

    The data were collected through a large-scale online psychological screening project conducted at Xinxiang Medical University during the COVID-19 pandemic [27]. The project aimed to assess the mental health status of college students. The study used a web-based application accessible via QR codes or URLs, allowing participants to complete the survey voluntarily on their personal devices without restrictions on time or location, although all data collection was confined to the campus due to pandemic restrictions. Participants included 24,367 undergraduate and graduate students, representing a convenience sample recruited through class-wide notifications distributed by counselors. The questionnaire comprised standard psychological screening instruments administered in a fixed sequence: the PHQ-9, GAD-7, PSS, and ISI, along with demographic items. These instruments maintained their original scoring systems. Specifically, the PHQ-9 consists of 9 items scored from 0 to 3 (total score range 0‐27), the GAD-7 includes 7 items scored from 0 to 3 (range 0‐21), the PSS comprises 14 items scored from 0 to 4 (range 0‐56), and the ISI contains 7 items scored from 0 to 4 (range 0‐28), with each question’s response time recorded to the nearest 0.01 seconds precision using timestamps. Prior to completing the psychological screening instruments, all participants were required to report demographic information, including age, sex, education, smoking, and drinking. Each variable was collected through a structured response format with predefined options. The anonymized dataset was published on Zenodo in comma-separated values format. To ensure data quality, the study implemented multiple controls: participants were required to complete all items (75 participants were excluded for missing responses), and response times were later screened for outliers using the median absolute deviation.

    Variables and Measurements

    CR was measured using the Page Time Index (PTI), which captures the proportion of items answered in less than 2 seconds per item—a widely accepted threshold for detecting CR [28-30]. While PTI is typically computed using page-level timestamps (eg, average time per item on a multi-item page), our dataset provided item-level response time, enabling more precise measurement. For each clinical item, we recorded the response duration from item presentation to answer submission. Items completed in less than 2 seconds were coded as “1” (careless), and those completed in 2 seconds or more were coded as “0” (careful). The CR was then calculated as the average of these binary indicators across all items, resulting in a continuous score ranging from 0 to 1 that reflects the proportion of careless responses.

    The independent variables included sex, age, education, smoking, and drinking. All variables were directly reported by participants through the demographic questions, with sex as a binary variable and age as a continuous variable. Education, smoking, and drinking were categorical variables. All variables were based on prior empirical evidence. Specifically, sex was included because previous studies have shown differences in response patterns between males and females [15], while age was incorporated due to its established relationship with attention span and survey engagement [31]. Education level was selected, as higher educational attainment is linked to more conscientious responding [32]. Behavioral factors such as smoking [17] and drinking [31] were included based on research suggesting these behaviors may influence response effort and impulsiveness.

    Statistical Analysis

    This study used secondary data, involving 24,367 college students. As the data were collected for screening rather than research purposes, the sample size was determined by the organizers of the original screening initiative rather than by formal power calculations. To evaluate whether this sample size was sufficient for our analytical objectives, we conducted post hoc power analyses using G*Power 3.1, developed by Faul et al [33,34]. With a sample size of 24,367 and 5 predictors, the achieved power was 100% to detect small effects (f²=0.02, α=.05). Sensitivity analyses confirmed 100% power for even smaller effects (f²=0.005). Therefore, the current sample size meets the statistical power requirements. To examine the factors associated with CR in online psychological assessments, we used a multiple linear regression model using SPSS Statistics (version 27.0; IBM). The multiple linear regression model is specified as follows:

    CR= β0+β1⋅Sex+β2⋅Age+β3⋅Educationbachelor+β4⋅Educationmaster+β5⋅Educationdoctor+β6⋅Smokingformer+β7⋅Smokingoccasional+β8⋅Smokingcurrent+β9⋅Drinkingformer+β10⋅Drinkingoccasional+β11⋅Drinkingcurrent+ε(1)

    The dependent variable was CR, operationalized as a continuous score ranging from 0 to 1. Independent variables included sex (binary: male or female), age (continuous), education (categorical: bachelor’s, master’s, or doctoral degree, with associate degree as the reference group), smoking (categorical: former, occasional, or current smoker, with never smoker as the reference group), and drinking (categorical: former, occasional, or current drinker, with never drinker as the reference group). The selection of these variables was based on theoretical considerations and prior empirical findings [15,17,31,32], which suggest their potential associations with attention, cognitive effort, and behavioral compliance in survey settings.

    To facilitate statistical analysis, we converted all categorical variables into dummy variables and included them in the multiple linear regression model. These dummy variables allow us to examine the unique effect of each category while controlling for other covariates. Given the relatively small number of predictors (5 variables) and their strong theoretical justification, we opted not to use stepwise selection methods, instead retaining all specified variables to avoid potential overfitting and maintain theoretical coherence [35]. Multicollinearity diagnostics using variance inflation factors (all <2.0) confirmed the absence of problematic correlations among predictors.

    Ethical Considerations

    The original data collection received ethical approval from the Ethics Committee of Xinxiang Medical College of Henan Province (XYLL-2020235) [27]. The dataset used in this study is publicly available, fully anonymized, and contains no personally identifiable information. Therefore, in accordance with prevailing guidelines for exempt research involving secondary anonymized data, no additional ethical review was required. Informed consent was obtained online from all participants prior to survey administration, including agreement for data sharing and dissemination for scientific research. This study ensured adherence to all necessary ethical considerations and informed consent procedures. To protect participants’ personal information, all data collected in the original screening were fully anonymized before analysis [27]. No explicit incentives or compensation were provided to participants, as the screening was a voluntary public health service intended to benefit students’ mental health.


    Participants

    The study initially collected 24,367 responses, from which 131 participants were excluded: 75 for missing values and 56 for implausible age entries. This yielded a final analytical sample of 24,236 participants, as detailed in Table 1. Female students predominated (64.0%, n=15,518), and most participants (84.7%, n=20,516) fell within the age range of 16‐22 years. Educational attainment showed a clear hierarchical distribution, with bachelor’s degree holders constituting 91.8% (n=22,259) of the sample, while graduate students were comparatively underrepresented (master’s: 2.8%, n=666 and doctoral: 0.1%, n=33). Most participants reported never smoking (92.7%, n=22,468) or consuming alcohol (67.5%, n=16,372), consistent with general population trends among Chinese university students.

    Table 1. Demographic and descriptive characteristics of survey participants.
    Number of respondentsValue
    All, N (%)24,236 (100)
    Sex, n (%)
    Male8718 (36)
    Female15,518 (64)
    Age (years), n (%)
    16‐2220,516 (84.7)
    23‐293643 (15)
    30‐3577 (0.3)
    Education, n (%)
    Associate degree1278 (5.3)
    Bachelor’s degree22,259 (91.8)
    Master’s degree666 (2.8)
    Doctoral degree33 (0.1)
    Smoking, n (%)
    Never smokes22,468 (92.7)
    Former smoker (cumulative smoking >10 packs), but not in the past year189 (0.8)
    Occasional smoker (cumulative smoking <10 packs)1039 (4.3)
    Current smoker (cumulative smoking >10 packs)540 (2.2)
    Drinking, n (%)
    Never drinks16,372 (67.5)
    Drank in the past (more than once a week), but not in the past year282 (1.2)
    Drinks occasionally (less than once a week)7344 (30.3)
    Current regular drinker (more than once a week)238 (1)

    Variable Definitions

    This study examines the following independent variables: sex, age, education, smoking, and drinking. Detailed variable coding is provided in Textbox 1.

    Textbox 1. Variable definitions.

    Variable and assignment instructions

    • Sex: 0=male; 1=female.
    • Age: age in years.
    • Education: 1=associate degree; 2=bachelor’s degree; 3=master’s degree; and 4=doctoral degree.
    • Smoking: 1=never smokes; 2=former smoker (cumulative smoking >10 packs), but not in the past year; 3=occasional smoker (cumulative smoking <10 packs); and 4=current smoker (cumulative smoking >10 packs).
    • Drinking: 1=never drinks; 2=drank in the past (more than once a week), but not in the past year; 3=drinks occasionally (less than once a week); and 4=current regular drinker (more than once a week).

    Regression Results of Careless Response

    To ensure data quality, we implemented a rigorous outlier removal protocol, excluding responses with completion times exceeding 3 SDs from the mean. This procedure resulted in slightly varied final sample sizes across instruments: PHQ-9 (n=23,984), PSS (n=24,133), ISI (n=24,152), and GAD-7 (n=24,165). The multiple linear regression results for the PHQ-9, PSS, ISI, and GAD-7 scales are presented in Tables 2-5.

    Table 2. Regression results of careless reactions on the Patient Health Questionnaire-9 scale.
    VariableStandardized coefficients β (95% CI)P value
    Sex (reference=male)
     Female−0.172 (−0.104 to −0.089)<.001
    Age (years)−0.036 (−0.007 to −0.003)<.001
    Education (reference=associate degree)
     Bachelor’s degree0.057 (0.041 to 0.071)<.001
     Master’s degree0.098 (0.136 to 0.188)<.001
     Doctoral degree0.028 (0.109 to 0.291)<.001
    Smoking (reference=never smokes)
     Former smoker, but not in the past year0.001 (−0.035 to 0.043).84
     Occasional smoker0.002 (−0.015 to 0.02).75
     Current smoker−0.022 (−0.064 to −0.016).001
    Drinking (reference=never drinks)
     Drank in the past, but not in the past year−0.046 (−0.147 to −0.084)<.001
     Drinks occasionally−0.163 (−0.103 to −0.087)<.001
     Current regular drinker−0.035 (−0.132 to −0.060)<.001
    Table 3. Regression results of careless reactions on the Perceived Stress Scale.
    VariableStandardized coefficient β (95% CI)P value
    Sex (reference=male)
     Female−0.234 (−0.162 to −0.144)<.001
    Age (years)−0.01 (−0.004 to 0.001).15
    Education (reference=associate degree)
     Bachelor’s degree0.028 (0.014 to 0.049)<.001
     Master’s degree0.061 (0.087 to 0.147)<.001
     Doctoral degree0.028 (0.128 to 0.339)<.001
    Smoking (reference=never smokes)
     Former smoker, but not in the past year0.018 (0.019 to 0.109).005
     Occasional smoker0.019 (0.01 to 0.050).003
     Current smoker−0.003 (−0.034 to 0.022).68
    Drinking (reference=never drinks)
     Drank in the past, but not in the past year−0.018 (−0.088 to −0.015).006
     Drinks occasionally−0.112 (−0.086 to −0.067)<.001
     Current regular drinker−0.015 (−0.088 to −0.005).03
    Table 4. Regression results of careless reactions on the Insomnia Severity Index scale.
    VariableStandardized coefficients β (95% CI)P value
    Sex (reference=male)
     Female−0.207 (−0.13 to −0.114)<.001
    Age (years)−0.036 (−0.007 to −0.003)<.001
    Education (reference=associate degree)
     Bachelor’s degree0.065 (0.051 to 0.083)<.001
     Master’s degree0.089 (0.127 to 0.182)<.001
     Doctoral degree0.031 (0.144 to 0.335)<.001
    Smoking (reference=never smokes)
     Former smoker, but not in the past year0.008 (−0.015 to 0.065).23
     Occasional smoker0.007 (−0.009 to 0.027).32
     Current smoker−0.003 (−0.031 to 0.019).64
    Drinking (reference=never drinks)
     Drank in the past, but not in the past year−0.038 (−0.134 to −0.068)<.001
     Drinks occasionally−0.135 (−0.091 to −0.074)<.001
     Current regular drinker−0.033 (−0.131 to −0.057)<.001
    Table 5. Regression results of careless reactions on the Generalized Anxiety Disorder-7 Scale.
    VariableStandardized coefficient β (95% CI)P value
    Sex (reference=male)
     Female−0.177 (−0.108 to −0.093)<.001
    Age (years)−0.053 (−0.009 to −0.005)<.001
    Education (reference=associate degree)
     Bachelor’s degree0.056 (0.04 to 0.07)<.001
     Master’s degree0.091 (0.125 to 0.178)<.001
     Doctoral degree0.021 (0.064 to 0.249).001
    Smoking (reference=never smokes)
     Former smoker, but not in the past year0.014 (0.005 to 0.083).03
     Occasional smoker0.002 (−0.015 to 0.020).80
     Current smoker−0.014 (−0.051 to −0.002).03
    Drinking (reference=never drinks)
     Drank in the past, but not in the past year−0.040 (−0.133 to −0.069)<.001
     Drinks occasionally−0.144 (−0.094 to −0.077)<.001
     Current regular drinker−0.030 (−0.12 to −0.048)<.001

    Table 2 shows that female students demonstrated lower levels of CR than male students when completing the PHQ-9 (P<.001). Older participants showed significantly lower levels of CR (P<.001). Participants with higher education levels had significantly higher levels of CR compared to the associate degree group (P<.001). Compared to the nonsmokers, current smokers demonstrated lower levels of CR (P=.001). Participants with a history of alcohol consumption also demonstrated lower levels of CR than those who never drank (P<.001).

    Table 3 reveals that when completing the PSS questionnaire, female students again demonstrated lower levels of CR than male students (P<.001). Participants with bachelor’s (P<.001), master’s (P<.001), or doctoral degrees (P<.001) showed higher levels of CR compared to those with associate degrees. Both former smokers (P=.005) and occasional smokers (P=.003) were more prone to CR than nonsmokers. Participants with a history of alcohol consumption—whether past drinkers (P=.006), occasional drinkers (P<.001), or frequent drinkers (P=.03)—demonstrated lower levels of CR than those who never drank.

    Table 4 reveals that female undergraduates demonstrated lower levels of CR than their male counterparts when completing the ISI (P<.001). For each unit increase in age, the level of CR on the ISI scale significantly decreased (P<.001). That is, older individuals were less prone to CR. Relative to associate degree holders, progressively higher education levels—bachelor’s (P<.001), master’s (P<.001), and doctoral (P<.001)—showed higher levels of CR. All drinking categories—former drinkers (P<.001), occasional drinkers (P<.001), and frequent drinkers (P<.001)—demonstrated lower levels of CR relative to nondrinkers.

    Table 5 indicates a pronounced sex difference emerged, with female students demonstrating lower levels of CR than males when completing the GAD-7 (P<.001). Older participants showed significantly lower levels of CR (P<.001). Relative to associate degree holders, participants with bachelor’s (P<.001), master’s (P<.001), and doctoral degrees (P=.001) consistently exhibited higher levels of CR. Former smokers showed significantly higher levels of CR compared to their nonsmoking peers (P=.03), and current smokers demonstrated lower levels of CR (P=.03). All categories of drinkers—including former drinkers (P<.001), occasional drinkers (P<.001), and regular drinkers (P<.001)—demonstrated lower levels of CR than nondrinkers.

    Multimedia Appendix 1 presents several consistent and divergent patterns in CR across the 4 psychological scales. Most notably, female participants demonstrated lower levels of CR than males in all questionnaires (P<.001 for all scales). Age was associated with lower levels of CR on the PHQ-9 (P<.001), ISI (P<.001), and GAD-7 (P<.001), but not on the PSS (P=.15). Educational attainment consistently demonstrated higher levels of CR, with advanced degree holders showing higher levels than associate degree holders across all measures. Smokers exhibited varied patterns, with current smokers demonstrating lower levels of CR on the PHQ-9 and GAD-7, but occasional smokers demonstrating higher levels of CR on the PSS than nonsmokers. Interestingly, college students with a history of drinking demonstrated lower levels of CR across all scales compared to those who had never consumed alcohol.

    Pattern Identification

    While the study has identified factors influencing CR in mental health screening questionnaires, the specific behavioral patterns underlying careless responses remain unclear. Understanding these patterns is crucial for improving questionnaire design and ultimately enhancing the validity and reliability of psychological assessments. To explore this, based on Erdem Kara’s study [30], we classified users whose PTI exceeded 0.67 seconds as careless respondents. The average response times with error bars for each question were calculated separately for careless and careful responders across the PHQ-9, PSS, ISI, and GAD-7 scales. The findings are presented in Figures 1-4.

    ‎
    Figure 1. Response time for individual questions of the Patient Health Questionnaire-9.
    ‎
    Figure 2. Response time for individual questions of the Generalized Anxiety Disorder-7 Scale.
    ‎
    Figure 3. Response time for individual questions of the Insomnia Severity Index.
    ‎
    Figure 4. Response time for individual questions of the Perceived Stress Scale.

    As illustrated in Figures 1 and 2, both careless and careful responders exhibit similar trends in response time per question across the PHQ-9 and GAD-7 scales. However, careless responders demonstrate significantly shorter response times compared to careful responders. This pattern suggests that careless responders tend to skim through questions rapidly, without engaging in thorough reading or thoughtful consideration. In contrast, Figures 3 and 4 reveal distinct behaviors for the ISI and PSS scales: careful responders display variability in response times per question, while careless responders maintain nearly constant response times. This finding suggests that careless responders may skip reading the questions altogether, opting instead to select answers automatically.

    This behavior can be attributed to the varying levels of importance participants place on different psychological conditions. For the PHQ-9 and GAD-7 scales, which assess depression and anxiety symptoms, even careless responders may still skim through the questions due to the perceived significance of these conditions. In contrast, for the ISI and PSS scales, which measure insomnia severity and perceived stress, participants may perceive these conditions as less critical. As a result, they may disregard the questions entirely, selecting answers to complete the task with minimal effort. This suggests that perceived importance influences the depth of engagement, with less salient conditions such as stress and insomnia receiving significantly less attention.


    Principal Findings

    The study found that female participants demonstrated lower levels of careless responding than male participants when filling out psychological screening questionnaires. This is consistent with previous research findings [31]. This phenomenon can be explained from the perspectives of emotional expression differences and social expectations. First, women demonstrated stronger emotional expression and empathy [36], enabling them to better understand the emotional nuances of the questionnaire and take mental health–related tasks more seriously. Second, social expectation theory suggests that women are more inclined to conform to social norms [37], particularly in tasks involving emotions and mental health.

    Older participants demonstrated significantly lower levels of CR on the PHQ-9, ISI, and GAD-7, whereas no significant age-related difference emerged for the PSS. This pattern aligns with previous findings, such as those of Carstensen and DeLiema [38], who suggest that older adults exhibit enhanced emotional regulation and task focus—traits that may reduce CR on symptom-focused questionnaires. The absence of age effects for the PSS could stem from its unique assessment of perceived stress, as the PSS measures acute stress responses to daily challenges rather than chronic emotional states, potentially eliciting similar levels of attention across age groups.

    Individuals with higher education levels showed more careless response behaviors when filling out psychological screening questionnaires. This finding contradicts conventional wisdom, as higher education is typically associated with greater conscientiousness in task completion. The tendency may stem from several factors. First, their deeper awareness of the stigma surrounding mental illness [39,40] could lead them to avoid negative labels by responding carelessly. Second, perfectionistic tendencies among this group [41,42] may make them overly sensitive to questionnaire items, potentially prompting careless responses.

    Smokers exhibited varied patterns, with current smokers demonstrating lower levels of CR to the PHQ-9 and GAD-7, but occasional smokers showing higher levels of CR to the PSS than nonsmokers.

    Drinkers consistently demonstrated lower levels of CR than nondrinkers. A potential explanation is that individuals with a history of drinking may be more attuned to their mental state, leading them to complete the questionnaires more carefully.

    When filling out the PHQ-9 and GAD-7 questionnaires, careless responders were characterized by quickly skimming through the questions, whereas when filling out the PSS and ISI questionnaires, careless responders appeared to skip some questions and directly select answers. These findings provide important references for questionnaire designers and distributors, suggesting that targeted intervention measures should be taken.

    Comparison With Previous Work

    Previous studies primarily examined the influence of external factors (eg, external motivation [16], social contact [17], and environmental distractions [18]) and internal factors (eg, intrinsic motivation [19], interest [15], and personality traits [20,22,32,43]) on careless responses. In contrast, this study delves deeper into how individual factors manifest differently across various types of psychological screening questionnaires. For instance, while prior research generally suggested that individuals with higher education levels are more likely to complete tasks conscientiously [44,45], this study found that highly educated individuals exhibited greater careless response behaviors when filling out psychological screening questionnaires. This discrepancy may be attributed to the intrusive nature of certain questionnaire items—highly educated individuals, driven by strong self-esteem, may be reluctant to disclose mental health concerns. In addition, this study identified variations in how age, smoking, and drinking behaviors influence careless responses across different questionnaire types. These findings diverge from previous conclusions, highlighting the need to account for the interaction between questionnaire type and individual characteristics when analyzing careless response behaviors.

    Clinical and Practical Implications

    Our findings underscore critical implications for improving mental health screening practices, particularly regarding respondents’ differential engagement across questionnaire types. First, the lower conscientiousness observed for PSS (stress) and ISI (insomnia) measures compared to depression and anxiety scales suggests that many participants may underestimate the clinical significance of stress and sleep disturbances. To address this, we strongly recommend incorporating clear educational messaging during questionnaire administration that emphasizes the serious health consequences of unmanaged stress and insomnia (eg, increased risks for cardiovascular disease, diabetes, and cognitive decline). Second, for mental disorder screening, our results indicate the need for additional validity checks when assessing highly educated populations, who may demonstrate greater careless response potentially due to stigma concerns. Third, older adults showed lower levels of CR with depression, sleep, and anxiety items, highlighting a critical need for early mental health education. Proactive interventions, such as psychoeducation campaigns targeting younger demographics, could help mitigate the long-term consequences of untreated symptoms, which often compound with age.

    Limitations and Future Directions

    This study has several limitations that could be studied in future work. First, our operationalization of CR relied on response time, which may not capture all forms of inattentive responding. Future work should combine response time metrics with other validated approaches such as eye-tracking methodologies and postsurvey self-reports of engagement. Second, the homogeneous sample—college students from a single Chinese medical university—raises concerns about generalizability. Future work could examine whether these conclusions generalize to diverse domains, such as international contexts with varying cultural norms. Third, while we controlled for major demographic and behavioral factors, unmeasured variables (eg, personality traits) may influence CR.

    Conclusion

    This study analyzed data from college students who completed 4 common psychological screening questionnaires online, identifying the individual factors that influence careless response behaviors and their heterogeneous effects across different questionnaire types. The findings indicate that sex, age, education level, smoking, and drinking habits significantly impact CR, with these effects differing across screening tools. Moreover, the study revealed distinct patterns of careless responses depending on the questionnaire type. When completing the PHQ-9 and GAD-7, careless responders tended to skim through questions quickly, whereas in the PSS and ISI, they were more likely to skip questions and select answers without careful consideration.

    These findings offer important guidance for both clinical practice and future research. For clinicians administering online mental health screenings, the results suggest implementing questionnaire-specific quality control measures, such as response time monitoring for PHQ-9 and GAD-7 assessments, and mandatory question-answering features for PSS and ISI measures. The identified demographic risk factors (eg, male sex and higher education) can help target these interventions to high-risk groups. For researchers, the study highlights the need to (1) develop more sophisticated detection algorithms that account for these instrument-specific response patterns, (2) investigate the psychological mechanisms underlying the observed demographic differences, and (3) examine whether these findings generalize to clinical populations and other cultural contexts.

    Acknowledgments

    This research was funded by the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (72321001). No generative AI tools were used in any part of the manuscript writing.

    Data Availability

    The datasets generated or analyzed during this study are available in the Zenodo repository [46].

    Authors' Contributions

    HK and JK conceptualized the study. JK curated the data and acquired funding. HK conducted the formal analysis and led the investigation. HY contributed to the methodology. JK managed the overall project. LZ provided resources and developed the software. JK supervised the study. ZZ and BJ validated the findings and created the visualizations. The original draft was written by HK, JK, and HY.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Regression results of careless responses across different questionnaires.

    DOCX File, 15 KB
    1. Herrman H, Patel V, Kieling C, et al. Time for united action on depression: a Lancet–World Psychiatric Association Commission. Lancet. Mar 2022;399(10328):957-1022. [CrossRef]
    2. Patel V, Saxena S, Lund C, et al. The Lancet Commission on global mental health and sustainable development. Lancet. Oct 2018;392(10157):1553-1598. [CrossRef] [Medline]
    3. Mack DL, DaSilva AW, Rogers C, et al. Mental health and behavior of college students during the COVID-19 pandemic: longitudinal mobile smartphone and ecological momentary assessment study, part II. J Med Internet Res. Jun 4, 2021;23(6):e28892. [CrossRef] [Medline]
    4. COVID-19 pandemic triggers 25% increase in prevalence of anxiety and depression worldwide. World Health Organization. 2022. URL: https:/​/www.​who.int/​news/​item/​02-03-2022-covid-19-pandemic-triggers-25-increase-in-prevalence-of-anxiety-and-depression-worldwide [Accessed 2025-02-22]
    5. Schnyder N, Panczak R, Groth N, Schultze-Lutter F. Association between mental health-related stigma and active help-seeking: systematic review and meta-analysis. Br J Psychiatry. Apr 2017;210(4):261-268. [CrossRef] [Medline]
    6. Santomauro DF, Mantilla Herrera AM, Shadid J, et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. The Lancet. Nov 2021;398(10312):1700-1712. [CrossRef]
    7. Vigo D, Thornicroft G, Atun R. Estimating the true global burden of mental illness. Lancet Psychiatry. Feb 2016;3(2):171-178. [CrossRef] [Medline]
    8. Bufano P, Laurino M, Said S, Tognetti A, Menicucci D. Digital phenotyping for monitoring mental disorders: systematic review. J Med Internet Res. Dec 13, 2023;25:e46778. [CrossRef] [Medline]
    9. Tutun S, Topuz K, Tosyali A, Bhattacherjee A, Li G. Explainable artificial intelligence for mental disorder screening: a computational design science approach. J Manag Inf Syst. Oct 2024;41(4):958-981. [CrossRef]
    10. Whitton AE, Hardy R, Cope K, et al. Mental health screening in general practices as a means for enhancing uptake of digital mental health interventions: observational cohort study. J Med Internet Res. Sep 16, 2021;23(9):e28369. [CrossRef] [Medline]
    11. Kingston D, Austin MP, Veldhuyzen van Zanten S, et al. Pregnant women’s views on the feasibility and acceptability of web-based mental health e-screening versus paper-based screening: a randomized controlled trial. J Med Internet Res. Apr 7, 2017;19(4):e88. [CrossRef] [Medline]
    12. Brühlmann F, Petralito S, Aeschbach LF, Opwis K. The quality of data collected online: an investigation of careless responding in a crowdsourced sample. Methods Psychol. Nov 2020;2:100022. [CrossRef]
    13. Ward MK, Meade AW, Allred CM, Pappalardo G, Stoughton JW. Careless response and attrition as sources of bias in online survey assessments of personality traits and performance. Comput Human Behav. Nov 2017;76:417-430. [CrossRef]
    14. Portillo-Van Diest A, Mortier P, Ballester L, et al. Ecological momentary assessment of mental health problems among university students: data quality evaluation study. J Med Internet Res. Dec 10, 2024;26:e55712. [CrossRef] [Medline]
    15. Meade AW, Craig SB. Identifying careless responses in survey data. Psychol Methods. Sep 2012;17(3):437-455. [CrossRef] [Medline]
    16. Gadiraju U, Kawase R, Dietze S, Demartini G. Understanding malicious behavior in crowdsourcing platforms: the case of online surveys. Presented at: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems; Apr 18-23, 2015; Seoul Republic of Korea. [CrossRef]
    17. Johnson JA. Ascertaining the validity of individual protocols from web-based personality inventories. J Res Pers. Feb 2005;39(1):103-129. [CrossRef]
    18. Carrier LM, Cheever NA, Rosen LD, Benitez S, Chang J. Multitasking across generations: multitasking choices and difficulty ratings in three generations of Americans. Comput Human Behav. Mar 2009;25(2):483-489. [CrossRef]
    19. Brower CK. Too long and too boring: the effects of survey length and interest on careless responding. Wright State University; 2018. URL: http://rave.ohiolink.edu/etdc/view?acc_num=wright1524614764948287 [Accessed 2025-07-22]
    20. Maniaci MR, Rogge RD. Caring about carelessness: participant inattention and its effects on research. J Res Pers. Feb 2014;48:61-83. [CrossRef]
    21. Bowling NA, Huang JL, Bragg CB, Khazon S, Liu M, Blackmore CE. Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. J Pers Soc Psychol. Aug 2016;111(2):218-229. [CrossRef] [Medline]
    22. McKay AS, Garcia DM, Clapper JP, Shultz KS. The attentive and the careless: Examining the relationship between benevolent and malevolent personality traits with careless responding in online surveys. Comput Human Behav. Jul 2018;84:295-303. [CrossRef]
    23. Costantini L, Pasquarella C, Odone A, et al. Screening for depression in primary care with Patient Health Questionnaire-9 (PHQ-9): a systematic review. J Affect Disord. Jan 15, 2021;279:473-483. [CrossRef] [Medline]
    24. Cotiga AC, Zanfirescu Ş, Iliescu D, Ciumăgeanu M, Gotca I, Popa CO. Psychometric characteristics of the Romanian adaptation of the GAD-7. J Psychopathol Behav Assess. Sep 2023;45(3):691-701. [CrossRef]
    25. Morin CM, Belleville G, Bélanger L, Ivers H. The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response. Sleep. May 2011;34(5):601-608. [CrossRef] [Medline]
    26. Harris KM, Gaffey AE, Schwartz JE, Krantz DS, Burg MM. The perceived stress scale as a measure of stress: decomposing score variance in longitudinal behavioral medicine studies. Ann Behav Med. Sep 13, 2023;57(10):846-854. [CrossRef] [Medline]
    27. Su Z, Liu R, Wei Y, et al. Temporal dynamics in psychological assessments: a novel dataset with scales and response times. Sci Data. Sep 27, 2024;11(1):1046. [CrossRef] [Medline]
    28. Bowling NA, Huang JL, Brower CK, Bragg CB. The quick and the careless: the construct validity of page time as a measure of insufficient effort responding to surveys. Organ Res Methods. Apr 2023;26(2):323-352. [CrossRef]
    29. Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. Detecting and deterring insufficient effort responding to surveys. J Bus Psychol. Mar 2012;27(1):99-114. [CrossRef]
    30. Erdem Kara B. Identifying the careless responders: a cross-country comparison on PISA 2018 dataset. Kastamonu Edu J. 32:396-407. [CrossRef]
    31. Jones A, Gillespie SM, Pennington CR, Strickland JC, Robinson E. Careless responding in online studies is associated with alcohol use: a mega-analysis. Psychol Addict Behav. Feb 2024;38(1):56-64. [CrossRef] [Medline]
    32. Curran PG. Methods for the detection of carelessly invalid responses in survey data. J Exp Soc Psychol. Sep 2016;66:4-19. [CrossRef]
    33. Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. May 2007;39(2):175-191. [CrossRef] [Medline]
    34. Faul F, Erdfelder E, Buchner A, Lang AG. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods. Nov 2009;41(4):1149-1160. [CrossRef] [Medline]
    35. Olusegun AM, Dikko HG, Gulumbe SU. Identifying the limitation of stepwise selection for variable selection in regression analysis. AJTAS. 2015;4(5):414. [CrossRef]
    36. Zahn-Waxler C, Hulle C. Empathy, guilt, and depression: when caring for others becomes costly to children. In: Oakley B, Knafo A, Madhavan G, Wilson DS, editors. Pathological Altruism. Oxford University Press; 2012:321-344. [CrossRef]
    37. Rudman LA, Glick P. The Social Psychology of Gender: How Power and Intimacy Shape Gender Relations. Guilford Publications; 2021. ISBN: 9781462546794
    38. Carstensen LL, DeLiema M. The positivity effect: a negativity bias in youth fades with age. Curr Opin Behav Sci. Feb 2018;19:7-12. [CrossRef] [Medline]
    39. Corrigan PW, Rao D. On the self-stigma of mental illness: stages, disclosure, and strategies for change. Can J Psychiatry. Aug 2012;57(8):464-469. [CrossRef] [Medline]
    40. Pescosolido BA, Martin JK, Lang A, Olafsdottir S. Rethinking theoretical approaches to stigma: a Framework Integrating Normative Influences on Stigma (FINIS). Soc Sci Med. Aug 2008;67(3):431-440. [CrossRef] [Medline]
    41. Curran T, Hill AP. Perfectionism is increasing over time: a meta-analysis of birth cohort differences from 1989 to 2016. Psychol Bull. Apr 2019;145(4):410-429. [CrossRef] [Medline]
    42. Smith MM, Sherry SB, Chen S, et al. The perniciousness of perfectionism: a meta-analytic review of the perfectionism-suicide relationship. J Pers. Jun 2018;86(3):522-542. [CrossRef] [Medline]
    43. Ward MK, Meade AW. Applying social psychology to prevent careless responding during online surveys. Appl Psychol. Apr 2018;67(2):231-263. URL: https://iaap-journals.onlinelibrary.wiley.com/toc/14640597/67/2 [CrossRef] [Medline]
    44. Roivainen E, Veijola J, Miettunen J. Careless responses in survey data and the validity of a screening instrument. Nord Psychol. Apr 2, 2016;68(2):114-123. [CrossRef]
    45. Lundberg I, Damström Thakker K, Hällström T, Forsell Y. Determinants of non-participation, and the effects of non-participation on potential cause-effect relationships, in the PART study on mental disorders. Soc Psychiatry Psychiatr Epidemiol. Jun 2005;40(6):475-483. [CrossRef] [Medline]
    46. Data from: temporal dynamics in psychological assessments: a novel dataset with scales and response times. Zenodo. URL: https://zenodo.org/records/10423537 [Accessed 2025-07-22]

    ‎
    CR: careless responding
    GAD-7: Generalized Anxiety Disorder-7 Scale
    ISI: Insomnia Severity Index
    PHQ-9: Patient Health Questionnaire-9
    PSS: Perceived Stress Scale
    PTI: Page Time Index
    WHO: World Health Organization

    Edited by Taiane de Azevedo Cardoso; submitted 13.03.25; peer-reviewed by Precious Usuemerai, Zain Ramzan; final revised version received 03.06.25; accepted 04.06.25; published 31.07.25.

    Copyright

    © Huawei Kuang, Lichao Zhu, Haonan Yin, Zihe Zhang, Biao Jing, Junwei Kuang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 31.7.2025.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.