The final sample consisted of 5430 adolescents (2769 [51%] females) at time 0 and 2117 (1109 [52.4%] females) at time 1 (approximately 7 months later). The participants were enrolled in Spanish secondary education grades, equivalent to US middle and high school, from grades 7 (12-13 years) to 12 (17-18 years). The average age of the sample at time 0 was 14.17 years (SD 1.50) and of the sample that participated at time 0 and time 1 was 13.99 years (SD 1.39).

This research used a non-experimental, transversal/longitudinal, quantitative, and descriptive-correlational design [34,35]. The UMH Project Evaluation Committee approved the study (reference no. DPS.JPR.02.17). Once the project was approved, quota sampling was carried out in 2 areas of southeastern Spain: the province of Alicante (PA) belonging to the Valencian Community, and the Autonomous Community of the Region of Murcia (RM), making a random selection of secondary schools based on ownership (public/nonpublic schools) and regional geographical areas (9 areas in PA and 21 in RM). After 100 schools were contacted, 13 from PA and 21 from RM agreed to participate, resulting in a total of 34 secondary schools (22 [65.2%] public and 12 [34.8%] nonpublic schools; 30 [87%] secular and 4 [13%] catholic schools).

Once the schools agreed to participate, signed informed consent was requested in writing from the parents/legal guardians of the adolescent participants and from the adolescents themselves, accepting participation in the research. The data collection was carried out in the schools and supervised by the research staff in person. The self-reporting assessment protocol was individually applied through the online survey tool LimeSurvey (LimeSurvey GmbH, Hamburg, Germany). Participation was voluntary, and the adolescents did not receive any incentive for their collaboration, while each school received a feedback report, including results by class group.

All analyses were conducted using IBM SPSS Statistics version 25 and Mplus 8.4 (Muthén & Muthén). Confirmatory factor analysis (CFA) was conducted to test the structural validity. Figure 1 represents the correlated 3-factor solution tested. We used a diagonally weighted least squares means and variance adjusted (WLSMV) model estimator due to a number of alternative responses and the nonnormality distribution [36]. We tested the model’s goodness of fit using the comparative fit index (CFI), the Tucker-Lewis index (TLI), and the root-mean-square error of approximation (RMSEA). A CFI of >0.90 and a TLI of >0.95 indicate an acceptable and an optimal fit, respectively [37], and RMSEA values of ≤0.10 indicate an acceptable fit [38].

Later, we tested whether the PSC-17-Y exhibits an invariant structure across gender and across time through longitudinal and multigroup measurement invariance analysis. In particular, 3 levels of invariance were tested: (1) configural (test whether all items load on the proposed factor), (2) metric (test whether item-factor loadings are similar across groups), and (3) scalar (test whether unstandardized item thresholds are similar across groups). In addition, within longitudinal measurement, invariance residuals covariances between the same item over time (eg, time 0 with time 1) were included. Thus, to indicate a significant decrement in fit when testing for measurement invariance, we used model comparison criteria of ΔCFI/ΔTFI ≥ 0.01 (ie, a decrease indicating the worst fit) [39] and ΔRMSEA ≥ 0.015 (ie, an increase indicating the worst fit) [40].

When there is scalar measurement invariance, the comparison of factor means across groups is permissible [41]. Consequently, we calculated gender differences. We also estimated the Cohen d index (standardized mean difference), which allows evaluating the effect size (ES) of the obtained differences [42].

The Cronbach α and the McDonald omega (ω) [43] were used to estimate the internal consistency of the PSC-17-Y since the McDonald ω is a better estimator of reliability than the Cronbach α [44].

Convergent and criterion validity was evaluated by calculating the correlation coefficients between the scores on the PSC-17-Y and different, well-established measures. Specifically, we tested the convergent validity with measures of distress (SEDS-S) and criterion validity with measures of well-being, QoL, and socioemotional skills (MHC-SF, KIDSCREEN-10 Index, SEHS-S). The Cohen criteria were used to estimate the ES of the correlations [42,45].

Finally, normative data for the PSC-17-Y were presented in the form of percentiles (75th and 90th). We also calculated the cut-off point of 15 for OVR, 5 for INT symptoms, 7 for EXT symptoms, and 7 for ATT, as proposed by Gardner et al [16,17], because these scores have not received evidence-based support in Spanish adolescents.

As the sample size determination for psychometric validation studies lacks clear recommendations [46], we determined the required sample size by allocating several observations 5-10 times greater than the variables [47]. Accordingly, the sample size needed ranged between 85 and 170 participants based on the number of items in the PSC-17-Y. Furthermore, according to the subject-to-item ratio method, a sample size of ≥1000 to perform exploratory factor analysis (EFA) or CFA would be excellent [46].

Findings of CFA and measurement invariance over time and across gender groups are summarized in Table 1. Since item 13 (“Me cuesta mucho cansarme”/“Acts as if driven by a motor”) had a low loading at the ATT subscale (0.127), we also tested the 16-item model, which slightly improved the fit. An optimal fit index was observed for both the 17- and 16-item models, with loadings from 0.348 (item 5) to 0.858 (item 6) and from 0.346 (item 5) to 0.859 (item 6), respectively. Longitudinal and multigroup measurement invariance was found for the 17- and 16-item models, which means that the structure, loadings, and intercepts of the PSC-17-Y were invariant over time and across gender groups. Although the 16-item version was slightly stronger than the 17-item version, we concluded that the minimal improvement in accuracy was less important than being able to use the 17-item model, which is the internationally recognized version and facilitates comparisons with other studies. For this reason, subsequent analyses were performed with a 17-item version.

The correlation matrix of the items can be seen in Table 2.

As can be seen in Tables 3 and 4, the reliability coefficients for all measures ranged from .64 to .76 (McDonald ω). Regarding gender differences among total scores, females showed significantly higher scores on general social-emotional distress (d=0.34) and INT (d=0.42) scales than males, with a small-to-medium ES. Concerning the differences in the EXT scale, males showed more symptoms of EXT problems than females, with a small ES (d=0.12). Although the scores for well-being and socioemotional competencies were lower in females than in males, the ESs were low (d ranged from 0.08 to 0.15). Finally, males showed higher HRQoL levels than females, with an ES of 0.35.

According to Table 5, the intercorrelation between the PSC-17-Y subscales was moderate, indicating that this measure is composed of 3 differentiated and mutually associated factors. Regarding the relationship between PSC-17-Y subscales and the remaining measures, the highest positive correlations were between the INT subscale and the measure of distress (large ES), while the association of the ATT and EXT subscales with distress was medium. Similarly, the correlation was higher (and negative) between the INT subscale and the HRQoL (large ES) than between different types of well-being and socioemotional competencies (moderate to large). However, the correlation between ATT and measures of well-being, socioemotional competencies, and the HRQoL was significant and negative, with a small-to-medium magnitude. Finally, the EXT subscale presented a small-to-medium association with the different positive measures.

Concerning normative information for PSC-17-Y subscales, each subscale is scored by the sum of its items. The adolescent’s score on the scale can then be used to obtain the corresponding percentile score. The normative information for each of the 3 PSC-17-Y subscales and total scores for the whole sample are shown in Table 6. We also included percentile scores for PSC-17-Y OVR scores to facilitate international comparisons.

The values obtained in our study using the international cut-off point of 15 for OVR, 7 for ATT, 5 for INT symptoms, and 7 for EXT symptoms, as proposed by Gardner et al [24], were 20.7%, 15.1%, 29.7%, and 5.1%, respectively. The 90th percentile indicated that 11.6% of the sample scored above this cut-off point of 17 for OVR on the PSC-17-Y, 26.3% of the participants exceeded the 75th percentile for the PSC-17-Y total score. Regarding specific symptoms, 15.1%, 10.2%, and 10.6% of the participants scored above the 90th percentile on ATT, INT symptoms, and EXT symptoms, respectively. The data at the 75th percentile or quartile 1 indicated that 29.1%, 28.7%, and 32.0% of the participants exceeded the cut-off points for ATT, INT symptoms, and EXT symptoms, respectively.

This study aimed to delineate the psychometric properties of the PSC-17-Y in Spanish adolescents. As expected, this study found evidence of a 3-factor solution, as in the original English version, and also gave evidence of reliability and validity (structural, convergent, and criterion) to assess several psychopathology symptoms among adolescents. However, item 13 showed a lower loading (0.127) than the remaining items. This finding is consistent with Bergmann et al. [22], who validated the PSC-17-Y in English and reported a relatively low factor loading (0.233) for this item. Following the same logic as these authors, we maintained item 13 in the final set of items for the PSC-17-Y in Spanish, since it is important to keep the measure as simple as possible for respondents and clinicians to complete, score, and interpret and since the parent- and youth-reported short forms were identical except for this 1 item. We found that the inclusion of item 13 had a negligible impact on the psychometric properties of the PSC-17-Y in Spanish. As a result, we elected to add item 13 to the 16-item model and recommend a 17-item short form of the PSC-Y that uses the same 17 items on the same 3 subscales as the parent-reported PSC-17. Furthermore, we hypothesized that the problem with this item could be the wording and that future studies might review it to improve the saturation in the factor. The PSC-17-Y also showed an invariant structure across both genders, again consistent with Bergman et al [22].

Additionally, our study provided the first evidence, as far we know, for longitudinal invariance, indicating that the PSC-17-Y in Spanish adolescents is stable over time. This finding implies that it is reasonable to conclude that growth or development in observed scores over time can be attributed to actual development or changes in the construct under investigation, not measurement problems [48]. Further studies that replicate our study over more extended periods are, of course, needed.

Once gender and longitudinal invariance were tested, this study provided new evidence on gender-attributable differences in the PSC-Y scales. The gender differences found in this study were consistent with the overwhelming prior research establishing that females are more likely to express INT symptoms and males are more likely to express EXT symptoms [49-51]. Furthermore, the small ES found in these gender differences is also consistent with other ones, which highlighted the small magnitude of gender differences in INT problems among children and adolescents [52]. These findings suggest that gender differences should be considered when pediatric and mental health professionals interpret PSC-17-Y results. Furthermore, gender is a crucial variable in the relationship between INT symptoms and suicide among adolescents, increasing this risk in females [53].

Regarding reliability evidence, our study showed McDonald ω values between .64 and .76, which are slightly lower than those shown by previous studies. Gardner et al [16] reported high internal consistency (.79 for INT symptoms, .83 for EXT symptoms, and .83 for ATT), and in the same order, Bergmann et al [22], with 16 items, found consistency values of .81, .74, and .69, respectively. The lowest internal consistency value was ATT (.65 and .64 for females and males, respectively). The removal of item 13 could improve the internal consistency of the ATT subscale slightly to values of .70 and .71 for females and males, respectively but at the cost of losing the original 17-item structure of the questionnaire.

Concerning other sources of convergent and criterion validity evidence of the PSC-17-Y, all 3 subscales correlated positively with the measure of distress and negatively with well-being, HRQoL, and social-emotional competencies, indicating a higher correlation between PSC-17-Y INT problems and the remaining convergent and criterion validity measures. This finding is consistent with previous studies on the PSC-17-Y, such as Parker et al [23], who reported screening validity of the PSC-17-Y in terms of higher scores on the PSC ATT and INT subscales among youth with any lifetime mental health diagnosis, as well as that ATT and INT subscale scores (but not EXT) are significantly correlated with psychosis, depression, and anxiety disorders. Thus, both our study and Parker et al’s [23] provide support for the convergent and discriminant validity of the PSC-17-Y.

Finally, regarding percentile scores, score distributions showed a positive asymmetric distribution, but these normative data can help locate specific and general psychopathological problems among adolescents. Our rates would be suggestive of adolescents presenting scores compatible with mental health symptoms likely being in the clinical range. These data are equivalent to those reported in previous studies using different versions and cut-off scores for the PSC, PSC-17, and PSC-17-Y: 5%-25% of children were screened positive [15,16,54-56]. For example, a study using the 90th percentile score on the PSC found that 10.4% of children had problems on the OVR scale [54]. Additionally, the prevalence rates found in our study are consistent with wide international reviews on estimates of mental health prevalence among adolescents [1].

The absence of an equivalent, well-established measure of INT, EXT, and ADHD measures; the absence of data on the area under the curve at optimal cut-off points in this study; and the sample's representativeness because we recruited the sample from the southeast of Spain exclusively were limitations of this study. In addition, there were only 2 waves of assessment in a short period of time (7 months), so it is recommended that future studies replicate the longitudinal invariance findings over longer intervals.

This study showed that the PSC-17-Y is a useful, reliable, and valid ultrabrief screening measure for detecting mental health problems in adolescents and can be administered over the internet. More specifically, this study provided evidence of the reliability and validity (structural and convergent-discriminant) of the Spanish version of the PSC-17-Y for adolescents.

Finally, these findings are significant for the scientific community. Therefore, this work has allowed us to extend the evidence of the validity of the PSC-17-Y to another language and country (Spain) in a large sample of adolescents, where scores were invariant over time irrespective of gender. This is a requirement that few instruments meet or for which evidence has been provided. All of this supports the reliability of the PSC-17-Y’s assessments and its use in clinical contexts, such as monitoring the development of symptomatology. The fact that the PSC-17-Y is a tool that is easy to administer is another support for its use in clinical contexts.

Having instruments such as the PSC-17-Y with established reliability meets an especially important need during COVID-19 times, which have been characterized by an increase in mental health problems among children and adolescents [3-5] and a possibly greater need for case identification and outcome measurement.

Lastly, the results also support the use of the PSC-17-Y in longitudinal research, for example, for the study of the temporal trajectories of psychopathology in children, facilitating, among other things, reliability in the evaluation of the effectiveness of treatments. In addition to its usefulness in research, the PSC-17-Y is an instrument with applicability in the clinical setting, specifically in both primary care and specialty mental health units, as a screening tool for mental health problems in children and adolescents that is valid for monitoring changes in functioning over time.