This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Early and effective identification of developmental disorders during childhood remains a critical task for the international community. The second highest prevalence of common developmental disorders in children are language delays, which are frequently the first symptoms of a possible disorder.
This paper evaluates a Web-based Clinical Decision Support System (CDSS) whose aim is to enhance the screening of language disorders at a nursery school. The common lack of early diagnosis of language disorders led us to deploy an easy-to-use CDSS in order to evaluate its accuracy in early detection of language pathologies. This CDSS can be used by pediatricians to support the screening of language disorders in primary care.
This paper details the evaluation results of the “Gades” CDSS at a nursery school with 146 children, 12 educators, and 1 language therapist. The methodology embraces two consecutive phases. The first stage involves the observation of each child’s language abilities, carried out by the educators, to facilitate the evaluation of language acquisition level performed by a language therapist. Next, the same language therapist evaluates the reliability of the observed results.
The Gades CDSS was integrated to provide the language therapist with the required clinical information. The validation process showed a global 83.6% (122/146) success rate in language evaluation and a 7% (7/94) rate of non-accepted system decisions within the range of children from 0 to 3 years old. The system helped language therapists to identify new children with potential disorders who required further evaluation. This process will revalidate the CDSS output and allow the enhancement of early detection of language disorders in children. The system does need minor refinement, since the therapists disagreed with some questions from the CDSS knowledge base (KB) and suggested adding a few questions about speech production and pragmatic abilities. The refinement of the KB will address these issues and include the requested improvements, with the support of the experts who took part in the original KB development.
This research demonstrated the benefit of a Web-based CDSS to monitor children’s neurodevelopment via the early detection of language delays at a nursery school. Current next steps focus on the design of a model that includes pseudo auto-learning capacity, supervised by experts.
The early detection of neurodevelopmental disorders in childhood is a key task to support diagnosis and treatment processes [
Estimates of disability predominance in childhood vary due to differences in definition and the wide range of methodologies and existing measuring instruments [
The mental health, social, and behavioral developmental needs of very young children have gained awareness in the last 10 years [
Most children achieve good verbal communication by the age of 3 years [
Ygual et al discovered a significant correlation between teachers’ observations and criteria scores on intelligibility, literal understanding of sentences, grammatical expression, and lexical richness [
The main objective of this research was to evaluate the deployment of a Web-based system for efficient screening of language disorders at the early stages of a child’s development. The implemented solution is a Clinical Decision Support System (CDSS), called “Gades”, whose use was widely tested in a nursery school. This paper discusses the results obtained from the Gades validation to provide professionals with real-time knowledge on early identification of 146 children with possible language disorders. The previous publication of Gades’ user requirements, implementation, deployment, and validation showed high success from a usability point of view [
The development of a knowledge base (KB) [
Gades KB is based on an ontology that integrates a child’s language acquisition information according to age. It has over 100 rules to generate alerts and/or alarms in case deviations from the child’s development stage are detected. The initial version of Gades KB was built between September 2011 and April 2013, according to the experience of a multidisciplinary team. Two language therapists, a neuropediatrician, a neonatologist, and three engineers supplied inputs for Gades KB, updating original versions of the Denver Developmental Screening Test. The team used CommonKADS (CK) methodology [
The Gades KB evaluates four areas of speech and language development: Sensory Reception (SR), Speech Perception (SP), Speech Production (SPD), and Pragmatic (P). The Gades system relies on Gades KB to support early detection of language disorders.
The Gades system aims to enhance early detection of children with language disorders. Its evaluation process involved 146 children attending Legamar Nursery School. The Gades KB integrates all the knowledge and logic associated with the decisions supported by the system. The potential outcome is the suggestion of early referral to specialists if a child under 6 years old may have a language disorder.
Gades home page and main functionality.
A Nursery School Language Therapist (NSLT), employed at Legamar Nursery School, Madrid, evaluated the Gades system in the spring of 2013. A total of 63 boys and 83 girls participated in the study; 94 children from 0 to 3 years old and 52 children in the 4-6 years stage. The number of enrolled children in the 0-3 years group was higher since the early detection of language disorders is a research priority in this developmental period. The entire staff of educators at Legamar observed and evaluated the behavior of the children, by following the questions suggested by Gades. The average age of the 12 teachers at Legamar was 34 years old. All the educators and the NSLT were women with little background in information technology.
The study started 6 months after the beginning of the school year to ensure that teachers had enough information about their pupils.
Evaluation method for early detection of language disorders.
In terms of age, the number of subjects involved in the study at age 3 was higher (48 children from the sample of 146 subjects), since there was a higher number of children enrolled at nursery school at this age. From a prevention point of view, the detection of language disorders before the age of 3 years is a key issue as it directly influences the Quality of Life related to Health (QoLrH). Hence, 100% of the children between 0 and 3 years at Legamar participated in the research. However, only 37.1% (52/140) of the children between 4-6 years were included, as early detection is not significant at this stage. In the 0-3 years stage, a population of 94 children participated in the study, and the distribution of children who attended the nursery school was 59% (55/94) girls and 41% (39/94) boys. In the 4-6 years stage, 52 children of the nursery school received language evaluation by Gades: 54% (28/52) were girls and 46% (24/52) were boys.
Number of language evaluations conducted by age and sex.
|
Year 1 | Year 2 | Year 3 | Year 4 | Year 5 | Year 6 | Total |
Boys | 5 | 14 | 20 | 7 | 14 | 3 | 63 |
Girls | 6 | 21 | 28 | 14 | 8 | 6 | 83 |
Total | 11 | 35 | 48 | 21 | 22 | 9 | 146 |
The following section details the language evaluation results, obtained from the feedback supplied by the NSLT and teachers who participated in the research.
Although the NSLT agreed with all the questions for several months, she pointed out the need to refine the KB in order to improve Gades’ decisions. For example, discrepancies arose between Gades’ decision and the NSLT—the therapist recommended postponing some questions, which are not yet required for some months in the second year. At years 4, 5, and 6, the NSLT requested adding questions related to the articulation of the language and pragmatic ability. The KB refinement will require cooperation between the NSLT at Legamar and other speech therapists.
A key result obtained from the Gades evaluation was the identification of possible language delays in 7 children who had not previously caused alarm to either the NSLT or his/her educator. These cases require a formal diagnosis process in order to compare the system’s decision with traditional methods. These children had been enrolled for a few months at Legamar and the speech therapist had not noticed any delays.
The NSLT identified discrepancies in cases 1, 2, and 3, described in
Number of language evaluations performed and therapist decisions.
Stage | Months of the questions | Year (age) of the child | Total number of cases | Number of cases where NSLTa accepted Gades decision (%) | Number of cases where NSLT did not accept Gades decision (%) |
0-3 years | 0-12 | Year 1 | 11 | 11 (100.0) | 0 (0.0) |
13-24 | Year 2 | 35 | 28 (80.0) | 7 (20.0)b | |
25-36 | Year 3 | 48 | 48 (100.0) | 0 (0.0) | |
|
Total |
|
94 | 87 (92.6) | 7 (7.4) |
4-6 years | 37-48 | Year 4 | 21 | 13 (61.9) | 8 (38.1)c |
49-60 | Year 5 | 22 | 21 (95.5) | 1 (4.5)c | |
61-72 | Year 6 | 9 | 1 (11.1) | 8 (88.9)c | |
|
Total |
|
52 | 35 (67.3) | 17 (32.7) |
Total |
|
|
146 | 122 (83.6) | 24 (16.4) |
aNLST: nursery school language therapist.
bThe age in months of some questions is incorrect. Therefore, the NSLT believed that some questions should be delayed.
cIncorporating additional questions related to the articulation of language and pragmatics is required.
Decision of the NSLTa on 7 new cases of children with possible language delays.
Case | Age and sex of the child | NSLT opinion |
1 | 21 months – Girl | She walked at 19 months and she is very shy and inhibited. |
2 | 18 months – Girl | She was brought to early attention. She was detected with a motor delay. |
3 | 26 months – Girl | She had begun motoric treatment with 8 months. |
4 | 34 months – Girl | After Gades evaluation she started speech and language intervention. |
5 | 39 months – Boy | After Gades evaluation he started speech and language intervention. |
6 | 36 months – Boy | After Gades evaluation he started speech and language intervention. |
7 | 42 months – Boy | After Gades evaluation he started speech and language intervention. |
aNLST: nursery school language therapist.
The NLST accepted Gades’ decisions in 93% (87/94) of the 0-3 years cases and 67% (35/52) at the 4-6 years stage.
The accuracy of the KB questions after the language evaluation process at Legamar is grouped by year in
There are more questions at the 0-3 years stage, because early detection of language disorders is critical at this developmental period. The child evolves very quickly at this stage and the KB requires higher accuracy to analyze the evolution status. There are not questions for all months in the 4-6 years stage, because the therapists determined during the process of KB construction different age ranges to support a specific assessment. Questions are structured according to evaluative items at 42, 45, 46, 48, 54, 60, 66, 69, and 72 months.
The fourth column of
The NSLT indicates that the separation between speech perception and pragmatic is minimal. Besides, pragmatic disorders often coexist with other language problems such as vocabulary development or grammar. Pragmatic problems have lower social acceptance. The NSLT considers that the correct evaluation of pragmatics is important to avoid, or to treat as early as possible, a future neurological disorder.
Accuracy and refinement of the knowledge base questions.
Year of the questions | Months of the questions | Number of KB questions | Evaluated areas | NSLTa opinion |
Year 1 | 0-6 | 18 | SRb - SPc - Pd | Questions OK |
7-12 | 23 | SP - SPDe - P | Questions OK | |
Year 2 | 13-18 | 17 | SP - SPD - P | Questions OK |
19-24 | 18 | SP - SPD - P | Disagree with some questions | |
Year 3 | 25-30 | 13 | SP - SPD | Add questions of P |
31-36 | 10 | SP - SPD | Add questions of P | |
Year 4 | 37-42 | 5 | SP - SPD - P | In SP more questions of articulation language |
43-48 | 11 | SP - SPD - P | Questions OK | |
Year 5 | 49-54 | 3 | SP - SPD | In SP more questions of articulation language |
55-60 | 4 | SP - SPD - P | Add questions of P | |
Year 6 | 61-66 | 3 | SP - SPD - P | Add questions of SP and SPD |
67-72 | 9 | SP - SPD - P | Questions OK |
aNSLT: nursery school language therapist
bSR: sensory reception
cSP: speech perception
dP: pragmatic
eSPD: speech production
Knowledge base accuracy by stage.
Knowledge base accuracy by age of the child.
The language evaluation identified a total of 88 cases with a normative result (OK), a total of 35 cases with a referral to a specialist (Alarm), and a total of 23 with a follow-up pediatric visit (Alert). According to the sex of the child, a total of 83 girls were evaluated with the following results: 13 cases (16%) with Alarm, 17 cases (20%) with Alerts, and 53 (64%) cases with OK. A total of 63 boys were evaluated with the following results: 10 cases (16%) with Alarm, 18 cases (29%) with Alerts, and 35 cases (56%) with OK.
Evaluations percentage by result and sex.
There were more language evaluation cases of children with normal development (OK) in the third year as
The maximum number of language evaluation cases of children with an Alert happened in the second year. The percentage of Alerts in years 1, 2, and 3 is higher than in years 4, 5, and 6. The higher percentage of Alarms at the 4-6 years stage is not significant because the NSLT detected some semantic mistakes in some KB questions. The current refinement of the KB is taking into account the opinion of the group of experts who originally participated in the KB construction and the evaluation results of Gades presented in this work. For this reason, the KB questions will not be reviewed until they can include the review from all experts, according to the analysis of the results of the Gades evaluation process summarized in
Evaluations percentage by result and age.
Although previous studies relate positive experiences about the educator role in the process of language acquisition [
Language evaluation performed by Gades is consistent with the higher incidence of language impairment between boys and girls identified in scientific studies [
The higher accuracy of Gades at 0-3 years than at the 4-6 years stage is directly related to the research rationale that gave priority to earlier and more precise detection of language disorders from 1 to 3 years old. This outcome does not invalidate the use of the system from 4 to 6 years old but paves the way for better performance at this stage after the KB refinement. Furthermore, the comparison between Gades’ results and the expert’s feedback shows that linguistic functionality from 30 months cannot be clearly formalized though a specific item. The NSLT pointed out the need to enhance the analysis of pragmatic skills at each contextual scenario from 3 years onward. The deeper specific review of Gades KB, including more specific questions of pragmatic evaluation, will require a second wide scale evaluation to measure the improvements of effectiveness and reliability for language evaluation results in the 4-6 years stage.
The definition of two incremental phases in the evaluation method of Gades’ capacity for early detection of language disorders helped to provide users with requested information packages at the stage of need. The design of traditional questionnaires for parents and teachers, adapted from Gades KB, made it easier to assess the language use of the child in different interaction environments [
The early detection of language disorder tools has well-documented limitations in the specialized literature [
The limitations of the aforementioned issues do not invalidate the current research as a high number of studies verified the existence of a significant correlation between the observations of teachers with diverse linguistic skills and the punctuations obtained by his/her pupils in different standardized evaluation tests. All of them used questionnaires focused on general or punctual aspects of linguistic processing as Gades inputs did [
Although the Legamar school is a private entity, its demographic data show a realistic potential to scale up the trials and results of this research to other public and private centers. It assists children from middle-class families with a normal distribution of gender, age, and parental income. The extrapolation of the study to other classes in public or private schools does not require methodological changes or a team of professionals to be involved. A higher number of children with language delays is expected to be obtained in a center where speech therapists do not belong to the regular staff. If the school does not have an NSLT, the method presented in this study cannot be applied equally and the NSLT functionality could fall to educators or another professionals.
Consulted language experts stated that the extrapolation of the study to another region where other dialects or languages are spoken may obtain similar results. Children all over the world learn more than one language without developing speech or language problems. Even though bilingual children develop language skills just as other children do [
Finally, we have not detected false negatives in any stage (0-3 or 4-6 years). The false positive rate in stage 0-3 years has been low. However, we had a high ratio of false positives in the 4-6 years stage. The NSLT detected the main causes of this ratio to be related to semantic mistakes in the questions involved. We are currently in a refinement process to solve this situation.
This research details an innovative solution to support knowledge-based detection of language disorders in children aged 0 to 6 years at nursery schools. The solution provides nursery school educators with a monitoring tool to assess the degree of language acquisition in their students. In spite of the additional workload faced by the educators, the school highlights the benefits of this type of monitoring for children.
The results of the evaluation at the Legamar Nursery School show that several children identified by Gades as having a possible language delay had not previously caused alarm to either the school therapist or to his/her educator. Further, a large number of children identified by Gades were also identified by the NSTL, especially in the 0-3 years stage. These results lead us to conclude that this kind of Web-based CDSS can undertake early detection of language delays in children at a nursery school with the support of their teachers, thus improving the neurodevelopmental follow-up.
In the process of early detection of language disorders, it is necessary to have not only a very specific knowledge, but also, a capacity for suitable observation. Therefore, we can summarize that Gades can be an effective CDSS for use by speech therapists and school psychologists in the rapid detection of children with difficulties in language development; Gades guides educators in the observation required for detection and also promotes the stimulation of skills aimed at diminishing and even preventing the appearance of these disorders; and Gades can be a collaboration tool involving parents and primary care pediatricians in the process of language evaluation.
Other conclusions of this research suggest the need to include supervised learning capacities in Gades. The learning functionality requires the definition of a specific model that allows a proper mix of automation and experts’ supervision. Experts will be able to update Gades KB easily, taking into account the suggestions triggered automatically by the system. These suggestions will come from significant samples of real use cases. The following complementary proposed actions will improve the capacity of Gades detection in order to promote better health status of children. First, questions related to difficulties in the sound articulation domain can be incorporated. There can be situations in which children do not have a problem in language development, rather the problem derives from difficulties with sound discrimination. These questions will be studied by a multidisciplinary team of experts skilled in the relevant areas. Second, complementary evaluation in other areas outside of the school can be included. The observation capacity of the teachers, though it is considerable, does not include all aspects that would be desirable at the time of establishing a proper diagnosis. To be able to analyze other contexts outside of school, there is a version being adapted for primary care pediatricians and analysis has also begun for a possible version for parents. Third, new questions to improve aspects in the language domain can be refined and added. In the 4-6 years stage, the need for major refinement was detected in questions related to the pragmatics. Currently, we are working on it with the NSLT at Legamar.
Furthermore, the authors have defined a new concept called “Internet of Toys”. It deals with the possibility of obtaining information about the child’s development through his or her natural interaction with toys. This new interaction paradigm might provide Gades with the capacity to acquire real-time data in order to improve its reasoning performance. Thus, the system could improve its effectiveness thanks to the very earliest utilization of information related to the behavior of the child. Data monitored via the expected interaction of children with certain toys could enhance Gades’ reliability with more critical information.
Attention-Deficit/Hyperactivity Disorder
Children's Communication Checklist
clinical decision support system
CommonKADS
knowledge base
Language Intervention Centre
nursery school language therapist
pragmatic
Quality of Life related to Health (QoLrH)
speech perception
speech production
sensory reception
World Health Organization
The authors wish to thank María Peñafiel, principal at Legamar School (Madrid, Spain); Mª Teresa Ferrando, MD, neuropediatrician at the Quirón Hospital of Madrid; Paloma Tejeda of the Language Intervention Centre (LIC) Universidad Autónoma de Madrid; and José Arizcun, MD, expert neonatologist in developmental disorders and child disability.
This article is part of research we’re conducting in the Talisec+ project (a framework for knowledge-based management of accessible security guarantees for personal autonomy; TIN2010-20510-C04-01), supported by the Ministry of Education and Science of Spain through the National Plan for R+D+I (research, development, and innovation).
None declared.