This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
DESDE-LTC (Description and Evaluation of Services and DirectoriEs for Long-Term Care) is an international classification system that allows standardized coding and comparisons between different territories and care sectors, such as health and social care, in defined geographic areas. We adapted DESDE-LTC into a computer tool (DESDE-AND) for compiling a directory of care services in Andalucia, Spain.
The aim of this study was to evaluate the maturity of DESDE-AND. A secondary objective of this study is to show the practicality of a new combined set of standard evaluation tools for measuring the maturity of health technology products.
A system for semiautomated coding of service provision has been co-designed. A panel of 23 domain experts and a group of 68 end users participated in its maturity assessment that included its technology readiness level (TRL), usability, validity, adoption (Adoption Impact Ladder [AIL]), and overall degree of maturity [implementation maturity model [IMM]). We piloted the prototype in an urban environment (Seville, Spain).
The prototype was demonstrated in an operational environment (TRL 7). Sixty-eight different care services were coded, generating fact sheets for each service and its geolocation map. The observed agreement was 90%, with moderate reliability. The tool was partially adopted by the regional government of Andalucia (Spain), reaching a level 5 in adoption (AIL) and a level 4 in maturity (IMM) and is ready for full implementation.
DESDE-AND is a usable and manageable system for coding and compiling service directories and it can be used as a core module of decision support systems to guide planning in complex cross-sectoral areas such as combined social and health care.
COVID-19 has dramatically exposed the cracks of otherwise highly connected systems such as social and health services. For example, nursing homes are both health-related organizations and social housing facilities for highly vulnerable people [
However, the aggregation and comparison of services from a whole system’s perspective is not an easy task, as shown by previous developments in disability or functional dependency in Europe [
The difficulty of comparability between services across sectors is related to a number of factors including [
DESDE-LTC (Description and Evaluation of Services and DirectoriEs for Long-Term Care) is a standardized coding system for care services to be used across sectors and tools such as health and social directories, atlases of care, and decision support systems [
Despite its uniqueness and tested psychometric properties, the applicability of DESDE-LTC is limited due to the significant effort in data gathering that requires surveys and direct interviews to managers of local services, as well as an intensive training for coders [
This study evaluates the maturity of DESDE-AND, an online computer system to produce a semiautomatic classification of services for compiling standard directories of care based on DESDE-LTC. “Maturity” has been defined in software development as the potential capability of the process of implementation of a new product and the consistency with which it could be applied in projects throughout the target organization [
This study follows a co-design/hybrid approach. The co-design process was based on information provided by a domain expert panel and key end users. The initial expert panel consisted of 28 professionals from different social and health areas of the regional public Agency for Social Services and Dependency of Andalucia (ASSDA). This group included experts in information technologies, policy planning, management, and service research. Training on the DESDE-LTC coding and on the use of the DESDE-AND prototype (version 0.0) was provided to all the experts. From this group, 23 domain experts participated in the panel discussion of the maturity study. The group of end users included the managers of the 68 services that participated in the demonstration study.
The tool has been designed in a collaboration partnership between Psicost Research Association and the public ASSDA. Within the regional Ministry of Equity and Social Policies of Andalucia (Spain), ASSDA coordinates the social public and private care and the welfare system for the 8.5 million inhabitants living in this Spanish region [
The DESDE-LTC is a tool developed from the European Service Mapping Schedule [
The implementation maturity model (IMM) uses the 5 maturity levels from the capability maturity model to assess and determine the degree of maturity of implementation processes in software engineering [
Readiness is the level of preparedness for the application of a new scientific knowledge for commercialization or generalized use in the real world. The technology readiness levels (TRLs) are a systematic measurement that supports assessments of the maturity of a particular technology during the acquisition phase of a program. Nine levels are considered [
The Usability Checklist consists of 15 items that assess relevance (meaningfulness, novelty, and potentiality of the new scientific knowledge to the target audience), acceptability (the degree to which a new application or product is agreed or approved by the target audience), functionality, security, practicality (related to implementation, training requirements, and complexity of evaluation, and the analysis, interpretation, and reporting of the data), efficiency (practicality in relation to effort, time, and costs), training, and agreement. It uses a 10-point Likert scale, in which 0 is none, and 10 is the maximum score of each assessed variable. The questionnaire can be adapted to the characteristics of any new tool and the target end users. It has been used in the assessment of decision support systems [
The Adoption Impact Ladder (AIL) is an inventory for evaluating the level to which the target organization has taken the new knowledge or its application as its own. It uses a quasi-ordinal scale with 7 categories, namely, (0) no adoption; (1) awareness; (2) assimilation; (3) conversion (or translation); (4) allocation; (5) provision; and (6) routinization (or monitoring). It has been previously used in Australia for assessing the impact on the practice of a state policy program (Ed-LinQ Program in Queensland) [
This study consists of 4 stages comprising 11 steps and 4 measurement instruments (
Stages of the DESDE-AND study methodology. AIL: Adoption Impact Ladder; IMM: implementation maturity model; TRL: technology readiness level.
The tool was built as an online computer system that guides “end users” step by step. The end user of this computer tool was defined as the manager or other decision maker of a service depending on the regional Ministry of Equality, Social Policies and Conciliation of Andalucia.
The development of the computer tool used architectural standards and was specifically programmed using JAVA. The software developer company (Guadaltech S.L.) prepared the user/administrator manuals, installation guides, implementation manual, and source code.
Two additional computer modules were developed to explore the combined use of DESDE-AND as a core module of a DSS (a data export system linked to geographic information system [GIS] for visualization and spatial analysis) and to assess its usability.
The validation process of the DESDE-AND was part of level 4 of the technology readiness. A mini-Delphi panel of 23 domain experts produced the expert’s knowledge base [
This stage included a pilot study and the assessment of the DESDE-AND practicability and manageability. The pilot study was carried out in the Seville metropolitan area. A total of 188 services providing care authorized from the regional Ministry of Equality, Social Policies and Conciliation of Andalucia were identified in the target area and invited to participate in the study. This list included 13 services located outside the target area that provided types of care that were not available in the city of Seville but that could be used by their residents. As much as 73 services accepted to participate but 5 of them were not able to complete the information requested. Finally, 68 services participated in the demonstration study.
In order to perform the coding, the managers of the services (end users) identified the basic social and health care units of each service and automatically assigned them a “main type of care” code according to the decision tree system offered. This code was subsequently confirmed by the personnel responsible for public administration.
The panel of experts carried out the analysis of the results of the coding in the selected services, which allowed the preparation of a report with recommendations for improvement of the instrument.
The recommendations by the panel of experts allowed modifications and improvements to be made for the development of the instrument in its final version (version 1.0).
The Psicost Usability Checklist, which was first used to assess the feasibility of the prototype by the 23 domain experts, was completed at the end of the demonstration phase by the 68 end users to assess the usability of the tool. The adoption by the target organization was measured by 3 experts in service evaluation and in the use of the decision support tools (FA-T, MG-C, and JS-P). The level of adoption was assessed in the individual services and in the target organization using AIL.
Every stage of the development process of the tool corresponded to a TRL. At the completion of the final version (DESDE-AND version 1.0), the group of domain experts assessed the level of technological maturity reached by the tool (TRL), which was compared with the one provided by the core team. Finally, the overall level of maturity was assessed using the IMM.
The schemas and algorithms necessary for the development of the computer-administered version 0.0 (prototype) were designed in the first stage. The classification process used specific question answering algorithms with 2 or more logical answers in a decision tree. Subsequently, a user interface for the web application was created with a user-friendly design, using a set of images and graphic objects to represent the available actions (
User interface of DESDE-AND web application.
A modular system with minimal interconnection between modules was used to develop the software and user interfaces.
The modular operability of DESDE-AND was tested with a supplementary module designed for exporting georeferenced information from the coding generated in the service inventory to any GIS and online cartographic viewers. This has been used to demonstrate the capacity of DESDE-AND data to be exported to other modules within a composite decision support system. GIS is the most common visualization system in DSS [
The feasibility of the tool was assessed using the Usability Checklist. A total of 23 domain experts completed the instrument (
Experts’ responses to the usability questionnaire. Each cell represents the qualitative score or answer of a different expert, and the last column shows the mean of their quantitative score by question and domain.
The interrater reliability analysis was performed after the pilot study in the 68 services. We assessed the coding concordance between ratings provided by the 68 service managers and the experts. The agreement was 0.89 with a Kappa index of 0.33 (95% CI). We also calculated the predictive validity of the tool (sensitivity, specificity, and predictive values), and observed high results in specificity (89.39%; 95% CI 81.21-97.58), sensitivity (100%), and predictive value (100%).
The pilot study included 68 services that agreed to participate and were classified using the DESDE-AND. The remaining services did not respond to the request, because they were closed or because they did not have adequate human or technical resources. Six of them started the process but did not complete the coding because they did not have personnel with basic training in information technology and communication. The 68 coded services were designed for different target groups: child and adolescents (n=5, 7%), older adults (n=40, 59%), persons with disabilities (n=16, 24%), drug addiction problems (n=5, 7%), and community social services (n=2, 3%).
The assessment of usability of the final version of DESDE-AND was performed by the service managers for the 68 services of the pilot study. Average scores were 7.6 for relevance, 7.8 for functionality, 7.8 for usability, 6.9 for efficiency, and 2.4 for resistance of the staff to perform the coding. More than half of the of the professionals (13/23, 57%) who coded the services considered that training was necessary at the user level in order to use the DESDE-AND.
We received 63 improvement suggestions and comments on the prototype to produce version 0.1. Of these, 52 recommendations were accepted and, among them, the corrections considered were performed to ensure gender equality. Eleven proposals were discarded due to technical issues, programming, or errors in the use of the tool.
The results of the pilot study in Seville were integrated into the open GIS of Google Maps [
Screenshots of the web-mapping application (Google Maps) for the geographic visualisation of the services coded through DESDE-AND tool for social care planning in Andalucia (Spain).
The level of adoption was assessed in the individual services registered for the study as well as in the main target organization (ASSDA) by 3 experts (FA-T, MRG-C, and JAS-P). According to these experts, DESDE-AND reached an AIL level of 5 (impact on the budget, financing, and allocation of resources for testing the tool in the target environment without routine use within the organization).
The overall maturity level of the tool was moderate-high or Managed (IMM 4): the process of implementing is documented throughout the organization rather than per aspect, and projects are carried out under the guidance of project operation standards and an implementation strategy [
This process has generated a computer tool with a semiautomated core module to obtain the standard directory of the health and social services available in a given area, based on an international classification.
It uses a multiaxial system that allows typify entities into differentiated groups using a string of codes or “axes” corresponding to the main attributes of the entity from other groupings in the same system [
This classification can be used by managers with the expert support provided by the computer tool, simplifying the complex task of coding in real time, and unifying the taxonomic and semantic criteria. It has been developed using a co-design approach focused on end users (managers, planners, and other decision makers) [
DESDE-AND is a system that allows various functions and different levels of web surfing/navigation and use: (1) a simple easy navigation for consumers; (2) a second advanced level of web surfing for case managers to improve care continuity; and (3) an advanced level that can be incorporated to decision support tools for social and health planning. This tool facilitates the analysis of health care ecosystems [
This system has been developed using a co-design approach. Co-design is largely used to improve service quality and delivery [
The original instrument DESDE-LTC has demonstrated its use in national and international comparisons of service directories and atlases of health and social care [
The information related to the coding of services can be exported to a GIS. This modular function facilitates data filtering by regions or provinces, municipalities, sectors of care, and types of services. It enables rapid geo-positioning of services with operational identification criteria and their specific characteristics. Henceforth, DESDE-AND facilitates the development of social and health care atlases of care [
The basic information obtained also allows the development of service indicators classified by geographic study areas, such as rates of services availability per inhabitants, beds (placement capacity), and professionals (workforce capacity); the analysis of the diversity of care in an area; and the balance between health and social care [
The final implementation of the tool in Andalusia depends on a series of regulatory steps. The Andalusian Social Services Law of 2017 entails the Map of Social Services of Andalusia and its update. The DESDE coding system will be included in the next version of the map.
The Map of Social Services of Andalucia is in operation. It is a computerized map of social services that allows the geolocation of the centers and services in the Catalogue of Social Services of Andalucia [
Finally, and as a secondary result, this study illustrates the use of standard measures and instruments for the quantitative assessment of maturity, a key phase in implementation research.
First, this computer tool uses an internet-based algorithm adapted to the specific conditions of the Autonomous Community of Andalucia, and to the information data sets of the Andalusian Social Services. The transferability to other territories has not been tested. However, the study describes the process and tools required to produce similar semiautomated tools adapted to other environments. Second, there is a lack of international agreement and guidelines for conducting impact analysis of digital health care [
Although the maturity assessment instruments used in this study were developed for self-rating and monitoring within organizations, they can provide useful information on the calibration of impact analysis across different projects, areas, and sectors in health technology assessment. We used a mixed approach combining evidence and expert knowledge, and by applying a co-design process [
DESDE-AND is a usable and manageable computer tool for guiding evidence-informed decision making in health and social planning. This tool reduces ambiguity and increases semantic interoperability in health and social care. It improves a previous tool that has demonstrated its usability for detecting gaps and inequities in the provision of health and social care. DESDE-AND is a relevant contribution to establish a common terminology, classification, and coding of health and social care services in the national context, as well as a standardized procedure for data collections and comparisons.
Adoption Impact Ladder
Description and Evaluation of Services and DirectoriEs for Long-Term Care
implementation maturity model
technology readiness level
We acknowledge the Health Information Systems Group (SICA-CTS-553) (University of Cádiz), TIC Department of Universidad Loyola Andalucía (Sevilla), and Guadaltech Soluciones Tecnológicas (Sevilla). We specially thank the contribution of the late I Fernandez de Bobadilla to this study. Other members of the DESDE-AND Group are A Rabadán, A Mora, A García-Lupato, C Bejar Reyes, C García-Alonso, C García Pérez, C López Mata, D Jiménez-Calvo, E Motrico, FJ Delgado Leal, F Romero-Alfaro, FJ Ortiz, I González Seco, I Carmona, J Almenara, JA García-Morales, JA Galera, JD Soto-Alba, J Limón, J Álvarez, M Poole, M Martínez-Domene, MC González-Montero, MF Raposo Triano, MJ Abellán, M Rabadán, P Recio, P Sánchez Villegas, R Vaquero López, R Vera Gálvez, R López-Medel, and S Rodríguez Gómez. We also acknowledge Assistant Professor T. Chen (Canberra University, Australia) for his advice and feedback on co-design and co-creation. The DESDE-AND project has been funded by the Andalusian Regional Government: call for biomedical and health sciences research (BOJA 77), Ref. PI-0075: “DESDE AND: Adaptación, validación y pilotaje del instrumento DESDE-LTC para la clasificación estandarizada de servicios de atención sociosanitaria de la Comunidad Autónoma de Andalucía”.
All authors have participated in different parts of the project DESDE-AND: LS-C and FA-T carried out the design and coordination of the DESDE-AND project; JAS-P, JLA-A, SPP, J-LG-C, CR-L-A, and MRG-C participated in the revision of the project’ materials; and CR-L-A and LS-C in the writing of the final manuscript. All authors read, contributed, and approved the final version of this text.
The authors are members of Psicost, a nonprofit research organization that shares the Official Registry of Computer Program Authorship of DESDE-AND together with the Government of Andalucia (Junta de Andalucía).