The relationship between humans and horses has long been recognized as a powerful source of emotional healing and psychological support. Research demonstrates that equine-assisted interventions, including equine-assisted learning (EAL) programs, have shown promise in supporting the mental health and emotional development of children and adolescents [1]. Through structured human-horse interactions, participants often experience reduced anxiety and depression, improved self-esteem, greater emotional resilience, and better quality of sleep [2-4]. These therapeutic relationships create powerful opportunities for healing and growth in structured therapeutic settings. These programs also foster a sense of emotional safety, personal security, and connectedness, particularly for at-risk children and young people who may struggle with low self-esteem or unstable learning environments [5,6]. Central to these outcomes is the horse’s unique socioemotional intelligence, which enables deep, nonverbal communication and trust-building with humans [7], making EAL a powerful tool for healing and growth in therapeutic settings. The Yawardani Jan-ga (“horse helping”) [8] project in Western Australia’s Kimberley region exemplifies this approach by integrating evidence-based EAL methodologies within a culturally secure, community-led framework. The program works with Aboriginal children and young people aged 6-25 years and is designed to respond to the unique educational and social challenges faced by Indigenous youth in remote communities. The program supports educational engagement and achievement by addressing underlying social and emotional well-being barriers that often hinder learning and participation [9,10].

Traditional evaluation tools in EAL settings rely on behavioral observations, structured interviews, standardized questionnaires, and physiological measurements conducted during sessions [11,12]. Despite their widespread use, these approaches present several limitations, including challenges with standardization, high time demands, and vulnerability to observer bias, which could lead to low interrater reliability across practitioners [1]. Physiological monitoring methods, such as heart rate or cortisol analysis, can provide objective indicators; however, the required equipment may be intrusive and risk disrupting natural child-horse interactions, potentially influencing the behaviors being measured [13]. There remains a lack of standardized, noninvasive tools that can objectively support the evaluation of processes and outcomes in EAL programs [14,15].

In practice, photographs are frequently collected during EAL sessions to capture key moments of child-horse interaction, supporting reflection, storytelling, practitioner documentation, and evaluation activities [16]. However, there has been limited research on systematic methods for analyzing such image-based data, despite its potential to provide complementary, evidence-informed insights into interaction patterns, emotional responses, and subtle behavioral cues that may not be readily captured through traditional observational methods. With advances in computational power and data storage capability, AI techniques have evolved significantly over the past decade. These techniques are now widely applied to a range of computer vision tasks, including image classification, object detection, and scene understanding [17-19]. More recently, vision-language models (VLMs) have emerged as a breakthrough in AI. Trained on large-scale datasets comprising paired images and text, VLMs bridge computer vision and natural language processing, enabling models to understand and generate natural language descriptions of visual content [20]. The ability of VLMs to generate text-based explanations from visual data offers new opportunities for more interpretable image-based analysis, facilitating human judgment and downstream analytical tasks. For example, in medical imaging, VLMs trained on extensive clinical datasets have enabled applications such as automated medical report generation from radiological scans and pathology slides, visual question answering for extracting diagnostic insights from medical images, and multimodal analysis combining text and imaging data for more comprehensive patient assessment. These applications demonstrate VLMs’ ability to interpret complex visual information and generate clinically relevant insights [21,22]. However, applications of AI to human-animal interactions in EAL settings remain limited. Existing studies have primarily focused on AI-based systems for monitoring equine behavior in broader contexts [23], with little research exploring the use of VLMs to analyze child-horse interactions in EAL programs.

Motivated by these gaps, we developed a VLM-assisted AI framework for analyzing images collected during EAL activities, in collaboration with the Aboriginal-led and culturally secure Yawardani Jan-ga EAL program in Western Australia, which supports the social and emotional well-being of Aboriginal children and young people. Training and testing datasets were curated from images collected during EAL sessions within the program. Co-designed with program domain experts and certified EAL practitioners, the framework uses VLMs to achieve three analytical objectives: (1) activity classification: automatically identify and categorize 9 therapeutic activities (eg, grooming and leading) from EAL session images; (2) proxemics analysis: analyze spatial relationships between children and horses by assigning proximity categories to document observed engagement patterns; and (3) body posture analysis: describe and classify children’s body language and postural indicators through transparent and interpretable reasoning. To the best of our knowledge, this is the first study to apply VLMs to child-horse interactions in EAL settings, combining few-shot learning, generative reasoning, and multimodal chain-of-thought (MCoT) prompting, and self-consistency to support interpretable image analysis with minimal labeled data. The framework provides structured outputs through 3 parallel analytical modules, which could assist EAL practitioners in documenting and reviewing observable interactions across activity types, proximity patterns, and body postures. By investigating specialized VLM-based image analysis tools in EAL settings, this study illustrates the potential of AI- and VLM-based approaches to complement practitioner-led documentation and support more structured analysis of child-horse therapeutic interaction patterns in programs that promote the social and emotional well-being of Aboriginal youth.

This section reviews research areas relevant to the design of our framework for analyzing child-horse interactions in EAL settings.

Building on prior work, our framework integrates VLMs, few-shot learning, and MCoT reasoning to address key tasks that facilitate the analysis of EAL interactions by enabling visual interaction analysis, supporting practical use with limited labeled data, and promoting interpretable and reliable reasoning grounded in visual evidence.

We propose a new VLM-based framework for understanding human-horse interactions in EAL using images and expert-verified label descriptions. The framework was developed in collaboration with the Yawardani Jan-ga program and co-designed with EAL practitioners and researchers to ensure alignment with practice-based needs. The Yawardani Jan-ga program integrates structured learning activities involving horses, such as grooming and leading, to support the social and emotional well-being of Aboriginal children and young people. The program seeks to capture human-horse interactions by identifying critical observable interactions through analyses of photo and video footage collected during EAL sessions, to quantify and rate equine-human dyadic interactions using coding schemes [10]. To support the analytic objective, the framework consists of three core learning modules: (1) an activity classification module that uses a VLM with few-shot learning to identify 9 therapeutic activities; (2) a proxemics analysis module that classifies spatial relationships between the child and horse, including physical contact; and (3) a body posture analysis module that applies MCoT prompting and self-consistency techniques to classify children’s body posture into confidence-related categories with transparent reasoning. The overall framework is illustrated in Figure 1. To address practical requirements related to data privacy and deployment feasibility in remote EAL service areas where network connectivity may be limited, all VLMs used in this framework were selected from open-source models that support local deployment. The following subsections describe each module in detail.

The Yawardani Jan-ga EAL program is structured around 10 therapeutic themes (eg, noticing, awareness, boundaries, and healthy relationships) grounded in the neurosequential model of therapeutics principles [10]. Each theme encompasses multiple activities, and a single activity can serve more than one theme. For this study, 9 representative activities were selected in consultation with EAL practitioners for this activity classification module. Below is a textual description of each activity.

Activity classification in EAL presents unique challenges, as different therapeutic activities may exhibit similar visual features while differing substantially in therapeutic intent and clinical relevance. For example, grooming and touching may both visually appear as a child’s hand in contact with a horse; however, grooming typically involves using tools such as brushes, whereas touching involves direct hand contact. These subtle yet meaningful distinctions motivate the use of VLMs that can align visual features with semantically precise textual descriptions to support more accurate activity interpretation. Accordingly, we adopted an image-text similarity–based approach using Sigmoid Loss for Language-Image Pre-training 2 (SigLIP2; google/siglip2-giant-opt-patch16-384) [26] as the foundational visual-linguistic embedding model. SigLIP2 provides strong alignment between visual and textual representations, supports fine-grained visual semantics, and demonstrates effectiveness in 0-shot and few-shot scenarios, making it well-suited for EAL settings characterized by limited labeled data and context-dependent activity definitions. We compared SigLIP2 against OpenAI’s CLIP-ViT-L-14 [24] to validate this model.

As illustrated in Figure 2, the activity classification module operates in 2 phases. During initialization, predefined textual descriptions for 9 activity categories are encoded using SigLIP2’s text encoder, and a small number of reference images per activity category are encoded using SigLIP2’s vision encoder and averaged to form visual prototypes. The text and image encoders produce embeddings that are aligned in a shared semantic space, enabling direct cosine similarity computation between visual and textual representations. In the classification phase, a query image is encoded using the same vision encoder and compared against both textual and visual prototype representations using cosine similarity. To integrate semantic and visual evidence, we introduce a weighted fusion strategy, defined as:

where score_c denotes the score for each activity class c, v_q represents the visual embedding of the query image. v_t,c represents the textual embedding of the predefined description for the activity category c. v_img,c denotes the visual prototype for class c calculated as the mean embedding of the n reference images provided during the initialization phase, and α controls the relative contribution of textual and visual similarities. The value of α was tuned separately using grid search over values from 0.0 to 1.0 in increments of 0.1 on the validation set, with macro F₁-score used as the selection criterion. Finally, softmax normalization yields probability distributions and confidence scores for different activities.

The proxemics analysis investigates spatial relationships between children and horses during therapeutic interactions. Interactions are classified into two therapeutically meaningful categories: (1) “physical contact,” where the child directly touches the horse through hands, arms, or body; (2) “no physical contact,” where the child is positioned near the horse without direct physical touch. Prior work has shown that contrastive VLMs, such as CLIP and SigLIP, have limited capacity to model spatial relationships [47], and our preliminary experiments similarly indicated that these models struggled to reliably distinguish contact-based interactions in our study settings. Following initial testing, we adopted the open-source Qwen2.5-VL-7B-Instruct model for proxemics analysis. Unlike contrastive VLMs, which rely on embedding similarity, Qwen2.5-VL is a generative VLM that performs joint image-text reasoning, enabling more explicit interpretation of spatial relationships and interaction context [29]. The 7B parameter configuration was selected because smaller models did not consistently support coherent multistep reasoning, whereas larger variants offered only limited performance gains at substantially higher computational cost [48].

As illustrated in Figure 3, the framework leverages few-shot in-context learning by pairing each preconfigured proximity category with expert-validated reference examples. For each query image, a structured multimodal template is constructed by combining the n-shot reference examples with the query image. The reference examples are formatted as user-assistant dialogue pairs that demonstrate correct interpretations for each category, followed by the query image and a task-specific prompt instructing the model to classify the interaction. The model jointly processes the reference and query images alongside the text instructions, reasoning over the visual content and instructional context to generate its response. The model generates a free-form natural language response describing the observed interaction. This output is then parsed to extract the primary binary label (physical contact vs no physical contact). When physical contact is identified, the module extracts the number of hands involved in the contact, such as 1 hand or 2 hands. When no physical contact is present, the module extracts the estimated distance between the child and the horse, such as very close or far away. Hand count and estimated distance are descriptive subattributes intended to provide additional contextual information for practitioners. The primary evaluated output of the proxemics module is the binary contact classification. We compared Qwen2.5-VL-7B-Instruct against LLaVA-OneVision-7B to further evaluate the proxemics classification task.

We further adopt Qwen2.5-VL-7B-Instruct as the foundational VLM model for body posture analysis. In addition, we explore 2 complementary techniques inspired by MCoT reasoning [42] and self-consistency [43] to support more interpretable and stable analysis. MCoT uses a 2-stage reasoning process that encourages the model to first describe observable visual evidence before making predictions. The self-consistency approach improves robustness by identifying predictions that remain stable across multiple independent reasoning paths. Our method separates the analysis into a 2-stage inference process and repeats this process across multiple stochastic runs (K=5). For computational efficiency, each input image is encoded once, and the resulting visual embeddings are reused across all K runs, so that output variability is primarily driven by the stochastic text generation. In stage 1 (rationale generation), the model is prompted to describe observable posture elements, including head position, shoulder alignment, spine curvature, and stance, without assigning a label. A higher sampling temperature is used to encourage variation in the generated rationales across runs. In stage 2 (classification), the model is conditioned on both the image and the stage 1 rationale to classify the posture as “confident,” “not confident,” or “indifferent,” using a low temperature to promote stable classification. Figure 4 depicts the 2-stage prompts. By adopting this “describe-then-decide” workflow, practitioners can assess not only the final prediction but also the reasoning that supports it.

A majority vote across K runs is used to determine the final classification. To evaluate the model’s reliability, we also calculate a combined confidence score as the product of the majority-vote agreement rate (the proportion of K runs that agree with the majority-vote class) and the rationale coherence (measured using SentenceBERT all-mpnet-base-V2), to help practitioners identify low-confidence predictions that may require manual review. Figure 5 illustrates this parallel sampling and semantic verification process for body posture analysis.

Ethics approval for this study was obtained from the Western Australian Aboriginal Health Ethics Committee (WAAHEC; HREC926). This study used deidentified data collected through the original Yawardani Jan-ga program and evaluation. Participants, or their parents/carers, where required, provided informed consent for the original data collection and for their deidentified data to be used for approved research purposes, including secondary analysis. Young people were also asked for assent or consent in an age-appropriate way. No additional consent was required for this secondary analysis. Participant privacy and confidentiality were protected throughout the study. The dataset did not contain directly identifiable personal information, and all analyses were conducted using deidentified records. No images or supplementary materials included in the manuscript allow identification of individual participants. No additional participant compensation was provided for this secondary analysis.

We evaluated the activity classification module on the curated dataset and compared its performance with the CLIP model as a baseline. To assess the effect of contextual evidence, we examined 0-shot through 4-shot configurations. The results are summarized in Table 1.

Across all shot settings, SigLIP2 outperformed CLIP across all evaluation metrics, with macro F₁-score performance gaps ranging from 28.7% to 37.1% (Table 1). SigLIP2 macro F₁-score ranged from 89.6% in the 0-shot configuration to 91.2% in the 4-shot configuration. CLIP performance remained substantially lower across all settings, ranging from 52.5% to 61.6% macro F₁-score. Although SigLIP2 4-shot achieved a slightly higher macro F₁-score, 2-shot was selected because it achieved the same accuracy with fewer reference images. Because SigLIP2 and CLIP-ViT-L-14 were evaluated on the same test images, a McNemar test with continuity correction was used to assess whether their paired classification errors differed significantly across all learning configurations. As shown in Table 2, SigLIP2 significantly outperformed CLIP-ViT-L-14 across all settings (P<.001).

The category-level confusion matrix performance is detailed in Figures 6 and 7. SigLIP2 diagonal values ranged from 0.74 to 1.00 across categories, with perfect classification for “ball” and “washing.” CLIP diagonal values ranged from 0.00 to 0.94, with particularly low performance for “leading the horse” at 0.00, “leading with obstacles” at 0.59, and “painting” at 0.44.

On the validation set under the 2-shot configuration, performance for both models increased as the text weight α increased. SigLIP2 achieved peak performance at α=.9, while CLIP peaked at α=.8 (Figure 8). Consequently, these optimal weights were applied during the final test set evaluation.

We benchmarked our approach against the LLaVA-OneVision-7B model on the test dataset across all shot learning configurations, as detailed in Table 3. Qwen2.5-VL-7B outperformed LLaVA-OneVision across all learning configurations. Qwen2.5-VL-7B achieved its best macro F₁-score of 88.5% in the 1-shot configuration, with an accuracy of 88.6%, macro precision of 89.0%, and macro recall of 88.0%. In the 2-shot configuration, accuracy was 87.9%, macro precision was 87.0%, macro recall was 87.5%, and macro F₁-score was 87.3%. LLaVA-OneVision’s accuracy was 80.3% at 0-shot and 65.5% at 2-shot, with macro F₁-scores of 78.3% and 53.9%, respectively. McNemar test with continuity correction was applied to assess whether the paired classification errors between Qwen2.5-VL-7B and LLaVA-OneVision differed significantly across all learning configurations. As shown in Table 4, Qwen2.5-VL-7B significantly outperformed LLaVA-OneVision across all settings (P=.01 for 0-shot and P<.001 for few-shot configurations).

To further examine how each model behaves at the individual-class level, Table 5 presents per-category performance for both “physical contact” and “no physical contact” across all learning methods. For Qwen2.5-VL-7B, 0-shot learning yielded a physical contact recall of 72.0% and an F₁-score of 82.3%, which shifted to 85.0% recall and 86.0% F₁-score under 1-shot learning. The “no physical contact” class achieved 91.0% recall and 91.0% F₁-score in the 1-shot setting. Under 2-shot learning, physical contact metrics were 84.0% precision, 86.0% recall, and 85.0% F₁-score, while no physical contact metrics were 90.0% precision, 89.0% recall, and 89.5% F₁-score.

We also note that, unlike Qwen2.5-VL-7B, LLaVA-OneVision showed reduced stability when additional reference examples were provided. For LLaVA-OneVision, the “physical contact” F₁-score dropped from 70.9% in the 0-shot setting to 39.9% and 30.7% in the 1-shot and 2-shot settings, respectively, with recall falling from 59.4% to 19.0%. In contrast, “no physical contact” recall remained above 94% across all settings. The F₁-score gap between the 2 classes widened from 14.7% in the 0-shot setting to 46.4% in the 2-shot setting, suggesting that predictions became increasingly biased toward the majority class as more reference examples were added. This pattern may reflect difficulty in consistently using additional visual reference examples for fine-grained proximity classification. Similar 0-shot to few-shot performance declines have been reported for LLaVA-family models [52], while stronger few-shot stability has been observed in larger multimodal models such as Qwen2.5-VL-32B [53]. Our findings are consistent with these reports and further indicate that, at the 7B scale, Qwen2.5-VL-7B remained more stable than LLaVA-OneVision-7B on this task.

To evaluate model reliability, we compared our full model, Qwen2.5-VL-7B + MCoT + SC, with 2 baseline inference strategies using 4 internal consistency metrics. Three strategies were evaluated: (1) single-stage CoT (simple CoT), which generates the reasoning and classification in a single pass; (2) multimodal 2-stage CoT (MCoT), which first produces a descriptive visual rationale and then conditions the final classification on both the image and the generated rationale; and (3) MCoT with self-consistency (MCoT+SC), which further aggregates multiple MCoT inference paths using majority voting to obtain a consensus prediction. Each method was evaluated across 5 independent experimental repeats. Within each repeat, the MCoT+SC strategy performed K=5 stochastic inference paths for each image. Internal consistency metrics were computed for each image and then summarized across the test set. The reported means and SDs were calculated across the 5 independent experimental repeats.

As shown in Table 6, simple CoT achieved a rationale coherence score of 0.913, an agreement rate of 0.104, and a vote entropy of 0.653 across 5 independent runs. The MCoT approach improved output consistency, with an agreement rate of 0.528, vote entropy of 0.300, and a rationale coherence score of 0.931. The MCoT+SC approach achieved the highest agreement rate of 0.705 and the lowest vote entropy of 0.174, indicating more stable classification outputs across runs, although its rationale coherence score was lower (0.857) than MCoT (0.931).

It should be noted that the consistency metrics reported here assess the stability of the model’s outputs, not the factual correctness of the posture classifications. Without ground-truth posture annotations, these metrics should be understood as indicators of reliability rather than accuracy. Validation against expert-annotated labels remains essential before this module can be used in clinical applications.

The evaluation of the AI-assisted VLM framework suggests that few-shot learning may support the classification of observable activities and spatial interactions in EAL settings using limited labeled data. For activity classification, SigLIP2 achieved stronger overall performance than CLIP-ViT-L-14 across the evaluated settings, suggesting that the choice of vision-language backbone is important for recognizing EAL activity categories in specific settings. The validation-based optimization of the text-image fusion weight (α) suggested that textual activity descriptions contributed useful discriminative information alongside visual prototypes within the proposed fusion strategy. The peak α = 0.9 for SigLIP2 indicates that the model relies predominantly on well-defined textual descriptions, where carefully specified activity definitions provide a strong discriminative signal for classification. The visual prototypes derived from reference images play a complementary role and may offer additional contextual grounding, altogether improving performance over the 0-shot setting. In the proxemics classification task, Qwen2.5-VL-7B showed relatively stable performance across learning configurations, whereas LLaVA-OneVision performed less consistently when additional examples were provided. For body posture analysis, the MCoT+SC approach improved average consistency, agreement rate, and vote entropy, indicating more stable classification outputs across repeated runs. However, its rationale coherence score was lower. This may be partly due to the higher sampling temperature used during rationale generation, which was intended to encourage diverse reasoning paths across runs but may also reduce semantic similarity among the generated rationales.

The framework may provide complementary support for documenting observable EAL interactions. The activity classification module identifies the activity type and provides a confidence score, which may help practitioners organize session records according to activity categories. The proxemics module classifies whether physical contact is present, providing one observable indicator of child-horse spatial interaction patterns. The body posture module classifies observable postural states and provides a natural-language rationale describing the posture elements that informed the classification, which may improve transparency in how the model reached its output. Together, these outputs may complement practitioner-led observational notes and support a more structured review of interaction patterns over time. These findings should not be interpreted as evidence that the framework can assess therapeutic progress or clinical outcomes; such applications would require further validation against practitioner ratings, participant goals, and standardized well-being or therapeutic outcome measures.

Several limitations of this study warrant discussion. First, the availability of large, diverse datasets in this specialized EAL context remains limited, and this study relied on relatively small real-world datasets. Expanding dataset size and diversity may improve generalizability. Second, the McNemar test assumes independent paired observations, but the dataset includes multiple images from the same participants, which may partially violate this assumption, potentially affecting the reported significance. Third, image quality, occlusion, and visually complex outdoor environments, which are possible in real-world EAL sessions, may affect performance across all 3 modules. Fourth, the body posture module lacks ground-truth labels, and model reliability was assessed through internal consistency measures rather than supervised evaluation. These metrics capture output stability but should not be interpreted as evidence of classification accuracy. Fifth, the framework operates on static images rather than continuous video and therefore does not capture dynamic interaction patterns such as posture transitions or evolving engagement. Sixth, the framework primarily analyses the child’s observable behaviors and captures only limited aspects of the horse’s responses. Although the proxemics module describes child-horse spatial interaction by estimating distance categories during noncontact interactions, deriving accurate spatial measurements from a single uncalibrated image remains technically challenging. Seventh, the body posture module is the most computationally demanding component, as it has 5 inference runs per image to support self-consistency, which may increase processing latency and limit practical deployment in remote service areas with constrained infrastructure. Eighth, the pretrained VLMs used in this study were trained on generalized, large-scale datasets and may carry culturally specific assumptions about body language and nonverbal behavior. This is particularly important for the body posture module, where classifications such as “confident” or “not confident” may depend on culturally specific visual cues that may not have the same meaning across communities. Therefore, body posture outputs should be interpreted as exploratory analytical outputs rather than culturally validated assessments, and reviewed cautiously alongside practitioner judgment, cultural knowledge, and the broader therapeutic context. Ninth, the framework relies on precise textual descriptions that draw sharp boundaries between activities, such as defining grooming as involving tools and touching as involving bare hands. While this design supports consistent annotation and evaluation, interactions in practice are fluid and may involve overlapping or transitional behaviors. Since the framework assigns a single label per image, it does not capture cases where 2 activities co-occur or when a child transitions between them. Finally, although the same prompt template was consistently applied to all images in this study, systematic testing of multiple prompt formulations was beyond our scope.

Several directions for future work follow from these limitations. First, future work should prioritize developing expert-annotated posture datasets to enable supervised evaluation of prediction performance. Larger and more participant-diverse datasets would support more robust performance evaluation and statistical analysis. Depth-aware or multiview approaches could also be explored to provide more reliable estimates of child-horse distance and spatial positioning. More efficient deployment strategies should be investigated, such as quantization, model distillation, reduced self-consistency sampling, and smaller, locally deployable models. Culturally informed adaptation and validation should be prioritized in collaboration with Aboriginal communities and EAL practitioners, including co-designing posture categories and interpretive criteria before their therapeutic application. Performance on more ambiguous interactions may differ from the results reported here, and future work could investigate multilabel classification, uncertainty-aware outputs, and temporal analysis across image sequences to better capture the dynamic nature of EAL interactions. Prompt robustness and the degree to which variations in wording may influence classification outputs should also be examined. Finally, future work should expand expert-annotated datasets, develop video-based analysis of EAL interactions, incorporate horse behavioral responses, and conduct clinical validation to examine whether framework outputs align with practitioner ratings and standardized therapeutic or well-being outcome measures.

This study proposed a VLM-based AI framework for EAL settings, comprising 3 modules: activity classification, proxemics analysis, and body posture analysis. The findings suggest that AI- and VLM-based approaches have the potential to support the analysis of images captured during EAL sessions. This study represents an initial step toward more structured documentation and analysis of observable child-horse therapeutic interactions in EAL settings. Such digital health tools may reduce practitioner burden, complement practitioner expertise and clinical judgment, and facilitate data-driven research into equine-assisted interventions.