Borderline personality disorder (BPD) is a serious mental disorder affecting approximately 1% of the adult population, 12% of the adult outpatient population, and 22% of the adult inpatient population [1]. BPD is characterized by frequent and intense moment-to-moment changes in affect, behavior, identity, and interpersonal relationships, which typically result in a significant deterioration of the person’s overall functioning and well-being [2]. It is 1 of the 10 personality disorders listed in the latest edition of the Diagnostic and Statistical Manual for Mental Disorders (DSM)–Fifth Edition [3]. Even though new international diagnostic classification systems, such as the International Classification of Diseases 11th Revision, removed specific types of personality disorder in favor of a general diagnosis of personality disorder, BPD was retained as a specific pattern [4].

According to established diagnostic criteria, BPD is marked by severe, unpredictable, and intense feelings; vulnerability to perceived rejection or alienation in the context of interpersonal hypersensitivity; behavioral dysregulation involving suicidality and often impulsiveness; and an impaired sense of self, usually contributing to identity disturbance, a poor sense of direction, poor self-esteem, and cognitive disorders [5]. All these areas of core BPD psychopathology are typically characterized by high instability over time [6,7]. Because of the high variability in BPD symptoms, it is desirable to study moment-to-moment variability to capture the true extent of the disorder. The ecological momentary assessment (EMA; sometimes referred to as experience sampling) [8] uses systematic and often very frequent self-report diaries for evaluating context, symptoms, stressors, and other factors as they occur in everyday life and in nonlaboratory or research settings.

EMA studies are characterized by the use of repeated, often very frequent assessments that are delivered to participants in real-life settings and in real time using handheld devices capable of registering responses to short self-report questions in daily life. These repeated real-time measurements provide many methodological advantages over conventional assessment strategies, as highlighted in the study by Shiffman et al [9]. First and foremost, in the setting of highly and rapidly variable phenomena, recall biases encountered in conventional retrospective survey methods are diminished by the momentary evaluation of participants’ current experiences or behaviors. Second, measurements taken in the moment and in the context in natural settings yield information that is more pertinent to the existing social or physical contexts, providing information that is more ecologically valid. Third, the daily intensive repeated measurements capture within-day, within-person behavior as well as experience alterations over time, enabling studies of immediate causes and effects of behavior in real time [10].

The EMA methodology is being increasingly used to study psychopathology processes in BPD, mainly because these are considered very unstable and variable over time, thus benefiting from frequent sampling schedules. Given the variability of BPD symptoms, several momentary ratings that minimize retrospective bias and have strong ecological validity can be useful in modeling the dynamic core psychopathology features of the disorder [11]. This has led to important scientific and clinical discoveries by expanding our understanding of diagnosis, symptomatic variability, and disease mechanisms [12]. For example, the studies by Gordon and Laws [13] and Sadikaj et al [7] reported using EMA that patients with BPD exhibit high emotional variability in their responses to interpersonal stressors. The study by Stiglmayr et al [14] found that dissociation was correlated with distress in patients with BPD, which varied over time. The study by Zeigler-Hill and Abraham [15] reported interesting findings revealing that patients with BPD show increased self-esteem variability, especially in response to stressful interpersonal situations with peers. The study by Links et al [16] examined the link between emotional disturbances (specifically, the intensity of negative mood, mood amplitude, mood dyscontrol, and mood triggering) and suicide attempts. Therefore, EMA appears to be an appropriate tool for measuring the variability in BPD symptoms. Another vital contribution of EMA to BPD research is the possibility of studying the causes and effects of behavior in real time [17]. This could be useful for the therapeutic management of risk situations [18], as has been developed in the dialectical behavior therapy chain analysis technique [19].

Even though BPD is characterized by severely debilitating symptoms, participant compliance with EMA protocols appears to be relatively high, suggesting that symptom severity may not per se affect compliance [20]. It is possible that in the case of BPD, the impact of severe symptoms on compliance is counteracted by a beneficial effect of testing reactivity. Testing reactivity refers to the extent to which the behavior of interest is modified by momentary questions [21]. Few studies have examined the specific effects of testing reactivity on BPD symptom severity and intensity, but available models suggest that through the processes of active reflection and self-monitoring, social desirability, or feedback processes, the intensity of symptoms may decrease as a result of unusual attention on target symptoms [22]. In fact, active and frequent self-monitoring may be a generic active ingredient across a variety of evidence-based psychosocial interventions for BPD (eg, the use of diary cards to identify triggers and patterns of affective dysregulation in dialectical behavior therapy) [23].

EMA studies offer numerous advantages over traditional retrospective reporting in the study of highly variable phenomena over time; however, they typically place a high demand on participants, which can, in turn, lead to low response compliance. Low response compliance occurs when the ratio of answered prompts in relation to the theoretical maximum number of prompts planned in the study is low and has a direct bearing on data quality [16]. If compliance is low, sparsely collected data are unlikely to be a valid measure of the intensity and variability of the phenomena of interest measured in daily life, eliminating all relative advantages of using experience sampling. Even though missingness is a frequent occurrence in EMA studies, when compliance falls critically low or when patterns of noncompliance occur systematically at specific measurement occasions (eg, a participant always misses prompts delivered when she is experiencing intense emotional distress), the validity of the study is compromised [14]. Valid inferences about moment-to-moment experiences reported by participants using EMA methodologies must yield sufficient data to be robust and valid. Therefore, EMA studies typically include different design features and use strategies for enhancing compliance with the study protocol. For instance, it is common for EMA researchers to offer some kind of financial compensation to participants, which may even be contingent on a certain percentage of compliance (eg, the study by Berenson et al [24]). Other strategies include briefing the participants about the intensive nature of the study and even arranging telephone calls when a participant engages in increased noncompliance [25].

Two recent meta-analyses examined design features that broadly affect response compliance in EMA research. The study by Wrzus and Neubauer [26] examined this issue in EMA research across research fields by sampling 477 articles including 677,536 participants. They found that most EMA studies involved 6 assessments per day and lasted 7 days, although the number of assessments was not related to response compliance. Across studies, a compliance rate of 79% was obtained for participants with BPD, and providing financial incentives significantly increased response compliance. Other design features had little or no effect on compliance. The authors also emphasized the high heterogeneity in study design and reporting and called for more standardized procedures for EMA planning and reporting. The study by Vachon et al [27] explored the issue of compliance with EMA protocols in studies that included different mental disorders. The authors sampled data from 8013 participants. Typically, EMA designs included an average of 6.9 assessments per day for an average of 11.2 days. The study by Vachon et al [27] found that compliance rates were significantly lower for studies that included a higher proportion of male participants and participants with psychotic disorders. Compliance seemed to increase when researchers used fixed sampling schemes (ie, prompts were delivered in preplanned intervals across the day), used higher financial incentives, and spaced successive assessment occasions more over time (ie, less intensive protocols). The study by Smyth et al [28] covered a different aspect and investigated the relationship between the intention to participate in EMA studies and aspects of design features (such as prompt frequency and study duration). They found that willingness to participate was higher in simpler and less intense protocols and suggested that future EMA studies explore how different combinations of design features may influence participation. Taken together, previous studies underline the vast heterogeneity in EMA planning and reporting and call for unified standards.

EMA appears to be particularly well suited for the study of basic BPD psychopathology. To capture all the benefits of using this methodology to advance the field and construct more robust and nuanced dynamic psychopathology models, data must be of the highest quality, and compliance with study protocols is a crucial component of data quality in EMA research. However, to date, there are no systematic reviews and meta-analyses examining compliance in EMA research of BPD psychopathology, including the role of standard design features in enhancing compliance. In this setting, this review had three specific goals: (1) to provide a narrative overview of the EMA methodologies used to investigate dynamic BPD psychopathology; (2) to obtain and report overall compliance rates and quantitatively examine the effect of regular study design features (ie, the number of daily prompts, measurement period of the study, number of items in each prompt, etc) and study procedures (ie, the use of financial incentives, the device type, etc) on compliance rates; and (3) to optimize the advantages of EMA techniques in investigating BPD, for which this study provides recommendations for future studies that will use mobile devices to collect real-time self-reported data from BPD participants.

Bibliographic literature searches were conducted in 4 electronic databases (PsycINFO, PubMed, Scopus, and Web of Science) on November 12, 2021, without any time or language restrictions. Keywords based on Medical Subject Headings terms were used to identify peer-reviewed journal articles reporting on the use of EMA among the population with BPD. The search terms used in the individual databases are presented in Table S1 in Multimedia Appendix 2.

The search results for both the systematic review and meta-analysis were screened independently by 3 authors (AD, DH, and SS) using Covidence (Veritas Health Innovation) [31], a web-based review management tool for screening and data extraction. The screening process was carried out in 2 stages: title and abstract stage and full-text stage. Discrepancies were resolved through discussion according to the eligibility criteria previously defined.

A total of 3 authors (AD, DH, and SS) independently extracted the study characteristics as well as EMA design features and procedures from each included study. The extracted study characteristics were location, BPD sample size, mean age of the BPD sample (in years), sex (percentage of female participants), and dropout rates of participants with BPD (in percentage). Furthermore, the extracted EMA design features and procedures were study length (in days), daily prompting frequency, extra assessment (ie, if there was a daily extra measurement in addition to the prompts), number of items (ie, items to be answered in each prompt), average time to answer a prompt (in minutes), assessment window in minutes (ie, how long each prompt is available to be answered before it is considered missed), device type (ie, smartphone or others), sampling scheme (ie, time-based or self-initiated prompts), prompt scheme (ie, fixed or random prompts), and the incentive given (ie, fixed or incremental incentive). Finally, if possible, data on the compliance rate were extracted. If the compliance rate was not explicitly reported, the number of total prompts sent and the number of answered prompts were extracted from each study, for the BPD sample and (if available) the non-BPD sample. Discrepancies in author coding assignments were discussed and resolved until a consensus was reached. If needed, attempts were made to contact the authors of the studies with incomplete or insufficient data.

We were unable to obtain full data from 7 studies that did not report separate results for control and BPD groups [32-38], which were, therefore, excluded from our study.

The methodologic quality of the included studies was independently rated by 3 authors (AD, DH, and SS) using the Newcastle-Ottawa Scale [39].

This tool was specifically developed for the quality assessment of nonrandomized studies in systematic reviews and meta-analyses. Using this tool, each study is judged on 8 items, which are categorized into three broad domains: (1) the selection of the study groups (comprising the items “Representativeness of the Exposed Cohort,” “Selection of non-Exposed Group,” “Ascertainment of Exposure,” and “Demonstration of Absence of Outcome at Start of Study”), (2) the comparability of the cohort groups (comprising the item “Comparability Based on the Design or Analysis”), and (3) the ascertainment of either the exposure or outcome of interest for case-control or control studies, respectively (comprising the items “Assessment of Outcome,” “Follow-up Long Enough for Outcomes to Occur,” and “Adequacy of Follow-up of Cohorts”). To rate the comparability of the study cohort groups, the Newcastle-Ottawa Scale requires the authors to predetermine 1 or 2 key control variables. We included age as the first control variable and sex as the second control variable. If 2 of the predefined control variables were met, the criterion “Comparability Based on the Design or Analysis’ was rated with 2 scores. For the items “Follow-up Long Enough for Outcomes to Occur” and “Adequacy of Follow-up of Cohorts,” the follow-up interval was defined as the time between the first EMA measurement point (ie, baseline) and the last EMA measurement point (ie, follow-up).

Each study was rated in terms of 8 criteria (7 criteria were scored 0 or 1, and 1 criterion was scored 0, 1, or 2), resulting in a maximum score of 9 for each study. Discrepancies in author coding assignments were discussed and resolved until consensus was reached. The quality of a study was assessed as high if 7 to all the 9 criteria were met, moderate if 4 to 6 criteria were met, and low if ≤3 criteria were met.

The “meta” package in R (version 4.0.2, R Foundation for Statistical Computing) was used for all analyses and plots [40]. We used a 2-sided significance level of P<.05 to indicate statistical significance.

First, compliance rates were calculated by dividing the total number of prompts delivered by the study by the number of answered prompts at the end of the study for the BPD sample in each study. We chose a random-effects model for assessing the effect sizes rather than fixed-effects models because significant variability between studies was assumed [41]. We assessed the between-study heterogeneity of the results by calculating the Q statistic, I², and τ² [42-44]. In general, I² can be interpreted as follows: a proportion of 25% is assumed to indicate a low heterogeneity; 50%, a moderate heterogeneity; and 75%, a high heterogeneity [43].

A total of 4 categorical design features and procedures of EMA studies (ie, instrument, device type, quality rating, and extra assessment) were tested on the effects of compliance rates in the BPD sample using univariate analyses with random-effects models. Analyses testing the effects of continuous design features and procedures on compliance rate (ie, study length, prompts per day, and total quality score) were omitted, as the included subgroups contained <10 studies. As is true for primary studies, which require an appropriately large ratio of participants to form meaningful subgroups, meta-analyses require an appropriately large number of studies. Therefore, the use of meta-regression is generally not recommended when the number of studies is small (ie, n<10) [43].

In total, the systematic literature search revealed 995 potentially relevant articles. The screening and full-text assessment resulted in 28 peer-reviewed EMA studies [16,17,24,45-69] including a total sample of 2052 participants (N=1158 for the BPD group, 56.43%; N=894 for the control group, 43.57%; refer to Figure 1 for a flowchart of study inclusion). All studies examined BPD symptoms and selected participants based on either DSM criteria (26/28, 93%) or International Classification of Diseases criteria (2/28, 7%). For a complete list of the included studies and their characteristics, refer to Tables 1 and 2.

The total quality scores of the eligible studies ranged from 5 (4/28, 14%) to 8 (8/8, 29%) out of 9 (total scores; refer to Tables 3 and 4). A total of 36% (10/28) of studies were of moderate quality, 29% (8/28) of studies were of high quality, and none of the included studies were rated as having low quality. All studies met the quality criterion “Ascertainment of Exposure” of the selection dimension and the criteria “Assessment of Outcome,” “Follow-up Long Enough for Outcomes to Occur,” and “Adequacy of Follow-up of Cohorts” of the outcome dimension, but the quality criterion “Demonstration of Absence of Outcome at Baseline” was met by none of the included studies.

To our knowledge, this is the first systematic review, narrative review, and meta-analysis describing EMA study design features and procedures as they are used in current BPD research and estimating the effects of EMA study features and procedures on the compliance rates of participants diagnosed with BPD. The aim of the narrative review was to describe the characteristics of EMA protocols (study features and procedures) with patients with BPD using a systematic review approach. High heterogeneity was found with respect to study features and procedures across the different studies. Our results regarding the heterogeneity of features and procedures in BPD research using EMA are in line with the general trends in broad EMA research identified previously by Wrzus and Neubauer [26] and Vachon et al [27]. Considering the above, we can point out that similar pitfalls and challenges related to such heterogeneity apply to EMA studies in BPD. These challenges are primarily related to the problem of integrating and aggregating data stemming from studies that use very different design features and procedures. In short, similar to the broad EMA use in health research, a more homogeneous set of features and procedures could allow for more direct comparisons and benefit the overall advancement of the field, further highlighting the benefits of using EMA for BPD psychopathology research.

Considering the need for more standardized EMA designs and the crucial importance of compliance rates to obtain robust data sets in EMA research of BPD, the aim of the meta-analysis was to estimate the effects of specific design features and procedures on participant compliance rates. Across the studies included in the meta-analysis, the pooled prevalence rate for compliance among participants with BPD was estimated to be 79%, which is very similar to the 80% compliance rate recommended by Stone and Shiffman [8]. This suggests that, on average, EMA research is feasible for participants with BPD. Nevertheless, the reported compliance rates were high across the included studies and varied from 52% [45] to 95% [54]. This high variability suggests that some design features or procedures may have a bearing on compliance rates and thus justifies a specific exploration of such potential effects. However, contrary to our predictions, we could not identify the features or procedures of EMA designs that have an effect on compliance. It is important to note that only a small set of studies was included in the analysis, and only a limited number of design features and procedures could be statistically tested. On the basis of an analysis of the available design features and procedures, study characteristics, and outcomes, it is possible that compliance is robust and stable across varying EMA prompt calibrations and designs with participants with BPD. Moreover, our analysis suggests that compliance of participants with BPD appears to exhibit significant tolerance for a variety of design features and procedures of EMA studies, including prompt intensities, the measurement period of the study, the number of items in each prompt, the estimated time for the completion of each prompt, and the time window from the prompt signal to answering the prompt. No effect was found for other relevant variables such as the nature of the incentive for participants and technology used (device type). However, further investigation is warranted once more studies are available. In such studies, it would be useful if authors would add additional information regarding procedures and design features, including the methods for enhancing participation and compliance, such as phone calls, intensive monitoring, and variable incentives. It would also be essential for future EMA research of BPD to have more standardized criteria for the reporting of the technical aspects, design features, and procedures. Such standardized criteria would certainly allow for a more focused and comprehensive testing of variables that may have a bearing on compliance. Thus, EMA design can be strengthened and yield a full range of benefits to our field of study.

The third aim of this study was to provide recommendations to foster compliance by choosing design features and procedures of EMA that can enhance compliance for future studies that adopt EMA for research in BPD psychopathology [67]. Our study found that EMA studies use vastly varying setups. These include divergent within-day prompt frequencies, measurement periods, compensation schedules, number of items, required answering times, assessment windows, device types, and extra assessment plans. EMA is extremely useful for understanding complex dynamics in BPD psychopathology; however, the variety of designs and setups currently preclude the aggregation of results across studies and learning which study setups are best to foster compliance. Thus, it is rather difficult to draw stable conclusions regarding EMA findings. Shiffman and Stone [9] have established some general guidelines for designing and reporting in EMA studies, but they did not provide recommendations for the actual reporting of the features and procedures of EMA designs in the studies or in supplemental materials. Some reporting guidelines have been widely adopted in other types of studies. For example, STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) [68] is a well-known checklist for observational studies. It consists of 22 items related to each section of the article (ie, title, abstract, introduction, methods, results, and discussion) with the aim of improving the quality of the reports. Following the same principle, a recent systematic review of methods and procedures used in EMA of nutrition and physical activity research in youths [69] developed a comprehensive checklist of specific items to be reported in EMA studies: Checklist for Reporting EMA Studies. More recently, Trull et al [71] made an effort to fill the gap of standardization in how data and procedures are reported in EMA studies related to psychopathology and published a review of recommended reporting guidelines and current practices for ambulatory assessment research in psychopathology. The authors reviewed publications from the last 7 years (2012-2018) from 3 major mental health journals (Journal of Abnormal Psychology, Psychological Medicine, and Clinical Psychological Science) and integrated the information on how the various studies reported the methods used in EMA with some previously existing guidelines from EMA study reports, making an effort to provide some recommendations for future EMA studies in mental health. Despite the growing number of EMA studies in BPD, there is still a lack of such specific guidelines. Santangelo et al [67] have already made an effort to synthesize the EMA literature on BPD symptomatology. They discussed the different characteristics of EMA studies, providing an overview of EMA findings in BPD based on DSM-IV criteria. The authors also mentioned different challenges of EMA and provided recommendations to enhance them. However, they did not specifically focus on the technical aspects of EMA as this study did. It is likely that to produce specific recommendations, as suggested by Santangelo et al [67], different setups need to be compared with each other in terms of their relative benefits for compliance, including additional variables and procedures used to enhance compliance. Therefore, standard reporting of design features and procedures of EMA studies is required to fully understand how to optimize participation and obtain robust data sets that can then be combined and compared across EMA studies that examine BPD psychopathology.

Although this study is innovative and unique in its contribution to EMA research on BPD psychopathology, the results must be interpreted with certain limitations in mind. First, despite the fact that there are numerous published studies that use EMA to investigate BPD (we found 79 studies in our literature search), only approximately one-third of all published studies (28 of the 79 studies identified in our literature search, 35%) that use EMA for BPD psychopathology actually provide specific information that allows us to extract relevant design features and procedures. Thus, our narrative review was based on a limited number of studies and a limited number of design features and procedures.

Second, available studies were even fewer when we attempted to quantitatively analyze the predictors of compliance rates. Only about a quarter of all studies identified (18 of the 79 studies identified in the search, 23%) provided enough information to calculate compliance rates (eg, they provided the theoretical number of prompts to be answered and the actual average number of answered prompts among participants in the study). Trull et al [71] have already warned that systematic evaluation and aggregation of the findings of EMA studies by meta-analysis critically depends on the clear and explicit reporting of study characteristics. Among the studies included in this review, most studies did not report several design features and procedures of EMA, including the number of items, required answer time, assessment window, and dropout. Therefore, we were unable to perform quantitative analyses for these variables using the included studies to examine their influence on compliance.

Third, we found high heterogeneity in compliance rates, which was not explained by selected predictors (ie, study design features and procedures) or subgroup analyses (ie, age groups and sex). Furthermore, we were not able to examine the effect of patient-level data in our meta-regression analysis (ie, percentage of females and mean age of sample) because the results of a meta-regression based on the study means of patient-level variables are prone to bias (ie, ecological fallacy) [72].

Fourth, additional variables that were not examined in our study may have an influence on compliance. For example, it is possible that monitoring compliance and calling participants to improve participation may foster compliance. An important point to be noted is that there are only a few studies that we could include because other studies lacked the design variables examined in this systematic review and meta-analysis. In addition, within the included studies, a considerable amount of key information to be analyzed as moderators of compliance was missing (eg, the number of items and required answer time).

Finally, regarding compliance, there may be a systematic publication bias. Studies with very low compliance likely do not yield valid results, so publication is reasonably withheld. This means that there may be a restriction in the reported range of compliance rates. Therefore, we can say that the conclusions of this study apply to EMA setups that have been relatively successful in securing participation across studies, which, by definition, raises the average compliance rates. Indeed, much could be learned from studies that did not yield acceptable compliance rates, but this literature is typically not available in published form.

This systematic review and meta-analysis described the design features and procedures used in EMA studies investigating BPD and reported that overall compliance rates among the included studies appeared to be stable and acceptable across different EMA calibrations and study designs. In addition, our findings emphasize the importance of moving toward standardized reporting of EMA designs (eg, prompt frequency) and findings (eg, variability indices) to improve and aggregate our knowledge of basic BPD psychopathology research using EMA. As early indications are that compliance might be robust and stable across varying setups, we (1) propose that future EMA studies should consider other relevant design features and procedures, such as indices of variability, when designing EMA protocols to investigate BPD; (2) hypothesize that there may be nonspecific factors of EMA study design that may have an impact on compliance rates, which should be addressed in future EMA studies of patients with BPD; and finally, (3) suggest that standard reporting of the design features and procedures of EMA is thus required for future studies.