This project was the product of a collaboration between the NIMH and BCH. We represent diverse specialties in child and adolescent behavioral and mental health (irritability, chronic pain, and substance use) and a range of experiences with EMA platforms (users and developers).

Think about the location of the developer relative to your study and institution. The developer’s location may influence the language or languages of the participant and researcher platform interfaces and instructional materials. In addition, developer location may impact the timing of support from developers and dictate the location of the data storage.

Determine whether each platform is available in the language or languages of your sample and study team. It is not only necessary to confirm that survey items and response options are programmable in the language or languages of your study, but it is also important that participant- and researcher-facing features are available in the appropriate language or languages. Can the platform accommodate languages with non-Roman alphabets (eg, Arabic and Greek) or languages that read from right to left (eg, Hebrew and Farsi)? Some developers are willing to build new languages into their platforms (potentially for a fee). Furthermore, developers may offer knowledge bases, manuals, or other informational materials that will help you use the platform, and it is important that these resources are accessible to your team.

If developer support with quick turnaround time is a priority, determine when your study will be live, when developers will be available, and how quickly developers will respond to service requests. Consider the time zone differences between the developer and study sites. Additional coordination may be necessary if there are gaps in developer coverage. For example, in previous research, our team at BCH scheduled a weekly check-in and troubleshooting meeting with developers outside the United States to circumvent response delays due to time zone differences.

Determine whether your country and institution impose restrictions on the location of your data. For example, the General Data Protection Regulation stipulates how the data of European Union citizens and residents must be collected, stored, processed, and disseminated (eg, individuals must have the ability to erase their personal data), and platforms that do not comply may not be used [61]. Regarding institutional policies on data storage, early conversations with your data security officer or IRB are critical (see the section Consideration 9: Security, Privacy, and Data Management). If data storage outside of your country is prohibited by your institution, some developers may collaborate on customizing data storage locations.

Although some studies are straightforward and require low maintenance, others are more complex and require more frequent communication with developers. Relative to the technical skill levels of your team members, consider the support (type and quantity) you will need from developers at the beginning and throughout the life span of your study.

Developers offer varying levels of support when initiating a new study, and accordingly, our teams encountered a range of support at the outset of our evaluation. Some developers programmed the study on our behalf, in part or in whole, after receiving our protocol. Other developers provided access to knowledge bases, manuals, and other informational materials to facilitate the setup led by our teams. The cost of developer involvement in the study setup varies. Some platforms offer unlimited support or build a fixed number of training sessions into their pricing. Other platforms support users on a fee-per-session basis.

Developers offer varying levels of support throughout the life span of a study. In our evaluation, developer response times spanned hours to weeks. Typically, we requested support by emailing a prespecified point of contact or submitting “help tickets” on the web. If support is provided via email, work with developers to determine an alternate point of contact for vacation and sick days. If support is provided via help tickets, inquire about the average time from receipt to resolution. Regardless, be informed and develop a backup plan. Plan ahead, as most technical malfunctions tend to occur when studies are launched.

The characteristics of your sample may inform the optimal participant user experience (UX) for your platform.

Intuitive and engaging UX is a goal for developers, regardless of sample characteristics. Still, depending on factors, including participant age range and level of cognitive functioning, it may be critical to implement an easy-to-navigate and visually appealing UX. For example, the NIMH prioritized a child-friendly UX. Of the platforms we evaluated, a few were characterized by child-friendly UX (eg, large fonts and bright colors), and some (but not all) allowed for user customization. Gamification and rewards may increase survey compliance and minimize dropout. Of the platforms that we evaluated, one offered awards for survey completion and another offered a virtual “shop” to unlock incentives such as fun facts. Creative response options may make items clearer and more engaging. In our evaluation, one platform offered emoji response options and another platform enabled researchers to upload images as response options.

If recruiting a high-risk sample (eg, individuals at risk for suicidal ideation), you may consider platforms that allow for responses (eg, thoughts of death or dying) to be “flagged” and forwarded to a point of contact on your team (eg, clinician) for follow-up. Some platforms leverage natural language processing to identify flagged terms in free-response items. In addition, some platforms can be programmed to automatically generate follow-up questions or messages for a flagged response.

Determine whether you will enroll participants in person or virtually, factoring in the technical skills of your team members and sample, and assess the relative fit of platform onboarding options.

Consider how participants will enroll in your study. Although participants may benefit from in vivo technical support, virtual onboarding may be less resource-intensive (eg, lower cost, easier to schedule). Both the technical skills of your sample (considering age, cognitive functioning, and clinical status) and research team are factors to consider. Some platforms autogenerate onboarding invitations that are sent to participants via email or SMS text message. Notably, email invitations may be sent to spam folders, emphasizing the importance of detailed study protocols, including guidance on troubleshooting technical issues during onboarding. Other platforms direct participants to scan a QR code for study enrolment. The level of automaticity and researcher time and effort in onboarding varies by platform. For example, when evaluating one platform, our team was responsible for generating mobile codes for enrollment.

Some platforms allow for the administration of a researcher-developed e-consent form as part of onboarding. Understand whether this feature is available if needed and can be disabled if you decide to consent participants through another method. In addition, given passive sensing integrations to some platforms, participants may be prompted to provide permission to access smartphone features and sensors (eg, microphones). In our evaluation, multiple participants expressed concerns about privacy upon receiving smartphone sensor permission requests from the platform, even though we instructed participants to refuse those requests and did not collect passive data.

Determine which design features are required for your surveys to be maximally effective, user-friendly, and tailored to your sample.

Determine whether you require your surveys to display text, images, audio, or video to the user. Depending on your sample, visual or auditory cues may be beneficial (see the section Consideration 3: Sample Characteristics). Although text display is ubiquitous across platforms, options to present images, audio, and video vary by platform. In addition, consider how you would like users to move through your survey; some platforms display multiple items on a single page (ie, via scrolling), other platforms display one item per page, and still others allow for customization. Think about whether participants should be able to skip items and return to previous items and determine if those features are modifiable in the platform.

Determine whether your response options will be a single-choice, multiple-choice, Likert-scale, or free-text response or something else. For example, you might ask participants to respond through captured images, audio, video, time stamps, or geotags. In addition, consider your need for “branching logic” or custom paths based on user responses. Many platforms allow participant responses to dictate the items to follow, but fewer platforms allow responses to trigger future surveys (eg, later in the day).

Determine the sampling scheme for your study and whether the platform supports your sampling scheme. Consider the other platform features you may need for successful survey deployment and completion.

Decide whether your surveys will be interval contingent, signal contingent, event contingent, or some combination thereof. Some platforms also allow for pairing with wearable devices and passive sensors, such that geolocation or another target might trigger a survey. Inquire as to whether multiple surveys can be programmed and deployed or if the platform can only manage a single survey. In addition, understand the method for developing the sampling scheme structure within the platform; our team found the evaluation to be critical in testing the researcher-facing platform design.

Alarms signaling users to complete a survey might use sound, vibration, a banner, or a push notification. Some platforms allow for reminder notifications if a survey is not completed. Understand the level of customization offered by your platform. Greater flexibility will allow you to develop a system tailored to your sample such that participants remain engaged but not overwhelmed with notifications.

Some degree of personalization in survey scheduling may increase survey compliance. For example, the morning survey for participant A (an early riser) may be deployed randomly between 7 AM and 8 AM, and the morning survey for participant B (a late sleeper) may be deployed randomly between 8:30 AM and 9:30 AM. Determine whether the platform allows for personalization or if survey scheduling must be held constant across your sample. In addition, when enrolling participants across time zones (eg, multisite studies), understand whether the survey will be deployed at the local time of the participant or researcher.

Understand the impact of participant phones being “offline” (eg, survey deployment, response, and upload to the server). For example, some platforms are native apps that allow users to access and respond to surveys even offline, and other platforms store responses locally until Wi-Fi or cellular data are available. In addition, consider the impact of the platform on data use. Significant data use may be prohibitive if participants use their own phones. Passive sensing may rely heavily on data.

Dashboards can be used for data monitoring. Given that dashboards are not universal to platforms, consider whether a dashboard is a priority.

Dashboards can be programmed to report high-level information about your sample, specific information about individual participants, or both. When determining your dashboard needs, consider which team members will need to view your data (eg, research coordinator, clinician, or principal investigator). Then, consider which variables each team member will need to view (eg, compliance and symptoms), including level (eg, full sample and individual participants) and interval (eg, daily and weekly). In addition, consider how the data should be visualized (eg, graphs and tables). For example, clinicians may want to track depression symptom scores weekly over the course of a clinical trial using a time-series graph. Some groups may be interested in data collection on dyads, making platforms with dyadic linking and dashboards with side-by-side data visualization optimal. If multiple users require dashboards for distinct purposes (eg, clinicians and researchers), inquire whether the platform supports multiple dashboards and users in a single study.

Consider how and how often data should be uploaded to the server and downloaded by your research team. In addition, consider your preferences for structuring your data export.

Consider how (eg, Wi-Fi and cellular data) and how often (eg, immediately and daily) data are uploaded to the server. Some platforms require Wi-Fi or cellular data to deliver surveys and subsequently upload data. Other platforms are accessible offline and do not require connections to Wi-Fi or cellular data for functionality. Still other platforms deliver surveys offline, and data are stored on the platform and uploaded when connected to Wi-Fi or cellular data. Data upload considerations may vary in importance depending on your study design (eg, use of participant phones compared with study phones).

In many cases, platforms export data as comma-separated values files. Some platforms offer additional formats (eg, JavaScript Object Notation) and application programming interfaces (eg, SPSS and R). In addition, some platforms export individual participant data (which may be less feasible for data collection on large samples) and others export data for the full sample.

Platforms vary in how their data exports are structured and whether (and to what extent) their data exports are customizable. Consider whether you will need your data in a long or wide format, how missing data should be reported, and which fields are necessary (eg, skipped items, response time, and summary statistics). Inconsistency between data exports and your needs will yield extra time and effort in preprocessing.

Consider security, privacy, and data management requirements as they relate to your country and institution. The presence or absence of certain platform features (eg, 2-factor authentication; 2FA) may render a platform incompatible with those requirements. Depending on your institution and familiarity with their policies, consider meeting or corresponding with your data security officer or IRB before consulting with developers. Our team at the NIMH met with our data security officer to understand the requirements of IT applications and services (Multimedia Appendix 2). We presented the corresponding questions to developers when we met to discuss their platforms. On the basis of developer responses, our data security officer assigned platforms to varying levels of security clearance for NIMH-specific standards, which informed the selection of our platform. Our team at BCH was not required to have an initial conversation with data security. Instead, our IRB application included questions about our candidate platform (Multimedia Appendix 2), and we met with the data security team following submission.

Inquire about whether and how the platform secures access, including requiring a personal identification number, a password, or 2FA. Although 2FA by some means (eg, email) may complicate, slow, and impede survey responses, other means may be simpler, quicker, and more feasible (eg, Touch ID, Face ID, and text). Some of the platforms we evaluated offered (and some required) 2FA. Other platforms did not offer 2FA, stating that study accounts do not store user phone numbers to enable 2FA. In addition, inquire about whether and how data are stored on the platform and mobile device and how data are encrypted. All the platforms evaluated encrypted data as they were transported between phones and servers, and most platforms used Amazon Web Services for cloud storage.

Ask developers what kind of information is stored about the participant (eg, log-in information, survey data, and location), where (eg, on the platform and in a server), and for how long (eg, days, months, duration of the contract, or in perpetuity). Follow up to determine how data are deidentified, if at all, and where the key to the data is stored. Inquire as to whether the platform is compliant with the laws and regulations of your country that protect an individual’s data or information (eg, Health Insurance Portability and Accountability Act in the United States and General Data Protection Regulation in the European Union). In our evaluation, platforms typically temporarily store survey data on phones in an encrypted format not visible in phone storage or accessible to other apps. These data were deleted from the phone once they were uploaded to the server. Some platforms avoid storing user information by not creating user accounts. Other platforms create user accounts but do not require personal information. Still others use information such as email addresses to create accounts.

Determine whether the data will be stored in the country of research or abroad (see the section Consideration 1: Location) and on a physical server or the cloud. Determine how long the platform will retain the data, along with whether and when long-term data storage will incur an additional cost. During our evaluation, one platform maintained data storage for the license period and another platform maintained data storage for 6 months following the license period to allow researchers time for data analysis.

Before meeting with developers, know your budget. The platforms we evaluated offered complimentary project consultations and estimates, which were key in selecting among candidates.

Platforms vary widely in their pricing and cost structures. Platforms may charge a flat fee for a user license with varying levels (basic and premium), or platforms may provide quotes that are team and project specific, varying depending on your estimated number of participants, the size of your team, the roles of your team members (eg, researchers and clinicians), your institution, and required features (eg, basic EMA vs EMA plus passive sensing). Some platforms offer free, basic versions of their platform. For example, one platform in our evaluation offered a basic package for clinicians to use in private practice with their clients while researchers were asked to purchase a premium package. Another platform offered discounted costs with the purchase of a multiple-year license. Costs may also vary based on whether you are part of an academic institution or a government institution and whether your project is single or multisite. Academic institutions are typically lower in cost than private entities, and single-site projects are lower in cost than multisite projects.

Consider how necessary, desired, and unnecessary features may impact platform costs. Sophisticated tools such as passive sensing, pairing with wearable devices, dyadic linking of participants, or compatibility with health care systems are costly to develop and maintain. Furthermore, images and videos may be expensive for developers to store. One of the platforms that we evaluated offered advanced features, including the ability to access electronic health records and facilitate biospecimen collection, which resulted in high overall costs for the platform. If your project requires only EMA, choose a platform with basic offerings or ask if you can opt out of extra features.

The next steps for the platform (eg, financial viability and developments) and your study and research program (eg, the need for additional features) should be considered before you partner with developers.

Consider the length of your study and explicitly ask developers their expectation for being operational for that duration and longer. A single, reliable platform is critical for stable data collection. Funding sources may inform the long-term stability of a platform. For example, if a platform is grant funded, inquire about plans for financing the platform when the funding period elapses. In addition, ask whether and when developers plan to launch upgrades to, or new versions of, the platform and how those developments might interface with an ongoing study. These considerations are especially critical for longitudinal studies.

As you plan your research program (eg, integration of additional measures delivered via EMA, passive sensing, and wearable devices), consider whether the platform has or can develop the capacity to support these plans. At the same time, it may be appropriate to select a particular platform for one phase of research with the understanding that the platform may be reevaluated in the future.