The following 3 focus groups of parents were assembled from the study population (see above): (1) four English-speaking parents with lower health literacy and/or lower educational achievement (high school education or less), (2) seven English-speaking parents with at least a college education and higher health literacy, and (3) four Spanish-speaking parents with diverse health literacy. Experienced moderators (authors BL and RB) followed focus group scripts to encourage discussion of (1) words used in ADHD care, (2) medications, (3) forms that parents are asked to fill out, (4) use of computers to input health data, and (5) parents’ preferences and ideas for how behavioral data and medication data might be displayed using preliminary design sketches as examples. Our process was consistent with the methods espoused by Gibbs [24].

For analysis of the focus group data, we employed a variation on content analyses [20] commonly used in the social sciences, such as grounded theory [25] and narrative explanatory model [26] methodologies. We first assembled and reviewed the collected data (ie, audio recordings, notes, and transcripts) generated for each focus group. Data for each group session were examined to identify majority opinions, dissenting opinions, examples, and anecdotes from the 5 topic areas that framed the discussion for that particular group. Summative opinions and recurring terms were then compared across the 3 focus groups to identify any difference in the pattern of collected data. A group of 3 individuals (the 2 moderators and the observer for the sessions) discussed the data and arrived at consensus for what findings represented a uniform result or a difference based on language or health literacy. The consensus findings were then separately reviewed by the first author to confirm their validity against the interview transcripts. No formal reliability measures were employed during the analysis of this qualitative data.

We offer the following demonstration of how the categorization of the focus group results informed the development of the initial prototype. Asked to explain the term “impulsive,” parents uniformly gave a consistent description. In contrast, presented the term “inattentive,” parents with lower health literacy stated “argumentative” and “provocative” were similar to “inattentive,” while parents with higher health literacy stated that “can’t focus” was similar to “inattentive.” This discrepancy in the interpretation of a core ADHD symptom reinforced the need for the interface to use language familiar to all users.

Specific displays for the user-interface (the viewer) of a prerelease version of the Indivo platform were analyzed by a single physician-informatician trained in human factors and information design. The main menu page and the medication data entry page were examined from the perspective of a parent who would engage the system for the first time with the intent of initiating documentation/tracking/management for ADHD. The informatician examined each display for its information density, structure, semantics, and navigational cues with the intent of discovering opportunities to improve how the system would support culturally diverse parents with a range of skills in the areas of technology and health literacy.

In response to the finding of the needs assessment and heuristic evaluation, detailed in the “Results” section, the prototype was developed within a prerelease version of Indivo, an open-source PCHR with strong security features and robust interoperability for data exchange between electronic health data sources [4,15]. To create the capacity for childhood ADHD clinical information management within Indivo, we initiated the development of a personal health application to support patient-driven data entry as a critical first step in designing an information management system that would demonstrate value longitudinally.

The initial prototype was tested in July 2007 at the Bentley Design and Usability Center (Bentley University, Waltham, MA, USA) using a cognitive walk-through protocol in which subjects were “instructed to verbalize their thoughts as they perform a particular experimental task” [27]. A trained facilitator led each subject through the protocol using a standard script that invited participants to enter data about their child’s ADHD medications and behaviors beginning with the phrase, “I will ask you to perform several tasks using the computer to enter information about your child.” (The full script of the test is available in Multimedia Appendix 1). During testing, we collected observations made about usability issues such as how subjects chose to enter data (eg, typed medication names as free text or selected from a drop-down list). While using the initial prototype, subjects were invited to describe their reactions to it [20,27]. For example, subjects entering data might ask themselves, “Am I receiving adequate confirmation of my actions?” The facilitator further explored subjects’ emotional responses and opinions generated by entering data. To expose problems that would otherwise not be captured, subjects were asked to explain comment or intents of initiated actions. Collected data included audiovisual recording of users’ verbal reactions and facial expressions, and on-screen mouse movements. Each testing session was conducted individually and lasted approximately 1.5 hours.

Using well-accepted social science analytic methods [28], we analyzed and summarized the collected data (ie, video recordings, notes, and NASA index ratings) to identify the usability issues and their causes, which included areas of confusion, unfamiliar or ambiguous terminology, lack of consistency across screens, navigation issues, and the like. After the multidisciplinary team analyzed the rich qualitative data generated by following the think-aloud protocol during the initial usability tests, the team prioritized the findings before generating recommendations for revising the personal health application. Subjects’ computer experience was determined by subjects’ reports and observations made by investigators during testing sessions [22,23], which was then categorized as high, moderate, or low (ie, does not use computers).

In response to comments from initial user testing (see “Results” section and Table 3, which summarizes the changes made to the initial prototype in response to the identified problems), the prototype was refined before it underwent performance testing. Of note, the medication module was significantly redesigned with specific attention to semantics around dose and frequency (see Figure 2).

We found that parents attached a range of descriptors to ADHD as a condition. Some descriptors were specific to the formal medical definition (eg, hyper, disorganized, and hard to control), while others addressed comorbid conditions (eg, anxious, angry, and fixation on certain topics).

Parents were shown alternative presentation formats for a behavioral questionnaire with 55 items. Formats displayed included one question per screen, groups of questions, and a longer list that would require scrolling. This questionnaire included the previously validated Vanderbilt Parent Assessment Scale, [30] which collects data on ADHD behaviors necessary for medication-related management decisions. Overall, participants preferred questions presented as one long list, and cited the efficiency of that approach. Of note, only one parent in the lower education, lower health literacy group preferred the one-question-per-screen approach.

Parents were shown two approaches for selection of a medication name: (1) a text box that captured spelling and sounded features of entered letters that would match to a medication name, and (2) a vertically-oriented alphabetical list of all prescription ADHD medications with an index of horizontal letters and a scroll bar to facilitate rapid localization of a given medication name. Parents endorsed the list for selecting medications, recognizing that the names were already spelled-out correctly. Parents also recognized the value of the dynamic text box that could react to the likely name being entered.

The information-sharing goal for the parent was to provide (or confirm) enough information on behaviors, current medication use, and potential side effects to allow for inferences regarding disease control and potential management options. The main menu display emphasized storage and retrieval of information at the summative level but did not show a clear roadmap for data entry. Users challenged by health literacy or technology experience would ask, “Where do I start?” and “What can I do with each of these labels in the index?”

The view of the main menu of the prerelease Indivo displayed the label “medication” in the left index; selecting the “add new medication” function led to a single screen with 7 data elements (either text box or drop-down menu) awaiting data entry (Figure 1). The accuracy of the data entered into the text box depended on the users’ keyboard-specific skills, medical knowledge, and comprehension of the labels. For example, prior research with parents identified the concept of “dose” as problematic and subject to multiple interpretations [31].

The focus groups and heuristic evaluation each confirmed the need for a disease-specific personal health application that provides a tailored data entry environment for the parent of a child with ADHD. Table 1 summarizes the main design features of the initial prototype proposed for an ADHD personal health application based on issues identified during the needs analysis. Given the importance of accurate capture of dose-specific information on ADHD medications, a structured and hierarchical approach to entry of data in this domain was designed in which data entry began with name and then moved to form, then strength, then frequency/dose, and lastly to review and edit (see Figure 3 for dose screen). Successful navigation and data entry required users to recognize and interact with drop-down menus and scroll bars using a mouse and/or keyboard.

Subjects’ overall reaction to the concept of the personal health application was positive, as emphasized by CW-1: “I think it’s awesome! It would save time by entering information before the visit; the doctor would be better prepared.” All subjects found the progression of data entry tasks intuitive, and most liked the feedback they received after completing each step. Problems identified during the usability tests are presented in Table 3. While most found the homepage easy to use, two felt it was visually unpleasant (eg, “bland,” and “clinical”). The purpose of the computer system was unclear to some (ie, they did not know whether it was to get or give information). Some subjects felt there should be a help button on the menu.

Subjects endorsed the concept of a medication summary page. Notably, 8 of 10 subjects failed the medication data entry task for which success required a parent to enter all attributes of the medicine (ie, name, form, strength, dose, and frequency), review the summarized data on-screen, and endorse it as correct. The attributes of dose and frequency created the most confusion. For example, a subject with higher health literacy and with moderate computer experience (CW-1) remarked: “I don’t understand…what [does] it mean by ‘dose?’” before leaving the field blank. While entering data on dose, a well-educated subject with higher health literacy and high computer experience (CW-5) commented: “Oh, wait a minute…two doses (shakes head)…two tablets; go back (smiles)…one tablet twice a day; there you go (smiles while laughing).” Of note, two parents (CW2 and CW8) entered both name and strength information (ie, Concerta 27 mg) into the dynamic text search box, which caused an error in returned matches and ultimate frustration for subjects.

Participants were divided about whether they preferred the dynamic text box or the alphabetized list. A subject with lower health literacy, little computer experience, and no high school degree (CW-3) noted: “You know how some people can’t really spell the names of the medications? I would like to see the list because you would be able to say, ‘yeah, that’s the medicine.’” Terminology about medication strength introduced confusion for users—the drop-down list for combination medicines displayed two numbers that represented the strength of each component and was not intuitive. For example, a well-educated subject with higher health literacy and high computer experience (CW-9) noted: “I am not a pharmacist; how am I supposed to know what 10/2.5 is?”

Of the 7 subjects, 5 successfully completed all 3 data entry tasks; subject PT-6 failed all three tasks, and PT-2 failed the behavioral symptom task but successfully completed the other two tasks. An example of the importance of providing users feedback about completing an interaction task, PT-2 stated: “It was just confusing, because it doesn’t say ‘thank you for completing the above three steps; you’re finished.’ You are just sort of wondering, ‘Did I miss a question?’” In addition to having lower health literacy, less than high school education, and low computer experience (see Table 1), PT-6’s poor performance was influenced by poor screen-scrolling skills that resulted in his not viewing all the questions.

The mean of all 7 subjects’ total time on task was 11.9 minutes (SD 6.5 minutes, range 7 to 26 minutes, median 9 minutes), with the subject who failed all 3 tasks (PT-6) having the longest time on task (26 minutes) and the subject who failed 1 task (PT-6) having the shortest time on task (7 minutes). Users’ previous computer experience was highly correlated with their time on task (r = .86, P = .01). For the 5 subjects who successfully finished all 3 data entry tasks, the mean task completion time (TCT) was 8.8 minutes (SD 1.5, range 7 to 11 minutes, median 9 minutes).

Completing the behavioral module took subjects the most time. After restricting analyses to those who completed each task, the mean TCT for the behavioral task was 5.5 minutes (median 4 minutes, range 2 to 5 minutes) compared with the medication module’s mean completion time of 2.2 minutes (median 3 minutes, range 1 to 5 minutes) and the side-effect module’s mean of .72 minutes (median 1 minute, range 0 to 1 minute).

Time on task was not associated with successfully completing all 3 tasks (P = .27), but subjects who successfully completed the behavioral module trended toward having briefer time on task (mean 3.6 minutes) than the 2 who didn’t complete this task (mean 10 minutes, P = .07). The amount of time subjects reported having spent on completing the 3 tasks (as measured by the temporal subsection of the NASA Task Load Index) was correlated with the actual time they spent on the 3 tasks (r = .74, P = .06).

The aggregate NASA Task Load Index score was not associated with successfully completing all 3 tasks (P = .92), nor were scores on the effort and frustration NASA Task Load Index subscales associated with successfully completing all 3 tasks (P = .36 and P = .67, respectively) . Similarly, users’ previous computer experience was not correlated with their total NASA Task Load Index score (r = .18).

Figure 4 presents the task burden experienced by subjects during data entry. Subjects’ responses confirmed the low physical task burden of using the computer even for the individual who did not master the mechanics of the mouse. Although some subjects experienced high frustration, their scores on the frustration subsection of the NASA Task Load Index were not correlated with time demands of the interface (ie, time on task) (r = .19, P = .69). The two users with the highest frustration scores were college-educated and had high health literacy, with one having high and the other moderate computer experience. No clear differences were evident for median task scores between subjects who did or did not successfully complete all 3 tasks.