Background: An established and well-known method for usability assessment of various human-computer interaction technologies is called heuristic evaluation (HE). HE has been adopted for evaluations in a wide variety of specialized contexts and with objectives that go beyond usability. A set of heuristics to evaluate how health information technologies (HITs) incorporate features that enable effective patient use of person-generated health data (PGHD) is needed in an era where there is a growing demand and variety of PGHD-enabled technologies in health care and where a number of remote patient-monitoring technologies do not yet enable patient use of PGHD. Such a set of heuristics would improve the likelihood of positive effects from patients’ use of PGHD and lower the risk of negative effects.
Objective: This study aims to describe the development of a set of heuristics for the design and evaluation of how well remote patient therapeutic technologies enable patients to use PGHD (PGHD enablement). We used the case of Kinect-based stroke rehabilitation systems (K-SRS) in this study.
Methods: The development of a set of heuristics to enable better use of PGHD was primarily guided by the R3C methodology. Closer inspection of the methodology reveals that neither its development nor its application to a case study were described in detail. Thus, where relevant, each step was grounded through best practice activities in the literature and by using Nielsen’s heuristics as a basis for determining the new set of heuristics. As such, this study builds on the R3C methodology, and the implementation of a mixed process is intended to result in a robust and credible set of heuristics.
Results: A total of 8 new heuristics for PGHD enablement in K-SRS were created. A systematic and detailed process was applied in each step of heuristic development, which bridged the gaps described earlier. It is hoped that this would aid future developers of specialized heuristics, who could apply the detailed process of heuristic development for other domains of technology, and additionally for the case of PGHD enablement for other health conditions. The R3C methodology was also augmented through the use of qualitative studies with target users and domain experts, and it is intended to result in a robust and credible set of heuristics, before validation and refinement.
Conclusions: This study is the first to develop a new set of specialized heuristics to evaluate how HITs incorporate features that enable effective patient use of PGHD, with K-SRS as a key case study. In addition, it is the first to describe how the identification of initial HIT features and concepts to enable PGHD could lead to the development of a specialized set of heuristics.
Varying Effects of Person-Generated Health Data
Person-generated health data (PGHD) are produced from technologies that allow people to access and utilize health data that they themselves generate outside of a health clinic setting, and to share these data with health care providers and others, typically via the internet. These technologies are designed to generate data about a range of health conditions and in pursuit of a range of health outcomes for remote patient monitoring [, ]. Example technologies include home-based, web-based, mobile, and wearable apps, which cater to a range of health conditions such as irritable bowel syndrome [ ], diabetes [ , ], and bipolar disorder [ ]; text reminders for asthma management [ ]; and social networking sites for the mental well-being of cancer patients [ ].
It is possible for PGHD utilization to have positive, negative, or nil effects on patients who use these technologies. PGHD use has been reported to provide patients with a range of benefits, such as increasing interest in their own health care processes [, , ], allowing them to monitor and manage their own health status [ ], and motivating them to undergo positive behavior change [ , ]. However, PGHD utilization can also cause patients to feel frustrated and discouraged [ ], and some patients may feel excluded from the benefits of PGHD [ ]. This highlights the need to consider patients’ perspective in the design and development of health technologies [ ], particularly those that generate PGHD [ ].
Measuring PGHD Effects
The varying effects of PGHD use on patients necessitates the development of a patient-reported outcome measure (PROM) of utilizing PGHD, or PROM-PGHD , to provide a standardized way of measuring PGHD outcomes and build empirical evidence about PGHD [ ]. PROMs are used to assess the health status or experience of health services and interventions by patients themselves [ - ] and have been shown to improve the precision of evaluating health information technologies (HITs) [ ]. Similarly, PROMs-PGHD would measure patients’ health outcomes or status as a result of accessing and using PGHD. Moreover, they could complement other clinical health outcome indicators, similar to how PROMs are used alongside other biometric measurements of patient health [ ].
The authors previously developed a PROM-PGHD that entailed establishing health outcomes that should be measured for a specific technology category and a health condition [, ]. The process identified some features that a domain or type of PGHD-enabled technology, that is, stroke rehabilitation systems, should have to evaluate their effectiveness in producing positive effects on patients who use PGHD to self-monitor or manage a specific health condition. For example, simulated stroke rehabilitation technologies should provide patients with real time information on the remaining exercise repetitions they need to do for them to self-manage their limited energy [ ].
Evaluation of Technologies Using Heuristics
Using a list of features based on predetermined criteria to assess technology categories is called heuristic evaluation (HE). First introduced by Nielsen and Molich , HE is an established and well-known method for the usability assessment of various human-computer interaction technologies [ ]. Using a set of guidelines, called heuristics, expert evaluators are able to quickly and efficiently identify issues that could affect the usability of technology artifacts [ ]. In practice, it could also be conducted during the design process to identify potential problems before launch [ , ].
HE has since been adopted for evaluations in a wide variety of specialized contexts and with objectives that go beyond usability. This has resulted in specialized heuristics, which could be used to direct evaluators to focus on assessing technologies according to specific concepts of interest . Nielsen’s original set of heuristics has been adopted or augmented to assess the quality of novice programing systems [ ], identify common mistakes caused by novice and experienced users of a web-based health record system for nurses [ ], and assess the safety of a HIT [ ]. Such specialized heuristic evaluations have identified problems beyond the original scope of the original heuristics [ ] and could therefore increase the effectiveness of evaluations in varied contexts [ ] and improve the quality of insights gained [ ].
Heuristics for PGHD Enablement
Developing a set of heuristics for use in the formative stages of development of patient and consumer health technologies could ensure more deliberate PGHD-enabling designs. This would improve the likelihood of positive effects from patients’ use of PGHD and lower the risk of negative effects. Such heuristics would also offer health care providers a standard way to evaluate technologies as part of selecting and implementing PGHD programs with their patients . HE would complement other in-depth PGHD evaluation methods [ , ] to explore PGHD outcomes [ , ], integration with clinical tools [ ], and assessment of quality management [ ]. A set of heuristics to evaluate how HITs incorporate features that enable effective patient use of PGHD is needed in an era where there is a growing demand and variety of PGHD-enabled technologies in health care and where a number of remote patient-monitoring technologies do not yet enable patient use of their PGHD. Furthermore, research into the outcomes and benefits of PGHD has not kept up [ , , ]. Specifically, methods are only emerging for systematic measurement of patient-reported health outcomes from accessing and utilizing PGHD [ , ]. In contrast to the already strong standardization of PROMs in many aspects of health care [ ], PGHD outcome evaluation methods are idiosyncratic and reported research is fragmented [ ].
In this study, we describe the development of a set of heuristics for the design and evaluation of how well remote patient therapeutic technologies enable patients to use PGHD (PGHD enablement). We used the case of Kinect-based stroke rehabilitation systems (K-SRS) to illustrate how the development of PROM-PGHD has led to the creation of these heuristics.
An important use case for evaluating PGHD effects is in simulated rehabilitation technologies for stroke, in particular those using body-tracking technology Kinect (Microsoft) . Key considerations within the context of stroke rehabilitation include the inherent complexity of care necessary to improve health outcomes [ , ], the requirement for stroke survivors to frequently conduct repetitive movement exercises [ , ], and the difficulty of accessing therapy accompanied by the high cost of care over an extended period [ ].
K-SRS offer a more convenient and cost-effective option . They allow stroke survivors to undertake prescribed movement exercises simulating activities of daily living (ADL) [ - ]. As survivors use these systems, they produce PGHD in the form of data indicating their therapeutic progress. These data, generally designed to be used by clinicians, have the potential to be used by stroke survivors themselves for health self-monitoring [ , , ].
Methodology to Develop Specialized Heuristics
A number of heuristic development processes have been employed in the literature. This includes literature reviews, analysis of usability problems, mixed processes, use of guidelines, interviews, and analysis of theories related to a specific domain . Of these, only one methodology known as the R3C [ ] has been used extensively (14 times) [ , , ]. The authors have since improved it [ ]. Another widespread approach is to point out the limitations of existing heuristics, in particular [ ], and build on these by exploring other aspects that need to be considered for the target domain. The original set of usability heuristics did not detail the steps taken to develop them [ , ]. Subsequently reported processes for developing heuristics for specific application domains have varied.
Procedure to Develop Heuristics for PGHD Enablement
The development of a set of heuristics for PGHD enablement was primarily guided by the R3C methodology most utilized and applied in the literature . However, a closer inspection of the methodology reveals that neither its development [ ] nor its application to a case study [ ] were described in explicit detail, leaving room for flexibility and interpretation for future implementations. There are also overlaps between its steps and the other heuristic development processes described earlier [ ]. For instance, step 1 explores and describes the target technology domain, followed by step 2, where the meaning of usability is reexamined within the context of the target technology domain. Thus, a literature review may be conducted for these steps [ ]. Moreover, although those steps do not specify conducting qualitative activities such as interviews and focus groups, their goals may indeed be augmented by such activities.
Thus, the development of a specialized set of heuristics for K-SRS technologies was guided by the R3C . Where relevant, each step was grounded through the best practice activities in the literature, that is, analysis of the context of use through a literature review, interviews, and focus groups, and by using the original heuristics [ ] as a basis for determining the new set of heuristics. As such, this study builds on the R3C methodology, and the implementation of a mixed process is intended to result in a robust and credible set of heuristics. Moreover, the detailed implementation of each step is intended to aid future developers of specialized heuristics for PGHD enablement within various contexts of health conditions and technology domains.
The steps of R3C are described in. The objective of this study was to develop an initial set of heuristics for PGHD enablement; thus, only steps 1 to 4 are implemented in this study. Evaluation of the heuristic set (step 5) and further refinement (step 6) is an area for future research.
Methodology to develop a specialized set of heuristics.
Steps of the R3C methodology to develop a specialized set of heuristics [, , ]
- Step 1: Exploratory stage
- Examine literature related to the main topics or technology domain of the research.
- Step 2: Descriptive stage
- Highlight key concepts from step 1. Connect the information, and assign weights.
- Step 3: Correlational stage
- Identify the characteristics that the heuristics should have. Original heuristics [ ] may be used as a basis. Clarify the need to consider the target technology domain.
- Step 4: Specification stage
- Formally specify the set of proposed specialized heuristics, using a standard template. Prioritize heuristics, and report any missing elements that need to be added.
- Step 5: Validation or experimental stage
- The new set of heuristics is tested on the target domain, and usability problems or issues found are compared with those that would be found using traditional heuristics.
- Step 6: Refinement stage
- The new set of heuristics is refined or improved following analysis of the results of step 5. However, this occurs iteratively, that is, if no more changes are recommended after step 5 then the development ends there.
Step 1: Exploratory Stage
A literature review was conducted as part of the PROM-PGHD development process to understand how K-SRS have been designed to enable PGHD and what benefits, if any, were recorded  using them. Similar to how this step was applied for this case [ ], the review also analyzed features of this domain, such as the effectiveness of K-SRS. Moreover, the types of PGHD generated by K-SRS were reviewed as well as how people were able to access those PGHD, the intended users and uses of PGHD, and any effects on patients from using PGHD [ ].
The literature review described the context under which PGHD is produced and used, relevant for step 4 when the heuristics are specified. As part of this step, factors that may influence the outcomes resulting from accessing and using PGHD were analyzed. The review has shown that PGHD was given to patients to guide them, for example, change movement behavior or actions to perform correct exercise movements. PGHD were also provided to help them achieve their short- and long-term goals . However, although people were generally provided with some form of PGHD as feedback, they did not have access to all their PGHD, which clinicians have access to. Other uses of PGHD include clinicians analyzing them to tailor rehabilitation programs for patients; researchers analyzing the effects on patients when they use a K-SRS; and to assess a variety of K-SRS, for example, how effective and reliable they are, and even compare them with other domains of technologies [ ]. The review also identified one study that reported that a patient’s daily access to her PGHD caused her to remember them over time and be motivated to improve [ ].
The review also highlighted the lack of patient-centered design in the development of K-SRS, given that the focus of providing data back to patients was for prescribing tasks and not to allow patients to access and make sense of their PGHD themselves. This is a missed opportunity to encourage patients to be more involved in their own health care , as patients who have direct access to their PGHD may become more engaged and thus improve their health outcomes [ , , , ]. Indeed, the lack of patient participation in the design of such PGHD systems may undermine patients’ rehabilitation experience [ ]. This is indicative of the need for developers of technology-based rehabilitation systems to consider PGHD in their design, implementation, and evaluation and allow patients to access them [ ].
Step 2: Descriptive Stage
This step is described as having 3 tasks: highlighting, connecting, and assigning weights to the concepts found in step 1. Step 2 is applied for this study as follows. First, concepts found through the literature review that could enable the effective patient use of PGHD were identified. Second, the concepts were categorized based on their inherent meaning according to 5 different reported effects of PGHD use thematically derived from the literature. Articles from a significant journal special issue on PGHD were inductively analyzed as an efficient and targeted way to identify reported effects of PGHD utilization from a range of HITs for different health conditions . This process identified 5 different effects that may result from patients’ use of their PGHD. PGHD may influence health-related behavioral or attitude changes, influence patients’ management of their own care owing to changes in how they feel about their health status, influence interest in their health care processes, facilitate their personal care goals, and influence their relationship with care providers. Concepts that did not match any of these effects were assigned a different category.
shows the categorized concepts highlighted from the literature review. More example quotes are shown in . The Weight column is discussed in the following paragraphs.
In the third activity of this stage, the concepts were assigned weights through qualitative studies. As part of the PROM-PGHD development process, the input of target users (stroke survivors) and experts (physiotherapists) was elicited through focus groups and interviews . On the basis of the health and technology case under consideration, stroke survivors with experience interacting with a leading K-SRS called Jintronix [ , ] were asked to respond to open-ended questions around the effects resulting from their access to and utilization of PGHD. Meanwhile, the clinicians were asked how they thought that access and utilization of PGHD would affect the stroke survivors they were working with. Participant recruitment was conducted in Australia at 3 different sites, with ethics approval granted by the Human Research Ethics Committees of Deakin University (2017-087), Austin Health (HREC/17/Austin/492), and the University of Melbourne (1852259.1). A total of 10 stroke survivors (7 females and 3 males) were recruited through Deakin University; 5 clinicians (4 females and 1 male) and 6 stroke survivors (1 female and 5 males) were recruited through the Austin Health hospital; and 1 stroke survivor (female) was recruited through Headway ABI in Queensland, Australia.
The qualitative studies enabled the authors to receive direct feedback from PGHD users to assign weights to the concepts found from step 1. In addition, the studies also bridged any relevant gaps between the concepts described in the literature, those perceived by clinicians, and those reported as important by stroke survivors themselves . New PGHD outcomes were documented that were not previously identified through the review and analysis of the literature described earlier [ ], leading to the identification of new features that should be incorporated in technologies to enable effective patient use of PGHD.
Concepts identified through the literature review but not described by either stroke survivors or clinicians were assigned a weight of 0, concepts described by clinicians only were assigned a weight of 1, concepts described by stroke survivors only were assigned a weight of 2, and concepts described by both stroke survivors and clinicians were assigned a weight of 3.shows the key concepts identified through the literature review assigned with weights and example quotes describing the concept, if any. shows new concepts identified through the qualitative studies, also assigned with weights and listing example quotes.
The categories were initially applied by GD and then independently reviewed by the coauthors. Any disagreements in the categories applied were discussed in a meeting until agreement was reached for each concept. More example quotes are shown in.
|Stroke survivor health-related behaviors||PGHDa can guide stroke survivors to make appropriate movement behavior or action changes to perform an exercise correctly|
|Stroke survivor personal care goals||PGHD can help stroke survivors achieve their short- and long-term goals||0—Not described|
|Functional effectiveness of therapy||PGHD can help inform clinicians, to analyze the functional effects of simulated rehabilitation therapy, and tailor programs for stroke survivors||0—Not described|
|Evaluation of the PGHD-enabled K-SRSb||PGHD can be used to assess the effectiveness and reliability of a K-SRS, compared with other types of simulated rehabilitation technologies||0—Not described|
|Stroke survivor interest in care processes|
aPGHD: person-generated health data.
bK-SRS: Kinect-based stroke rehabilitation systems.
|Stroke survivor health-related behaviors|
|Stroke survivor personal care goals|
|Feelings about health status|
|Stroke survivor interest in care processes|
|Stroke survivor relationship with care provider(s)|
|Relationship with family and carers|
aPGHD: person-generated health data.
Step 3: Correlational Stage
In this stage, the characteristics that the new set of heuristics should have are defined, which are later specified in step 4.
The key concepts identified through the first 2 steps are used as the basis to define the characteristics. As opposed to concepts, characteristics read as features that K-SRS should have to enable stroke survivors to use PGHD. These characteristics are also matched with the original heuristics  to identify any similarities. Some of the concepts were defined with similar characteristics and, therefore, combined. To further increase the credibility of the resulting heuristics, any characteristic matching an original heuristic is reworded using the original heuristic as a guide. Moreover, concepts with a weight of 0, that is, not described by stroke survivors or clinicians, were not defined as a characteristic and correspondingly not specified as a heuristic.
The characteristics were initially defined by GD and then independently reviewed by the coauthors. Any disagreements were discussed in a meeting until agreement was reached for each characteristic.
lists the characteristics defined from the concepts identified previously that match an original heuristic [ ], and as we have reworded them. lists the characteristics that did not have a matching original heuristic [ ]. The numbering continues from through to , to indicate which characteristics were combined in the next step. To see which concepts were defined as which characteristics, please see .
|Number||Characteristics||Reworded using the original heuristic as a guide|
aPGHD: person-generated health data.
Characteristics defined from key concepts identified from steps 1 and 2, which do not have a matching original heuristic.
- Person-generated health data (PGHD)–enabled systems should present PGHD that indicate negative or decreasing therapy progress carefully and in a form that elicits a stroke survivors’ competitiveness with the self.
- PGHD-enabled systems should highlight PGHD indicating positive or improving therapy progress more and providing them with more frequency than negative or decreasing progress.
- PGHD-enabled systems should provide PGHD to stroke survivors for increased understanding about how their rehabilitation is contributing to their functional ability.
- PGHD-enabled systems should provide stroke survivors with the option to share their PGHD with loved ones in a secure manner.
Step 4: Specification Stage
In this stage, the characteristics defined in step 3 are specified as a heuristic, following a structured format: ID, name, definition, explanation of how the heuristic was developed, example(s) of when a system being evaluated complies with or violates the heuristic, expected benefits if the system complies with the heuristic, and anticipated problems of heuristic misunderstanding .
The ID applied to the heuristics was structured as [Number]-PGHD-W[Weight]. The weights were indicated to provide implementers an idea of the process that went through developing the heuristics. The heuristic names were written succinctly, in a similar fashion to the original heuristics . The characteristics were written as the heuristic definition. The explanation of each heuristic described how they were developed from one or more heuristic and provided example descriptions of those concepts by stroke survivors, clinicians, or both. The example and benefits of each heuristic were described based on the literature review [ ] and qualitative studies [ ]. The anticipated problems indicated where heuristic definitions may be quite close and may be misunderstood.
A number of characteristics were either similar or tightly complemented each other and were therefore combined into one heuristic. Characteristics 3 and 12 described the need for K-SRS to match the health and therapy context of the stroke survivors. Characteristics 5, 2, 6, and 7 described the need for K-SRS to always provide PGHD to stroke survivors. Finally, characteristics 8 and 9 both described the need for K-SRS to provide options for stroke survivors to seek more information, contact their clinicians, and allow their clinician to contact them.
A necessary augmentation is conducted in this step. In combining the characteristics, considerations had to be taken with the individual weights and the categories of those characteristics. Where the weights of the characteristics being combined were different, the highest weight was indicated for the resulting heuristic’s ID. Meanwhile, the categories were added as tag(s) under each heuristic and reworded to indicate how the heuristic informs implementers to enable PGHD use. This preserves the categories identified in step 2 as metadata that could aid implementers in understanding the concepts underlying each heuristic. Interestingly, the R3C methodology did not describe how the categories were relevant to the resulting heuristics .
The heuristics were initially specified by GD and then independently reviewed by the coauthors. Any disagreements were discussed in a meeting until agreement was reached for each specified heuristic. Moreover, steps 5 and 6 are expected to refine the example, expected benefits, and anticipated problems of each heuristic. Therefore, the heuristics presented here may still be revised later on .
The next section shows the new, initial set of heuristics for PGHD enablement.
Design Heuristics to Enable Better Use of PGHD for Testing
Eight new heuristics for PGHD enablement in K-SRS were created. Heuristics with higher weights are presented first, to highlight heuristics that should be prioritized. The first 6 heuristics have a weight of 3, and the last 2 heuristics have a weight of 2. There are 6 tags used: improve health-related behaviors, increase positive feelings about health status, facilitate positive personal care goals, increase interest in care processes, improve relationships with care providers, and improve relationships with family and caregivers.
For brevity, only the ID, name, tag(s), definition, and explanation are presented in. This could also be the short form that implementers may use when evaluating HITs, similar to how brief definitions are presented for the original heuristics [ ]. However, as with the original heuristics, implementers ought to understand the underlying concepts and anticipated problems of each heuristic and should, therefore, view the full specifications in .
Short form of the specified heuristics for person-generated health data enablement.
1-PGHD-W3: Encouraging person-generated health data (PGHD)
- Tag(s): Improve health-related behaviors; increase positive feelings about health status
- Definition: The system should highlight PGHD indicating positive or improving therapy progress and provide them with more frequency than negative or decreasing progress.
- Explanation: This heuristic was formed from 3 concepts identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors and clinicians commented on how PGHD can cause positive or negative emotions, correlated with whether the PGHD is positive or negative, for example, when survivors frequently see their PGHD improve, it encourages them to do more of the exercises. A clinician described how seeing PGHD could help survivors be more compliant. On the other hand, if survivors see their PGHD decline, they might lose faith in the system and not perform their exercises.
2-PGHD-W3: Evoking competitiveness with self
- Tag(s): Facilitate positive personal care goals
- Definition: The system should present PGHD that indicates negative or decreasing therapy progress carefully and in a form that elicits a stroke survivors’ competitiveness with the self.
- Explanation: This was a concept identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors described how seeing their PGHD can motivate them to try harder, sometimes even when their PGHD indicates that they are not doing very well with their therapy exercises, because it can feel like a competition with themselves. Clinicians agree that it can give survivors something to keep them striving to be better.
3-PGHD-W3: Understandable health data
- Tag(s): Increase positive feelings about health status
- Definition: PGHD-enabled systems should ensure that PGHD is clearly understandable to stroke survivors. The system should speak their language, with words, phrases, and concepts familiar to them, rather than system-oriented terms. They should follow real-world conventions, making information appear in a natural and logical order.
- Explanation: This was a concept identified through a literature review and confirmed by stroke survivors and clinicians. A stroke survivor found that simply having percentages as feedback for performing an exercise could be confusing, as it could be misunderstood as being compared with people who have not had stroke. Clinicians also agreed that PGHD may not be understood by survivors.
4-PGHD-W3: Visibility of health progress
- Tags: Increase positive feelings about health status; increase interest in care processes
- Definition: PGHD-enabled systems should always provide PGHD to stroke survivors to keep them informed about what is going on with their health status, through appropriate feedback within reasonable time.
- Explanation: This heuristic was formed from 4 concepts identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors described how PGHD could help them understand their rehabilitation progress, to determine whether their functional progress was headed in the right direction and learn when they have made a mistake. PGHD could also help them self-manage their energy as they perform their exercises and increase their trust in their therapy process.
5-PGHD-W3: Help and support
- Tag(s): Improve relationship with care providers
- Definition: Even though it is better if the system can be used without additional help or documentation, it may be necessary to provide them. PGHD-enabled systems should provide stroke survivors the option to contact a clinician about their PGHD; allow their clinicians to contact them; or at least provide survivors with the option of viewing functional, action-based suggestions for them to achieve their therapy goals. Any such information should be easy to search, focused on the survivors’ exercises, list concrete steps to be carried out, and not be too lengthy.
- Explanation: This heuristic was formed from 2 concepts identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors described how they would want to contact their therapists, particularly if they have not been doing well to ask for assistance and ask for advice on what they could do better. Clinicians also described how it would be good if therapists could contact survivors when they are not being compliant. They also described how stroke survivors would likely ask for more explanation and for advice on how they can do things differently to improve their PGHD.
6-PGHD-W3: Communication of health data
- Tag(s): Improve relationship with family and carers
- Definition: PGHD-enabled systems should provide stroke survivors with the option to share their PGHD with concerned parties, for example, loved ones, in a secure manner.
- Explanation: This was a concept identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors described how it could help them communicate their health status to people they would like to share it with, particularly when they are tired or when they “run out of words.” Clinicians described how survivors might want to share their PGHD with family, particularly with the younger generation who are more technologically inclined.
7-PGHD-W2: Guide for correct exercise movement or actions
- Tag(s): Improve health-related behaviors
- Definition: The system should help stroke survivors in performing more exercises or actions and to recognize, understand, and recover from errors they make. Guidance or error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
- Explanation: This was a concept identified through a literature review and confirmed by stroke survivors. A stroke survivor commented on how PGHD can help them work out what they were doing wrong and adjust accordingly.
8-PGHD-W2: Match between system PGHD and real-world context of stroke survivors
- Tag(s): Facilitate positive personal care goals; increase interest in care processes
- Definition: PGHD-enabled systems should present PGHD in a format that matches the real-world context, therapy progress, and goals of the stroke survivors. When presenting PGHD, for example, as a score, percentage, or a graph, they should be matched to the therapy goals of the survivors. These goals may change over the course of a survivors' therapy, from gaining functional progress in the first few months or years to maintaining function to perform ADL when their progress starts to plateau [ ]. PGHD needs to be connected with the needs of the stroke survivors and where they are clinically.
- Explanation: This heuristic was formed from 2 concepts identified through a literature review and confirmed by stroke survivors and clinicians. Stroke survivors described how receiving a percent score that did not indicate a 100% could be a negative motivator. Achieving 100% given their condition could be very difficult or even impossible. Clinicians also described how PGHD should help survivors understand how their therapy is contributing to improved performance of ADL.
This study is the first to develop a new set of specialized heuristics to evaluate how HITs incorporate features that enable effective patient use of PGHD, with K-SRS as a key case study. In addition, it is the first to describe how the identification of initial HIT features and concepts to enable PGHD could lead to the development of a specialized set of heuristics. As such, it uncovers a valuable dimension to the PROM-PGHD development method, which considers the sociotechnical context of HITs [, ]. This context is the complex nature of interactions between people’s unique health conditions and behavior and that of the technologies and tools they use within their environment [ ]. The prior development of a PROM-PGHD for the key case of K-SRS was guided by this context [ ]. Although PROMs have long been known to support the evaluation of health interventions, which may be supported by HITs [ ], they are developed to consider only the health condition of patients. Conversely, the nature of PGHD requires the development of a PROM-PGHD to consider the sociotechnical context of the patient experience, that is, both the health condition and technology category [ ].
Subsequently, this study highlights the value of considering the sociotechnical context of HITs in their design and evaluation. Although unexpected consequences and even failures can occur from design flaws and technical limitations, they are also often the result of sociotechnical factors . Thus, HIT usability testing and evaluation can benefit from frameworks, standards, and guidelines that consider the sociotechnical context of those technologies [ , ]. In particular, PROMs-PGHD [ ], heuristics for PGHD enablement, and cognitive work analysis [ ] ensure that understanding of the complexities of care is incorporated in HIT design and implementation [ ].
The R3C methodology  guided the heuristic development process; however, it was not applied without some difficulty. The methodology’s development, description, and application to a case study [ , ] were not described in explicit detail, which left too much room for flexibility. More specifically, it did not specify how the concepts identified from step 1 were going to be connected in step 2; how weights were going to be determined and applied to those concepts, and what influence those weights would have on the succeeding steps; how the characteristics were going to be identified in step 3 and how they differed from the concepts previously identified; and how the original heuristics were going to be “used” as a basis, after matching them with the defined characteristics. Moreover, although R3C presented a structured template for the specification of the heuristics in step 4, the resulting case heuristics only presented summaries or abbreviated versions [ , , ]. Subsequently, implementations of R3C’s steps to develop other specialized heuristics have varied [ - ].
A step-by-step implementation of the R3C methodology was presented. A systematic and detailed process was applied in each step of heuristic development, which bridged the gaps described earlier. It is hoped that this would aid future developers of specialized heuristics, who could apply the detailed process of heuristic development for other domains of technology, and additionally for the case of PGHD enablement for other health conditions. Throughout the process, it was observed that the concepts and characteristics may need to be combined. This study presents a way to retain the weights and categories applied to them as heuristic metadata to aid implementers in understanding their underlying concepts. Although the last 2 heuristics have a weight of 2, implementers should still aim to conduct the evaluation using all 8 heuristics. However, in case of any resource constraint, the first 6 heuristics should be prioritized.
The R3C methodology  was also augmented through the use of qualitative studies with target users (stroke survivors) and domain experts (physiotherapists) to determine the weights, discover new concepts, and ensure the defined characteristics and resulting heuristics were as close to the experiences of the users as possible. This is intended to result in a robust and credible set of heuristics, before the validation stage (step 5), after which the specified heuristics may be improved through the refinement stage (step 6). This is especially valuable, as the R3C methodology’s recommended iteration of the process leads from step 6 back to step 4 [ ], implying that steps 1 to 3 need to be rigorously implemented. Subsequently, the initial set of heuristics presented here will be validated and refined following steps 5 and 6.
Data gathered from the qualitative studies were used as an important augmentation in the heuristic development process. However, as those studies were primarily meant to support the development of a PROM-PGHD, it is possible that participants were not asked all the relevant questions around PGHD enablement features. In the event, many concepts found through the literature review were supported by qualitative studies, and indeed, most of the resulting heuristics were formed from new concepts found in those studies. Therefore, although it might appear to be a limiting factor, it actually indicates the robustness of the process followed and highlights the importance of this suggested augmentation. Moreover, the validation stage (step 5) still allows for possible additional heuristics to be identified.
The flexibility of the R3C methodology  meant that it is possible that the process followed in this study deviated from the original intent of the authors. However, we believe that the detailed, step-by-step implementation of R3C and the suggested augmentation have in effect modeled a way to build on and strengthen it.
The new set of heuristics for PGHD enablement, following a detailed, systematic development process augmented from best practice that we have presented, could serve as a guide for future developers of specialized heuristics in general and specifically for developers of heuristics for PGHD enablement of a variety of technology domains and health conditions. The new set of heuristics is needed in a period of rising demand for, supply of, and variety of PGHD-enabled technologies in health care. It offers health care providers a standard way to evaluate technologies as part of selecting and implementing PGHD programs with their patients , complementing other in-depth PGHD evaluation methods [ , ], and has a broader relevance for the design and implementation of HITs. In addition, an interesting dimension to the PROM-PGHD development process was discovered, and it highlights the value of considering the sociotechnical context of HITs in their design and evaluation.
GD would like to acknowledge the Melbourne School of Engineering through which his research scholarship is provided and his organizational sponsor—Newman College (University of Melbourne).
Conflicts of Interest
Steps 1-4, transforming concepts to heuristics.XLSX File (Microsoft Excel File), 19 KB
- Dimaguila G, Gray K, Merolli M. Patient-reported outcomes of utilising person-generated health data in simulated rehabilitation technology: perceptions of stroke survivors. Stud Health Technol Inform 2019 Aug 21;264:993-997. [CrossRef] [Medline]
- Rosenbloom ST. Person-generated health and wellness data for health care. J Am Med Inform Assoc 2016 May;23(3):438-439. [CrossRef] [Medline]
- Karkar R, Zia J, Vilardaga R, Mishra SR, Fogarty J, Munson SA, et al. A framework for self-experimentation in personalized health. J Am Med Inform Assoc 2016 May;23(3):440-448 [FREE Full text] [CrossRef] [Medline]
- Mamykina L, Levine ME, Davidson PG, Smaldone AM, Elhadad N, Albers DJ. Data-driven health management: reasoning about personally generated data in diabetes with information technologies. J Am Med Inform Assoc 2016 May;23(3):526-531 [FREE Full text] [CrossRef] [Medline]
- Kumar RB, Goren ND, Stark DE, Wall DP, Longhurst CA. Automated integration of continuous glucose monitor data in the electronic health record using consumer technology. J Am Med Inform Assoc 2016 May;23(3):532-537 [FREE Full text] [CrossRef] [Medline]
- Abdullah S, Matthews M, Frank E, Doherty G, Gay G, Choudhury T. Automatic detection of social rhythms in bipolar disorder. J Am Med Inform Assoc 2016 May;23(3):538-543. [CrossRef] [Medline]
- Johnson KB, Patterson BL, Ho Y, Chen Q, Nian H, Davison CL, et al. The feasibility of text reminders to improve medication adherence in adolescents with asthma. J Am Med Inform Assoc 2016 May;23(3):449-455 [FREE Full text] [CrossRef] [Medline]
- Erfani SS, Blount Y, Abedin B. The influence of health-specific social network site use on the psychological well-being of cancer-affected people. J Am Med Inform Assoc 2016 May;23(3):467-476. [CrossRef] [Medline]
- Sanger PC, Hartzler A, Lordon RJ, Armstrong CA, Lober WB, Evans HL, et al. A patient-centered system in a provider-centered world: challenges of incorporating post-discharge wound data into practice. J Am Med Inform Assoc 2016 May;23(3):514-525 [FREE Full text] [CrossRef] [Medline]
- Woods SS, Evans NC, Frisbee KL. Integrating patient voices into health information for self-care and patient-clinician partnerships: veterans affairs design recommendations for patient-generated data applications. J Am Med Inform Assoc 2016 May;23(3):491-495. [CrossRef] [Medline]
- Demiris G, Iribarren SJ, Sward K, Lee S, Yang R. Patient generated health data use in clinical practice: a systematic review. Nurs Outlook 2019;67(4):311-330. [CrossRef] [Medline]
- Shaw RJ, Steinberg DM, Bonnet J, Modarai F, George A, Cunningham T, et al. Mobile health devices: will patients actually use them? J Am Med Inform Assoc 2016 May;23(3):462-466 [FREE Full text] [CrossRef] [Medline]
- Constand MK, MacDermid JC, dal Bello-Haas V, Law M. Scoping review of patient-centered care approaches in healthcare. BMC Health Serv Res 2014 Jun 19;14:271 [FREE Full text] [CrossRef] [Medline]
- Dimaguila GL, Gray K, Merolli M. Person-generated health data in simulated rehabilitation using kinect for stroke: literature review. JMIR Rehabil Assist Technol 2018 May 8;5(1):e11 [FREE Full text] [CrossRef] [Medline]
- Dimaguila GL, Gray K, Merolli M. Measuring the outcomes of using person-generated health data: a case study of developing a PROM item bank. BMJ Health Care Inform 2019 Aug;26(1):e100070 [FREE Full text] [CrossRef] [Medline]
- Cresswell KM, McKinstry B, Wolters M, Shah A, Sheikh A. Five key strategic priorities of integrating patient generated health data into United Kingdom electronic health records. J Innov Health Inform 2019 Jan 4;25(4):254-259 [FREE Full text] [CrossRef] [Medline]
- US Food and Drug Administration. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claimsF. 2009. URL: https://www.fda.gov/media/77832/download [accessed 2020-06-19]
- PROMs background document. Canadian Institute for Health Information. 2015. URL: https://www.cihi.ca/sites/default/files/proms_background_may21_en-web_0.pdf [accessed 2020-06-19]
- Walsh TR, Irwin DE, Meier A, Varni JW, DeWalt DA. The use of focus groups in the development of the PROMIS pediatrics item bank. Qual Life Res 2008 Jun;17(5):725-735 [FREE Full text] [CrossRef] [Medline]
- Basch E, Spertus J, Dudley RA, Wu A, Chuahan C, Cohen P, et al. Methods for developing patient-reported outcome-based performance measures (PRO-PMS). Value Health 2015 Jun;18(4):493-504 [FREE Full text] [CrossRef] [Medline]
- Williams K. Patient-reported outcome measures: Literature review. Australian Health Services Research Institute, University of Wollongong.: -; 2016. URL: https://www.safetyandquality.gov.au/sites/default/files/migrated/PROMs-Literature-Review-December-2016.pdf [accessed 2020-06-19]
- Dimaguila GL, Gray K, Merolli M. Patient-reported outcome measures of utilizing person-generated health data in the case of simulated stroke rehabilitation: development method. JMIR Res Protoc 2020 May 7;9(5):e16827 [FREE Full text] [CrossRef] [Medline]
- Nielsen J, Molich R. Heuristic Evaluation of User Interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1990 Presented at: CHI'90; April 21-26, 1990; Los Angeles, California, USA. [CrossRef]
- Kientz J, Choe EK, Birch B, Maharaj R, Fonville A, Glasson C, et al. Heuristic evaluation of persuasive health technologies. In: Proceedings of the 1st ACM International Health Informatics Symposium. -: -; 2010 Presented at: ACM International Health Informatics Symposium; November 2010; Arlington Virginia USA p. 555-564. [CrossRef]
- Liu L, Hayes G. Heuristic Evaluation of Personal Health Records Systems. 2010 Presented at: CHI 2010; April 2010; Atlanta, Georgia, USA.
- Kölling M, McKay F. Heuristic evaluation for novice programming systems. ACM Trans Comput Educ 2016 Jun 27;16(3):1-30. [CrossRef]
- Scandurra I, Hägglund M, Moström D, Koch S. Heuristic evaluation extended by user analysis: a fast and efficient method to identify potential usability problems in health information systems. J Inform Technol in Healthc 2006;4(5):317-325 [FREE Full text]
- Borycki E, Kushniruk A, Nohr C, Takeda H, Kuwata S, Carvalho C, et al. Usability methods for ensuring health information technology safety: evidence-based approaches. Contribution of the IMIA working group health informatics for patient safety. Yearb Med Inform 2013;8:20-27. [Medline]
- 10 Usability Heuristics for User Interface Design. Nielsen Norman Group: UX Training, Consulting, & Research. 1994. URL: https://www.nngroup.com/articles/ten-usability-heuristics/ [accessed 2019-07-28]
- Khowaja K, Salim SS. Correction: heuristics to evaluate interactive systems for children with autism spectrum disorder (ASD). PLoS One 2015;10(8):e0136977 [FREE Full text] [CrossRef] [Medline]
- Abdolkhani R, Borda A, Gray K. Quality management of patient generated health data in remote patient monitoring using medical wearables - a systematic review. Stud Health Technol Inform 2018;252:1-7. [Medline]
- Mendis S. Stroke disability and rehabilitation of stroke: World Health Organization perspective. Int J Stroke 2013 Jan;8(1):3-4. [CrossRef] [Medline]
- Cardiovascular diseases (CVDs). World Health Organization. 2016. URL: https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds) [accessed 2020-06-19]
- Albiol-Perez S, Palacios- Navarro G, Guerrón-Paredes N, Gil-Gómez JA, Quilis JA, Gil-Gómez H, et al. The Perfetti Method, a Novel Virtual Fine Motor Rehabilitation System for Chronic Acquired Brain Injury. In: Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare. 2014 Presented at: PervasiveHealth'14; May 20-23, 2014; Oldenburg, Germany. [CrossRef]
- Byblow WD, Stinear CM, Barber PA, Petoe MA, Ackerley SJ. Proportional recovery after stroke depends on corticomotor integrity. Ann Neurol 2015 Dec;78(6):848-859. [CrossRef] [Medline]
- Mayo NE, MacKay-Lyons MJ, Scott SC, Moriello C, Brophy J. A randomized trial of two home-based exercise programmes to improve functional walking post-stroke. Clin Rehabil 2013 Jul;27(7):659-671. [CrossRef] [Medline]
- Lohse KR, Hilderman CG, Cheung KL, Tatla S, van der Loos HF. Virtual reality therapy for adults post-stroke: a systematic review and meta-analysis exploring virtual environments and commercial games in therapy. PLoS One 2014;9(3):e93318 [FREE Full text] [CrossRef] [Medline]
- Darekar A, McFadyen BJ, Lamontagne A, Fung J. Efficacy of virtual reality-based intervention on balance and mobility disorders post-stroke: a scoping review. J Neuroeng Rehabil 2015 May 10;12:46 [FREE Full text] [CrossRef] [Medline]
- Moreira MC, de Amorim Lima AM, Ferraz KM, Benedetti Rodrigues MA. Use of virtual reality in gait recovery among post stroke patients--a systematic literature review. Disabil Rehabil Assist Technol 2013 Sep;8(5):357-362. [CrossRef] [Medline]
- Ogourtsova T, Souza Silva W, Archambault PS, Lamontagne A. Virtual reality treatment and assessments for post-stroke unilateral spatial neglect: a systematic literature review. Neuropsychol Rehabil 2017 Apr;27(3):409-454. [CrossRef] [Medline]
- Quiñones D, Rusu C. How to develop usability heuristics: a systematic literature review. Comput Stand Inter 2017 Aug;53:89-122. [CrossRef]
- Figueroa I, Jiménez C, Allende-Cid H, Leger P. Developing usability heuristics with PROMETHEUS: a case study in virtual learning environments. Comput Stand Inter 2019 Jul;65:132-142. [CrossRef]
- Jiménez C, Rusu C, Roncagliolo S, Inostroza R, Rusu V. Evaluating a methodology to establish usability heuristics. -: -; 2012 Presented at: Conference of the Chilean Computer Science Society; November 2012; Valparaiso, Chile p. 51-59. [CrossRef]
- Rusu C, Roncagliolo S, Rusu V, Collazos C. A methodology to establish usability heuristics. ACHI 2011: The Fourth International Conference on Advances in Computer-Human Interactions. Wilmington, USA: IARIA; 2011. URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.684.7653&rep=rep1&type=pdf [accessed 2020-06-19]
- Quiñones D, Rusu C, Roncagliolo S, Rusu V, Collazos C. Developing usability heuristics for grid computing applications: lessons learned. Prog Adv Comput Intell Eng 2016:485-495. [CrossRef]
- Nielsen J, Molich R. Teaching user interface design based on usability engineering. SIGCHI Bull 1989 Aug;21(1):45-48. [CrossRef]
- Rusu C, Roncagliolo S, Tapia G, Hayvar D, Rusu V, Gorgan D. Usability heuristics for grid computing applications. In: The Fourth International Conference on Advances in Computer-Human Interactions. 2011 Presented at: ACHI 2011: The 4th International Conferences on Advances in Computer-Human Interactions ACHI; 2011; France.
- Pastor I, Hayes H, Bamberg S. A Feasibility Study of an Upper Limb Rehabilitation System Using Kinect and Computer Games. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2012 Presented at: EMBC'12; August 28-September 1, 2012; San Diego, CA, USA. [CrossRef]
- Brennan P, Valdez R, Alexander G, Arora S, Bernstam EV, Edmunds M, et al. Patient-centered care, collaboration, communication, and coordination: a report from AMIA's 2013 Policy Meeting. J Am Med Inform Assoc 2015 Apr;22(e1):e2-e6. [CrossRef] [Medline]
- Kristensen HK, Tistad M, Koch LV, Ytterberg C. The importance of patient involvement in stroke rehabilitation. PLoS One 2016;11(6):e0157149 [FREE Full text] [CrossRef] [Medline]
- DeWalt DA, Rothrock N, Yount S, Stone AA, PROMIS Cooperative Group. Evaluation of item candidates: the PROMIS qualitative item review. Med Care 2007 May;45(5 Suppl 1):S12-S21 [FREE Full text] [CrossRef] [Medline]
- Johnson L, Bird M, Muthalib M, Teo W. Innovative STRoke Interactive Virtual thErapy (STRIVE) online platform for community-dwelling stroke survivors: a randomised controlled trial protocol. BMJ Open 2018 Jan 9;8(1):e018388 [FREE Full text] [CrossRef] [Medline]
- Wozney LM, Baxter P, Fast H, Cleghorn L, Hundert AS, Newton AS. Sociotechnical human factors involved in remote online usability testing of two ehealth interventions. JMIR Hum Factors 2016 Feb 3;3(1):e6 [FREE Full text] [CrossRef] [Medline]
- Percival J, McGregor C. An evaluation of understandability of patient journey models in mental health. JMIR Hum Factors 2016 Jul 28;3(2):e20 [FREE Full text] [CrossRef] [Medline]
- Asan O, Yang Y. Using eye trackers for usability evaluation of health information technology: a systematic literature review. JMIR Hum Factors 2015 Apr 14;2(1):e5 [FREE Full text] [CrossRef] [Medline]
- St-Maurice JD, Burns CM. Modeling patient treatment with medical records: an abstraction hierarchy to understand user competencies and needs. JMIR Hum Factors 2017 Jul 28;4(3):e16 [FREE Full text] [CrossRef] [Medline]
- Solano A, Rusu C, Collazos C, Roncagliolo S, Arciniegas JL, Rusu V. Usability Heuristics for Interactive Digital Television. 2011 Presented at: The Third International Conference on Advances in Future Internet (AFIN 2011); August 2011; Nice, France.
- Sanz F, Galvez R, Rusu C, Roncagliolo S, Rusu V, Collazos C, et al. A set of usability heuristics and design recommendations for u-learning applications. Proc Int Conf Inf Technol New Gener 2016:983-993. [CrossRef]
- Schefer R, Bezerra MS, Zaina LA. Supporting the Development of Social Networking Mobile Apps for Deaf Users: Guidelines Based on User Experience Issues. In: Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion. 2018 Presented at: DSAI'18; June 20-22, 2018; Thessaloniki, Greece. [CrossRef]
- Munoz R, Chalegre V. Defining Virtual Worlds Usability Heuristics. In: Ninth International Conference on Information Technology - New Generations. 2012 Presented at: ITNG'12; April 16-18, 2012; Las Vegas, NV, USA. [CrossRef]
- Díaz G, del Rio CM. A Proposal of Usability Heuristics Oriented to E-Banking Websites. In: International Conference of Design, User Experience, and Usability. 2018 Presented at: DUXU'18; July 15-18, 2018; Las Vegas, NV, USA. [CrossRef]
|ADL: activities of daily living|
|HE: heuristic evaluation|
|HIT: health information technology|
|K-SRS: Kinect-based stroke rehabilitation system|
|PGHD: person-generated health data|
|PROM: patient-reported outcome measure|
|PROM-PGHD: patient-reported outcome measure of utilizing person-generated health data|
Edited by C Gibbons, G Eysenbach; submitted 20.11.19; peer-reviewed by PA Silva, L Novak; comments to author 30.03.20; revised version received 29.04.20; accepted 13.05.20; published 28.07.20Copyright
©Gerardo Luis Dimaguila, Kathleen Gray, Mark Merolli. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.07.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.