Person-generated health data (PGHD) are produced from technologies that allow people to access and utilize health data that they themselves generate outside of a health clinic setting, and to share these data with health care providers and others, typically via the internet. These technologies are designed to generate data about a range of health conditions and in pursuit of a range of health outcomes for remote patient monitoring [1,2]. Example technologies include home-based, web-based, mobile, and wearable apps, which cater to a range of health conditions such as irritable bowel syndrome [3], diabetes [4,5], and bipolar disorder [6]; text reminders for asthma management [7]; and social networking sites for the mental well-being of cancer patients [8].

It is possible for PGHD utilization to have positive, negative, or nil effects on patients who use these technologies. PGHD use has been reported to provide patients with a range of benefits, such as increasing interest in their own health care processes [5,9,10], allowing them to monitor and manage their own health status [11], and motivating them to undergo positive behavior change [1,12]. However, PGHD utilization can also cause patients to feel frustrated and discouraged [1], and some patients may feel excluded from the benefits of PGHD [11]. This highlights the need to consider patients’ perspective in the design and development of health technologies [13], particularly those that generate PGHD [14].

The varying effects of PGHD use on patients necessitates the development of a patient-reported outcome measure (PROM) of utilizing PGHD, or PROM-PGHD [15], to provide a standardized way of measuring PGHD outcomes and build empirical evidence about PGHD [16]. PROMs are used to assess the health status or experience of health services and interventions by patients themselves [17-19] and have been shown to improve the precision of evaluating health information technologies (HITs) [20]. Similarly, PROMs-PGHD would measure patients’ health outcomes or status as a result of accessing and using PGHD. Moreover, they could complement other clinical health outcome indicators, similar to how PROMs are used alongside other biometric measurements of patient health [21].

The authors previously developed a PROM-PGHD that entailed establishing health outcomes that should be measured for a specific technology category and a health condition [15,22]. The process identified some features that a domain or type of PGHD-enabled technology, that is, stroke rehabilitation systems, should have to evaluate their effectiveness in producing positive effects on patients who use PGHD to self-monitor or manage a specific health condition. For example, simulated stroke rehabilitation technologies should provide patients with real time information on the remaining exercise repetitions they need to do for them to self-manage their limited energy [1].

Using a list of features based on predetermined criteria to assess technology categories is called heuristic evaluation (HE). First introduced by Nielsen and Molich [23], HE is an established and well-known method for the usability assessment of various human-computer interaction technologies [24]. Using a set of guidelines, called heuristics, expert evaluators are able to quickly and efficiently identify issues that could affect the usability of technology artifacts [25]. In practice, it could also be conducted during the design process to identify potential problems before launch [24,25].

HE has since been adopted for evaluations in a wide variety of specialized contexts and with objectives that go beyond usability. This has resulted in specialized heuristics, which could be used to direct evaluators to focus on assessing technologies according to specific concepts of interest [24]. Nielsen’s original set of heuristics has been adopted or augmented to assess the quality of novice programing systems [26], identify common mistakes caused by novice and experienced users of a web-based health record system for nurses [27], and assess the safety of a HIT [28]. Such specialized heuristic evaluations have identified problems beyond the original scope of the original heuristics [29] and could therefore increase the effectiveness of evaluations in varied contexts [30] and improve the quality of insights gained [26].

Developing a set of heuristics for use in the formative stages of development of patient and consumer health technologies could ensure more deliberate PGHD-enabling designs. This would improve the likelihood of positive effects from patients’ use of PGHD and lower the risk of negative effects. Such heuristics would also offer health care providers a standard way to evaluate technologies as part of selecting and implementing PGHD programs with their patients [11]. HE would complement other in-depth PGHD evaluation methods [2,15] to explore PGHD outcomes [14,16], integration with clinical tools [2], and assessment of quality management [31]. A set of heuristics to evaluate how HITs incorporate features that enable effective patient use of PGHD is needed in an era where there is a growing demand and variety of PGHD-enabled technologies in health care and where a number of remote patient-monitoring technologies do not yet enable patient use of their PGHD. Furthermore, research into the outcomes and benefits of PGHD has not kept up [2,11,14]. Specifically, methods are only emerging for systematic measurement of patient-reported health outcomes from accessing and utilizing PGHD [11,15]. In contrast to the already strong standardization of PROMs in many aspects of health care [21], PGHD outcome evaluation methods are idiosyncratic and reported research is fragmented [11].

In this study, we describe the development of a set of heuristics for the design and evaluation of how well remote patient therapeutic technologies enable patients to use PGHD (PGHD enablement). We used the case of Kinect-based stroke rehabilitation systems (K-SRS) to illustrate how the development of PROM-PGHD has led to the creation of these heuristics.

An important use case for evaluating PGHD effects is in simulated rehabilitation technologies for stroke, in particular those using body-tracking technology Kinect (Microsoft) [15]. Key considerations within the context of stroke rehabilitation include the inherent complexity of care necessary to improve health outcomes [32,33], the requirement for stroke survivors to frequently conduct repetitive movement exercises [34,35], and the difficulty of accessing therapy accompanied by the high cost of care over an extended period [36].

K-SRS offer a more convenient and cost-effective option [14]. They allow stroke survivors to undertake prescribed movement exercises simulating activities of daily living (ADL) [37-40]. As survivors use these systems, they produce PGHD in the form of data indicating their therapeutic progress. These data, generally designed to be used by clinicians, have the potential to be used by stroke survivors themselves for health self-monitoring [14,38,39].

A number of heuristic development processes have been employed in the literature. This includes literature reviews, analysis of usability problems, mixed processes, use of guidelines, interviews, and analysis of theories related to a specific domain [41]. Of these, only one methodology known as the R3C [42] has been used extensively (14 times) [41,43,44]. The authors have since improved it [45]. Another widespread approach is to point out the limitations of existing heuristics, in particular [23], and build on these by exploring other aspects that need to be considered for the target domain. The original set of usability heuristics did not detail the steps taken to develop them [23,46]. Subsequently reported processes for developing heuristics for specific application domains have varied.

The development of a set of heuristics for PGHD enablement was primarily guided by the R3C methodology most utilized and applied in the literature [45]. However, a closer inspection of the methodology reveals that neither its development [44] nor its application to a case study [47] were described in explicit detail, leaving room for flexibility and interpretation for future implementations. There are also overlaps between its steps and the other heuristic development processes described earlier [41]. For instance, step 1 explores and describes the target technology domain, followed by step 2, where the meaning of usability is reexamined within the context of the target technology domain. Thus, a literature review may be conducted for these steps [47]. Moreover, although those steps do not specify conducting qualitative activities such as interviews and focus groups, their goals may indeed be augmented by such activities.

Thus, the development of a specialized set of heuristics for K-SRS technologies was guided by the R3C [45]. Where relevant, each step was grounded through the best practice activities in the literature, that is, analysis of the context of use through a literature review, interviews, and focus groups, and by using the original heuristics [23] as a basis for determining the new set of heuristics. As such, this study builds on the R3C methodology, and the implementation of a mixed process is intended to result in a robust and credible set of heuristics. Moreover, the detailed implementation of each step is intended to aid future developers of specialized heuristics for PGHD enablement within various contexts of health conditions and technology domains.

The steps of R3C are described in Textbox 1. The objective of this study was to develop an initial set of heuristics for PGHD enablement; thus, only steps 1 to 4 are implemented in this study. Evaluation of the heuristic set (step 5) and further refinement (step 6) is an area for future research.

This step is described as having 3 tasks: highlighting, connecting, and assigning weights to the concepts found in step 1. Step 2 is applied for this study as follows. First, concepts found through the literature review that could enable the effective patient use of PGHD were identified. Second, the concepts were categorized based on their inherent meaning according to 5 different reported effects of PGHD use thematically derived from the literature. Articles from a significant journal special issue on PGHD were inductively analyzed as an efficient and targeted way to identify reported effects of PGHD utilization from a range of HITs for different health conditions [15]. This process identified 5 different effects that may result from patients’ use of their PGHD. PGHD may influence health-related behavioral or attitude changes, influence patients’ management of their own care owing to changes in how they feel about their health status, influence interest in their health care processes, facilitate their personal care goals, and influence their relationship with care providers. Concepts that did not match any of these effects were assigned a different category.

Table 1 shows the categorized concepts highlighted from the literature review. More example quotes are shown in Multimedia Appendix 1. The Weight column is discussed in the following paragraphs.

In the third activity of this stage, the concepts were assigned weights through qualitative studies. As part of the PROM-PGHD development process, the input of target users (stroke survivors) and experts (physiotherapists) was elicited through focus groups and interviews [51]. On the basis of the health and technology case under consideration, stroke survivors with experience interacting with a leading K-SRS called Jintronix [14,52] were asked to respond to open-ended questions around the effects resulting from their access to and utilization of PGHD. Meanwhile, the clinicians were asked how they thought that access and utilization of PGHD would affect the stroke survivors they were working with. Participant recruitment was conducted in Australia at 3 different sites, with ethics approval granted by the Human Research Ethics Committees of Deakin University (2017-087), Austin Health (HREC/17/Austin/492), and the University of Melbourne (1852259.1). A total of 10 stroke survivors (7 females and 3 males) were recruited through Deakin University; 5 clinicians (4 females and 1 male) and 6 stroke survivors (1 female and 5 males) were recruited through the Austin Health hospital; and 1 stroke survivor (female) was recruited through Headway ABI in Queensland, Australia.

The qualitative studies enabled the authors to receive direct feedback from PGHD users to assign weights to the concepts found from step 1. In addition, the studies also bridged any relevant gaps between the concepts described in the literature, those perceived by clinicians, and those reported as important by stroke survivors themselves [1]. New PGHD outcomes were documented that were not previously identified through the review and analysis of the literature described earlier [14], leading to the identification of new features that should be incorporated in technologies to enable effective patient use of PGHD.

Concepts identified through the literature review but not described by either stroke survivors or clinicians were assigned a weight of 0, concepts described by clinicians only were assigned a weight of 1, concepts described by stroke survivors only were assigned a weight of 2, and concepts described by both stroke survivors and clinicians were assigned a weight of 3. Table 1 shows the key concepts identified through the literature review assigned with weights and example quotes describing the concept, if any. Table 2 shows new concepts identified through the qualitative studies, also assigned with weights and listing example quotes.

The categories were initially applied by GD and then independently reviewed by the coauthors. Any disagreements in the categories applied were discussed in a meeting until agreement was reached for each concept. More example quotes are shown in Multimedia Appendix 1.

In this stage, the characteristics that the new set of heuristics should have are defined, which are later specified in step 4.

The key concepts identified through the first 2 steps are used as the basis to define the characteristics. As opposed to concepts, characteristics read as features that K-SRS should have to enable stroke survivors to use PGHD. These characteristics are also matched with the original heuristics [23] to identify any similarities. Some of the concepts were defined with similar characteristics and, therefore, combined. To further increase the credibility of the resulting heuristics, any characteristic matching an original heuristic is reworded using the original heuristic as a guide. Moreover, concepts with a weight of 0, that is, not described by stroke survivors or clinicians, were not defined as a characteristic and correspondingly not specified as a heuristic.

The characteristics were initially defined by GD and then independently reviewed by the coauthors. Any disagreements were discussed in a meeting until agreement was reached for each characteristic.

Table 3 lists the characteristics defined from the concepts identified previously that match an original heuristic [23], and as we have reworded them. Textbox 2 lists the characteristics that did not have a matching original heuristic [23]. The numbering continues from Table 3 through to Textbox 2, to indicate which characteristics were combined in the next step. To see which concepts were defined as which characteristics, please see Multimedia Appendix 1.

Eight new heuristics for PGHD enablement in K-SRS were created. Heuristics with higher weights are presented first, to highlight heuristics that should be prioritized. The first 6 heuristics have a weight of 3, and the last 2 heuristics have a weight of 2. There are 6 tags used: improve health-related behaviors, increase positive feelings about health status, facilitate positive personal care goals, increase interest in care processes, improve relationships with care providers, and improve relationships with family and caregivers.

For brevity, only the ID, name, tag(s), definition, and explanation are presented in Textbox 3. This could also be the short form that implementers may use when evaluating HITs, similar to how brief definitions are presented for the original heuristics [29]. However, as with the original heuristics, implementers ought to understand the underlying concepts and anticipated problems of each heuristic and should, therefore, view the full specifications in Multimedia Appendix 1.

This study is the first to develop a new set of specialized heuristics to evaluate how HITs incorporate features that enable effective patient use of PGHD, with K-SRS as a key case study. In addition, it is the first to describe how the identification of initial HIT features and concepts to enable PGHD could lead to the development of a specialized set of heuristics. As such, it uncovers a valuable dimension to the PROM-PGHD development method, which considers the sociotechnical context of HITs [15,22]. This context is the complex nature of interactions between people’s unique health conditions and behavior and that of the technologies and tools they use within their environment [53]. The prior development of a PROM-PGHD for the key case of K-SRS was guided by this context [15]. Although PROMs have long been known to support the evaluation of health interventions, which may be supported by HITs [20], they are developed to consider only the health condition of patients. Conversely, the nature of PGHD requires the development of a PROM-PGHD to consider the sociotechnical context of the patient experience, that is, both the health condition and technology category [15].

Subsequently, this study highlights the value of considering the sociotechnical context of HITs in their design and evaluation. Although unexpected consequences and even failures can occur from design flaws and technical limitations, they are also often the result of sociotechnical factors [54]. Thus, HIT usability testing and evaluation can benefit from frameworks, standards, and guidelines that consider the sociotechnical context of those technologies [53,55]. In particular, PROMs-PGHD [15], heuristics for PGHD enablement, and cognitive work analysis [56] ensure that understanding of the complexities of care is incorporated in HIT design and implementation [56].

The R3C methodology [45] guided the heuristic development process; however, it was not applied without some difficulty. The methodology’s development, description, and application to a case study [44,47] were not described in explicit detail, which left too much room for flexibility. More specifically, it did not specify how the concepts identified from step 1 were going to be connected in step 2; how weights were going to be determined and applied to those concepts, and what influence those weights would have on the succeeding steps; how the characteristics were going to be identified in step 3 and how they differed from the concepts previously identified; and how the original heuristics were going to be “used” as a basis, after matching them with the defined characteristics. Moreover, although R3C presented a structured template for the specification of the heuristics in step 4, the resulting case heuristics only presented summaries or abbreviated versions [47,57,58]. Subsequently, implementations of R3C’s steps to develop other specialized heuristics have varied [59-61].

A step-by-step implementation of the R3C methodology was presented. A systematic and detailed process was applied in each step of heuristic development, which bridged the gaps described earlier. It is hoped that this would aid future developers of specialized heuristics, who could apply the detailed process of heuristic development for other domains of technology, and additionally for the case of PGHD enablement for other health conditions. Throughout the process, it was observed that the concepts and characteristics may need to be combined. This study presents a way to retain the weights and categories applied to them as heuristic metadata to aid implementers in understanding their underlying concepts. Although the last 2 heuristics have a weight of 2, implementers should still aim to conduct the evaluation using all 8 heuristics. However, in case of any resource constraint, the first 6 heuristics should be prioritized.

The R3C methodology [44] was also augmented through the use of qualitative studies with target users (stroke survivors) and domain experts (physiotherapists) to determine the weights, discover new concepts, and ensure the defined characteristics and resulting heuristics were as close to the experiences of the users as possible. This is intended to result in a robust and credible set of heuristics, before the validation stage (step 5), after which the specified heuristics may be improved through the refinement stage (step 6). This is especially valuable, as the R3C methodology’s recommended iteration of the process leads from step 6 back to step 4 [45], implying that steps 1 to 3 need to be rigorously implemented. Subsequently, the initial set of heuristics presented here will be validated and refined following steps 5 and 6.

Data gathered from the qualitative studies were used as an important augmentation in the heuristic development process. However, as those studies were primarily meant to support the development of a PROM-PGHD, it is possible that participants were not asked all the relevant questions around PGHD enablement features. In the event, many concepts found through the literature review were supported by qualitative studies, and indeed, most of the resulting heuristics were formed from new concepts found in those studies. Therefore, although it might appear to be a limiting factor, it actually indicates the robustness of the process followed and highlights the importance of this suggested augmentation. Moreover, the validation stage (step 5) still allows for possible additional heuristics to be identified.

The flexibility of the R3C methodology [44] meant that it is possible that the process followed in this study deviated from the original intent of the authors. However, we believe that the detailed, step-by-step implementation of R3C and the suggested augmentation have in effect modeled a way to build on and strengthen it.

The new set of heuristics for PGHD enablement, following a detailed, systematic development process augmented from best practice that we have presented, could serve as a guide for future developers of specialized heuristics in general and specifically for developers of heuristics for PGHD enablement of a variety of technology domains and health conditions. The new set of heuristics is needed in a period of rising demand for, supply of, and variety of PGHD-enabled technologies in health care. It offers health care providers a standard way to evaluate technologies as part of selecting and implementing PGHD programs with their patients [11], complementing other in-depth PGHD evaluation methods [2,15], and has a broader relevance for the design and implementation of HITs. In addition, an interesting dimension to the PROM-PGHD development process was discovered, and it highlights the value of considering the sociotechnical context of HITs in their design and evaluation.