One of the most serious side effects of cytotoxic chemotherapy and immunotherapy is neutropenia—a decrease in neutrophils, the most common type of white blood cell (WBC) and the most important cell needed to prevent bacterial infection. The primary clinical consequence of neutropenia is an elevated risk of life-threatening bacterial infection that typically requires immediate admission to the emergency department, hospitalization, and treatment [1-3]. Every year, approximately 850,000 patients with cancer start chemotherapy treatments in the United States [4], and 140,000 (17%) [5] will endure at least one episode of febrile neutropenia (FN), or neutropenia accompanied by a fever. FN typically requires an admission of over 1 week, costing approximately US $30,000 per episode [6,7], with associated mortality rates between 7% to 10% [8]. The timely detection and awareness of severe neutropenia (ie, <500 absolute neutrophil count/µL) [9] can be crucial to prevent and manage FN in the outpatient setting [10,11] and the emergency department [12,13].

In the current standard of care, the risk of FN is evaluated by using a priori scores, such as the Multinational Association for Supportive Care in Cancer score [14], to indicate primary prophylaxis with growth colony stimulating factors or by patients regularly monitoring their temperature at home to seek emergency care when fever ensues [15]. Despite these existing methods, FN still has an important economic and clinical impact in cancer care. The early detection of neutropenia could be used to prevent FN by triggering an early administration of granulocyte colony-stimulating factor or antibiotics [16-19]. Unfortunately, current neutropenia-monitoring options rely on venipunctures in the clinical setting or finger-prick blood samples at the point of care [20]. These technologies either require laboratory infrastructure limited to the hospital setting or are impractical as they cannot be operated by minimally trained users to achieve accurate and reliable results [21-24]. To address this unmet need, this paper presents a usability evaluation of a novel, noninvasive technology that allows automated and frequent neutropenia monitoring by patients from the home setting with minimal training.

Assessing the usability for this kind of technology is crucial in ensuring the accuracy of the results, driving adoption, and improving patient compliance and adherence [25]. According to the International Organization for Standardization (IEC 62366-1:2015), usability is defined as a “characteristic of the user interface that facilitates use and thereby establishes effectiveness, efficiency and user satisfaction in the intended use environment” [26]. These metrics can be measured by gaining insight into patient perspectives regarding user performance, satisfaction, and acceptability while using an intervention [27]. For this study, the standardized System Usability Scale (SUS) survey was chosen as a quantitative method of assessing subjective usability due to evidence that it can be used to assess any technology [28] and has successfully been used in the medical domain to assess home medical devices [29-31].

The early detection of FN risk is essential as it can be associated with a higher chance of survival, more successful treatment, and improved quality of life. Therefore, the need for these technologies to be user friendly to the majority of the patient population subsequently increases, as this can impact the patients’ perception of the technology and their decision to take the test [32]. Additionally, technology-based solutions such as the one presented in this paper can help strengthen the relationship and communication between patients and their doctors, empower the patients’ well-being, and help doctors make better and more informed decisions [33].

We hypothesized that novice users will consider PointCheck (Leuko Labs), the first noninvasive optical technology for screening severe neutropenia, to be easy to use. The primary study objective was to evaluate the usability of PointCheck with the goal of identifying potential user interface (UI), functionality, and design issues from the perspective of untrained, first-time users in a simulated home environment. The primary end point for the study, defined a priori, was a group mean score of 80.8 on a standardized SUS, indicating a favorable perception of usability and a higher likelihood of adoption.

PointCheck is the first noninvasive device (Figure 1) designed to screen for severe neutropenia in the home setting [34]. By imaging the blood flowing through the capillaries in the finger, PointCheck enables real-time remote monitoring of WBC levels based on optical imaging and without a blood draw [35,36].

The device consists of an optics and illumination system, on-board computing electronics, an 8.9-cm touch screen UI, a power cord, and disposable finger cartridges (Figure 1). It uses a camera microscopy system and LEDs to image capillaries in the nailfold region of the finger—typically the nondominant 4th (ring) finger, which has been shown by previous literature to contain the most intact and visible capillaries when compared to other fingers [37]. The finger cartridge is a disposable component that is prefilled with mineral oil and allows for effective optical refractive index coupling to ensure transdermal imaging quality [38,39]. The finger cartridge is designed for 1-time use. The hardware system design resembles the methods used in standard nailfold video capillaroscopy, which is an established technique used by rheumatologists to evaluate capillary morphology and microcirculation [40].

The UI on the touch screen provides a guided walk-through to facilitate the correct use of the device. It prompts the user to warm their hands; open up a new, unused cartridge; properly place the cartridge into the device; and insert their nondominant 4th (ring) finger all the way into the cartridge while properly supporting their arm on a flat, stable surface (Figure 2). A final checklist ensures that the most critical steps have been completed and the user is able to start the 1-minute measurement. The version of the device used in this study was a beta prototype (version 4).

Usability data was gathered from a cohort including both healthy volunteers and outpatients with cancer receiving chemotherapy. Patients were recruited at both the Boston Medical Center and Hospital Universitario 12 de Octubre before their routine chemotherapy administration. The healthy volunteers were recruited at the Massachusetts Institute of Technology’s Center for Clinical and Translational Research via advertisements displayed on Massachusetts Institute of Technology’s campus and social media and via email lists. The study visits took place in a simulated home environment, and testing was conducted without supervision from a medical professional. No participants had prior experience with the tested device.

A total of 154 participants (85 patients and 69 volunteers) participated in this study. According to standard usability sample size models, this sample size provides a 99% chance of detecting errors with the probability of occurrence of 3% at least once [41].

To ensure the generalizability of the results, we included younger (aged <65 years) and older (aged ≥65 years) adults, patients with diverse cancer types (lymphoma, leukemia, and myeloma, among other tumor types), men and women, and different education levels (≥8th grade or <8th grade). This allowed us to better understand the links between certain characteristics of the potential patients (ie, age, education, technophilia, and health literacy) and the usability [42,43].

Institutional review board (IRB) approvals were obtained from the Boston Medical Center IRB (H-39964), Hospital Universitario 12 de Octubre IRB (20/049), and the New England IRB (1290027) to conduct the study prior to recruitment. Participants provided written consent before agreeing to participate in the study according to good clinical practice guidelines (ICH E6:R2) [44].

We used a mixed method approach to gather usability data through (1) user observation, (2) a short-answer qualitative e-questionnaire, and (3) a standardized quantitative SUS to assess perceived usability and satisfaction.

Regarding user observation, study coordinators observed participants while they used the device to document any errors that could potentially lead to imaging errors on the device. For example, an unsupported arm or incorrect hand placement could result in too much movement during a measurement and cause an error in the reading. Study coordinators also observed participants to identify and document any points of confusion during the walk-through steps that could be improved. All documented observations were collated into a list to be manually categorized by the type and frequency of occurrence (see Qualitative Results).

A subset (n=120) of the participants were given the opportunity to give feedback and document their thoughts, feelings, and experience using the device through an e-questionnaire containing 4 questions (Multimedia Appendix 1). We used this questionnaire to assess any potential confusion or difficulties participants may have had using the device or the UI, their attitude toward the product, and any potential features they would like to see added to improve user friendliness. The feedback from the questionnaires was collated into a spreadsheet to be manually categorized into themes (see Qualitative Results).

Finally, the SUS survey was used as a method of assessing subjective usability. The SUS is a Likert-type questionnaire comprising 10 questions with 5 response options ranging from “strongly disagree” to “strongly agree,” allowing for a subjective assessment of usability [45]. Scores range from 0 to 100 with higher scores indicating favorable user perceptions of the device and lower scores indicating low usability. The success criterion for a favorable evaluation in the SUS was defined by a mean group score of greater than 80.8 (see Quantitative Results). This threshold was selected based on previously published cutoffs to define the promoters of a technology [46].

Baseline assessments conducted by research staff included a brief physical examination and collection of demographic information. Study coordinators read a short script that provided the participants with information about how to use the device and emphasized that the aim of the study was to test the user friendliness of the device and not the participants’ ability to use the device correctly (Multimedia Appendix 2). In addition, the participants were provided with a 1-page guide containing device instructions before attempting to take a measurement on their own (Figure 3).

Participants were then asked to follow the instructions presented to them on the device screen, guiding them through the critical steps required to obtain high-quality measurements. The study coordinators did not intervene or answer questions related to device use to reproduce the conditions of unsupervised home use. A second observer monitored and recorded a subset of visits either in person or through the Zoom teleconferencing platform (Zoom Video Communications). Observers documented participant errors, feedback, and tendencies. After completing the initial measurement, participants were immediately asked to complete the SUS and questionnaire to evaluate their first impressions about the user friendliness of the system to prevent any bias introduced from repeating measurements and becoming familiar with the measurement process. Participants performed additional trials, each lasting about 1.5 minutes (1-minute measurement plus 30-second setup and walk-through) for a total of 2 to 6 repeat measurements to evaluate the device precision. These subsequent trials were not used for the perceived usability evaluation and are not reported here.

Basic demographic characteristics were summarized using descriptive statistics. A final SUS score was computed in accordance with Brooke [47], and responses to the e-questionnaire were tokenized for their content and categorized into themes for qualitative analysis. Statistical comparisons were made between the different group categories stratified by age and literacy to evaluate usability differences using nonparametric techniques (Mann-Whitney U test). All quantitative data were processed using RStudio (version 1.3.1093; RStudio Team) [48].

Table 1 represents the breakdown of study participant characteristics by age, education level, gender, and race. Of the 154 participants, 118 (76.6%) were aged <65 years, with an average age of 44.8 (range 18-88) years. A majority (n=102, 66.2%) of the participants had an education level exceeding 8th grade, whereas 43 (27.9%) participants had an education level below 8th grade, and 9 (5.8%) did not provide educational level information.

The average SUS score across all participants was 86.1. In total, 70.1% (108/154) of the participants scored above the goal of 80.8 (Tables 2 and 3), which indicated that they would be early promoters and more likely to recommend the device to a friend [49]. When stratifying the SUS results by education level, we found that participants exceeding the 8th grade level scored slightly higher than those with an 8th grade level education and below—but only by a margin of 2.5 points, which was not found to be statistically significant (P=.27; Table 4). When stratifying SUS results by age categories, we found that participants aged <65 years also scored higher than participants aged ≥65 years by a margin of 3.4 points, showing a nonstatistically significant trend (P=.06). Both groups had a mean score above the predefined threshold of 80.8 (Table 5). When evaluating the SUS results by the individual survey questions, we found that 96% (145/151) of the participants that completed the survey found PointCheck easy to use and felt that they could learn to use it very quickly (Figure 4).