This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Patients with cancer undergoing cytotoxic chemotherapy face an elevated risk of developing serious infection as a consequence of their treatment, which lowers their white blood cell count and, more specifically, their absolute neutrophil count. This condition is known as neutropenia. Neutropenia accompanied by a fever is referred to as febrile neutropenia, a common side effect of chemotherapy with a high mortality rate. The timely detection of severe neutropenia (<500 absolute neutrophil count/μL) is critical in detecting and managing febrile neutropenia. Current methods rely on blood draws, which limit them to clinical settings and do not allow frequent or portable monitoring. In this study, we demonstrated the usability of PointCheck, a noninvasive device for neutropenia screening, in a simulated home environment without clinical supervision. PointCheck automatically performs microscopy through the skin of the finger to image the blood flowing through superficial microcapillaries and enables the remote monitoring of neutropenia status, without requiring venipuncture.
This study aimed to evaluate the usability of PointCheck, a noninvasive optical technology for screening severe neutropenia, with the goal of identifying potential user interface, functionality, and design issues from the perspective of untrained users.
We conducted a multicenter study using quantitative and qualitative approaches to evaluate the usability of PointCheck across 154 untrained participants. We used a mixed method approach to gather usability data through user testing observations, a short-answer qualitative questionnaire, and a standardized quantitative System Usability Scale (SUS) survey to assess perceived usability and satisfaction.
Of the 154 participants, we found that 108 (70.1%) scored above 80.8 on the SUS across all sites, with a mean SUS score of 86.1 across all sites. Furthermore, the SUS results indicated that, out of the 151 users who completed the SUS survey, 145 (96%) found that they learned how to use PointCheck very quickly, and 141 (93.4%) felt very confident when using the device.
We have shown that PointCheck, a novel technology for noninvasive, home-based neutropenia detection, can be safely and effectively operated by first-time users. In a simulated home environment, these users found it easy to use, with a mean SUS score of 86.1, indicating an excellent perception of usability and placing this device within the top tenth percentile of systems evaluated for usability by the SUS.
ClinicalTrials.gov NCT04448314; https://clinicaltrials.gov/ct2/show/NCT04448314 (Hospital Universitario 12 de Octubre registration) and NCT04448301; https://clinicaltrials.gov/ct2/show/NCT04448301 (Boston Medical Center registration)
One of the most serious side effects of cytotoxic chemotherapy and immunotherapy is neutropenia—a decrease in neutrophils, the most common type of white blood cell (WBC) and the most important cell needed to prevent bacterial infection. The primary clinical consequence of neutropenia is an elevated risk of life-threatening bacterial infection that typically requires immediate admission to the emergency department, hospitalization, and treatment [
In the current standard of care, the risk of FN is evaluated by using a priori scores, such as the Multinational Association for Supportive Care in Cancer score [
Assessing the usability for this kind of technology is crucial in ensuring the accuracy of the results, driving adoption, and improving patient compliance and adherence [
The early detection of FN risk is essential as it can be associated with a higher chance of survival, more successful treatment, and improved quality of life. Therefore, the need for these technologies to be user friendly to the majority of the patient population subsequently increases, as this can impact the patients’ perception of the technology and their decision to take the test [
We hypothesized that novice users will consider PointCheck (Leuko Labs), the first noninvasive optical technology for screening severe neutropenia, to be easy to use. The primary study objective was to evaluate the usability of PointCheck with the goal of identifying potential user interface (UI), functionality, and design issues from the perspective of untrained, first-time users in a simulated home environment. The primary end point for the study, defined a priori, was a group mean score of 80.8 on a standardized SUS, indicating a favorable perception of usability and a higher likelihood of adoption.
PointCheck is the first noninvasive device (
The PointCheck device and its main components.
The device consists of an optics and illumination system, on-board computing electronics, an 8.9-cm touch screen UI, a power cord, and disposable finger cartridges (
The UI on the touch screen provides a guided walk-through to facilitate the correct use of the device. It prompts the user to warm their hands; open up a new, unused cartridge; properly place the cartridge into the device; and insert their nondominant 4th (ring) finger all the way into the cartridge while properly supporting their arm on a flat, stable surface (
Screenshots of PointCheck’s user interface depicting the user walk-through tutorial in English for taking a measurement and device function via the touch screen interface. Language support for Spanish- and Haitian-speaking populations was implemented to translate the instructions.
Usability data was gathered from a cohort including both healthy volunteers and outpatients with cancer receiving chemotherapy. Patients were recruited at both the Boston Medical Center and Hospital Universitario 12 de Octubre before their routine chemotherapy administration. The healthy volunteers were recruited at the Massachusetts Institute of Technology’s Center for Clinical and Translational Research via advertisements displayed on Massachusetts Institute of Technology’s campus and social media and via email lists. The study visits took place in a simulated home environment, and testing was conducted without supervision from a medical professional. No participants had prior experience with the tested device.
A total of 154 participants (85 patients and 69 volunteers) participated in this study. According to standard usability sample size models, this sample size provides a 99% chance of detecting errors with the probability of occurrence of 3% at least once [
To ensure the generalizability of the results, we included younger (aged <65 years) and older (aged ≥65 years) adults, patients with diverse cancer types (lymphoma, leukemia, and myeloma, among other tumor types), men and women, and different education levels (≥8th grade or <8th grade). This allowed us to better understand the links between certain characteristics of the potential patients (ie, age, education, technophilia, and health literacy) and the usability [
Institutional review board (IRB) approvals were obtained from the Boston Medical Center IRB (H-39964), Hospital Universitario 12 de Octubre IRB (20/049), and the New England IRB (1290027) to conduct the study prior to recruitment. Participants provided written consent before agreeing to participate in the study according to good clinical practice guidelines (ICH E6:R2) [
We used a mixed method approach to gather usability data through (1) user observation, (2) a short-answer qualitative e-questionnaire, and (3) a standardized quantitative SUS to assess perceived usability and satisfaction.
Regarding user observation, study coordinators observed participants while they used the device to document any errors that could potentially lead to imaging errors on the device. For example, an unsupported arm or incorrect hand placement could result in too much movement during a measurement and cause an error in the reading. Study coordinators also observed participants to identify and document any points of confusion during the walk-through steps that could be improved. All documented observations were collated into a list to be manually categorized by the type and frequency of occurrence (see Qualitative Results).
A subset (n=120) of the participants were given the opportunity to give feedback and document their thoughts, feelings, and experience using the device through an e-questionnaire containing 4 questions (
Finally, the SUS survey was used as a method of assessing subjective usability. The SUS is a Likert-type questionnaire comprising 10 questions with 5 response options ranging from “strongly disagree” to “strongly agree,” allowing for a subjective assessment of usability [
Baseline assessments conducted by research staff included a brief physical examination and collection of demographic information. Study coordinators read a short script that provided the participants with information about how to use the device and emphasized that the aim of the study was to test the user friendliness of the device and not the participants’ ability to use the device correctly (
One-page quick start guide provided to participants before attempting to take a measurement on their own.
Participants were then asked to follow the instructions presented to them on the device screen, guiding them through the critical steps required to obtain high-quality measurements. The study coordinators did not intervene or answer questions related to device use to reproduce the conditions of unsupervised home use. A second observer monitored and recorded a subset of visits either in person or through the Zoom teleconferencing platform (Zoom Video Communications). Observers documented participant errors, feedback, and tendencies. After completing the initial measurement, participants were immediately asked to complete the SUS and questionnaire to evaluate their first impressions about the user friendliness of the system to prevent any bias introduced from repeating measurements and becoming familiar with the measurement process. Participants performed additional trials, each lasting about 1.5 minutes (1-minute measurement plus 30-second setup and walk-through) for a total of 2 to 6 repeat measurements to evaluate the device precision. These subsequent trials were not used for the perceived usability evaluation and are not reported here.
Basic demographic characteristics were summarized using descriptive statistics. A final SUS score was computed in accordance with Brooke [
Basic demographics of participants. Educational, race, and ethnicity level data were missing for 9 (5.8%), 11 (7.1%) and 10 (6.5%) out of 154 participants, respectively.
Demographic | Participants (N=154) | |
|
||
|
Mean (SD) | 44.8 (20.5) |
|
Median (range) | 38.3 (18.0-88.5) |
|
||
|
Male | 67 (43.5) |
|
Female | 87 (56.5) |
|
||
|
<8th grade | 43 (27.9) |
|
≥8th grade | 102 (66.2) |
|
Missing | 9 (5.8) |
|
||
|
American Indian or Alaska Native | 1 (0.6) |
|
Asian | 28 (18.2) |
|
Black or African American | 28 (18.2) |
|
More than 1 race | 5 (3.2) |
|
Unknown | 11 (7.1) |
|
White | 81 (52.6) |
|
||
|
Hispanic or Latino | 49 (31.8) |
|
Not Hispanic nor Latino | 95 (61.7) |
|
Unknown | 10 (6.5) |
The average SUS score across all participants was 86.1. In total, 70.1% (108/154) of the participants scored above the goal of 80.8 (
Quantitative System Usability Scale (SUS) results across all participants. SUS surveys were incomplete for 3 participants and could not be computed.
SUS score (range 0-100) | Overall (N=154) |
Mean (SD) | 86.1 (12.2) |
Median (range) | 87.5 (20.0-100) |
Missing, n (%) | 3 (1.9) |
Total percent of promoters (defined as participants scoring >80.8 points on the System Usability Scale). System Usability Scale surveys were incomplete for 3 participants and could not be computed.
Cutoff used (points) | Overall (N=154), n (%) |
≤80.8 | 43 (27.9) |
>80.8 | 108 (70.1) |
Missing | 3 (1.9) |
System Usability Scale (SUS) results stratified by educational level (<8th grade and ≥8th grade). The results were generated using the data available from a total of 145 participants. Educational level data was missing for 9 participants (N=154, 5.8%).
SUS score (range 0-100) | <8th Grade (N=43) | ≥8th Grade (N=102) | Overall (N=154) |
Mean (SD) | 84.0 (12.7) | 86.5 (12.2) | 86.1 (12.2) |
Median (range) | 85.0 (57.5-100) | 87.5 (20.0-100) | 87.5 (20.0-100) |
System Usability Scale (SUS) results stratified by age category (<65 years and ≥65 years).
SUS score (range 0-100) | <65 years (N=118) | ≥65 years (N=36) | Overall (N=154) |
Mean (SD) | 86.9 (12.5) | 83.5 (11.0) | 86.1 (12.2) |
Median (range) | 90.0 (20.0-100]) | 82.5 (57.5-100) | 87.5 (20.0-100) |
SUS survey responses assessed individually. Percentage values are calculated using available data from a total of 151 participants. SUS surveys were incomplete for 3 participants. SUS: System Usability Scale.
Error observation notes were collated into a list and then manually categorized by the type and frequency of occurrence. The primary error sources included skipping or misreading instructions and on-screen instruction accessibility. The majority (70/86, 81%) of these errors occurred only on the first use and were shown to be correctable by interventional guidance. Such guidance was given after the SUS survey had been completed, and improvement in most cases was demonstrated in subsequent trials. This shows that although the device performs well in independent use, the monitoring of first use by an experienced operator may have further benefit for catching usability errors.
The feedback from the questionnaires was collated into a spreadsheet and then manually categorized into the following themes: UI/user experience, aesthetic design/logical design, hand rest, cartridge, cleaning/sanitation, and software/bugs. The themes were then broken down into the following subthemes: confidence in use, training effectiveness, UI design/clarity of UI instructions, ergonomic design, foreseeable home use issue, and accessibility. The instances of feedback falling within these subthemes were counted and generated the 3 overarching themes: pretraining effectiveness, user friendliness of PointCheck (related to ease of use, accessibility, and clarity of UI elements), and ergonomic design.
A portion of participants initially expressed some uncertainty when using the device for the first time (Participants #38 and #18;
Illustrative quotes for the 3 overarching themes.
Theme/category | Illustrative quote |
Pretraining effectiveness |
“Would like a YouTube channel/clip to watch in advance that will explain the device.” (Participant #38) “The most difficult step was probably removing the cartridge from the box. I was not sure if I had to keep the cartridge clean for measurements and the instructions did not tell how I should be holding the cartridge or if I should even be careful or not about touching it too much and getting it dirty” (Participant #18) “The device was cumbersome (awkward) to use because it was the first time. After the first time, it would be easier to use.” (Participant #40) |
User friendliness of PointCheck |
“Others can’t see as well, may need others to help them if they have dementia or are forgetful.” (Participant #53) “Could not read font of the three step instructions.” (Participant #65) “If I were to use the device on a daily basis, I would be relatively annoyed by the fact the three repeat questions are timed lag to press yes.” (Participant #15) “One thing I was confused about was checking off ‘was my finger in all the way.’ It was just a circle and I was confused what I was supposed to do on this step.” (Participant #21) “The cartridge lid is shown to be peeled from the flat side (facing down) but then needs to be rotated to be put in with the flat side up. If the peel could be opened in the right orientation, that would have helped.” (Participant #24) “Very crisp, and clear how to start using it...the screen has good contrast, good font choice given resolution and size, and the purple/grey color scheme is also calming.” (Participant #30) “Even though I knew beforehand that I should rest my elbow, I forgot to do so after inserting my finger because that was my main focus, so I really appreciated the reminder to have my elbow rested right after the step where I inserted my ring finger.” (Participant #34) |
Ergonomic Design |
“The display seems angled a little high, considering that the machine needs to be placed a considerable distance to rest my arm. The steps were fairly intuitive.” (Participant #3) “I was expecting the device to be a bit smaller. I think the device is designed for a very big hand, I think it would be probably better to try to make it the size closer to a computer mouse.” (Participant #16) “My fingers are pretty small and narrow but the soft spiked insides of the capture cylinder still left marks on my finger afterwards. This was completely non-painful but just noting this here for other users who might have thicker fingers! It was a bit bigger than I expected (the size of the device) but it doesn’t impact usability. It also produced quite a loud hum but again, doesn’t impact usability.” (Participant #11) “I liked the brush-like texture inside the tube. It helped me feel that I had my finger in the right position.” (Participant #33) “My first impression was that it looked pretty compact and that it’s very straight to the point in its features--nothing fancy, just functional.” (Participant #34) |
Older participants (aged ≥65 years) discussed the need for improved screen readability, mentioning increasing the font size or needing additional assistance (Participants #53 and #65;
There were mixed opinions on the overall design of the UI, but the majority (145/151, 96%) of the participants found the overall system to be easy to use and that they could learn quickly. Some participants did comment on the elements of UI design, such as buttons, on-screen instructions, or color choices, that made them feel frustrated, confused, or uncertain about whether they were performing the measurement correctly (Participants #15, #21, and #24;
Finally, participants also addressed the changes they wished to see in the ergonomic design to better meet the needs of end users (Participants #3, #16, and #11;
In this study, we aimed to evaluate the usability and design of PointCheck, a novel technology for noninvasive, home-based neutropenia detection. Through a mixed method approach of user observation, questionnaires, and a SUS survey, we have validated the hypothesis that PointCheck is easy to use by first-time users in a simulated home environment with a mean SUS score of 86.1 (
Although the majority of first-time users expressed high satisfaction with the overall design and user friendliness of PointCheck (
In observing the use of the device in context, correct positioning during the use of the device may be more difficult for nonambulatory patients. Ideally, patients will have a training session with their health care professional prior to bringing this device home for normal use. This training would allow patients to familiarize themselves with the device beforehand, ask any questions related to use, and receive the support needed to ensure confidence in using the device alone for weeks at a time.
It is to be noted that a majority of study participants were aged <65 years and have an educational level of ≥8th grade level, both of which are factors that increase the likelihood of technological proficiency and willingness to adopt new technology [
Although this study aimed to evaluate a variety of usability factors in a simulated home environment, a single study cannot claim to assess these factors in all use cases and situations. The perceived usability of PointCheck should be tested further in real-world home environments with users who receive prior training to identify context-related issues in the future.
Overall, this study demonstrated that PointCheck, a novel digital device for noninvasive WBC monitoring, can be easy to use for unsupervised patients in the home setting. By enabling continuous home-monitoring for severe neutropenia, PointCheck has the potential to change the standard of care for patients with cancer and substantially improve their clinical outcomes.
Short-answer qualitative e-questionnaire.
Study coordinator training script.
febrile neutropenia
institutional review board
System Usability Scale
User Interface
white blood cells
Funding for the study was supported through the National Cancer Institute Small Business Innovation Research program (grant R44CA228920), the National Institutes of Health (grant U54HL143541), the Massachusetts Technology Collaborative Digital Health Sandbox Program, and the Centro de Desarrollo Tecnológico e Industrial in Spain (grant SNEO-20191188). We are also grateful to Catherine Ricciardi, DNP, ANP-BC (Nurse Director and Clinical Operations); and Tatiana Urman, RN, MSN (Clinical Research Nurse Coordinator) at the Massachusetts Institute of Technology Center for Clinical and Translational Research. We extend a special thank you to all volunteers and patients who participated in this study.
GL, APT, IB, NB, RB, AB, ASF, and CCG are current employees and holders of stock options in a privately held company, patents, and royalties of Leuko Labs, Inc. IB and CCG have membership on the board of directors or advisory committees of Leuko Labs, Inc. JML received research funding from Roche, Novartis, Incyte, Astellas, and BMS and consulted for Janssen, BMS, Novartis, Incyte, Roche, GSK, and Pfizer. None of these items are related to this work. JMS has membership on the board of directors or advisory committees of Pharmacosmos and Astra Zeneca and received honoraria from Abbvie and Stemline. None of these items are related to this work. All other authors declared no other conflicts of interest.