Too Much Trust? Wearables and the Accuracy Perception Gap

Blythe Karow is a MedTech strategist with over 25 years of experience in the wearables space. She is also the founder of The Karow Advisory Group and The Device Files Substack. This op-ed is the second in her series on the quantified self and how the wearables space is evolving.

In today’s wearable world, we can count heartbeats, check for atrial fibrillation, flag possible high blood pressure, look at our sleep stages, and even watch over sleeping infants. Some of these features have been cleared by the Food and Drug Administration (FDA). Some have not. Most people wearing them couldn’t tell you which is which, and I’m not convinced it would change how much they trust the numbers if they could.

Users generally trust their wearables—maybe too much.

We increasingly want our devices to tell us what the numbers mean, all the while taking for granted that those numbers are accurate. I have, for example, met multiple people who have stopped wearing their wearables because their sleep score was consistently low, contradicting their own sense of their sleep quality. But they didn’t ditch their wearables because they thought the data were wrong. They assumed their wearable was right, and it was stressing them out—what we now call “orthosomnia,” a wearable data–driven preoccupation with getting perfect sleep scores.

Worry about accuracy doesn’t seem to stop us from acting on the numbers. In a survey of almost 400 wearable users in Western Europe last year, 35% named data accuracy as a key concern, while 66% had nonetheless changed their exercise routines based on what their device told them and 29% had changed their sleep habits.

Even more interestingly, a 2017 case report documented the story of a 27-year-old woman who was feeling unrefreshed after nights her wearable scored as poor sleep. When a full overnight sleep study—involving cleared, validated medical device equipment—demonstrated that she was, in fact, getting substantial deep sleep, she still believed her wearable was more accurate than the sleep lab.

Two Kinds of Accuracy

Every measurement has a range of possible error, whether it comes from a hospital monitor, a home blood pressure cuff, or a ring. A reading of 72 beats per minute really means “somewhere around 72, give or take some amount.” How much error depends on the device, what you’re doing, how it’s being worn, and often on things you can’t control, like the color of your skin or its conductance.

When people talk about whether a wearable is accurate, they are usually blending two questions. The first is about the data going in: did the sensor capture the signal correctly? The second is about interpretation: did the algorithm draw the right conclusion from that signal? A device can get one right and the other wrong, or it can get bad data in and then provide a poor interpretation because of it.

Data In

Clinicians have a term for tools whose results depend on the skill of the person using them: operator-dependent. Ultrasound is the classic example: I couldn’t tell you what I was looking at on an ultrasound, but the tech can.

Consumer wearables are operator-dependent devices whose operators have no training and don’t read the instructions for use, so the device relies on the user’s intuition about how to wear and use it.

Fit, placement, and technique become major variables in accuracy, and most people don’t think about them at all. A loose band lets the optical sensor shift with every movement. Sam Sheng, Senior Director of Sensing Systems and Technologies at Google, notes that optical sensors—used to measure things like heart rate and blood oxygen level—work within a fairly narrow range of contact pressure, roughly 50 to 200 grams of force. Below that, the signal degrades; above it, blood flow under the sensor is cut off. A skin temperature sensor used in direct sunlight gets overwhelmed by infrared light, a limitation spelled out in instructions few people read. Activity matters too: in one study, absolute heart rate error during activity was, on average, 30% higher than when at rest.

Clinicians have a blind spot of their own when it comes to trust, and it points the other way. Nearly half (44%) of US physicians say regulatory approval is essential to, or strongly influences, their trust in a consumer device’s accuracy. The problem is that FDA clearance tells a clinician the feature met a specific bar in a controlled test environment. It says nothing about whether their patient used it correctly at home.

Sheng also described a case where someone wore their home blood pressure cuff on their elbow (yes, that is the wrong placement), reported the readings to their doctor, and was prescribed medication based on them. It’s not uncommon: when researchers filmed patients taking their own blood pressure, only 3% did it without error, and 60% made 3 or more errors.

‎

Data Interpretation

Interpretation brings its own errors, and how much those errors matter depends on what is being measured. Steps, for example, are an easy way to see if you’re keeping moving, but whether you get 5000 or 5500 doesn’t really matter and probably won’t drive any clinical decisions. The stakes get higher for measurements like heart rate and blood pressure—where real diagnostic decisions could be made based on inaccurate data.

Kenneth Civello, MD, MPH, a cardiac electrophysiologist at Our Lady of the Lake in Baton Rouge, described a patient with a history of atrial fibrillation who drove nearly 2 hours to the clinic after his watch showed an elevated heart rate, only to find a normal rhythm of about 60 beats per minute in the office. That’s what we would call a false positive.

False negatives—where a patient might trust a wearable to detect a condition like atrial fibrillation and it never alerts them—worry Civello most, particularly for blood pressure. Apple’s FDA-cleared hypertension feature correctly flagged 41.2% of people with blood pressure of 130/80 mm Hg or higher in its validation study, with 92.3% specificity. When it alerts, it is usually right. But when it stays silent, that does not necessarily mean blood pressure is normal. “I’d worry that they’d go, oh, my blood pressure is good,” Civello told me. “I don’t need to check my blood pressure.” Blood pressure features also don’t all work the same way. Some flag patterns without ever showing a number, while others display estimated readings. Users reasonably assume these features are comparable with cleared medical BP cuffs, and there is little public data to tell them otherwise.

What’s to Be Done?

Currently, users either don’t have access to or don’t see information on their wearable’s performance data (because no one really reads the instructions!).

Two suggestions for the wearables industry to help close the accuracy perception gap:

1. Published comparisons of how each wearable collects and displays data, with information on accuracy–both data in and interpretation.
2. Use the app interface as an opportunity to provide the details that currently hide in the instructions for use—not another disclaimer in the terms and conditions, but rather education for users while they dig around in the app to better understand their readings. Why not add accuracy bars or a summary on accuracy behind the little “info” icon?

Right now, the user who trusts every number and the clinician who trusts only those from cleared devices are both making assumptions that can significantly impact lives. Wearable companies are very good at designing experiences that keep people engaged with their numbers—the same skill could teach people how much to believe them.