This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
A decrease in the level of pulse oxygen saturation as measured by pulse oximetry (SpO2) is an indicator of hypoxemia that may occur in various respiratory diseases, such as chronic obstructive pulmonary disease (COPD), sleep apnea syndrome, and COVID-19. Currently, no mass-market wrist-worn SpO2 monitor meets the medical standards for pulse oximeters.
The main objective of this monocentric and prospective clinical study with single-blind analysis was to test and validate the accuracy of the reflective pulse oximeter function of the Withings ScanWatch to measure SpO2 levels at different stages of hypoxia. The secondary objective was to confirm the safety of this device when used as intended.
To achieve these objectives, we included 14 healthy participants aged 23-39 years in the study, and we induced several stable plateaus of arterial oxygen saturation (SaO2) ranging from 100%-70% to mimic nonhypoxic conditions and then mild, moderate, and severe hypoxic conditions. We measured the SpO2 level with a Withings ScanWatch on each participant’s wrist and the SaO2 from blood samples with a co-oximeter, the ABL90 hemoximeter (Radiometer Medical ApS).
After removal of the inconclusive measurements, we obtained 275 and 244 conclusive measurements with the two ScanWatches on the participants’ right and left wrists, respectively, evenly distributed among the 3 predetermined SpO2 groups: SpO2≤80%, 80%<SpO2≤90%, and 90%<SpO2. We found a strong association and a high level of agreement between the measurements collected from the devices, with high Pearson correlation coefficients of
In conclusion, the Withings ScanWatch is able to measure SpO2 levels with adequate accuracy at a clinical grade. No undesirable effects or adverse events were reported during the study.
ClinicalTrials.gov NCT04380389; http://clinicaltrials.gov/ct2/show/NCT04380389
According to the World Health Organization, respiratory diseases are medical conditions affecting the airways and other structures of the lungs. Three of these include chronic obstructive pulmonary disease (COPD), sleep apnea syndrome (SAS), [
Detecting SAS, COPD, and COVID-19 earlier in people can help reduce potential damage from these diseases, and accomplishing this requires a more thorough examination of oxygen levels. Oxygen saturation, defined as the fraction of oxygen-saturated hemoglobin relative to total blood hemoglobin, is measured either through an invasive method by sampling arterial blood to analyze the arterial oxygen saturation (SaO2) via a co-oximeter or with readings collected noninvasively using a pulse oximeter to measure the peripheral or pulse oxygen saturation (SpO2) level [
Smartwatches exhibit a high degree of satisfaction and growing popularity among the general population for health monitoring [
In particular, because the ScanWatch is a wrist-worn watch that can measure SpO2 continuously along with heart rate and, potentially, breathing rate [
Pulse oximetry relies on the differential absorption by blood of red and infrared light.
Absorption spectra of oxygenated and deoxygenated hemoglobin. Licensed from Adrian Curtin/CC BY-SA. Hb: deoxygenated hemoglobin; HbO2: oxygenated hemoglobin.
Two wavelengths are chosen so that the ratios of the absorption rates of Hb and HbO2 are respectively maximal and minimal. Thus, customary choices are red light (at 660 nm) and infrared light (at 940 nm). From the intensities of the transmitted or reflected light at these two wavelengths, one can deduce the concentrations of both forms of hemoglobin, [HbO2] and [Hb], and the functional oxygen saturation of arterial blood (neglecting carboxyhemoglobin and methemoglobin).
In practice, a ratio of ratios (also called modulation ratio), R, is computed as follows [
Alternating current (AC) refers to the amplitude of the pulsatile component of the PPG signal, and direct current (DC) refers to the value of its baseline. The ratio of AC/DC is commonly called the perfusion ratio.
An analytical relationship between the perfusion index R and the fraction of oxygenated hemoglobin SpO2 can be obtained with a model based on the Beer-Lambert law and a simplified model for the light path taken by both wavelengths. This simple model, which neglects the multiple scattering events of light in the skin, applies to some extent to pulse oximeters operating in transmission where tissues are located between the light-emitting diode (LED) and photodiode (PD). However, the model is unable to explain how reflective pulse oximeters operate, where the LED and PD are on the same side. Therefore, an experimental calibration of R to SpO2 during a hypoxia study is necessary.
Measuring SpO2 with a reflective sensor is more challenging on the wrist than on the fingertip. Due to the thickness of the wrist and the presence of bones, it is impossible to measure SpO2 in transmission. In reflectance mode, the signal-to-noise ratio (SNR) is smaller than typically found in transmission-mode finger pulse oximeters for two reasons. First, the light emitted by the LEDs penetrates the various layers of the skin, but only a small fraction will find its way back (upward) to the PD adjacent to the LED after multiple scattering events inside the skin. Second, blood perfusion is dramatically lower on the back of the wrist than on the finger. During our calibration studies, we compared blood perfusion on the wrist and on the finger using ScanWatch sensors, and we found empirically that the SNR for the measurements taken at the finger was approximately 10 times as high as the SNR for the measurements taken at the wrist.
A second challenge is that optical measurements on the wrist are more prone to artifacts. Motion artifacts are much more frequent than on the fingertip because of the presence of tendons and bones. Light-skin coupling of finger pulse oximeters is robust due to the clamp design. In the absence of a clamp, the optical sensor can lose contact with the skin more easily, both breaking the optical coupling and letting ambient light in; this tends to unpredictably modify the DC levels.
The main objective of this study was to clinically test and validate the accuracy of the reflective pulse oximeter function of the Withings ScanWatch to measure SpO2 levels during mild, moderate, and severe hypoxia compared to a co-oximeter, the ABL90 hemoximeter (Radiometer Medical ApS), in accordance with the ISO 80601-2-61:2017 standard and US Food & Drug Administration (FDA) guidance. The secondary objective was to confirm the safety of the device when used as intended.
This monocentric and prospective clinical study with a double-blind analysis was conducted on 14 healthy participants in a hypoxia laboratory at the University of California San Francisco in March 2020 at an altitude of 122 m and at a room temperature of 25 °C. Inclusion criteria were being between the ages of 18 and 50 years, having a healthy status with no evidence of any medical problems, and having both wrist circumferences between 14 and 22 cm. Current smokers, women who were pregnant, lactating, or trying to get pregnant, and participants with obesity (BMI >30 kg/m2) or who had an injury, deformity, or abnormality at the sensor sites and piercings that might cause air leaks during the test were excluded from the study. Participants with a known history of heart, lung, kidney, or liver disease; diabetes; clotting disorder; hemoglobinopathy or history of anemia; sensitivity to local anesthesia; or fainting or vasovagal response or any other serious systemic illness were also excluded, as well as those diagnosed with asthma, sleep apnea and Raynaud disease. Exclusion criteria also included participants with a resting heart rate of over 120 beats per minute, a systolic blood pressure over 150 mm Hg, a diastolic blood pressure over 90 mm Hg, a room air SpO2 under 94%, and a carboxyhemoglobin level over 3%.
All selected participants in the study met the inclusion criteria. Before starting the study, two Withings ScanWatch units were placed on each participant’s wrist to measure their SpO2, while a 22-gauge catheter was placed in their left radial artery to measure their SaO2. In addition, two reference finger pulse oximeters were placed on each participant to facilitate the identification of plateaus and any discrepancies between the hands. Each participant was then asked to lie in a semisupine position, to remain still, and to breathe a mixture of gas through a mouthpiece while a nose clip blocked their nose. After a 5-minute rest period, they were instructed to hyperventilate 2-3 times deeper and faster than normal during the runs to speed alveolar gas equilibration. The target SpO2 at each run was chosen to be evenly balanced over the 70%-100% SpO2 range.
On the initial step of the run, two blood samples of 1-2 mL were first collected 30 seconds apart from the participant at the same oxygen saturation level as the room air. Then, approximately 10 seconds later, the inspired oxygen was abruptly changed to reduce oxygen saturation to the next SpO2 target level. On the next steps of the run, two blood samples were collected 30 seconds apart from the participant when the reference finger pulse oximeters both measured a stabilized level of oxygen saturation, with less than 1% of difference for 35 seconds. Note that the second sampling was not performed if the oxygen saturation level was destabilized between measurements. Then, approximately 10 seconds later, inspired oxygen was abruptly changed again to reach the next SpO2 target level. Overall, these 75-second periods of stable oxygen saturation were defined as “plateaus,” and every participant was subjected to 5 plateaus (typically 92%, 87%, 82%, 77%, and 70%) before being brought back to a high oxygen saturation level (100% O2) by breathing oxygen-enriched air for 2 minutes. After this run was repeated a second time, the collected blood samples (approximately 20-25 samples per participant) were immediately analyzed with the co-oximeter (ABL90 multiwavelength hemoximeter), then compared to the data recorded with the Withings ScanWatches (
Comparison of SpO2 measured by the ScanWatch and hemoximeter for one subject. SpO2: pulse oxygen saturation as measured by pulse oximetry.
The study was registered at ClinicalTrials.gov (NCT04380389).
The Withings ScanWatch is an analog battery-operated watch consisting of (1) a metal case with a connected movement, (2) three hands, with two hands indicating the time and one hand indicating a cumulated activity level, (3) an adjustable silicone band to fit any user’s wrist between 14 and 22 cm, and (4) a reflective pulse oximeter composed of three LEDs (red, infrared, and green), one broadband photodiode, and one infrared-cut photodiode (
Front view (left) and back view (right) of the Withings ScanWatch.
In a reflective design, the positions of the emitters and receptors must be carefully chosen to maximize the SNR for both wavelengths. This is achieved with optical simulations using Monte Carlo methods [
On the other hand, increasing the LED-PD distance also comes with disadvantages. Because only a small fraction of the total light emitted by the LEDs actually reaches the photodiode, increasing this distance requires a higher light intensity, which consumes more energy. In addition, LEDs have a maximum current they can sustain without damage, so it becomes counterproductive to increase the LED-PD spacing above a certain value at which too little light would be received by the photodiode to provide a usable signal. Based on this trade-off, we found that spacings between 7 and 9 mm were optimal.
The Withings ScanWatch embeds a real-time self-contained algorithm implemented in C language designed to estimate SpO2. The machine learning components of the algorithm (the neural network and linear regressions) were trained using two hypoxia calibration studies, totaling 34 subjects. This algorithm comprises three parts, as described below.
First, a signal processing part estimates AC and DC for each wavelength as well as the modulation ratio. DC levels are obtained by applying a moving average filter on the raw signals, and AC levels are obtained by filtering the raw signals with a bandpass filter centered around 1 Hz and computing the moving standard deviation on the resulting signals. Thus, the perfusions (AC/DC ratios) can be computed for each wavelength, and the modulation ratio (perfusion in the red divided by the perfusion in the infrared) can be estimated. This modulation ratio is supposed to be linearly correlated with SpO2; however, in the case of wrist-worn oximeters, the frequent occurrence of movement and respiratory artefacts often cause that ratio to be unreliable.
To counteract the problem of the modulation ratio not being linearly correlated with SpO2, a second SpO2 estimator is used independently from the first part using a 1D convolutional neural network [
Finally, the modulation ratio computed in the first part and the SpO2 estimation calculated in the second part are merged via a linear regression to obtain a final SpO2 estimation.
Several indicators were used to determine whether a SpO2 measurement was conclusive or inconclusive. First, two algorithms use the PPG and accelerometer signals to assess if the watch is worn and if the user is still. These algorithms are heuristics that rely on simple filtering and thresholding, and they were calibrated on separate data sets acquired specifically for this purpose. The stillness of the user was derived from the absence of variation in the accelerometer signal, and the presence of a pulse on the PPG signal was the main factor to determine that the watch was worn.
In parallel, a second neural network with the same topology (8-second window over the three LED signals) detected and eliminated signals of poor quality. The calibration hypoxia studies were manually annotated to provide labels on which to train the neural network.
An SpO2 measurement was considered to be inconclusive if the watch was not worn, the user was moving, or the measurement was classified as being of poor signal quality.
A statistical analysis of the collected data was performed with the software Python 3.6.9 on a frozen database. A separate analysis was conducted for each wrist. The bias (mean error) and root mean square error (RMSE) between the SpO2 and SaO2 values were calculated for each range of SpO2 values (SpO2≤80%, 80%<SpO2≤90%, and 90%<SpO2) and for the whole range of 70%-100%. We used the Pearson correlation coefficient to determine the strength of the association between the SpO2 values collected from the Withings ScanWatch and the ABL90 hemoximeter. We used Bland-Altman plots to measure the agreement between the Withings ScanWatch and the ABL-90 hemoximeter.
Because a time offset exists between the Withings ScanWatch and the blood sample measured by the co-oximeter, due in part to physiological considerations (eg, distance between the arms, the wrists, and the depth of the arteries) and in part to the delay inherent to the Withings ScanWatch algorithm, we applied a plateau-matching algorithm in accordance with the recommendations of the ISO 80601-2-61:2017 standard before comparing the readings. We used a cross-correlation method to determine the delay between the measurements collected by the devices on each patient’s wrists and the blood samples.
Time offsets applied on the Withings ScanWatch for plateau matching. A negative offset indicates that Withings ScanWatch lags behind the co-oximeter.
Offset | Mean (s) | Median (s) | SD (s) |
Right hand | –4.2 | –4.5 | 9.5 |
Left hand | –5.2 | –6.5 | 10.9 |
The 14 participants in our study included 8 men and 6 women aged 23-39 years with various skin tones: fair, medium, and dark skin (
Demographic data of the study participants.
Subject | Age (years) | Gender | Skin pigmentation | BMI (kg/m2) | BP1a (mm Hg) | BP2b (mm Hg) |
1 | 25 | Male | Dark | 22.2 | 91/73 | 111/72 |
2 | 26 | Male | Medium | 22.0 | 119/63 | 112/67 |
3 | 23 | Female | Dark | 20.8 | 125/63 | 112/67 |
4 | 23 | Female | Medium | 23.0 | 112/57 | - |
5 | 26 | Female | Light | 21.7 | 123/61 | 126/74 |
6 | 26 | Male | Medium | 28.1 | 113/67 | 120/65 |
7 | 25 | Male | Medium | 22.7 | 117/57 | 121/73 |
8 | 39 | Male | Light | 28.1 | 106/71 | 112/81 |
9 | 28 | Female | Light | 22.7 | 107/63 | 101/67 |
10 | 26 | Male | Medium | 22.4 | 108/64 | 110/55 |
11 | 28 | Female | Medium | 25.4 | 127/70 | 111/67 |
12 | 30 | Male | Medium | 20.8 | 138/91 | 132/88 |
13 | 28 | Male | Dark | 23.5 | 128/70 | 120/68 |
14 | 26 | Female | Light | 25.4 | 109/60 | 107/70 |
aBP1: blood pressure before the study.
bBP2 : blood pressure after the study.
Of the 322 oxygen saturation measurements collected by the Withings ScanWatches placed on the participants’ right and left wrists, 275 (85.4%) and 244 (75.8%) samples, respectively, were classified as conclusive measurements and were subsequently included for further data analysis. The remaining samples, 47/322 (14.6%) and 78/322 (24.2%) taken from the participants’ right and left wrists, respectively, were classified as inconclusive measurements (poor signal quality, motion detected, or watch not worn) and were excluded.
The SpO2 levels collected from the conclusive measurements ranged from 70% to 100% and were evenly distributed into 3 groups: SpO2≤80%, 80%<SpO2≤90%, and 90%<SpO2 (
The correlation plots show a positive strong correlation between the SpO2 values collected from the participants’ right and left hands, with high Pearson correlation coefficients of
Distribution among the SpO2 groups of the conclusive measurements collected by the Withings ScanWatch. The corresponding distribution of SaO2 values given by the ABL90 hemoximeter are also reported.
Group | Values, n (%) | ||
|
Withings ScanWatch (SpO2a) | ABL90 hemoximeter (SaO2b) | |
|
Right wrist (n=275) | Left wrist (n=244) | Blood samples (n=322) |
SpO2≤80% | 87 (31.6) | 72 (29.5) | 103 (32.0) |
80%<SpO2≤90% | 95 (34.5) | 78 (32.0) | 109 (33.9) |
90%<SpO2 | 93 (33.8) | 94 (38.5) | 110 (34.2) |
aSpO2: pulse oxygen saturation as measured by pulse oximetry.
bSaO2: arterial oxygen saturation.
Bias and RMSE found from the Withings ScanWatches placed on the participants’ right and left wrists.
Group | Values (%) | |||
|
Right wrist | Left wrist | ||
|
Bias | RMSEa | Bias | RMSE |
SpO2≤80% | 0.75 | 3.29 | 2.25 | 3.75 |
80%<SpO2≤90% | 2.02 | 3.24 | 2.41 | 3.21 |
90%<SpO2 | 0.14 | 2.41 | 0.31 | 1.90 |
Total | 0.98 | 3.00 | 1.56 | 2.97 |
aRSME: root mean square error.
Correlation plots for the Withings ScanWatches versus the ABL90 hemoximeter from the participants’ right (A) and left (B) hands.
Bland-Altman plots for the Withings ScanWatches and the ABL90 hemoximeter from the participants’ right (A) and left (B) hands. BFSL: best fit straight line.
No undesirable effects or adverse events were reported during the study.
Since 2014, the popularity of smartwatches has grown considerably, particularly in the health care and biomedical industries [
In this study, we observed an inequality in the distribution of the 325 measurements collected, with a higher rate of inconclusive measurements taken from the participants’ left wrists (78/322, 24.2%) than from their right wrists (47/322, 14.6%). This imbalance was mainly caused by unwanted movements when taking blood from the participants’ left arms. We also observed that the interactions between the catheter and the device on the left wrist interfered with the collection of the conclusive measurements when SpO2 levels were below 90% (
Next, we examined the strength of the association and the agreement between the conclusive measurements taken from the reflective pulse oximeter function of the Withings ScanWatch and the ABL90 hemoximeter. We found high Pearson correlation coefficients of
In addition to the accuracy of 3% found in the hypoxia study, the ScanWatch SpO2 algorithm possesses adequate resolution and dynamics to identify apnea and hypopnea events when worn during sleep (
SpO2 measured by ScanWatch and a finger pulse oximeter during apnea/hypopnea events.
The clinical study was conducted in a controlled environment with a well-established protocol and methodology to collect SpO2 measurements during stable plateaus of SpO2 in healthy participants aged 23-39 years. Its design may limit the generalizability of the results to real-world situations. Indeed, the ability of the Withings ScanWatch reflective pulse oximeter to dynamically monitor the evolution of SpO2 in a subject is unknown in the real world because participants were exposed to stable plateaus of SpO2 between 70% and 100% in this study. Finally, given the risks induced by hypoxia on older subjects or patients with respiratory conditions, we were unable to test the ability of the Withings ScanWatch to measure SpO2 in these populations. In future work, the accuracy of the Withings ScanWatch reflective pulse oximeter should therefore be tested in real-life conditions (including in the home, at a hospital ward, and during rehabilitation), on a specific population such as patients with COPD or obstructive sleep apnea, or to diagnose and monitor patients with respiratory diseases.
FDA guidance and the ISO 80601-2-61:2017 standard require RMSEs below 3.5% and 4% for reflectance pulse oximeter approval, respectively [
alternating current
chronic obstructive pulmonary disease
direct current
US Food & Drug Administration
deoxygenated hemoglobin
oxygenated hemoglobin
light-emitting diode
photodiode
photoplethysmography
polysomnography
root mean square error
arterial oxygen saturation
sleep apnea syndrome
signal-to-noise ratio
pulse oxygen saturation as measured by pulse oximetry
This work was supported by Withings (Issy-les-Moulineaux, 92130, France). We thank Guillaume Ha, PhD, for his medical writing assistance.
Both authors are employees of Withings, which manufactures ScanWatch, the connected watch studied in this article.