My Comparison Of 3 Sleep Trackers With A Manual Diary

photo of the emfit beautyrest and fitbit versa sleep trackers

Have you ever used a sleep tracker that said you were asleep when you know you were in bed reading or watching TV? Or that you slept all night, when you know you got out of bed more than once?

How accurate are they really?

The accuracy of personal sleep trackers has long been in question. The manufacturers, of course, say they are accurate, perhaps with fine print that says ‘80%’ accurate in some cases.

But people all too often find serious errors in the data, and their faith in the device is slowly but surely eroded to zero.

So how do we know we can trust a sleep tracker?

It would be great to take a vast range into a sleep center and compare them to a more reliable polysomnogram over several nights.

That’s not what sleep centers are for though, and you couldn’t do it every time a new device came out. So as an alternative, I do my own tests with smaller selections of trackers.


Contents


How I ran this test


For this particular test, I compared 3 trackers for 5 nights. Importantly, I kept a manual sleep diary and noted the information which I could objectively record, including:

  • The time I got into bed.
  • Estimated time of falling asleep, based on the time I turned the light off and my memory of how quickly I fell asleep.
  • The exact time of any extended wakings or bed exits in the night.
  • The time I woke up in the morning.
  • The time I got out of bed.

Comparing the sleep stages data

Many trackers provide much more detailed data, such as sleep stages, heart and breathing rate, movements, and give you a sleep score etc.

But I can’t objectively monitor those. So my aim is simply to check that they get the basics right. And if they can do that, then perhaps I can be more optimistic that they get the sleep stages right too.

However, it is possible to compare what they each say about sleep stages. And that’s what you’ll find throughout this article.

My hope was that at least 2 of the 3 trackers would agree on the sleep stages, and preferably all 3. Sadly that wasn’t the case, even if they sometimes appeared the same on some measures.

The trackers I tested

For this particular test, I used 4 tracking methods:

  1. My personal sleep diary – a pen and paper next to my bed, along with a clock.
  2. Emfit QS – a dedicated sleep and sports recovery tracker that uses a measuring strip placed under the mattress.
  3. Beautyrest – a dedicated sleep tracker that uses a measuring pad under the mattress.
  4. Fitbit Versa – a sleep and activity tracker worn on the wrist.

Note that my intention isn’t to compare every single feature these trackers offer. It’s simply to compare their basic accuracy for sleep. You’ll find this article is long enough already!

photo showing the emfit, beautyrest and fitbit versa sleep trackers together

The Emfit and Beautyrest both work with a measuring strip under your mattress, connected to a small data collection device and connected to your Wifi. The Fitbit Versa is worn all night on the wrist.

 


Night 1 results


Get into bed

  • My sleep diary: 10 p.m.
  • Emfit: 10 p.m.
  • Beautyrest: doesn’t record this
  • Fitbit Versa: doesn’t record this

Fell asleep

  • My sleep diary: lights out at 12 a.m. and fell asleep soon after.
  • Emfit: 12.10 a.m.
  • Beautyrest: 12.06 a.m.
  • Fitbit Versa: 12.02 a.m.

All three trackers show that I fell asleep shortly after turning the light out at midnight. And I think just 8 minutes difference between them is pretty good.

However, there’s an issue with the Emfit. Although it got the 00.10 time right, it actually started tracking when I got into bed at 10 p.m.

And it recorded some of the time I was reading as light sleep – presumably because of my lower heart rate and lack of movement.

Total sleep

  • My sleep diary: Must be less than 8 hours
  • Emfit: 7 hrs 56
  • Beautyrest: 7 hrs 54
  • Fitbit Versa: 7 hrs 26

By my calculations, the total sleep time has to be less than 8 hours. The Emfit and Beautyrest both record just under 8 hours, meaning my nocturnal wakings would have been very brief.

The Versa is considerably more conservative, perhaps being tighter with the decisions about when I was awake or not.

Significant wakings in the night

  • My sleep diary: 5:00, 5:30
  • Emfit: 5:00, 5:20
  • Beautyrest: 5:00, 5:30
  • Fitbit Versa: 5:00, 5:30

All three trackers did a good job of recording my double waking – once for the bathroom and once for some water. The Emfit recorded one of them as 10 minutes early.

Time of waking

  • My sleep diary: 8.15 awake. A little later made coffee, went back to bed and read until 9.05.
  • Emfit: awake at 8.25. Got up at 9.05. Recorded some reading time as light sleep.
  • Beautyrest: awake at 8.18.
  • Fitbit Versa: awake at 8.27

The Beautyrest is pretty much spot on with the waking time. The Emfit and Versa both took a while to catch up, and only when I got out of bed to make coffee do they acknowledge that I wasn’t asleep.

The Emfit records some additional light sleep when I was in bed checking emails.

Sleep stages

Notes to understand the graphs

It’s difficult to compare the exact timings visually if the trackers have a different start and finish time. So it’s important to read my bullet points and notes below, and not rely on the visual look of the graphs alone.

And unfortunately, the writing tends to be very small on the graphs, especially when uploaded to a website article! So I apologize if you struggle to read them.

The blue and yellow on the Emfit (top) are for time awake. Later graphs also say ‘bed exit’ in tiny writing. The lowest height purple is deep sleep, middle height is light sleep, and the peaks are REM sleep.

The Fitbit and Beautyrest don’t show exits, but they do show awake time. This is represented by a person on the Beautyrest graph, and red peaks in the Fitbit graph. They both have a code next to them to explain their other stages.


Sleep stage graphs for night 1: 

image of the different sleep stage graphs provided by the emfit, beautyrest and fitbit versa on august 2 to compare the timings

REM

  • Emfit: 1.55 hrs
  • Beautyrest: 2.20 hrs
  • Fitbit Versa: 1.53 hrs

Deep

  • Emfit: 1.48 hrs
  • Beautyrest: 0.54 hrs
  • Fitbit Versa: 1.53 hrs

Light

  • Emfit: 4.13 hrs
  • Beautyrest: 4.40 hrs
  • Fitbit Versa: 3.45 hrs

Awake

  • Emfit: 2.42 hrs
  • Beautyrest: 0.28 hrs
  • Fitbit Versa: 1 hrs

It’s tempting to think the Emfit and Versa have quite similar sleep stage timings, at least for the REM and deep stages. And the Beautyrest has a different idea altogether.

However, if you look through the following nights’ data, there are many occasions when the times are very different.

What’s interesting is that the graphs of all three trackers show a large period of REM sleep just after 7 a.m. in the morning. And they also show more deep sleep in the first half of the night.

Both of those points follow the pattern that most people will experience, so that gives some room for optimism.

So perhaps the trackers are doing a reasonable job of spotting when I’m in different stages. They just can’t agree on exactly how long I spend in each. They all have different algorithms, so that’s no real surprise.


Night 2 results


Get into bed

  • My sleep diary:  10.40 p.m.
  • Emfit:  10.45 p.m.
  • Beautyrest: doesn’t say
  • Fitbit Versa: doesn’t say

Fell asleep

  • My sleep diary: 12.10 a.m. lights out. Fell asleep quickly.
  • Emfit: 12.10 a.m.
  • Beautyrest: 12.12 a.m.
  • Fitbit Versa: 12.13 a.m.

Again, all 3 trackers are quite accurate with the time I fell asleep.

Total sleep

  • My sleep diary: hard to say because of the number of wakings.
  • Emfit: 9 hrs 44
  • Beautyrest: 8 hrs 36
  • Fitbit Versa: 8 hrs 18

I wrote in my diary that it felt like a bad night’s sleep because I woke so often. Perhaps the trackers are right though, and despite waking a lot, I fell asleep quickly each time.

The Emfit has over an hour more sleep than the other two, which seems unlikely considering how I felt that it wasn’t a good night’s sleep.

Wakings in the night

  • My sleep diary: 2:00, 4:00, 4:30, then many up until 8 a.m.
  • Emfit: 4:25, 4:45
  • Beautyrest: 2:00, awake from 4:20 to 5:00
  • Fitbit Versa: 2:00, 4:00, 4:30, 4:45 + large number of short wakings

The Versa and Beautyrest both did a good job at reporting my wakings. The Emfit, however, recorded much of the time I was awake as light sleep, including times I was writing in my diary.

The Versa is the clear winner on this point though, correctly recording all of my significant wakings.

Time of waking

  • My sleep diary: 9:30 a.m. woke up, made coffee, checked emails in bed.
  • Emfit: 9:15 woke up. Later resumed sleep tracking when I was checking emails.
  • Beautyrest: 9:32 woke up – automatically stops tracking
  • Fitbit Versa: 9:31 woke up – automatically stops tracking

The Versa and Beautyrest are spot on again, and both stopped the sleep tracking even though I got back into bed with a coffee to check emails. The Emfit continued tracking until I finally left the bed 1.5 hours later (don’t judge me – it was all for the experiment!), and records some light sleep when I was awake.

Sleep stages

Sleep stage graphs for night 2: 

image of the sleep stage graphs provided by the emfit, beautyrest and fitbit versa on august 3

REM

  • Emfit: 1.35 hrs
  • Beautyrest: 2.18 hrs
  • Fitbit Versa: 2.08 hrs

Deep

  • Emfit: 2.34 hrs
  • Beautyrest: 0.58 hrs
  • Fitbit Versa: 2.04 hrs

Light

  • Emfit: 5.35 hrs
  • Beautyrest: 5.20 hrs
  • Fitbit Versa: 4.06 hrs

Awake

  • Emfit: 2.20 hrs
  • Beautyrest: 0.54 hrs
  • Fitbit Versa: 1.12 hrs

Once again, the exact time spent in each sleep stage is different for all three. There isn’t even a match between two of them that might be interesting to explore more.

Having said that, all three do show significant wakings around the same time. And even though the total time is different, they all show 5 phases of REM sleep, with more as the night progresses.

So it seems that they roughly agree on when different stages were happening, but not on how long was spent in each stage.


Night 3 results


Since only the Emfit tracks when I get into bed (and reliably so), I’ll skip straight to the more interesting sleep timings for the remaining nights.

Fell asleep

  • My sleep diary: 12.50 a.m. lights out. Not sure how long I was awake for.
  • Emfit: 11.55 p.m. (records an hour of reading as sleep time)
  • Beautyrest: 12.49 a.m.
  • Fitbit Versa: 12.51 a.m.

Total sleep

  • My sleep diary: hard to say because of the number of wakings.
  • Emfit: 7 hrs 46
  • Beautyrest: 5 hrs 11
  • Fitbit Versa: 4 hrs 30

I wrote in my diary that it felt like a very bad night’s sleep. The Beautyrest and Versa reflect this, although there are still 41 minutes difference between them. The Emfit is very generous with my sleep time.

Wakings / get up times

  • My sleep diary: 5:15, 5:25 to 6:10, made tea, stayed in bed until 7:25, made more tea, got out of bed at 8 a.m.
  • Emfit: 5:40, 6:05 to 6:25, 7:15 to 7:30, 7:55 to 8:10
  • Beautyrest: records awake from 5:37 a.m. through to 7:29 a.m. Then automatically stopped tracking.
  • Fitbit Versa: 01:39, 4:52, 5:35 to 5:40, 5:55 to 6:03. Automatically stopped tracking.

There were so many wakings on this night, and all 3 trackers rightly spot that, recording many wakings for extended periods. However, they don’t really agree on when the wakings took place.

What’s interesting is the point each tracker decided to stop the sleep period. That has a lot to do with their algorithm, but you also wonder how much is due to their sensitivity and accuracy generally.

Sleep stages

Sleep stage graphs for night 3: 

image of the sleep stage graphs provided by the emfit, beautyrest and fitbit versa on august 4

REM

  • Emfit: 1.57 hrs
  • Beautyrest: 0.55 hrs
  • Fitbit Versa: 0.50 hrs

Deep

  • Emfit: 1.17 hrs
  • Beautyrest: 0.59 hrs
  • Fitbit Versa: 0.41 hrs

Light

  • Emfit: 4.32 hrs
  • Beautyrest: 3.17 hrs
  • Fitbit Versa: 2.59 hrs

Awake

  • Emfit: 1.24 hrs
  • Beautyrest: 2.01 hrs
  • Fitbit Versa: 0.47 hrs

At first glance, it’s tempting to think that the Beautyrest and Versa appear quite similar on three of the measurements. But the differences are still marked enough to not convince me personally that they can be depended on.

Having said that, they both match quite well with the REM stage, similar to previous nights. So could there be a pattern developing in how similar those two are?


Night 4 results


Fell asleep

  • My sleep diary: 12.10 a.m. lights out. Fell asleep very quickly.
  • Emfit: 12.10 a.m.
  • Beautyrest: 11:50 p.m.
  • Fitbit Versa: 12.11 a.m.

This time it’s the Beautyrest’s turn to mistake reading as being asleep. The Emfit and Versa are both accurate though.

Total sleep

  • My sleep diary: I don’t remember waking at all, so should be between 7 and 8 hours.
  • Emfit: 8 hrs 28
  • Beautyrest: 8 hrs 21
  • Fitbit Versa: 7 hrs 19

The Emfit and Beautyrest both have an impossible time here. I didn’t even have my eyes closed for the time they say I was asleep. The Versa looks more reasonable.

Wakings in the night

  • My sleep diary: I don’t remember waking at all.
  • Emfit: none recorded
  • Beautyrest: 11:45, 12:04, 01:03
  • Versa: 00:21, 01:04, 02.59, 06.35, 08.27

This night is a good example of the difference in sensitivity. The Emfit seems to agree with me that I didn’t wake up at all. The Beautyrest and Versa show more wakings, with several that last only a minute.

So are they too sensitive, recording normal nocturnal movement as being awake? Or is the Emfit (and my memory!) not sensitive enough, and missing wakings?

It feels like a crucially important question. And frustratingly, it’s one I don’t have the answer to.

Time of waking

  • My sleep diary: 8:30 a.m.
  • Emfit: 8:35 a.m.
  • Beautyrest: 8:32 a.m.
  • Fitbit Versa: 8:31 a.m.

All 3 trackers do a very good job of recording the time I woke up. I got up straight away on this morning, rather than being a difficult customer and lying in bed reading. And that seems to have made life easier for the trackers.

Sleep stages

Sleep stage graphs for night 4:

image of the sleep stage graphs for night 4 of the experiment

REM

  • Emfit: 1.42 hrs
  • Beautyrest: 2.25 hrs
  • Fitbit Versa: 2.39 hrs

Deep

  • Emfit: 1.57 hrs
  • Beautyrest: 0.55 hrs
  • Fitbit Versa: 1.26 hrs

Light

  • Emfit: 4.49 hrs
  • Beautyrest: 5.01 hrs
  • Fitbit Versa: 3.24 hrs

Awake

  • Emfit: 0.35 hrs
  • Beautyrest: 0.31 hrs
  • Fitbit Versa: 1.01 hrs

As is the case on every night, there are some similar measurements between two trackers on one or two points. But the differences seem more apparent than the similarities on this night.


Night 5 results


Fell asleep

  • My sleep diary: probably between 12:00 and 12:20 a.m.
  • Emfit: 12:00 a.m.
  • Beautyrest: 12:18 a.m.
  • Fitbit Versa: 12:12 a.m.

All three trackers are in the right ballpark, and I wasn’t sure exactly when I fell asleep anyway.

Total sleep

  • My sleep diary: slept through the night. So total time should be no more than 7.55 hours
  • Emfit: 8.28 hrs
  • Beautyrest: 7.53 hrs
  • Fitbit Versa: 7.09 hrs

The Emfit gives an impossible total sleep time, with 33 minutes more time asleep than I even had my eyes closed for.

The Beautyrest might be right, but only if I literally had no small wakings at all – something I can’t confirm either way as I just don’t remember. The Versa is more conservative and suggests I was awake more often than I thought.

Wakings

  • My sleep diary: none
  • Emfit: has a bed exit at 8:15 then asleep again.
  • Beautyrest: none
  • Fitbit Versa: 01.53, 04.05, 06.13

The Beautyrest and Emfit agree that there were no significant wakings in the night. The Versa records three, each lasting more than 5 minutes. I don’t remember any of them though.

Time of waking

  • My sleep diary: woke at 8:00 am. Made coffee at 8:15 a.m and read in bed until 9 a.m.
  • Emfit: exit bed at 8:15 am. Records time in bed until 9 a.m. correctly. But adds 10 minutes REM sleep in the middle.
  • Beautyrest: Automatically stopped tracking at 8:11 a.m.
  • Fitbit Versa: awake at 8:07 am. Stopped tracking at 8:12 a.m.

The Beautyrest is 11 minutes late to spot my wake up time. The Versa is also a few minutes out. The Emfit gets it very wrong and records 10 minutes of reading time as REM sleep.

Sleep stages

Sleep stage graphs for night 5:

sleep stages graphs for night 5

REM

  • Emfit: 1.40 hrs
  • Beautyrest: 2.23 hrs
  • Fitbit Versa: 1.36 hrs

Deep

  • Emfit: 1.45 hrs
  • Beautyrest: 0.53 hrs
  • Fitbit Versa: 1.04 hrs

Light

  • Emfit: 5.03 hrs
  • Beautyrest: 4.37 hrs
  • Fitbit Versa: 4.29 hrs

Awake

  • Emfit: 0.51 hrs
  • Beautyrest: 0.09 hrs
  • Fitbit Versa: 0.51 hrs

There are a couple of significant times spent in a sleep stage that two or even all three trackers match on. All three have me in a deep sleep for a good chunk of time at roughly the same time.

But there are also points in the night when the trackers think I’m in different sleep stages.

 


Conclusions


1. Time fell asleep + reading in bed or sleeping?

One of the most obvious outcomes from the 5 nights is that the Emfit struggled to tell the difference between reading in bed and sleeping. On all 5 nights, it recorded sleep when I was most definitely awake.

It would get the time I turned the light out and fell asleep fairly right, which is good. But it would add extra sleep time before then mixed in with time shown as awake in bed.

Yes, you can manually adjust the time in the morning. But personally, I don’t feel that I should have to. It just sets off alarm bells that I can’t trust the rest of the data.

The Beautyrest made the same mistake once, but along with the Fitbit Versa, was mostly very good at automatically recording when I really fell asleep.

2. Total sleep

The Emfit tended to record more sleep than was even possible. For example, adding an extra hour to the total time I even had my eyes closed.

Part of the problem seems to be that it records from the second you get into bed. So if it mistakes your low heart rate and lack of movement when relaxing as light sleep, then the total time will also be wrong.

The Beautyrest cut this measure a little fine. It often had the total time very close to the total possible time, based on how long I had my eyes closed.

The Versa was either the most accurate or perhaps a little too conservative with the calculation. It’s hard for me to say.

But overall, even though they were sometimes different, at least the Versa and Beautyrest seemed reasonably accurate to me.

3. Wakings in the night

All 3 trackers did pretty well on this point I thought. None of them missed times I got up to use the bathroom or to prepare a drink. And even though they were sometimes out by 5 to 10 minutes in the actual timings, the overall picture of how restless the night was is fairly reliable.

On the whole, I think I’d be happy with any of them if I needed to monitor significant wakings and bed exits in the night. But perhaps not if I needed to know exactly how long they each lasted!

4. Time of waking

I thought the Versa was very good at recording my waking time and automatically stopping the tracking. The Beautyrest had a couple of blips, but wasn’t bad on the whole.

The Emfit wasn’t bad at spotting the first major wake up. But as with reading in bed at the start of the night, it again recorded some sleep if I stayed in bed drinking a cup of tea and checking emails.

5. Sleep stages

I really wanted at least two of the trackers to have the same sleep stage reports, and preferably all three. But that didn’t happen. Even on the nights when two trackers seemed to agree on REM or deep sleep, they would then disagree completely on light sleep and awake.

The only room for optimism is that all three sometimes agreed on the number of REM or deep sleep stages, and even roughly when they occurred in the night.

And this was often in line with what scientists know about normal sleep patterns. For example, more deep sleep earlier in the night, more REM sleep later in the night, and perhaps 4 or 5 cycles in total.

The problem is that they disagree on how long each stage lasted, and the exact time they started and finished.

So unfortunately, my conclusion is that I can’t really conclude anything! As most sleep experts continue to say, you should take the sleep stage reports with a pinch of salt.

Maybe one of the trackers was secretly very accurate, but I simply have no way of knowing from this test.


Overall conclusion


I was disappointed with the Emfit overall, and impressed by both the Fitbit Versa and Beautyrest.

I think the main stumbling block is the time spent in bed relaxing. All 3 trackers showed that when I don’t spend too long reading, or get up when I wake up, they can keep on top of things well.

But when the night is more complicated, with a long time in bed reading and multiple wakings, they quickly deviate from a place of agreement.

As I mention in the section below, the Emfit is also the least user-friendly. It’s more complex to set up, and doesn’t offer any coaching to help you sift through the vast amount of data it sends your way.

The Beautyrest really shines when it comes to turning the data into something you can understand and use. The Fitbit Versa is somewhere in the middle I think.

But on balance, I’d be happy to use either of those two on an ongoing basis. I can’t confirm if their sleep stage tracking is accurate, but neither can I say it isn’t.

So for now, I wouldn’t base any serious decisions on that data, but I would use it to get a good overall feel for how your sleep is looking, along with some helpful suggestions as to how you can improve it.


More information about the 3 trackers


As I mentioned at the start, my intention with this article was to compare accuracy. However, with the experience I have with the trackers, I’ll briefly cover a few points that I know readers are typically interested in.

Price

The prices do fluctuate, but at the time of writing (October 2018) the prices in the U.S. are:

  • Sleep diary – the price of a pen and paper!
  • Emfit – $250
  • Beautyrest – $178 (however, there are some premium features you need to sign up for. Basically, you just get the data presented better for you, but it’s no essential).
  • Fitbit Versa – $197

Setup and installation

Sleep diary

Easy with pen and paper. If you use an App, you just need to download it.

Emfit

The most complicated setup. It’s easy enough to put the measuring strip under your mattress – assuming you have the physical ability to move your mattress out of the way first.

If you don’t have good technical troubleshooting skills, you can run into problems if your home wifi isn’t already configured the way the tracker needs it to be. To be fair, mine was ok – just a little confusing to connect it to the Wifi. But it’s not guaranteed to work out of the box on all Wifi connections.

Registering on their website is fine, but does take a little time.

Beautyrest

Easier than the Emfit. Again, you need to move your mattress to attach the measuring strip.

Connecting to the Wifi is more straightforward, with simple video instructions on the phone App once you download it. There are some possible complications depending on the type of Wifi you have, so it’s worth double checking in advance.

Registering is also easy to do once you download the App. I had mine set up in under 10 minutes.

Fitbit Versa

Easy unless you’re one of the unlucky ones that has problems syncing with your smartphone!

You just charge the tracker out of the box, download the Fitbit App, register and sync via Bluetooth. It’s relatively easy to do, and the only problem is the occasionally unpredictable syncing.

Where do you get the sleep tracking data?

  • Sleep diary – n/a
  • Emfit – log into their website on your computer, tablet or smartphone.
  • Beautyrest – on an App, and also daily emails they send you.
  • Fitbit Versa – on an App. You can also access a desktop version if you prefer.

Ease of understanding the App data

  • Sleep diary – easy as long as you can read your handwriting in the middle of the night!
  • Emfit – overwhelming if you consider all the other heart rate and recovery data. If you focus on the sleep data, it’s nicely presented on the graphs.
  • Beautyrest – easy. Nice visual graphs, along with simple sleep scores and targets.
  • Fitbit Versa – easy. Nice graphs, and the important data is presented clearly.

Sleep coaching / interpretation of the data

  • Sleep diary – n/a
  • Emfit – Generic information to explain what the data means. No personalised interpretation or sleep coaching. You have to do the work yourself.
  • Beautyrest – the best for sleep coaching. They provide personalised interpretations, advice for improving your sleep, and take into account other data you provide such as stress levels and exercise.
  • Fitbit Versa – General information about the data. Interesting benchmarks against your own 30 day data, and people your gender and age. Some sleep tips from time to time on the App, but could be better.

Your view

Have you tried any of these trackers? What has your experience been of them, or any others?

And I’m open to suggestions about the way I’ve interpreted the results of this test, or ideas for making the next one better.

Leave a comment

Your email address will not be published.


Your message will only be visible after moderation.