Add WAKE-REM-NREM classifiers (#84)

* add wake-rem-nrem classifiers * fix number of training records * fix image links * actually fix links * Apply suggestions from code review Co-authored-by: Clemens Brunner <[email protected]> Co-authored-by: Clemens Brunner <[email protected]>
cbrnr · Mar 21, 2022 · 23f3636 · 23f3636
1 parent 31dc61e
commit 23f3636
Show file tree

Hide file tree

Showing 8 changed files with 2,651 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,7 @@
 - Add support for Python 3.10 ([#75](https://github.com/cbrnr/sleepecg/pull/75) by [Florian Hofer](https://github.com/hofaflo))
 - Add functions for classification ([#78](https://github.com/cbrnr/sleepecg/pull/78) by [Florian Hofer](https://github.com/hofaflo))
 - Add a WAKE-SLEEP classifier trained on MESA ([#78](https://github.com/cbrnr/sleepecg/pull/78) by [Florian Hofer](https://github.com/hofaflo))
+- Add two WAKE-REM-NREM classifiers trained on MESA ([#84](https://github.com/cbrnr/sleepecg/pull/84) by [Florian Hofer](https://github.com/hofaflo))
 
 ## [0.4.1] - 2022-01-14
 ### Fixed

diff --git a/doc/source/classification.md b/doc/source/classification.md
@@ -8,7 +8,19 @@ Note that the performance metrics are averages, so a classifier might perform mu
 
 |Identifier|Stages|Model|Training set (nights)|Test set (nights)|Acc|κ|Source|
 |-|-|-|-|-|-|-|-|
-|`ws-gru-mesa`|WAKE-SLEEP|[GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)|MESA (1971)|SHHS (1000)|0.83|0.60|[Link](https://github.com/cbrnr/sleepecg/blob/main/examples/classifiers/ws_gru_mesa.py)|
+|`ws-gru-mesa`|WAKE-SLEEP|[GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)|MESA (1970)|SHHS (1000)|0.83|0.60|[Link](https://github.com/cbrnr/sleepecg/blob/main/examples/classifiers/ws_gru_mesa.py)|
+|`wrn-gru-mesa`|WAKE-REM-NREM|[GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)|MESA (1970)|SHHS (1000)|0.75|0.54|[Link](https://github.com/cbrnr/sleepecg/blob/main/examples/classifiers/wrn_gru_mesa.py)|
+|`wrn-gru-mesa-weighted`|WAKE-REM-NREM|[GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit)|MESA (1970)|SHHS (1000)|0.70|0.53|[Link](https://github.com/cbrnr/sleepecg/blob/main/examples/classifiers/wrn_gru_mesa_weighted.py)|
+
+
+### Notes
+As REM is much less common than NREM and WAKE, the current models show limited performance in WAKE-REM-NREM classification.
+Using no sample weights, the `wrn-gru-mesa` model identifies 64% of actual REM stages as NREM as shown in the confusion matrix below (left).
+With sample weights inversely proportional to class frequency in `wrn-gru-mesa-weighted`, the confusion matrix (right) is more symmetrical.
+While the values normalized over true classes (given in parentheses) look promising, the number of actual NREM stages identified as REM is larger than the number of correctly identified REM stages (i.e. the model has low precision for REM).
+A weaker weighting approach is likely required to find the optimal middle way.
+
+![wrn-gru-mesa confusion matrix](./img/wrn-gru-mesa.svg)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;![wrn-gru-mesa-weighted confusion matrix](./img/wrn-gru-mesa-weighted.svg)
 
 
 ## Usage examples