Tag/detect drift

[cdemulde]

Write a function detecting drift in the data. The idea would be that the user gives:

• a range in which to apply the function (similar to other tagging functions)
• the maximum slope a signal is expected to have over a certain period (see below)
• the period over which a certain slope is allowed

This function could calculate the slope of the data in a certain given period (by for example fitting a line through it) and compare it with the maximum expected slope. In first instance, it would be interesting for the user to know if drift is present, secondly it would be good to be able to correct for it.

Some additional sources of information/inspiration:

• https://dsp.stackexchange.com/questions/27468/how-to-remove-or-filter-the-drift-problem-in-measured-strain-signal
• https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.detrend.html#scipy.signal.detrend

@jorasinghr could you have a look at this and let me know if things are unclear?

[cdemulde]

Looking good! Some ideas for further work on this:

• Apply the currently implemented detect_drift function iteratively over a complete dataset (so over day 0 to 5, over day 1 to 6, over day 2 to 7 etc.) to not only detect if there is drift in a certain period, but to detect any drift at all in a dataset. User output would then be something like: drift detected in period ... to ...
• Once the detection is done, the scipy.detrend function can be used in another function to actually remove the drift from a timeseries. When this is done, tags need to be added to the self.meta_filled dataset, indicating that the original values were changed at certain indexes.
• Since I don't expect drift to be present in the current testing dataset, might be good to create one where we introduce some artificial drift and see if the functions successfully pick it up/remove it.

[cdemulde]

Thanks for your work on this @jorasinghr! I'll have a closer look as soon as I can, but I can already tell this has been a great help!

[cdemulde]

Both tagging and removing function are good to go, only some additional testing still needs to be done:

• Downward drift
• Real data with drift
• Longer dataset