missing and bad data

[wwieder]

How should we handle missing or bad data?

Specifically NIWO has big data gaps that can't be filled (relative humidity missing for Dec 2018 - June 2019).

OR

Odd spikes in temperature and pressure in Jan & Feb 2019

[wwieder]

@ddurden I noticed the comment about RH here.

NCAR-NEON/TowerTools_ForcingData/flow.api.clm.R

Line 298 in 8e37bc3

varDp <- c("PRECTmms_MDS" = "SECPRE_30min", "rH" = "RH_30min", "FLDS_MDS" = "SLRNR_30min", "Rg" = "SLRNR_30min", "Pa_MDS" = "BP_30min") #Currently using the relative humidity from the soil array, tower top was not reporting data at HARV during this time

Is there another data product for relative humidity we could be using?

[wwieder]

There also must be other temperature data we can use. Big gaps, here from SCBI, as well as a few absurdly low values

[wwieder]

@ddurden said the new NEON data processing pipeline will improve the quality of flags that we can use for this. What's the timeline for this release, Dave?

[ddurden]

@wwieder , we just finished reprocessing all the NEON data. Additionally, I just pulled in a robust regression gap-filling algorithm that will fill data from replicate, adjacent, or similar streams.

[wwieder]

awesome, @ddurden is this ready to go now, or are you still working on the script?

[ddurden]

@wwieder , the new data are available and just added the regression gap-filling function. I'm working on applying this to all driver data streams. I'm still targeting Jan. for data files being in ECS. However, if you want access to the updated function ahead of time I can point you to it.

[wwieder]

I think this issue is outdated, with updates noted in #50