Japan data gaps

[russbiggs]

Data from Japan consistently has 2 hours of missing data every 24 hours.

We should prioritize fixing this gap and patching historical to ensure that daily averages are not bias significantly by this gap. I would also like ot ensure that this is not indicative or a larger datetime issue. @majesticio can we do an audit to ensure the measurement values we have are matching up with what Japan reports to rule out that type of issue and calirfy this is just missing data.

[majesticio]

I will audit the data to check for parity. If it is only on our end I will try to throttle Japan a little more to see if it resolves the issue.

[majesticio]

Strange finding today. I was looking at Japan right during the data gap. The provider is indeed sending back csv files, but they only contain the headers and no data.

Its is a url like this:
https://soramame.env.go.jp/data/sokutei/NoudoTime/01101520/today.csv?_=1707844422578

changing or removing the timestamp at the end did not improve the response.

I inspected another station endpoint on their live map (eg https://soramame.env.go.jp/soramame/download?DL_KBN=2&TDFKN_CD=09&SKT_CD=09207010) and it returned this:

09207010,2024/02/13,22,0,0,0.005,0.005,,0.029,0.08,2.02,2.10,0.005,0,,NNW,1.9,,
09207010,2024/02/14,01,0,0,0.004,0.004,,0.031,0.06,2.01,2.07,0.004,3,,NW,0.5,,

These are the last 2 rows of the csv. the first column is the station ID, then date, then hour. You can clearly see the data gap as it jumps from hour 22 to hour 1 of the following day.

I intend to inspect this again tomorrow during the data gap period, but it does seem like this is on the provider's end, not ours.

[majesticio]

I can confirm that the data gap is persistent and aligns with the data gaps we see on our platform. The csvs that are requested around midnight in Tokyo time only have headers, but no data. Oddly, the monthly download csv ( which does not include the last 24 hours ) is not missing those hours. Its clumsy, but I am suggesting we fill in the data gaps with a separate adapter and offset deployment that will process the monthly csv to fill the missing data retroactively

[russbiggs]

If using the monthly does that mean we wouldnt get the data until the end of the month? or is the monthly csv populated daily?

[majesticio]

The monthly csv is updated daily, but is always one day behind