Relax nanosecond datetime restriction in CF time decoding

[kmuehlbauer]

• Closes Interoperability with Pandas 2.0 non-nanosecond datetime #7493
• Closes DataArray constructor still coerces to np.datetime64[ns], not cftime in 0.11.0 #2587
• Tests added/changed
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

This is another attempt to resolve #7493. This goes a step further than #9580.

The idea of this PR is to automatically infer the needed resolutions for decoding/encoding and only keep the constraints pandas imposes ("s" - lowest resolution, "ns" - highest resolution). There is still the idea of a default resolution, but this should only take precedence if it doesn't clash with the automatic inference. This can be discussed, though. Update: I've implemented time-unit-kwarg a first try to have default resolution on decode, which will override the current inferred resolution only to higher resolution (eg. 's' -> 'ns').

For sanity checking, and also for my own good, I've created a documentation page on time-coding in the internal dev section. Any suggestions (especially grammar) or ideas for enhancements are much appreciated.

There still might be room for consolidation of functions/methods (mostly in coding/times.py), but I have to leave it alone for some days. I went down that rabbit hole and need to relax, too 😬.

Looking forward to get your insights here, @spencerkclark, @ChrisBarker-NOAA, @pydata/xarray.

Todo:

• floating point handling
• Handling in Variable constructor
• update decoding tests to iterate over time_units (where appropriate)
• ...

[kmuehlbauer]

Nice, mypy 1.12 is out and breaks our typing, 😭.

[TomNicholas]

Nice, mypy 1.12 is out and breaks our typing, 😭

Can we pin it in the CI temporarily?

[kmuehlbauer]

Can we pin it in the CI temporarily?

Yes, 1.11.2 was the last version.

[kmuehlbauer]

This is now ready for a first round of review. I think this is already in a quite usable state.

But no rush, this should be thoroughly tested.

[spencerkclark]

Sounds good @kmuehlbauer! I’ll try and take an initial look this weekend.

[ChrisBarker-NOAA]

create a nice example how to handle these difficulties?

Sure -- where would be a good home for that?

[kmuehlbauer]

Not sure, but https://docs.xarray.dev/en/stable/user-guide/time-series.html could have a dedicated floating point date section.

[kmuehlbauer]

I've added a kwarg time_unit into the decode_cf and subsequent functionality.

But instead of adding that kwarg we could slightly overload the decode_times to take one of "s", "ms", "us", "ns" with "ns" as default.

This would have the positive effect, that we wouldn't need the additional kwarg and have to distribute it through the backends.

• decode_times=None - directs to decode_times=True
• decode_times=False - no decoding
• decode_times=True - decode times with default value ("ns")
• decode_times="s" - decode times to at least "s"
• decode_times="ms" - decode times to at least "ms"
• decode_times="us" - decode times to at least "us"
• decode_times="ns" - decode times to "ns"

We could guard decode_times=None and decode_times=True with a DeprecationWarning and add our new defaults in the WarningMessage (eg. "us").

This methodology would be fully backwards compatible. It advertises the change via DeprecationWarning in normal operation and also if issues appear in the decoding steps.

If this is something which makes sense @shoyer, @dcherian, @spencerkclark, I'd add the needed changes to this PR.

[dcherian]

Alternatively, we could make small progress on #4490 and have

from xarray.coding import DatetimeCoder

ds = xr.open_mfdataset(..., decode_times=DatetimeCoder(units="ms"))

In the long term, it seems nice to have the default use the "natural" units i.e. "h" for units="hours since ..." and apparently even "M" for units=months since ... (!!)

https://numpy.org/doc/stable/reference/arrays.datetime.html#basic-datetimes
The date units are years (‘Y’), months (‘M’), weeks (‘W’), and days (‘D’), while the time units are hours (‘h’), minutes (‘m’), seconds (‘s’), milliseconds (‘ms’),

[kmuehlbauer]

Alternatively, we could make small progress on #4490 and have

from xarray.coding import DatetimeCoder

ds = xr.open_mfdataset(..., decode_times=DatetimeCoder(units="ms"))

This took a while to sink in 😉 Yes, that's a neat move. I'll incorporate this suggestion.

In the long term, it seems nice to have the default use the "natural" units i.e. "h" for units="hours since ..." and apparently even "M" for units=months since ... (!!)

As long as we use pd.Timestamp for parsing the time unit specification (eg. seconds since 1992-10-8 15:15:42.5 -6:00) we can only do this for those units pd.Timestamp supports (‘s’, ‘ms’, ‘us’, and ‘ns’). We could add some code that checks if the data can be represented in "days" or "hours" (as given in the time unit specification) and convert after the parsing. Not sure how much there is involved. And this won't work for indexes, as those are restricted to (‘s’, ‘ms’, ‘us’, and ‘ns’).

[spencerkclark]

+1 for fewer arguments to open_dataset, however that is achieved!

Indeed for those practical reasons I do not think it is worth trying to match the on-disk units of integer data any more closely. Second precision already allows for a time span of roughly +/- 290 billion years (many times older than the Earth), which I think is plenty for most applications :).

Monthly or yearly units are also somewhat awkward to deal with due to their different (albeit often violated) definition in the CF conventions.

[qq492947833]

Same question，the CMIP6 data have 2300 year dataset，but now xarray can just read data before 2262.So if you can fix this problem,it will be helpful to us.Thanks a lot!

[ChrisBarker-NOAA]

In the long term, it seems nice to have the default use the "natural" units i.e. "h" for units="hours since ..." and apparently even "M" for units=months since ... (!!)

While appealing, I think this is not a good idea. A couple points:

NOTE: I got a bit lost in the discussion, yoy all may have already come to these same conclusions, but I thought I 'd capture it here in one big post ;-)

"months" and "years" are NOT recommended by CF, as they are not clearly defined timespans. (thought UDUNITS does have a definition for them, which is the average -- e.g. 365.25 days to the year (or thereabouts...)

As for days (or even hours):

1. most of the time, these are used with a floating point type anyway.

2. if an integer type -- that does not mean that the unit is not necessarily the required / desired precision.

• One of the limitations with the CF encoding of time is that it it's inherently a continuum (to the precision used), and that doesn't change with the units.
• For instance, a user might want daily data (maybe a daily average temperature) or monthly, or ... -- but there is no way to actually express that directly with a CF time [*]. That is:
  unit: "days since 01-01-2024"
values: [0, 1, 2, 3, 4]

looks like it's it's expressing Jan 1, 2, 3, 4, ....

But what that actually means is:

01-01-2024T00:00:00
01-02-2024T00:00:00
01-03-2024T00:00:00
01-04-2024T00:00:00
...

That is, the zeroth time of each day -- i.e. a specific point in the time continuum.

And this maps to what all (that I know of) the datetime objects do too (python datetime.date notwithstanding -- and I don't think it does months)

if a user does want to have a way to express the "day", they might do:

unit: "days since 01-01-2024T12:00:00"
values: [0, 1, 2, 3, 4]

That is, noon of each day.

But then we can't use days as the unit with the fixed epoch of numpy datetime64.

Anyway, all this to say -- I don't think that there is ever a use case for using numpy datetime units longer than a second, certainly not by assuming something from the units of the time.

Using seconds as a default for any encoding of seconds or longer seems reasonable to me, though. But is there any real loss to using milliseconds?

[*] The way to express, e.g. a daily average, is to use "cell bounds", specifically defining the bounds of the average.

[kmuehlbauer]

Thanks @ChrisBarker-NOAA, I think we should move all these valuable comments in this PR into the docs somehow. I can take a look when this one is finalized.

[rabernat]

Properly supporting datetime intervals (rather than just instants) feels like it would solve so many semantic problems. We've been discussing that for years. I hope that it's now feasible post custom indexes refactor. But that's probably off topic for this thread...

[ChrisBarker-NOAA]

Properly supporting datetime intervals (rather than just instants) feels like it would solve so many semantic problem

Absolutely -- but yes, a whole other topic :-)