Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Negative date handling #127

Open
mikapfl opened this issue Sep 25, 2020 · 6 comments · May be fixed by #129
Open

Negative date handling #127

mikapfl opened this issue Sep 25, 2020 · 6 comments · May be fixed by #129
Labels
question Further information is requested

Comments

@mikapfl
Copy link
Member

mikapfl commented Sep 25, 2020

When I try to save a dataset with a cftime axis to CSV, it fails with ValueError: Ambiguous time values with time_axis = 'None'

It would be great if cftime would also just work with CSV, considering that the CSV format allows dates outside the range of pandas standard np.datetime64[ns].

I can reproduce the problem using a dataset from CMIP, I have attached a notebook which reproduces the error (in a zip to appease github). Using something with dates going back only to 1850, this works nicely.

Maybe I am also doing cftime axes wrong.

reproduce_scmdata_cftime_csv_bug.zip

@znicholls
Copy link
Collaborator

In the example, you save the dates as strings and have some of them be negative. I don't think that case is handled, I don't think any Python libraries handle negative dates in the way you want e.g.

In [1]: import datetime as dt

In [2]: dt.datetime(-1, 1, 1)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-0a3e58cf412e> in <module>
----> 1 dt.datetime(-1, 1, 1)

ValueError: year -1 is out of range

In [4]: from dateutil import parser

# this turns your negative into a positive so you end up with two timepoints being
# labelled as 0001-01-01 which causes all the issues
In [5]: parser.parse("-0001-01-01")
Out[5]: datetime.datetime(1, 1, 1, 0, 0)

If you get rid of the negative dates, it saves without an issue. Do you know of a library which can handle negative dates properly?

@znicholls znicholls added the question Further information is requested label Sep 26, 2020
@znicholls znicholls changed the title saving to CSV with crftime axis Negative date handling Sep 26, 2020
@mikapfl
Copy link
Member Author

mikapfl commented Sep 26, 2020

Hi,

ah, thanks for the investigation! There are several python libraries which are just fine with negative dates, for example, numpy does it:

In [1]: import numpy as np
In [2]: np.datetime64('-0001-01-01', 's')
Out[2]: numpy.datetime64('-001-01-01T00:00:00')

Also, xarray can use the cftime library to handle all kinds of weird dates, including negative dates and calendars with a year 0 etc.
See http://xarray.pydata.org/en/stable/weather-climate.html .

Cheers,
Mika

@znicholls
Copy link
Collaborator

Ok so the problem is here

return _ufunc_str_to_datetime(inp)
where we convert strings to datetime using the parser. This is probably another argument for swapping to an xarray backend because that gives us better time handling. Thoughts @lewisjared ?

@lewisjared
Copy link
Collaborator

Perhaps we use xarray.CFTimeIndex explicitly since generally, our dates span wider than pd.Timestamp can handle or weird years from idealised scenarios. We will still need some logic to handle decimal years which I'm not sure if xarray can use them.

@rgieseke
Copy link
Member

xclim has a xclim.core.calendar.datetime_to_decimal_year function

xclim.core.calendar.datetime_to_decimal_year(times: xarray.core.dataarray.DataArray, calendar: Optional[str] = None) → xarray.core.dataarray.DataArray[source]

Convert a datetime xr.DataArray to decimal years according to its calendar or the given one.

Decimal years are the number of years since 0001-01-01 00:00:00 AD. Ex: ‘2000-03-01 12:00’ is 2000.1653 in a standard calendar, 2000.16301 in a “noleap” or 2000.16806 in a “360_day”.

https://xclim.readthedocs.io/en/stable/api.html#calendar-handling-utilities

@lewisjared
Copy link
Collaborator

Great find. We don't want all the extra deps which come with xclim (GDAL, PROJ, etc), but the xclim.calendar might have some handy functionality https://xclim.readthedocs.io/en/stable/_modules/xclim/core/calendar.html#datetime_to_decimal_year

@lewisjared lewisjared linked a pull request Sep 29, 2020 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants