Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix time zone handling in index values and metadata #1866

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Fix time zone handling in index values and metadata #1866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b117618
Fix crash during normalization of time zones
G-D-Petrov Mar 8, 2024
b5776b5
Update the docs
G-D-Petrov Oct 2, 2024
e1c3887
Bring back a backwards compat check
G-D-Petrov Oct 2, 2024
8d03e9f
Normalize time zones for DatetimeIndex in _PandasNormalizer
G-D-Petrov Oct 2, 2024
e33b93a
- Update the documentation to clarify time zone handling in ArcticDB
G-D-Petrov Oct 2, 2024
fb8baed
Normalize time zones for DatetimeIndex in _PandasNormalizer
G-D-Petrov Oct 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/mkdocs/docs/faq.md
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't repro the crash in issue #1406 anymore

Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,15 @@ The handling of `NaN` in ArcticDB depends on the type of the column under consid
* For string columns, `NaN`, as well as Python `None`, are fully supported.
* For floating-point numeric columns, `NaN` is also fully supported.
* For integer numeric columns `NaN` is not supported. A column that otherwise contains only integers will be treated as a floating point column if a `NaN` is encountered by ArcticDB, at which point [the usual rules](api/arctic.md#LibraryOptions) around type promotion for libraries configured with or without dynamic schema all apply as usual.

### How does ArcticDB handle `time zone` information?

ArcticDB takes a more strict approach to time zone handling than Pandas. While Pandas support multiple types for storing time zone information, ArcticDB coerces all time zone information to `pytz.tzfile` as this is the default representation for Pandas as well. This way we can ensure that all time zone information is stored in a consistent manner, and that all time zone information is preserved when data is written to and read from ArcticDB.
Copy link
Collaborator

@poodlewars poodlewars Oct 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is pytz actually the default in Pandas? https://github.com/pandas-dev/pandas/blob/main/pyproject.toml#L33

It seems to have changed a lot in Pandas 2 https://github.com/pandas-dev/pandas/blob/2a10e04a099d5f1633abcdfbb2dd9fdf09142f8d/doc/source/whatsnew/v2.0.0.rst#L555


When writing data with time zone information to ArcticDB, we preserve the time name as string (e.g. "Europe/Amsterdam"). When reading data with time zone information from ArcticDB, this data is used to create a `pytz.tzfile` object.
Copy link
Collaborator

@poodlewars poodlewars Oct 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if the writer uses a timezone file that is not in the reader's zone database? How does this work across operating systems? I assume the database files are quite different across Linux and Windows, there are quite complex mappings between them. I think this makes it fundamentally impossible to round-trip timezones correctly - at least without relying on some standard database and validating that what is written is in there 🤔

Ah I see - this is all handled by pytz. It would be interesting to know how this stuff behaves for zones that are not in the Olson database. It would also be interesting to know how this behaves if the reader has an older pytz and the timezone is not in their database.


!!! note

This means that regardles of the timezone types being written in ArcticDB, all time zone types are normalised to `pytz.tzfile`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's important that we round-trip correctly - this could be quite a disruptive change otherwise. At least requiring a new major version - unless I'm misunderstanding? I notice the PR description says that this keeps the interface the same.

If you would like a another time-zone type back then string representation can be used to recreate it, e.g. `str(dt.tzinfo)`:
- For `ZoneInfo`, after Python 3.9+, you can use: `ZoneInfo(str(dt.tzinfo))`
Loading
Loading