Skip to content

Commit

Permalink
Fix crash during normalization of time zones
Browse files Browse the repository at this point in the history
Add another chak for the tests

Address PR comments and add more test cases

Fix edge case on Windows

Fix broken test on windows

Recreate the timestamp with timezone

Coerce all timezone to datetime.timezone

Add metadata to compat tests

Add meta to compat tests

Add a check for the buff len from denormalizing timestamps

Add a note in the docs about using the time zone info to create other time zone types

Add comment about the old PD_TIMESTAMP normalization

Handle case when we have a valid tz from a past version

Fix tz in meta in backwards compat test

Fix install of old protobuf version

Remove outdated assert

Add 0 offset case

Add write of meta data

Fix interop test

Add metadata to append

Rebase

Fix

Test with string only

Fix comparison of time zone information in test_normalization.py

Normalize time zone information in _normalization.py

Normalize time zone information in _normalization.py and fix crash during normalization of time zones

Fix some details

Don;t create libs with v2 encoding

Update test cases
  • Loading branch information
G-D-Petrov committed Oct 2, 2024
1 parent aa1eba3 commit b117618
Show file tree
Hide file tree
Showing 7 changed files with 560 additions and 138 deletions.
13 changes: 13 additions & 0 deletions docs/mkdocs/docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,16 @@ The handling of `NaN` in ArcticDB depends on the type of the column under consid
* For string columns, `NaN`, as well as Python `None`, are fully supported.
* For floating-point numeric columns, `NaN` is also fully supported.
* For integer numeric columns `NaN` is not supported. A column that otherwise contains only integers will be treated as a floating point column if a `NaN` is encountered by ArcticDB, at which point [the usual rules](api/arctic.md#LibraryOptions) around type promotion for libraries configured with or without dynamic schema all apply as usual.

### How does ArcticDB handle `time zone` information?

ArcticDB takes a more strict approach to time zone handling than Pandas. While Pandas support multiple types for storing time zone information, ArcticDB coerces all time zone information to `datetime.timezone`. This way we can ensure that all time zone information is stored in a consistent manner, and that all time zone information is preserved when data is written to and read from ArcticDB.

When writing data with time zone information to ArcticDB, we preserve the time zone offset and name. When reading data with time zone information from ArcticDB, this data is used to create a `datetime.timezone` object.

!!! note

This means that regardles of the timezone types being written in ArcticDB, all time zone types are normalised to `datetime.timezone`.
If you would like a another time-zone type back then tzname can be used to recreate it:
- For `pytz` you can use: `pytz.gettz(dt.tzname())`
- For `ZoneInfo`, after Python 3.9+, you can use: `ZoneInfo(dt.tzname())`
Loading

0 comments on commit b117618

Please sign in to comment.