-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add SQLite mixed-type data handling capabilities #50
Conversation
+ preset overrides
move from logging.warn to logging.warning (avoid deprecated calls)
Co-authored-by: Gregory Way <[email protected]>
Co-Authored-By: Gregory Way <[email protected]>
Co-Authored-By: Faisal Alquaddoomi <[email protected]>
Co-Authored-By: Faisal Alquaddoomi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good; I left a couple of comments. Cool to see you making the tool more robust.
Co-authored-by: Faisal Alquaddoomi <[email protected]>
…s/CytoTable into sqlite-mixed-types-handling
@falquaddoomi @d33bs - is this ready to merge? Will it solve #38? (@jenna-tomkinson and @MattsonCam are currently blocked by it for the JUMP single cell analysis) |
Update, @d33bs - just saw your comment in #38 (comment) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Excited to get this merged in.
Description
This change seeks to address #38 through dynamic handling of mixed-type SQLite column data when exceptions of this kind are encountered with DuckDB SQLite queries.
When mixed type exceptions are encountered, the design intent here strives to keep as much data as possible while assuming one or many values of mismatching type within the SQLite table being queried. SQLite3 SQL case statements are formed to convert mismatched storage class values to NULL's for every column in the table being queried. These SQLite NULL's are interpreted by
sqlite3
as PythonNone
's, which are then interpreted aspyarrow.null()
's for export to Parquet.For SQLite data sources where mixed typing exceptions are not encountered, the default capabilities through DuckDB function as usual.
What is the nature of your change?
Checklist
Please ensure that all boxes are checked before indicating that a pull request is ready for review.