Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix singer parsing for columns with dtype object #27

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

fix singer parsing for columns with dtype object #27

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8484456
fix singer parsing for columns with dtype object
keyn4 Feb 16, 2024
05f4d63
Merge branch 'master' into HGI-5304
keyn4 Sep 17, 2024
6c6f37e
not parse numbers, add flag to parse cols in get
keyn4 Sep 17, 2024
4a3ee82
bump version
keyn4 Sep 17, 2024
8bc055c
parse cols when reading parquet files
keyn4 Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion gluestick/etl_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,9 @@ def get(self, stream, default=None, catalog_types=False, **kwargs):
if catalog and catalog_types:
types_params = self.get_types_from_catalog(catalog, stream)
kwargs.update(types_params)
return pd.read_csv(filepath, **kwargs)
df = pd.read_csv(filepath, **kwargs)
df = parse_object_cols(df)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's flag this logic (add a param like parse_objects=False)

return df

def get_metadata(self, stream):
"""Get metadata from parquet file."""
Expand Down Expand Up @@ -650,3 +652,14 @@ def localize_datetime(df, column_name):
df[column_name] = df[column_name].dt.tz_convert('UTC')

return df[column_name]


def parse_object_cols(df):
object_columns = [column for column in df.columns if df[column].dtype == 'object']
for col in object_columns:
try:
df[col] = df[col].apply(lambda x: parse_objs(x))
except:
continue
return df

Loading