Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Mapping source column names to parquet-compatible destination column names #122

Open
RogerNoble opened this issue Dec 24, 2024 · 0 comments

Comments

@RogerNoble
Copy link

In the current ingestion pipeline of the CF.Cumulus project, data sources such as SQL Server allow column names that contain spaces. However, the Parquet destination does not support column names with spaces.

Proposed Solution:
Introduce a new column in the [ingest].[Attributes] table to allow users to specify a destination column name for each source column. This enhancement would function as follows:

  • Add a new column, e.g., AttributeTargetName, to the ingest metadata table that tracks column mappings.
  • Allow users to define AttributeTargetName for each source column. This field can be populated with a Parquet-compatible name (e.g., replacing spaces with underscores or removing spaces entirely).
  • If AttributeTargetName is left blank or null, the system will default to using the original source column name, ensuring backward compatibility and making this a non-breaking change.
  • Modify the [ingest].[GetDatasetPayload] stored procedure to generate the required TabularTranslator json expression to reference AttributeTargetName when writing to Parquet files. If not specified / NULL, the AttributeName column will be used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant