Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use TableSchema in BigQuery Column List #1293

Merged
merged 3 commits into from
Feb 13, 2025
Merged

Conversation

IanRFerguson
Copy link
Contributor

This fix patches an inefficiency in the GoogleBigQuery.get_columns_list() function ... writing a SELECT * FROM {{ TABLE }} LIMIT 1 in BigQuery isn't recommended as the entire table will be scanned in the process.

This PR updates the logic to leverage the underlying schema definition, rather than querying against the table.

@IanRFerguson
Copy link
Contributor Author

not sure why the AirMeet test isn't passing the ruff format check ... it wasn't passing locally either but appears to be now

image

Copy link

github-actions bot commented Feb 13, 2025

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  parsons/google
  google_bigquery.py 1197-1199
Project Total  

This report was generated by python-coverage-comment-action

@IanRFerguson
Copy link
Contributor Author

Removed AirMeet anyway, out of scope in this PR

Copy link
Collaborator

@austinweisgrau austinweisgrau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch

@IanRFerguson IanRFerguson merged commit cc42389 into main Feb 13, 2025
133 checks passed
@IanRFerguson IanRFerguson deleted the ian/cleanup-bq-column-list branch February 13, 2025 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants