Simplify Socrata API script and allow multiple assets to be updated #745
+94
−116
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wrridgeway
commented
Feb 26, 2025
| @@ -36,6 +36,8 @@ def parse_years(years): | |||
| Make sure the years environmental variable is formatted correctly. | |||
| """ | |||
|
|
|||
| if years == "": | |||
There was a problem hiding this comment.
Just want to add a bit more error handling. Probably won't ever be needed.
wrridgeway
commented
Feb 26, 2025
Comment on lines
-224
to
-233
| session.get( | ||
| ( | ||
| "https://datacatalog.cookcountyil.gov/resource/" | ||
| + asset_id | ||
| + ".json?$limit=1" | ||
| ), | ||
| headers={"X-App-Token": app_token}, | ||
| ).raise_for_status() | ||
|
|
||
| session.get(url=url, headers={"X-App-Token": app_token}).raise_for_status() |
There was a problem hiding this comment.
We were checking for a bad URL status twice.
wrridgeway
commented
Feb 26, 2025
Comment on lines
+240
to
+243
| overwrite = False | ||
| print(response.content) | ||
|
|
||
| # Return the updated count so that if this function is called in a loop | ||
| # the updated count persists. | ||
| return count | ||
| overwrite = False |
There was a problem hiding this comment.
Ensuring overwrite can never be true more than once per asset.
wrridgeway
commented
Feb 26, 2025
socrata/socrata_upload.py
Outdated
Comment on lines
132
to
136
| conditional if `years` is non-empty. Many of the CCAO's open data assets are | ||
| Build a dictionary of Athena compatible SQL queries and their associated years. Many of the CCAO's open data assets are |
There was a problem hiding this comment.
This is the meat of what's changing here.
wrridgeway
changed the title
wrridgeway
commented
Feb 27, 2025
dbt/models/open_data/exposures.yml
Outdated
| @@ -199,7 +199,7 @@ exposures: | |||
| name: Data Department | |||
| meta: | |||
| test_row_count: true | |||
| asset_id: vgzx-68gb | |||
| asset_id: 3sp4-s4qg | |||
There was a problem hiding this comment.
Needs to be reverted before merging.
wrridgeway
changed the title
wrridgeway
changed the title
wrridgeway
changed the title
wrridgeway
commented
Feb 27, 2025
Comment on lines
+9
to
+21
| Comma-separated Socrata assets to update. Options are: | ||
| Appeals | | ||
| Assessed Values | | ||
| Parcel Addresses | | ||
| Parcel Proximity | | ||
| Parcel Sales | | ||
| Parcel Status | | ||
| Permits | | ||
| Property Tax-Exempt Parcels | | ||
| Parcel Universe (Historic) | | ||
| Parcel Universe (Current Year) | | ||
| Residential Condominium Unit Characteristics | | ||
| Single and Multi-Family Improvement Characteristics |
There was a problem hiding this comment.
This is gross, but I can't think of a better way to inform users of their choices since multiple choice isn't an input type and the github actions drop down menu doesn't seem to respect yaml line break formatting.
wrridgeway
commented
Feb 28, 2025
| @@ -88,6 +101,9 @@ def get_asset_info(socrata_asset): | |||
| Simple helper function to retrieve asset-specific information from dbt. | |||
| """ | |||
|
|
|||
| if not os.path.isdir("./dbt"): | |||
There was a problem hiding this comment.
Only helpful locally, unnecessary for github deployment.
wrridgeway
commented
Feb 28, 2025
| # Comma separated list of years | ||
| type: string | ||
| description: Years to update or overwrite (comma-separated) | ||
| default: 'all' |
There was a problem hiding this comment.
Removed the default and explained user's choices.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This started as trying to allow this script to accept multiple assets, but working with it was a chore and it was clear it could be simplified and improved.
This PR really just shifts a lot of the looping and year handling to the query building and upload functions and removes a lot of
overwriteparameter handling involving acountvariable by only allowingoverwriteto be true the first time it's handled.