Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor daily_euro_ohlc DAG #553

Open
sydneynotthecity opened this issue Jan 2, 2025 · 0 comments
Open

Refactor daily_euro_ohlc DAG #553

sydneynotthecity opened this issue Jan 2, 2025 · 0 comments

Comments

@sydneynotthecity
Copy link
Contributor

This DAG pulls euro ohlc pricing information from coingecko (it’s configurable but in general this is what its purpose is). There are currently cloud functions that do the same thing. These should be consolidated.

The preference is to move all these into cloud functions as there are a few bugs/inefficiencies with daily_euro_ohlc_dag

daily_euro_ohlc_dag:

  • Doesn’t deduplicate so there are multiples of the exact same timestamp and ohlc data
  • Uploads to GCS then uploads to BQ whereas the cloud functions directly upload to BQ
  • Does not use our API key to get around rate limiting from the given website (TBD: need to check if this is true)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant