Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate cdp changes to ingest from datastore txmeta files #333

Merged
merged 28 commits into from
Apr 22, 2024
Merged

Integrate cdp changes to ingest from datastore txmeta files #333

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
7741fb3
Integrate cdp changes to ingest from datastore txmeta files
chowbao Mar 24, 2024
8ca2dfb
lint
chowbao Mar 24, 2024
17e0f78
ci/cd error
chowbao Mar 24, 2024
f3b9c24
Add missing var
chowbao Mar 24, 2024
bec0c14
fix param name
chowbao Mar 24, 2024
9c8279b
Merge branch 'master' into cdp-integration
sydneynotthecity Apr 3, 2024
7aa3a03
Update ci.yml
sydneynotthecity Apr 3, 2024
0a53c7d
Update ci.yml
sydneynotthecity Apr 3, 2024
256f0f8
Update ci.yml
sydneynotthecity Apr 5, 2024
8550cf7
Update ci.yml
sydneynotthecity Apr 5, 2024
a32fbe2
updated gcsfuse path reference
edualvess Apr 10, 2024
7ad4da4
Updated variables to new composer 2 environment
edualvess Apr 10, 2024
0075d02
Merge branch 'master' into cdp-integration
sydneynotthecity Apr 18, 2024
68d05c6
Updates and addressing comments
chowbao Apr 19, 2024
e848dac
Merge branch 'cdp-integration' of https://github.com/stellar/stellar-…
chowbao Apr 19, 2024
cb73317
Update vars
chowbao Apr 19, 2024
dc796be
Remove old cdp_ vars
chowbao Apr 19, 2024
65de79d
Update image
chowbao Apr 22, 2024
1eebc8f
Update params
chowbao Apr 22, 2024
90d92d7
Update airflow vars
chowbao Apr 22, 2024
795f692
Update date
chowbao Apr 22, 2024
530bc01
Update dag
chowbao Apr 22, 2024
75ed8bb
Update dags
chowbao Apr 22, 2024
52a7ee5
linting
chowbao Apr 22, 2024
7526ac4
Address comments
chowbao Apr 22, 2024
da2953c
Remove xcom from dbt tasks
chowbao Apr 22, 2024
d926e9d
Update vars
chowbao Apr 22, 2024
3562758
Update image
chowbao Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions airflow_variables_dev.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
"dbt_threads": 12,
"gcs_exported_data_bucket_name": "us-central1-test-hubble-2-5f1f2dbf-bucket",
"gcs_exported_object_prefix": "dag-exported",
"image_name": "stellar/stellar-etl:395f1f8",
"image_name": "chowbao/stellar-etl:cdp-test9",
"image_output_path": "/etl/exported_data/",
"image_pull_policy": "IfNotPresent",
"kube_config_location": "",
Expand Down Expand Up @@ -274,5 +274,7 @@
"public_source_schema": "test_crypto_stellar",
"slack_elementary_channel": "stellar-elementary-alerts",
"elementary_secret": "slack-token-elementary",
"dbt_elementary_dataset": "test_elementary"
"dbt_elementary_dataset": "test_elementary",
"use_captive_core": "False",
"txmeta_datastore_url": "gcs://ledger-exporter/ledgers/testnet"
}
6 changes: 4 additions & 2 deletions airflow_variables_prod.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@
"dbt_threads": 12,
"gcs_exported_data_bucket_name": "us-central1-hubble-2-d948d67b-bucket",
"gcs_exported_object_prefix": "dag-exported",
"image_name": "stellar/stellar-etl:395f1f8",
"image_name": "chowbao/stellar-etl:cdp-test9",
"image_output_path": "/etl/exported_data/",
"image_pull_policy": "IfNotPresent",
"kube_config_location": "",
Expand Down Expand Up @@ -363,5 +363,7 @@
"public_source_schema": "crypto_stellar",
"slack_elementary_channel": "alerts-hubble-data-quality",
"elementary_secret": "slack-token-elementary",
"dbt_elementary_dataset": "elementary"
"dbt_elementary_dataset": "elementary",
"use_captive_core": "False",
"txmeta_datastore_url": "gcs://ledger-exporter/ledgers/testnet"
chowbao marked this conversation as resolved.
Show resolved Hide resolved
}
12 changes: 3 additions & 9 deletions dags/dbt_enriched_base_tables_dag.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,8 @@
)

# Wait on ingestion DAGs
# NOTE: history_archive_without_captive_core is currently disabled in
# favor of history_archive_with_captive_core_combined;
# Update with ledger exporter project integration
# wait_on_cc = build_cross_deps(
# dag, "wait_on_ledgers_txs", "history_archive_without_captive_core"
# )
wait_on_cc = build_cross_deps(
dag, "wait_on_ledgers_txs", "history_archive_with_captive_core_combined_export"
wait_on_history_table = build_cross_deps(
dag, "wait_on_ledgers_txs", "history_table_export"
)
wait_on_state_table = build_cross_deps(dag, "wait_on_state_table", "state_table_export")

Expand All @@ -42,7 +36,7 @@
elementary = elementary_task(dag, "dbt_enriched_base_tables")

# DAG task graph
wait_on_cc >> enriched_history_operations_task
wait_on_history_table >> enriched_history_operations_task

wait_on_state_table >> current_state_task

Expand Down
311 changes: 0 additions & 311 deletions dags/history_archive_with_captive_core_combined_export_dag.py
View file
Open in desktop

This file was deleted.

Loading
Loading