Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet Sink for substreams #132

Merged
merged 48 commits into from
Aug 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
a344eeb
Adding code for parquet sink
Feb 27, 2023
9b59a7b
Parquet sink now working! Tested on EnsNames spkg and it works!
Mar 13, 2023
f4be8fe
Hooking up parquet sink to messari cli + big refactor
Mar 21, 2023
db256c9
Getting rid of extra parquet folder in output filepath
Mar 21, 2023
f9c1309
ewfiwn
Mar 23, 2023
6cc2c0f
Adding proper enum support. Also improved test setup to start generat…
Mar 24, 2023
0891251
Adding test data derive
Mar 24, 2023
b6c4362
Cleanup
Mar 24, 2023
f05e890
More cleanup
Mar 24, 2023
98da755
Delete .DS_Store
robinbernon Mar 24, 2023
8c54da7
Delete .DS_Store
robinbernon Mar 24, 2023
a7f6c10
Delete .DS_Store
robinbernon Mar 24, 2023
3e8ce50
Delete .DS_Store
robinbernon Mar 24, 2023
e4a06f6
Delete .DS_Store
robinbernon Mar 24, 2023
1d493d9
Delete .DS_Store
robinbernon Mar 24, 2023
4aea18e
Add support for multiple output files in sink, and also added general…
Mar 24, 2023
25ce07e
Adding oneof support and adding initial macro code for easy end to en…
Apr 21, 2023
b5e856b
Adding all the code necessary for setting up the pipeline for substre…
Apr 25, 2023
817dff5
Adding cmds to support config file accompanyment for pipeline spkg pr…
May 2, 2023
139f6a2
Unit test with derive macro now working end to end
May 9, 2023
614236f
Adding versioning to pipeline deployments
May 22, 2023
fea793a
Fixing internal type declaration issues with protos and also head blo…
May 23, 2023
a2a6142
Reducing spliting requirements by droping need to name field as "items"
May 23, 2023
e0a0c86
Last fix for getting synthetix substreams working
May 24, 2023
7f40ae7
Adding enum support to sink testing framework
May 29, 2023
f5857ea
Adding escrow reward and token balance tests for parquet sink
May 30, 2023
4f38b0d
Adding optional and repeated types to sink
Jun 3, 2023
8a294eb
Fix for option and repeated tests with extension for packed type testing
Jun 5, 2023
af8ec64
Fixing column writer issue for more than one file
Jun 6, 2023
c71cf3b
Adding oneof field support for sink
Jun 11, 2023
cd095a5
Getting rid of most build warnings
Jun 11, 2023
9fefdd8
Fixing install-cli issue
Jun 11, 2023
62f0a5c
Fixing syn crate conflict issue
Jun 11, 2023
093de71
First draft for substreams processing to DWH with dagster
Jun 15, 2023
f930eca
Refactoring
Jun 20, 2023
651332d
Fix for getting correct latest block from block output file
Jun 20, 2023
4fe3ae4
Small fix
Jun 20, 2023
021585e
Adding debugging printlns for columns and reducing the filesize
Jun 21, 2023
aeb7629
Switching to v2 endpoint
Jun 22, 2023
a303725
Clearing def and rep lvls after writing to file
Jun 22, 2023
2db34e4
Fixing optional split message issue
Jun 22, 2023
b9d968f
Fixing start block retreival when no files in s3. Also add AWAITING_U…
Jun 22, 2023
41eb496
minor fixes
jaimehgb Jun 26, 2023
09f7847
Cleanup unnecessary files & formatting
jaimehgb Jul 7, 2023
19d17bb
Merge branch 'master' into parquet
jaimehgb Jul 7, 2023
88069ac
Pass bucket as arg
jaimehgb Jul 31, 2023
ba36806
Merge branch 'master' into parquet
jaimehgb Jul 31, 2023
fd2524d
Merge branch 'master' into parquet
this-username-is-taken Aug 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ jobs:
~/.cargo/git/db/
target/
key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
- name: install-cli
run: make install-cli
- name: build all
run: |
chmod a+x /opt/hostedtoolcache/streamingfast/substreams/v1.0.1/linux-x64/substreams
Expand Down
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,17 @@ test:
.PHONY: install-cli
install-cli:
cargo install --path ./messari-cli

.PHONY: upload-cli-for-dagster
upload-cli-for-dagster:
$(MAKE) -C messari-cli build-dagster
messari upload-cli-to-aws ./messari-cli/dagster-cli/messari

.PHONY: pack-all
pack-all:
$(MAKE) -C erc20-price pack

.PHONY: upload-all
upload-all:
$(MAKE) -C erc20-price upload

1 change: 1 addition & 0 deletions messari-cli/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
target
Loading