Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexers v2 #132

Merged
merged 37 commits into from
Nov 11, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
c56719d
Create indexers-v2 docker baseline
marcus-snx Oct 23, 2024
5efd639
Fix package.json name
marcus-snx Oct 23, 2024
5ef68ea
Create separate services for indexers
marcus-snx Oct 23, 2024
1103017
Remove service
marcus-snx Oct 23, 2024
4811eed
Add cannon config for base
marcus-snx Oct 23, 2024
8676c18
Fix docker exiting when no migrations
marcus-snx Oct 23, 2024
7907660
Make services explicit for indexers-v2
marcus-snx Oct 24, 2024
412b0d5
Refactor logic
marcus-snx Oct 25, 2024
d9078aa
Handle Exceptions
marcus-snx Oct 29, 2024
ca9d08d
Fix arbitrum mainnet network config
marcus-snx Oct 29, 2024
7275dc9
Consolidate contract data
marcus-snx Oct 29, 2024
cde7a83
Fix sdk version
marcus-snx Oct 29, 2024
3623f3c
Create entrypoint bash script
marcus-snx Oct 29, 2024
3136887
Add block range
marcus-snx Oct 29, 2024
a786170
Fix entrypoint script
marcus-snx Oct 29, 2024
226f8a7
Update network config files
marcus-snx Oct 29, 2024
992ea2b
Use sdk logger
marcus-snx Oct 31, 2024
91ab6cf
Improve arg and config parsing
marcus-snx Nov 5, 2024
b7bb3ca
Add rate limit config
marcus-snx Nov 5, 2024
ea8f205
Fix base mainnet config file
marcus-snx Nov 5, 2024
7889047
Add config for base mainnet parquet
marcus-snx Nov 5, 2024
2976d5a
Set default config
marcus-snx Nov 5, 2024
69d2ec6
Fix packages
marcus-snx Nov 5, 2024
978b140
Add env var to entrypoint
marcus-snx Nov 5, 2024
981affb
Add custom config to indexer service
marcus-snx Nov 5, 2024
ef9fb76
Remove migration deps in squid commands
marcus-snx Nov 5, 2024
b18a47e
Add parquet export support
marcus-snx Nov 6, 2024
d3eea6f
Fix patches paths in Dockerfile
marcus-snx Nov 6, 2024
fe8a67c
Remove required for config_name arg
marcus-snx Nov 6, 2024
2eeaa5d
Patch gen tools for no archive url
marcus-snx Nov 6, 2024
1bd0517
Fix blockrange
marcus-snx Nov 7, 2024
5f1190a
Update indexers-v2/requirements.txt
Tburm Nov 8, 2024
3646184
Remove contract renaming
marcus-snx Nov 11, 2024
fbe4130
Remove postgres target
marcus-snx Nov 11, 2024
b0dd1d3
Merge branch 'feat/indexers-v2' of github.com:Synthetixio/data into f…
marcus-snx Nov 11, 2024
2f4b52d
Remove deprecated syncIntervalBlocks
marcus-snx Nov 11, 2024
bbeef8a
Merge branch 'main' into feat/indexers-v2
marcus-snx Nov 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 29 additions & 16 deletions indexers-v2/main.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
import json
import sys
import os
import argparse
import yaml
import logging
from synthetix import Synthetix

logging.basicConfig(level=logging.INFO)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can simplify a bit by leaving this out, and using snx.logger.info


def save_abi(abi, contract_name):
os.makedirs("abi", exist_ok=True)
with open(f"abi/{contract_name}.json", "w") as file:
json.dump(abi, file, indent=2)


def create_squidgen_config(rpc_url, archive_url, contracts_info):
def create_squidgen_config(rpc_url, archive_url, contracts_info, block_range):
config = {
"archive": archive_url,
"finalityConfirmation": 1,
Expand All @@ -27,6 +29,7 @@ def create_squidgen_config(rpc_url, archive_url, contracts_info):
contract_config = {
"name": name,
"address": address,
"range": block_range,
"abi": f"./abi/{name}.json",
"events": True,
"functions": False,
Expand Down Expand Up @@ -99,21 +102,29 @@ def load_network_config(path):

contracts = []

if "cannon_config" in network_config:
snx = Synthetix(
provider_rpc=rpc_endpoint,
network_id=network_config["network_id"],
cannon_config=network_config["cannon_config"],
)
else:
snx = Synthetix(
provider_rpc=rpc_endpoint,
network_id=network_config["network_id"],
)

block_range = {}
block_range["from"] = network_config["range"].get("from", 0)
if "to" in network_config["range"]:
if network_config["range"]["to"] == "latest":
block_range["to"] = snx.web3.eth.block_number
else:
block_range["to"] = network_config["range"]["to"]

if "contracts_from_sdk" in network_config:
contracts_from_sdk = network_config["contracts_from_sdk"]

if "cannon_config" in network_config:
snx = Synthetix(
provider_rpc=rpc_endpoint,
network_id=network_config["network_id"],
cannon_config=network_config["cannon_config"],
)
else:
snx = Synthetix(
provider_rpc=rpc_endpoint,
network_id=network_config["network_id"],
)

for contract in contracts_from_sdk:
name = contract["name"]
package = contract["package"]
Expand All @@ -133,12 +144,14 @@ def load_network_config(path):
save_abi(contract_data["abi"], name)
contracts.append({"name": name, "address": contract_data["address"]})

squidgen_config = create_squidgen_config(rpc_endpoint, archive_url, contracts)
squidgen_config = create_squidgen_config(
rpc_endpoint, archive_url, contracts, block_range
)
write_yaml(squidgen_config, "squidgen.yaml")

squid_config = create_squid_config(args.network_name)
write_yaml(squid_config, "squid.yaml")

print(
logging.info(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use snx.logger.info

f"squidgen.yaml, squid.yaml, and ABI files have been generated for {args.network_name}"
)
2 changes: 2 additions & 0 deletions indexers-v2/networks/arbitrum_mainnet/network_config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
arbitrum_mainnet:
archive_url: "https://v2.archive.subsquid.io/network/arbitrum-one"
range:
from: 218000000
contracts_from_sdk:
- name: CoreProxy
package: system
Expand Down
2 changes: 2 additions & 0 deletions indexers-v2/networks/base_mainnet/network_config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
base_mainnet:
archive_url: "https://v2.archive.subsquid.io/network/base-mainnet"
range:
from: 7500000
contracts_from_sdk:
- name: CoreProxy
package: system
Expand Down