yExporter #3

[0xBasically]

Scope

This is a budget request for maintaining all exporter-related tasks and adding new features if time permits. The exporter is a service that powers yearn.vision, calculates APYs for all our networks, serves data for those who would like to consume it, and is used to visualize current and historical vault, partner, treasury, and transaction data.

Highlights

The good

We built and integrated yETH into yearn.vision.
We have deployed Velodrome and Aerodrome apy previews to production so one can see yet to be deployed vault APYs for all these pools.
- Velodrome Previews APY
- Aerodrome Previews APY
We have updated the Yearn-Exporter-API but more optimizations on the API are needed.
Added Base network and info on yearn.vision
Added Aerodrome forward-looking APY
Made substantial progress on Bal forward-looking APY
Added Speed and Async Optimizations to Exporter in various areas included the apy "s3" script:

• fully implemented async code + dank on the main exporters, perf gains of 40-99.95% depending on exporter
• new disk caches in both ypm and eth_portfolio enable data sharing between containers and reduce load on node, further reducing node dependency. More to come.
• LogFilter class brings optimized log fetching / caching / decoding / reading / re-reading (big add tbh, for partners) as well as cleaner code
• dank now supports tenderly
• dank can now catch and handle 429 responses and now works with shitty public rpcs
• Give the optimizations above it might be possible to now run exporter on less performant nodes

The bad

Archive nodes cost will be 300$ per node per network for unlimited usage. Previously it had been 100$ per node per network for a single node for the provider we were using. They had node downtime affecting our team and data availability to both the v1 api and yearn.vision. Also we will be dropping support for Arbitrum and Fantom due to inactivity and limited budget and adding Polygon network for v3 development.

• Need to fix tests and add v3 support to exporter, we weren't able to finish this on the last budget request.

Plan

Responsibilities

• handle up status and monitoring for:

  • staging and prod environments for yearn.vision
  • staging and prod environments for yearn-exporter-api which is the base API for partners.yearn.finance
  • prod Yearn APY data
  • prod Zootroop revenue calculations
  • prod etherscan cache
  • prod sentry for exception tracing

• code maintenance for:

  • yearn-exporter
  • yearn-exporter-api
  • yMonitor
  • External python libraries
    • ypricemagic
    • dank_mids
    • eth-portfolio
    • toolcache

Tasks

• finish Balancer apys

  • Basically finished needs a few tweeks
  • Calculate forward-looking APYs like we do with curve and convex, for bal and aura strategies.
  • Calculate Balancer APY previews

• add yvaults v3 support

  • add support for networks v3 will be deploying on
  • build new v3 vault exporter
  • add v3 support to apy script & monitoring

• consolidate data layer for open data science:

  • We're using a mixture of victoria-metrics db and postgres db. A spike to replace victoria by timescale+promscale has been closed because promscale has been officially discontinued this year.
  • The current solution with victoria-metrics is stable and works but lacks flexibility for partial updates and is not easy to integrate into BI systems like metabase.
  • A new effort will be taken to create a datamodel with postgres-based timescale db which will enable us to have more control about the indexed web3 data and let us integrate it into 3rd party tools more easily. This will also help external data scientists to work with the exporter data more easily.

• grafana dashboard statistics and enhancements

  • Building a statistics and machine learning dashboard in Grafana.
    • Expand dashboard concepts and tune ML dashboards to lower timeframes.
  • Build on PoC Dash and pull in realtime data for them, calculate statistics in docker in R.
  • Enhance and update dashboards to include customized graphs with statistics and metrics each.

• fix tests

  • many older tests are failing which delays refactorings, leads to regressions and implies a lot of manual testing
  • migrate price-related tests to ypricemagic repo to reduce run time

• yearn-exporter-api updates and improvements for partners and apps

  • work with teams to optimize pulling data from the exporter API and databases.

• replace eth-brownie with ape once ape has async support

  • ape-dankmids

Deadline

2024-01-31

People

Crebsy and BobTheBuilder will take care of most of the maintenance, fixes, and if time permits, new feature additions to Exporter's codebase. 0xBasic is reviewing PRs, maintaining and improving grafana dashboards while CSdata1 works on integrating statistics and machine learning models into the Exporter. We will maintain and monitor the current APIs, the partner scripts, and data from Exporter.

Duration

This budget request is for a period of 3 months.

This budget request will let us run the exporter in "maintenance mode". This is a state where we dont add new features, unless time permits, and allows us to maintain the Exporter codebase and libraries with fixes and small improvements when needed. Thus lowering total costs while keeping everything working as normal: exporters, apis, monitoring.

Public Group

YFI Exporter - Telegram Group

Past budget request

yExporter #2

Money

Monthly breakdown: 41.67% cut for personnel

Type	Description	DAI
HOSTING	3x server	390
NODES	4x networks	1200
Personnel	3x Dev, 1x Data Analyst	13600
============	=============	========
	TOTAL	15190.00

Amount

45570 DAI

Wallet address

0xcD63C69f08bdDa7Fe96a87A2Ca3f56f3a6990a75

Reporting

Once