make epochs table instead of incremental

[andrewhong5297]

Thank you for contributing to Spellbook 🪄

Update!

Please build spells in the proper subproject directory. For more information, please see the main readme, which also links to a GH discussion with the option to ask questions.

Contribution type

Please check the type of contribution this pull request is for:

• New spell(s)
• Adding to existing spell lineage
• Bug fix

I'm making solana_utils.epoch a table to refresh fully each time versus incremental, because the incremental is messing with the row_number() to get first_block_epoch. because each time it runs, it will add a new first_block_epoch window function ordering which then messes up the query. Open to creative solutions here too if anyone thinks of one, too late in the night for me to find one haha

[andrewhong5297]

seems to be some solution to using incremental tables with window function columns but would make this sql maybe 100-200 lines longer lol. it's a cheap query so maybe we just do the full refresh each time. I'm sure we've run into this in the past with other models in here.

https://butterflymx.com/blog/dbt-incremental-models/

[0xRobin]

@andrewhong5297 a creative solution here is to manually look back more then the standard incremental window. So that the first result in the incremental window still has all the necessary rows to completely fill it's window.

So if an epoch here is ~2 days, we need to look back 1d (incr window) + 2d (epoch duration) on blocks, let's do 4d just to be safe.
So on solana.blocks add:

{% if is_incremental() %}
        WHERE time > now() - interval '4' day
{% endif %}

And on the final select we still add:

{% if is_incremental() %}
WHERE {{incremental_predicate('block_time')}}
{% endif %}

So we only merge the results in the standard incremental window.

[andrewhong5297]

But then wouldn't it overwrite the past epoch it runs into? i.e it might get the current epoch right but then mess up the past epoch.

[andrewhong5297]

Like basically using a day filter will always end up in the middle of one epoch or another. but also we still need to add the slots in the middle of an epoch, and update them too.

[jeff-dude]

lets merge for sake of time, but i need to revisit existing spells to find if we had a good solution to window functions / incremental. building an example one could be useful too.

[andrewhong5297]

agree, worth revisiting in the future as this is a simple table we can use as an example