Initial implementations of the TPU Vortexor

[lijunwangs]

Problem

Framework for Solana TPU vortexor which allows core validator to offload TPU receiving, sigverify and deduplications.

Summary of Changes

This is the first installment of the change. This is the framework part. It starts the TPU and forward stream based on the CLI configurations. In future PRs we will have the sigverify, forwarder, subscription management. The code has no impact to the existing validator.

Added a unit test for starting the vortexor, send some transactions and stops it.

Fixes #

[lijunwangs]

This is moved from nonblocking::quic::test to here to be reused for other testing

[behzadnouri]

Is there a design document or anything for this?
like what is the motivation and end goal here? how is the validator going to talk with this new binary? how does the binary obtain stakes to do stake-based QoS etc? does the binary need access to the private key of the validator? etc ...

[lijunwangs]

Is there a design document or anything for this? like what is the motivation and end goal here? how is the validator going to talk with this new binary? how does the binary obtain stakes to do stake-based QoS etc? does the binary need access to the private key of the validator? etc ...

The motivation is to make TPU streaming, sig verification more scalable. The readme captures this: https://raw.githubusercontent.com/anza-xyz/agave/11908fbbcfab7194f954177c2da0646249ee308a/vortexor/Readme.md. I have set up a review meeting to go over this to give more context. The vortexor will retrieve the stake map via regular RPC service from configured validator -- the RPC can be either pairing validator or any other trusted RPC validator in the network supporting the RPC pub/sub service. The Vortexor does not need the private key to the validator.

[behzadnouri]

The Vortexor does not need the private key to the validator.

don't we need the private key for QUIC handshake ?

[lijunwangs]

The Vortexor does not need the private key to the validator.

don't we need the private key for QUIC handshake ?

A private key is used to sign the TLS certificate. But the client does not care about if the key is the same as the one of the validator advertising the TPU address (The vortexor address). The client code does not check that and we have no use case to enforce that check -- this is the same currently using --public-tpu-address. Without requiring using the same private key making it more secure and simplify deployment.

[behzadnouri]

A private key is used to sign the TLS certificate. But the client does not care about if the key is the same as the one of the validator advertising the TPU address (The vortexor address). The client code does not check that and we have no use case to enforce that check

How does the client prune its connection cache when it becomes too big?
Isn't that by pubkey => stake mapping? and isn't the pubkey from QUIC handshake?

[lijunwangs]

A private key is used to sign the TLS certificate. But the client does not care about if the key is the same as the one of the validator advertising the TPU address (The vortexor address). The client code does not check that and we have no use case to enforce that check

How does the client prune its connection cache when it becomes too big? Isn't that by pubkey => stake mapping? and isn't the pubkey from QUIC handshake?

We randomly select a connection to prune -- the current connection cache is agnostic of the stake map.

[lijunwangs]

@pgarg66 @behzadnouri @bw-solana @sakridge -- can you follow up on this PR request? I have updated the readme.md with some of feedback I got in the table top review. Thanks

[pgarg66]

Hey @lijunwangs, could you post the comments/summary from the table-top review? It'll be good to have them captured here in this PR.

Also, there were some action items based on the review. It'll be great to have tracking issues for those, so that reviewers know what's coming down the line for Vortexor.

[bw-solana]

I don't have a ton of feedback on the code for now, but a few comments (most of which are just reiterating what came up during live discussion):

• I think it would be good to map out how Jito relayers and DZ networks might interface with the vortexor
• It would be good to call out if vortexor and core validator can be run on the same box
• It would be good to provide some details on how operators can upgrade from current validator to vortexor + core validator configuration.
• It would be good to specify details on how core validator ingest might work. Is this connection provided all of the bandwidth by default? Can core validator also ingest from other sources? How is throttling handled when dealing with multiple sources of ingress (e.g. multiple vortexors and public TPU connections)?

[lijunwangs]

I don't have a ton of feedback on the code for now, but a few comments (most of which are just reiterating what came up during live discussion):

• I think it would be good to map out how Jito relayers and DZ networks might interface with the vortexor
• It would be good to call out if vortexor and core validator can be run on the same box
• It would be good to provide some details on how operators can upgrade from current validator to vortexor + core validator configuration.
• It would be good to specify details on how core validator ingest might work. Is this connection provided all of the bandwidth by default? Can core validator also ingest from other sources? How is throttling handled when dealing with multiple sources of ingress (e.g. multiple vortexors and public TPU connections)?

Thanks for the comments. Updated the Readme.md with this.

[lijunwangs]

Hey @lijunwangs, could you post the comments/summary from the table-top review? It'll be good to have them captured here in this PR.

Also, there were some action items based on the review. It'll be great to have tracking issues for those, so that reviewers know what's coming down the line for Vortexor.

I have put the discussion point we had into the Readme.md -- as I think it is a design questions. I have created a project to track these efforts:

https://github.com/orgs/anza-xyz/projects/15/views/1

[pgarg66]

Some suggestions/comments.

In its current form it looks like a wrapper on top of part of existing QUIC and TPU functionality. The follow on PRs would add more specifics for Vortexor.

It looks in the right direction to me, but I would like another person to review it as well.