Skip to content

Commit

Permalink
Addressed some feedback from Pankaj and Brennan
Browse files Browse the repository at this point in the history
  • Loading branch information
@lijunwangs
lijunwangs committed Dec 19, 2024
1 parent 441088f commit a4b63f0
Show file tree
Hide file tree
Showing 5 changed files with 114 additions and 100 deletions.
81 changes: 54 additions & 27 deletions vortexor/Readme.md
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,47 @@
# Introduction
The vortexor is a service which can be used to offload receiving transaction
from the public, doing signature verifications and deduplications from the core
validator which can focus on processing and executing the transactions. The
filtered transactions can be forwarded to the validators linked with the
vortexor. This vortexor makes the TPU transaction ingestion more scalable
compared to single node solution.
from the public, performing signature verifications and deduplications from the
core validator enabling it to focus on processing and executing the
transactions. The verified and filtered transactions then will be forwarded to
the validators linked with the vortexor. This vortexor makes the TPU transaction
ingestion and verification more scalable compared with the single node solution.

The archietecture diagram of the Vorexor with the relationship to the validator.
# Archietecure
Figure 1 describes the archietecture diagram of the Vorexor with the
relationship to the validator.

+---------------------+
| Solana |
| RPC / Web Socket |
| Service |
+---------------------+
|
v
+--------------------- VORTEXOR ------------------------+
| |
| | |
| +------------------+ |
| | StakedKeyUpdater | |
| +------------------+ |
| | |
| v |
| +-------------+ +--------------------+ |
TPU --> | | TPU Streamer| -----> | SigVerifier/Dedup | |
/QUIC | +-------------+ +--------------------+ |
| | | |
| v v |
| | | |
| v v |
| +----------------+ +------------------------+ |
| | Subscription |<----| VerifiedPacketForwarder| |
| | Management | +------------------------+ |
| +----------------+ | |
+--------------------------------|----------------------+
^ |
heart beat/subscriptions | v
^ | (UDP/QUIC)
heart beat/subscriptions | |
| v
+-------------------- AGAVE VALIDATOR ------------------+
| |
| +----------------+ +-----------------------+ |
Validator Config-> | | Subscription | | VerifiedPacketReceiver| |
Admin RPC | | Management | | | |
Config-> | | Subscription | | VerifiedPacketReceiver| |
Admin RPC | | Management | | | |
| +----------------+ +-----------------------+ |
| | | |
| | v |
Expand All @@ -35,6 +51,7 @@ Admin RPC | | Management | | | |
| +--------------------+ +-----------+ |
+-------------------------------------------------------+

Figure 1.

The Vorexor is a new executable which can be deployed on to different nodes from
the core Agave validator. It can also be deployed onto the same node as the core
Expand All @@ -49,27 +66,22 @@ It has the following major components:
from the validator. Subscriptions action include subscription for
transactions and cancel subscriptions.
4. VerifiedPacketForwarder -- This is responsible for forwarding the verified
transaction packetsto the subscribed validators. We target use UDP/QUIC to
transaction packets to the subscribed validators. We target use UDP/QUIC to
send transactions to the validators. The validator has option to bind to
private address for receiving the verified packets.
The validators can also use firewall rules to allow transactions only from
the vortexor.
the chosen vortexor.
5. The Vortexor StakedKeyUpdater -- this service is responsible for retrieving
the stake map from the network and make it available to the TPU streamer
so that it can apply SWQOS.
so that it can apply stake-weighted QOS.

In the validator, there is new component which receives the verified packets
sent from the vortexor which directly sends the packets to the banking stage.
The validator's Admin RPC is enhanced to configure the peering vortexor. The
ContactInfo of the validator is updated with the address of the vortexor when it
is linked with the validator. There is periodic heartbeat messages sent from the
vortexor to the validator. If there are not transactions sent and no heartbeat
messages from the vortexor within configurable timeout window, the validator may
decide the vortexor is dead or disconnected it may choose to use another
vortexor or use its own native QUI TPU streamer by updating the ContactInfo
about TPU address.

Relationship of Validator and Vortexor
is linked with the validator.

# Relationship of Validator and Vortexor
The validator always broadcast one TPU address which will be served by a
vortexor. A validator can change its pairing vortexor to another. A vortexor
based on its performance can serve 1 or more validators. The architecture
Expand All @@ -78,15 +90,19 @@ to serve a validator to make the solution more scalable -- see blow.

Load Balancer
|
v
__________________________
| | |
| | |
Vortexor Vortexor Vortexor
| | |
| | |
__________________________
|
v
Validator

Figure 2.

When the validator is in the 'Paired' mode which it is either getting active
transactions from the corresponding vortexor or receiving heartbeat messages,
Expand All @@ -103,11 +119,18 @@ validator can still set the QOS against the vortexor. And it is expected the
validator will grant much higher bandwidth to the vortexor compared with regular
TPU clients in a regular validator's TPU configurations.

Deployment Considerations
There is periodic heartbeat messages sent from the vortexor to the validator.
If there are not transactions sent and no heartbeat messages from the vortexor
within configurable timeout window, the validator may decide the vortexor is
dead or disconnected it may choose to use another vortexor or use its own built-
in QUIC-based TPU streamer by updating the ContactInfo about its TPU address.

# Deployment Considerations
The vortexor while making the validator more scalable in handling transactions,
does have its drawbacks:

1. It increases the deployment complexity. By default, for validators which
do not use vortexors, there is no deployment impact and functional/performance
do not use vortexors, there is no deployment and functional/performance
impacts. For validators using vortexors, it requires addtional deployment task
to deploy the vortexor. For performance considerations, it is most likely the
vortexor will be running in a seperate node from its pairing validator. To
Expand All @@ -120,12 +143,14 @@ between the vortexor and the validator, the validator auto fallback to its
built-in streamer. In addition, the validator and the vortexor will support the
additional admin RPC to query the vortexor pairing states and mange pairing
relationship.

2. There is an extra hop from the original clients sending the transaction to
the leader validator. This is a trade-off between scalability and latency. The
latency can be minimized as the vortexor is expected to run on a node which
is on the private network with low latency to the validator. With this
consideration, the solution also supports pure UDP to forward transactions
from the vortexor to the validator.

3. The security implications, there is an implict trust relationship between
the validator and the vortexor. It is expected that the vortexor and the
validator to be running in the same private network. In addition, firewall rules
Expand All @@ -135,18 +160,20 @@ In the QUIC we can have rule to limit the connections from the known pubkey.
Finally, there can be an option to enforce the transaction to go through
sigverifications in the validator (default will be off due to its added
computing cost -- double verifications).

4. There is already some solution like jito-relayer being used by validators.
The solutions will be compatibile with jito-relayers in that it should not
impact validators already using jito validator and relayers. Also, we will keep
the arguments of vortexor CLI close to jito-relayer's CLI as possible to reduce
surprises for validators migrating to using vortexor from the jito-relayer.

5. The vortexor's networking setup. In the simplest format, The vortexor's
TPU and TPU forward port might be directly accessible to the internet. Or it
might be put behind the load balancer for security and performance.
The vortexor is encouraged to communicate with the validator using a private
network for performance and security considerations.

Uprade Considerations
# Uprade Considerations
It is up to the operators to decide if to adopt vortexors. Operators can either
use the vortexor or not use it without concerns if the rest of the network's
decision as the vorexor itself does not change the protocol of the network. When
Expand Down
30 changes: 24 additions & 6 deletions vortexor/src/cli.rs
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
use {
crate::vortexor::{MAX_QUIC_CONNECTIONS_PER_PEER, NUM_QUIC_ENDPOINTS},
clap::{crate_description, crate_name, App, AppSettings, Arg},
solana_clap_utils::input_validators::{is_keypair_or_ask_keyword, is_parsable},
solana_net_utils::{MINIMUM_VALIDATOR_PORT_RANGE_WIDTH, VALIDATOR_PORT_RANGE},
Expand All @@ -12,6 +11,9 @@ use {
},
};

pub const DEFAULT_MAX_QUIC_CONNECTIONS_PER_PEER: usize = 8;
pub const DEFAULT_NUM_QUIC_ENDPOINTS: usize = 8;

pub struct DefaultArgs {
pub bind_address: String,
pub dynamic_port_range: String,
Expand All @@ -30,7 +32,7 @@ impl Default for DefaultArgs {
Self {
bind_address: "0.0.0.0".to_string(),
dynamic_port_range: format!("{}-{}", VALIDATOR_PORT_RANGE.0, VALIDATOR_PORT_RANGE.1),
max_connections_per_peer: MAX_QUIC_CONNECTIONS_PER_PEER.to_string(),
max_connections_per_peer: DEFAULT_MAX_QUIC_CONNECTIONS_PER_PEER.to_string(),
max_tpu_staked_connections: MAX_STAKED_CONNECTIONS.to_string(),
max_tpu_unstaked_connections: MAX_UNSTAKED_CONNECTIONS.to_string(),
max_fwd_staked_connections: MAX_STAKED_CONNECTIONS
Expand All @@ -40,7 +42,7 @@ impl Default for DefaultArgs {
max_streams_per_ms: DEFAULT_MAX_STREAMS_PER_MS.to_string(),
max_connections_per_ipaddr_per_min: DEFAULT_MAX_CONNECTIONS_PER_IPADDR_PER_MINUTE
.to_string(),
num_quic_endpoints: NUM_QUIC_ENDPOINTS.to_string(),
num_quic_endpoints: DEFAULT_NUM_QUIC_ENDPOINTS.to_string(),
}
}
}
Expand Down Expand Up @@ -104,23 +106,39 @@ pub fn app<'a>(version: &'a str, default_args: &'a DefaultArgs) -> App<'a, 'a> {
.takes_value(true)
.default_value(&default_args.max_connections_per_peer)
.validator(is_parsable::<u32>)
.help("Controls the max concurrent connection per IpAddr."),
.help("Controls the max concurrent connections per IpAddr."),
)
.arg(
Arg::with_name("max_tpu_staked_connections")
.long("max-tpu-staked-connections")
.takes_value(true)
.default_value(&default_args.max_tpu_staked_connections)
.validator(is_parsable::<u32>)
.help("Controls the max concurrent connection per IpAddr."),
.help("Controls the max concurrent connections for TPU from staked nodes."),
)
.arg(
Arg::with_name("max_tpu_unstaked_connections")
.long("max-tpu-unstaked-connections")
.takes_value(true)
.default_value(&default_args.max_tpu_unstaked_connections)
.validator(is_parsable::<u32>)
.help("Controls the max concurrent connection per IpAddr."),
.help("Controls the max concurrent connections fort TPU from unstaked nodes."),
)
.arg(
Arg::with_name("max_fwd_staked_connections")
.long("max-fwd-staked-connections")
.takes_value(true)
.default_value(&default_args.max_fwd_staked_connections)
.validator(is_parsable::<u32>)
.help("Controls the max concurrent connections for TPU-forward from staked nodes."),
)
.arg(
Arg::with_name("max_fwd_unstaked_connections")
.long("max-fwd-unstaked-connections")
.takes_value(true)
.default_value(&default_args.max_fwd_unstaked_connections)
.validator(is_parsable::<u32>)
.help("Controls the max concurrent connections for TPU-forward from unstaked nodes."),
)
.arg(
Arg::with_name("max_connections_per_ipaddr_per_minute")
Expand Down
6 changes: 6 additions & 0 deletions vortexor/src/main.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ pub fn main() {
.expect("invalid bind_address");
let max_connections_per_peer = value_t_or_exit!(matches, "max_connections_per_peer", u64);
let max_tpu_staked_connections = value_t_or_exit!(matches, "max_tpu_staked_connections", u64);
let max_fwd_staked_connections = value_t_or_exit!(matches, "max_fwd_staked_connections", u64);
let max_fwd_unstaked_connections =
value_t_or_exit!(matches, "max_fwd_unstaked_connections", u64);

let max_tpu_unstaked_connections =
value_t_or_exit!(matches, "max_tpu_unstaked_connections", u64);

Expand Down Expand Up @@ -66,6 +70,8 @@ pub fn main() {
max_connections_per_peer,
max_tpu_staked_connections,
max_tpu_unstaked_connections,
max_fwd_staked_connections,
max_fwd_unstaked_connections,
max_streams_per_ms,
max_connections_per_ipaddr_per_min,
tpu_coalesce,
Expand Down
Loading

0 comments on commit a4b63f0

Please sign in to comment.