Skip to content

Commit

Permalink
Replace tokio-uring with io-uring (#993)
Browse files Browse the repository at this point in the history
This is a fairly major change to swap out `tokio-uring` with the lower level `io-uring`, which has some upsides and downsides.

### Upsides

In `tokio-uring`, _every_ udp [recv_from](https://github.com/tokio-rs/tokio-uring/blob/7761222aa7f4bd48c559ca82e9535d47aac96d53/src/io/recv_from.rs#L20-L54) and [send_to](https://github.com/tokio-rs/tokio-uring/blob/7761222aa7f4bd48c559ca82e9535d47aac96d53/src/io/send_to.rs#L22-L70) performs **3** heap allocations (maybe even more in other parts of the code?) which is extremely wasteful in the context of a proxy that can be sending and receiving many thousands of packets a second. Moving to `io-uring` means we need to take responsibility for the lifetimes of memory being written/read by the kernel during I/O, but means we can minimize/get rid of memory allocations since we have the full context. For example, the QCMP loop now doesn't use the heap at all in favor of just reusing stack allocations.

Additionally, the current code which forwards packets either [downstream](https://github.com/googleforgames/quilkin/blob/ee0b70f654f4c01b1ce027985b80d7d1133c7432/src/components/proxy/packet_router.rs#L67-L110) or [upstream](https://github.com/googleforgames/quilkin/blob/ee0b70f654f4c01b1ce027985b80d7d1133c7432/src/components/proxy/sessions.rs#L166-L202) only ever sends 1 packet at a time per worker/session, the new code takes advantage of not being async/await by just sending up to a few thousand packets concurrently, reducing a (probably minor) throughput bottleneck.

### Downsides

A lot more code, some of which is unsafe, though slightly less than it could have been, as now the session and packet_router both share the same implementation. The non-linux code is also now separated since they are no longer really compatible since the io uring loop is not async so we can't pretend the code is the same between linux and non-linux, which also contributes to code increase.

Overall, it's just simply more complicated relative to the old code, but does give us tighter control.
  • Loading branch information
Jake-Shadle authored Aug 15, 2024
1 parent ee0b70f commit 8d44088
Show file tree
Hide file tree
Showing 22 changed files with 1,439 additions and 440 deletions.
67 changes: 22 additions & 45 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 9 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ categories = ["game-development", "network-programming"]
edition.workspace = true
exclude = ["docs", "build", "examples", "image"]

[lints]
workspace = true

[[bench]]
name = "read_write"
harness = false
Expand Down Expand Up @@ -144,8 +147,10 @@ version = "0.1"
features = ["client", "client-legacy"]

[target.'cfg(target_os = "linux")'.dependencies]
io-uring = { version = "0.6", default-features = false }
libc = "0.2"
slab = "0.4"
sys-info = "0.9.1"
tokio-uring = { version = "0.5", features = ["bytes"] }
pprof = { version = "0.13.0", features = ["prost", "prost-codec"] }

[dev-dependencies]
Expand Down Expand Up @@ -225,3 +230,6 @@ fixedstr = { version = "0.5", features = ["flex-str"] }
parking_lot = "0.12.1"
schemars = { version = "0.8.15", features = ["bytes", "url"] }
url = { version = "2.4.1", features = ["serde"] }

[workspace.lints.clippy]
undocumented_unsafe_blocks = "deny"
3 changes: 3 additions & 0 deletions crates/agones/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ license = "Apache-2.0"
description = "End to end integration tests to be run against a Kubernetes cluster with Agones installed"
readme = "README.md"

[lints]
workspace = true

[dependencies]
base64.workspace = true
futures.workspace = true
Expand Down
3 changes: 2 additions & 1 deletion crates/macros/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ edition = "2018"
[lib]
proc-macro = true

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lints]
workspace = true

[dependencies]
proc-macro2 = "1.0.58"
Expand Down
3 changes: 2 additions & 1 deletion crates/quilkin-proto/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ version = "0.1.0"
edition.workspace = true
license.workspace = true

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lints]
workspace = true

[dependencies]
prost.workspace = true
Expand Down
3 changes: 3 additions & 0 deletions crates/test/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ version = "0.1.0"
edition = "2021"
publish = false

[lints]
workspace = true

[dependencies]
async-channel.workspace = true
once_cell.workspace = true
Expand Down
11 changes: 6 additions & 5 deletions crates/test/tests/proxy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -124,17 +124,17 @@ trace_test!(uring_receiver, {
config,
tx,
BUFFER_POOL.clone(),
shutdown_rx,
shutdown_rx.clone(),
),
}
.spawn()
.spawn(shutdown_rx)
.await
.expect("failed to spawn task");

// Drop the socket, otherwise it can
drop(ws);

sb.timeout(500, ready.notified()).await;
let _ = sb.timeout(500, ready).await;

let msg = "hello-downstream";
tracing::debug!("sending packet");
Expand Down Expand Up @@ -166,7 +166,7 @@ trace_test!(
config.clone(),
tx,
BUFFER_POOL.clone(),
shutdown_rx,
shutdown_rx.clone(),
);

const WORKER_COUNT: usize = 3;
Expand All @@ -179,12 +179,13 @@ trace_test!(
&sessions,
rx,
BUFFER_POOL.clone(),
shutdown_rx,
)
.await
.unwrap();

for wn in workers {
sb.timeout(200, wn.notified()).await;
let _ = sb.timeout(200, wn).await;
}

let socket = std::sync::Arc::new(sb.client());
Expand Down
Loading

0 comments on commit 8d44088

Please sign in to comment.