Gracefully shutdown on signals

[jyn514]

Right now, every time docs.rs is deployed systemd sends a SIGQUIT to the daemon, which immediately exits since it doesn't have a signal handler. Instead, it should gracefully refuse new connections with a 503 Try Again (with a very short try-again) and finish handling all old connections.

This would likely also fix #984.

[Nemo157]

What might be better than refusing new connections with 503 is to just instantly stop accepting new connections while draining the old ones. Combined with SO_REUSEPORT binding that would allow starting the new instance first, then shutting down the old instance when it's successfully started. Though surprisingly I haven't been able to see a nice way to do rolling restarts with systemd (I guess everyone that's going to that effort is using k8s instead of systemd 🤔).

[jyn514]

I don't see an option for SO_REUSEPORT in iron, hyper, or even TcpListener from std - is this something we'd need nix for?

[jyn514]

Yeah, here we go: https://docs.rs/nix/0.19.0/nix/sys/socket/fn.setsockopt.html, https://docs.rs/nix/0.19.0/nix/sys/socket/sockopt/struct.ReusePort.html

[Nemo157]

I believe socket2 is the state of the art in std::net wrappers, something like

let socket = socket2::Socket::new(Domain::ipv4(), Type::stream(), Some(Protocol::tcp())).unwrap();

socket.bind(&"127.0.0.1:3000".parse::<SocketAddr>()?.into())?;
#[cfg(unix)] {
  socket.set_reuse_port(true)?;
}
socket.listen(128)?;

let listener = socket.into_tcp_listener();

[pickfire]

How about something like listenfd + systemfd?

[pickfire]

@jyn514 Are you working on this? What @Nemo157 showed seemed possible but the Domain::ipv4(), how about ipv6?

[jyn514]

I am not currently working on this. We do not deal with IPv6 for docs.rs, we use an nginx proxy that barely touches IP at all (it's just going to 127.0.0.1).

[pickfire]

But if we use an nginx proxy why couldn't we do a load balancing between two ports, we could spawn a new instance on another port and then close the old one when the new instance is serving connections right?

[jyn514]

I would be concerned about state - I want to keep as little state in configuration files as possible, because it's not tested except in prod. If you know a way to do this without having to keep track of what port is currently open (maybe with systemd?) that would be ok though.

[pickfire]

Either way, I believe using the reuse port thing is better than spawning a second one and relying on the load balancer if we can do it ourselves. listenfd + systemfd can do it for dev, I think we can use a similar method. I feel interested to try this out.

[jyn514]

@pickfire go for it :) feel free to ask here or on Discord if you need help.

[pickfire]

I recall we either need to update hyper version of iron or change to another framework (I recall someone saying hyper). So I am not working on this anymore. Someone else feel free to take it.

[syphar]

Thinking about this, it would be solved when using a web framework, like warp .
These typically support graceful shutdown.

[jyn514]

@syphar I would be surprised if any of the major web frameworks handle signals - do you happen to have docs for that?

[syphar]

@syphar I would be surprised if any of the major web frameworks handle signals - do you happen to have docs for that?

it's on the must-have-list for any migration I'll make, either in the framework, or easy to glue to.

( axum has an example for this)

[syphar]

Solved with #1963