Skip to content

Commit

Permalink
[Tunnelbroker] ignore errors about closed connection when sending mes…
Browse files Browse the repository at this point in the history
…sage to device

Summary:
This is a fix for [ENG-10110](https://linear.app/comm/issue/ENG-10119/alarm-tunnelbrokerproductionwebsocketerroralarm-in-us-east-ohio).

The code here triggered an error because it kept sending messages probably before receiving [close](https://github.com/CommE2E/comm/blob/7649ca4931b29d3cabf0696271938a87651ecc40/services/tunnelbroker/src/websockets/mod.rs#L265) message.

Test Plan: This is complicated to reproduce so just carefully reading the code as we are using exactly the same approach as [here](https://github.com/CommE2E/comm/blob/ed9275b0a10a9baa463681d03489b60f508162ab/services/tunnelbroker/src/amqp.rs#L164). I am going to test it more after deploying to staging.

Reviewers: bartek

Reviewed By: bartek

Subscribers: ashoat, tomek

Differential Revision: https://phab.comm.dev/D14254
xsanm committed Jan 28, 2025
1 parent c8f92d2 commit 1a8de1b
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions services/tunnelbroker/src/websockets/session.rs
Original file line number Diff line number Diff line change
@@ -745,6 +745,10 @@ impl<S: AsyncRead + AsyncWrite + Unpin> WebsocketSession<S> {

pub async fn send_message_to_device(&mut self, message: Message) {
if let Err(e) = self.tx.send(message).await {
if should_ignore_error(&e) {
debug!("Ignored error when sending message to device: {e:?}");
return;
}
error!(
errorType = error_types::WEBSOCKET_ERROR,
"Failed to send message to device: {}", e
@@ -866,3 +870,18 @@ impl<S: AsyncRead + AsyncWrite + Unpin> WebsocketSession<S> {
Ok(())
}
}

fn should_ignore_error(err: &hyper_tungstenite::tungstenite::Error) -> bool {
use hyper_tungstenite::tungstenite::Error as E;
use std::io::ErrorKind;

match err {
E::ConnectionClosed | E::AlreadyClosed => true,
E::Io(io_error) => match io_error.kind() {
// The operation failed because a pipe was closed.
ErrorKind::BrokenPipe => true,
_ => false,
},
_ => false,
}
}

0 comments on commit 1a8de1b

Please sign in to comment.