Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix message acks on wrong rabbitmq channels #4358

Merged
merged 15 commits into from
Dec 11, 2024
Merged

Fix message acks on wrong rabbitmq channels #4358

merged 15 commits into from
Dec 11, 2024

Conversation

pcapriotti
Copy link
Contributor

@pcapriotti pcapriotti commented Dec 3, 2024

This PR fixes an issue with the way client acks are delivered to rabbitmq. When a channel dies and is replaced by a new one, it can happen that acks that were in flight are delivered to the wrong channel. This is bad, because it can cause the wrong message to be acked, or the channel to be forcibly closed by the rabbitmq server.

The fix is to simply control whether channel reconnection is enabled based on a new enableRetry flag in the connection pool. For cannon pools, this flag is always set to False, thereby disabling reconnection completely, and letting the client handle any rabbitmq disconnection by creating a new websocket connection.

Setting the flag to True can be useful if/when the rabbitmq connection pool is used in background worker or other services.

https://wearezeta.atlassian.net/browse/WPB-11491

Checklist

  • Add a new entry in an appropriate subdirectory of changelog.d
  • Read and follow the PR guidelines

@echoes-hq echoes-hq bot added the echoes/initiative: scale Enterprise Readiness Initiatives label Dec 3, 2024
@zebot zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Dec 3, 2024
@pcapriotti pcapriotti marked this pull request as ready for review December 3, 2024 14:28
services/cannon/src/Cannon/RabbitMq.hs Outdated Show resolved Hide resolved
charts/integration/templates/configmap.yaml Outdated Show resolved Hide resolved
integration/test/Test/Events.hs Outdated Show resolved Hide resolved
integration/test/Test/Events.hs Outdated Show resolved Hide resolved
services/cannon/src/Cannon/RabbitMq.hs Outdated Show resolved Hide resolved
Co-authored-by: Akshay Mankar <[email protected]>
@pcapriotti pcapriotti merged commit 8af8925 into develop Dec 11, 2024
11 checks passed
@pcapriotti pcapriotti deleted the rabbitmq-acks branch December 11, 2024 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
echoes/initiative: scale Enterprise Readiness Initiatives ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants