[PM-17562] Initial POC of Distributed Events #5323

brant-livefront · 2025-01-24T19:43:08Z

🎟️ Tracking

https://bitwarden.atlassian.net/browse/PM-17562

📔 Objective

This PR is an initial POC of a new way of handling events. The end goal will be to add an AMQP messaging for all platforms that will broadcast and fan-out events in addition to our normal handling of events. This allows multiple integrations to be built in the future (e.g a Slack message when X event occurs, etc). The Cloud version will ultimately be built on Azure Service Bus, but for this initial POC we wanted to just build something for local / self-hosted.

This PR adds a new opt-in flow for Events. Specifically:

A Docker Compose profile that will start a RabbitMQ container
An IEventWriteService implementation that writes events to the RabbitMQ exchange
A RabbitMQ subscriber that monitors for new events and writes those events to the DB (replicating what we had before, but introducing the RabbitMQ in the middle)
A RabbitMQ subscriber that will post JSON to a configurable URL
- This is included as a small proof of concept for the fan-out. It's a way of showing that we can add new subscribers that provide new integrations once this is in place.
New configurations that allow for specifying all of this.
- Everything here should be opt-in only. If you don't configure the RabbitMQ service, everything should operate as before.
- Same with the POST service - even if RabbitMQ is configured, it will only start up if there's a URL configured.

The following is copied from my PR in the contributing-docs repo. This is added to the Events page to document the new RabbitMQ additions and how to configure and start the service.

Distributed Events (optional)

There is a new system which will add support for distributing events via an AMQP nessaging system.
In the future this will enable new integrations by allowing for a means to subscribe to
events via messaging stream.

As an initial proof of concept, there is an optional RabbitMQ implementation that refactors the way
events are handled when running locally or self-hosted. Instead of writing directly to the Events table
via the EventsRepository, it will broadcast each event via a RabbitMQ exchange. A new
RabbitMqEventRepositoryListener then subscribes to the RabbitMQ exchange and writes to the
Events table via the EventsRepository. The end result is the same (events are stored in the
database), but this allows for other integrations to subscribe.

To illustrate this, there is also a RabbitMqEventHttpPostListener which subscribes to the RabbitMQ
events exchange and POSTs each event to a configurable URL. This is meant to be
a simple, concrete example of how multiple integrations are enabled by moving to
distributed events.

graph TD
	  subgraph With RabbitMQ
        B1[EventService]
        B2[RabbitMQEventWriteService]
        B3[RabbitMQ exchange]
        B4[RabbitMqEventRepositoryListener]
        B5[RabbitMqEventHttpPostListener]
        B6[Events Database Table]
        B7[HTTP Server]

        B1 -->|IEventWriteService| B2 --> B3
        B3--> B4 --> B6
        B3--> B5
        B5 -->|HTTP POST| B7
    end

    subgraph Without RabbitMQ
        A1[EventService]
        A2[RepositoryEventWriteService]
        A3[Events Database Table]

        A1 -->|IEventWriteService| A2 --> A3

end

To enable the new RabbitMQ-based event stream, take the following steps:

Running the RabbitMQ container

Verify that you've set a RabbitMQ username and password in the .env file (see .env.example
for an example)
Use Docker Compose to run the container with your current settings:
```
docker compose --profile rabbitmq up -d
```
This will run the RabbitMQ container with your username and password on localhost with the standard ports
To verify this is running, open http://localhost:15672 in a browser and login with the username and password in your .env file.

Configuring the server to use RabbitMQ for events

Add the following to your secrets.json file, changing the defaults to match your .env file:

    "eventLogging": {
      "rabbitMq": {
        "hostName": "localhost",
        "username": "bitwarden",
        "password": "SET_A_PASSWORD_HERE_123",
        "exchangeName": "events-exchange",
        "eventRepositoryQueueName": "events-write-queue",
      }
    },

(optional) To use the RabbitMqEventHttpPostListener, specify a queue name and destination
URL that will receive the POST inside of the RabbitMq object:
```
     "httpPostQueueName": "events-httpPost-queue",
     "httpPostUrl": "<HTTP POST URL>",
```
- Tip: RequetBin provides an easy to setup server that will
  receive these requests and let you inspect them.
Re-run the powershell script to add these secrets to each Bitwarden project:
```
pwsh setup_secrets.ps1
```
Start (or restart) all of your projects to pick up the new settings

With these changes in place, you should see the database events written as before, but you'll also see in the RabbitMQ management interface that the messages are flowing through the configured exchange.

⏰ Reminders before review

Contributor guidelines followed
All formatters and local linters executed and passed
Written new unit and / or integration tests where applicable
Protected functional changes with optionality (feature flags)
Used internationalization (i18n) for all UI strings
CI builds passed
Communicated to DevOps any deployment requirements
Updated any necessary documentation (Confluence, contributing docs) or informed the documentation team

🦮 Reviewer guidelines

👍 (:+1:) or similar for great changes
📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
❓ (:question:) for questions
🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed issue and could potentially benefit from discussion
🎨 (:art:) for suggestions / improvements
❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
⛏ (:pick:) for minor or nitpick changes

codecov · 2025-01-24T19:52:59Z

Codecov Report

Attention: Patch coverage is 5.84416% with 145 lines in your changes missing coverage. Please review.

Project coverage is 44.23%. Comparing base (f1c94a1) to head (3ca63f9).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...vices/Implementations/RabbitMqEventListenerBase.cs	0.00%	59 Missing ⚠️
...vices/Implementations/RabbitMqEventWriteService.cs	0.00%	38 Missing ⚠️
src/Events/Startup.cs	0.00%	13 Missing ⚠️
...s/Implementations/RabbitMqEventHttpPostListener.cs	0.00%	12 Missing ⚠️
...SharedWeb/Utilities/ServiceCollectionExtensions.cs	0.00%	10 Missing ⚠️
...Implementations/RabbitMqEventRepositoryListener.cs	0.00%	9 Missing ⚠️
src/Core/Settings/GlobalSettings.cs	69.23%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5323      +/-   ##
==========================================
- Coverage   44.32%   44.23%   -0.09%     
==========================================
  Files        1483     1487       +4     
  Lines       68416    68572     +156     
  Branches     6173     6179       +6     
==========================================
+ Hits        30325    30335      +10     
- Misses      36782    36928     +146     
  Partials     1309     1309

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2025-01-24T20:47:06Z

Checkmarx One – Scan Summary & Details – 2d239a14-682c-47d3-a77e-ddd9b95e21c5

New Issues (7)

Checkmarx found the following issues in this Pull Request

Issue	Source File / Package	Checkmarx Insight
CSRF	/src/Api/Auth/Controllers/TwoFactorController.cs: 406	details Method PostRecover at line 406 of /src/Api/Auth/Controllers/TwoFactorController.cs gets a parameter from a user request from model. This parameter ... Attack Vector
Privacy_Violation	/src/Core/NotificationHub/NotificationHubPushNotificationService.cs: 195	details Method PushAuthRequestAsync at line 195 of /src/Core/NotificationHub/NotificationHubPushNotificationService.cs sends user information outside the a... Attack Vector
Privacy_Violation	/src/Core/Auth/Services/Implementations/AuthRequestService.cs: 226	details Method UpdateAuthRequestAsync at line 226 of /src/Core/Auth/Services/Implementations/AuthRequestService.cs sends user information outside the appli... Attack Vector
Heap_Inspection	/src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs: 19	details Method RabbitMqEventWriteService at line 19 of /src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs defines Password, which... Attack Vector
Heap_Inspection	/src/Core/AdminConsole/Services/Implementations/RabbitMqEventListenerBase.cs: 29	details Method RabbitMqEventListenerBase at line 29 of /src/Core/AdminConsole/Services/Implementations/RabbitMqEventListenerBase.cs defines Password, which... Attack Vector
Heap_Inspection	/src/Core/Settings/GlobalSettings.cs: 268	details Method at line 268 of /src/Core/Settings/GlobalSettings.cs defines _password, which is designated to contain user passwords. However, while plaint... Attack Vector
Missing_CSP_Header	/src/Core/MailTemplates/Handlebars/AdminConsole/NotifyAdminDeviceApprovalRequested.html.hbs: 9	details A Content Security Policy is not explicitly defined within the web-application. Attack Vector

Fixed Issues (17)

Great job! The following issues were fixed in this Pull Request

Severity	Issue	Source File / Package
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationUsersController.cs: 106
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationUsersController.cs: 238
	CSRF	/src/Admin/AdminConsole/Controllers/OrganizationsController.cs: 375
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationsController.cs: 127
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationUsersController.cs: 264
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationUsersController.cs: 360
	CSRF	/src/Api/AdminConsole/Controllers/OrganizationUsersController.cs: 377
	CSRF	/src/Api/Vault/Controllers/CiphersController.cs: 110
	CSRF	/src/Api/Vault/Controllers/CiphersController.cs: 124
	CSRF	/src/Api/Vault/Controllers/CiphersController.cs: 944
	CSRF	/src/Api/Vault/Controllers/CiphersController.cs: 684
	Privacy_Violation	/src/Core/Auth/Services/Implementations/AuthRequestService.cs: 221
	Log_Forging	/src/Billing/Controllers/RecoveryController.cs: 38
	Log_Forging	/src/Billing/Controllers/RecoveryController.cs: 38
	Log_Forging	/src/Billing/Controllers/RecoveryController.cs: 38
	Log_Forging	/src/Billing/Controllers/StripeController.cs: 164
	Log_Forging	/src/Billing/Controllers/StripeController.cs: 164

justindbaur

Happened across this PR so maybe this feedback is a little early, if so sorry about that.

justindbaur · 2025-01-24T21:00:25Z

src/Core/AdminConsole/Services/Implementations/RabbitMqEventHttpPostListener.cs

+        var message = JsonSerializer.Serialize(eventMessage);
+        using var httpClient = new HttpClient();
+        var content = new StringContent(message, Encoding.UTF8, "application/json");
+        var response = await httpClient.PostAsync(_httpPostUrl, content);
+        response.EnsureSuccessStatusCode();


We shouldn't be creating short lived HttpClient's. We should instead created a named HttpClient in DI and retrieve it with IHttpClientFactory that gets injected here.

https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/http/httpclient-guidelines#recommended-use

I think this should be fixed now, but let me know if I need to refine it further. 👍

src/Core/AdminConsole/Services/Implementations/RabbitMqEventListenerBase.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventHttpPostListener.cs

dev/.env.example

dev/docker-compose.yml

Co-authored-by: Justin Baur <[email protected]>

withinfocus

Few things and some questions.

dev/docker-compose.yml

src/Core/AdminConsole/Services/Implementations/RabbitMqEventHttpPostListener.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventRepositoryListener.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventHttpPostListener.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventListenerBase.cs

withinfocus · 2025-01-27T21:51:54Z

src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs

+
+        await channel.ExchangeDeclareAsync(exchange: _exchangeName, type: ExchangeType.Fanout);
+
+        var body = JsonSerializer.SerializeToUtf8Bytes(e);


❓ Is JSON our best option for serialization?

I think so, at least to start. Were you thinking about something like Protobuf?

Perhaps ... just don't know best practices with Rabbit.

src/Core/Settings/GlobalSettings.cs

…from PR review

brant-livefront · 2025-01-28T18:26:49Z

If you're running this to test locally, note that in refactoring all of the global settings, you'll need to adopt the new secrets and replublish them. The sample secrets are included in the description above (I've updated it to match the new format). After adding those, you'll need to run:

```bash
pwsh setup_secrets.ps1 -clear
```

withinfocus

This makes sense to me as a first pass and we'll iterate on some additional topics as discussed.

withinfocus · 2025-01-28T19:20:31Z

src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs

+
+        await channel.ExchangeDeclareAsync(exchange: _exchangeName, type: ExchangeType.Fanout);
+
+        var body = JsonSerializer.SerializeToUtf8Bytes(e);


Perhaps ... just don't know best practices with Rabbit.

justindbaur

A few more thoughts.

src/Core/AdminConsole/Services/Implementations/RabbitMqEventHttpPostListener.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventRepositoryListener.cs

src/Core/AdminConsole/Services/Implementations/RabbitMqEventWriteService.cs

Co-authored-by: Justin Baur <[email protected]>

src/Core/AdminConsole/Services/Implementations/RabbitMqEventRepositoryListener.cs

jrmccannon

Looks good from my perspective.

@withinfocus Is this POC a step in the direction of decoupling services and moving toward an event based architecture? Or is there a different end goal for this?

withinfocus · 2025-01-30T17:04:40Z

Not necessarily, yet -- just allowing our events to be ingested in more places. It's definitely an implicit enabler of that though, for the eventing angle at least. We could use this same backbone for other interprocess communication.

Initial POC of Distributed Events

d386eee

brant-livefront requested a review from withinfocus January 24, 2025 19:43

brant-livefront requested a review from a team as a code owner January 24, 2025 19:43

brant-livefront requested a review from jrmccannon January 24, 2025 19:43

Merge branch 'main' into brant/self-hosted-amqp-service

21fe94c

justindbaur reviewed Jan 24, 2025

View reviewed changes

brant-livefront changed the title ~~Initial POC of Distributed Events~~ [PM-17562] Initial POC of Distributed Events Jan 24, 2025

withinfocus reviewed Jan 24, 2025

View reviewed changes

dev/.env.example Outdated Show resolved Hide resolved

dev/docker-compose.yml Show resolved Hide resolved

brant-livefront and others added 4 commits January 24, 2025 17:27

Apply suggestions from code review

8f2a013

Co-authored-by: Justin Baur <[email protected]>

Clean up files to support accepted changes. Address PR Feedback

ee19ff5

Merge branch 'main' into brant/self-hosted-amqp-service

2c9079b

Removed unneeded using to fix lint warning

1e5690c

brant-livefront mentioned this pull request Jan 27, 2025

[PM-17562] Add documentation for the initial Distributed Events POC bitwarden/contributing-docs#533

Merged

Merge branch 'main' into brant/self-hosted-amqp-service

7f8f1a9

withinfocus requested changes Jan 27, 2025

View reviewed changes

brant-livefront added 3 commits January 28, 2025 08:43

Merge branch 'main' into brant/self-hosted-amqp-service

e238bf9

Moved config into a common EventLogging top-level item. Fixed issues …

58b5c32

…from PR review

Merge branch 'main' into brant/self-hosted-amqp-service

ab68e43

withinfocus previously approved these changes Jan 28, 2025

View reviewed changes

vincentsalucci requested a review from eliykat January 28, 2025 19:52

justindbaur reviewed Jan 28, 2025

View reviewed changes

Optimized per suggestion from justinbaur

289daf6

Co-authored-by: Justin Baur <[email protected]>

brant-livefront dismissed withinfocus’s stale review via 289daf6 January 29, 2025 15:42

brant-livefront added 3 commits January 29, 2025 10:44

Merge branch 'main' into brant/self-hosted-amqp-service

5cc8483

Updated to add IAsyncDisposable as suggested in PR review

e567e18

Updated with suggestion to use KeyedSingleton for the IEventWriteService

2b3dca4

withinfocus reviewed Jan 29, 2025

View reviewed changes

src/Core/AdminConsole/Services/Implementations/RabbitMqEventRepositoryListener.cs Outdated Show resolved Hide resolved

Changed key case to lowercase

3ca63f9

jrmccannon approved these changes Jan 30, 2025

View reviewed changes

withinfocus approved these changes Jan 30, 2025

View reviewed changes

brant-livefront merged commit 5efd68c into main Jan 30, 2025
53 checks passed

brant-livefront deleted the brant/self-hosted-amqp-service branch January 30, 2025 17:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PM-17562] Initial POC of Distributed Events #5323

[PM-17562] Initial POC of Distributed Events #5323

brant-livefront commented Jan 24, 2025 •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading

github-actions bot commented Jan 24, 2025 •

edited

Loading

justindbaur left a comment

justindbaur Jan 24, 2025

brant-livefront Jan 27, 2025

withinfocus left a comment

withinfocus Jan 27, 2025

brant-livefront Jan 28, 2025

withinfocus Jan 28, 2025

brant-livefront commented Jan 28, 2025

withinfocus left a comment

withinfocus Jan 28, 2025

justindbaur left a comment

jrmccannon left a comment

withinfocus commented Jan 30, 2025


		await channel.ExchangeDeclareAsync(exchange: _exchangeName, type: ExchangeType.Fanout);

		var body = JsonSerializer.SerializeToUtf8Bytes(e);

[PM-17562] Initial POC of Distributed Events #5323

[PM-17562] Initial POC of Distributed Events #5323

Conversation

brant-livefront commented Jan 24, 2025 • edited Loading

🎟️ Tracking

📔 Objective

Distributed Events (optional)

Running the RabbitMQ container

Configuring the server to use RabbitMQ for events

⏰ Reminders before review

🦮 Reviewer guidelines

codecov bot commented Jan 24, 2025 • edited Loading

Codecov Report

github-actions bot commented Jan 24, 2025 • edited Loading

justindbaur left a comment

Choose a reason for hiding this comment

justindbaur Jan 24, 2025

Choose a reason for hiding this comment

brant-livefront Jan 27, 2025

Choose a reason for hiding this comment

withinfocus left a comment

Choose a reason for hiding this comment

withinfocus Jan 27, 2025

Choose a reason for hiding this comment

brant-livefront Jan 28, 2025

Choose a reason for hiding this comment

withinfocus Jan 28, 2025

Choose a reason for hiding this comment

brant-livefront commented Jan 28, 2025

withinfocus left a comment

Choose a reason for hiding this comment

withinfocus Jan 28, 2025

Choose a reason for hiding this comment

justindbaur left a comment

Choose a reason for hiding this comment

jrmccannon left a comment

Choose a reason for hiding this comment

withinfocus commented Jan 30, 2025

brant-livefront commented Jan 24, 2025 •

edited

Loading

codecov bot commented Jan 24, 2025 •

edited

Loading

github-actions bot commented Jan 24, 2025 •

edited

Loading