Use django-q2 for notifications

[hmpf]

Closes #400

Note that django-q2 won't even install on python 3.7.

Dependencies

There's a new one, so remember to install django-q2 in the venv.

Database

Depending on django_q2 adds new database tables, so migrations must be run:

python manage.py migrate

Redis

It currently uses redis, the same redis that is configured for websockets, so set the environment-variable ARGUS_REDIS_SERVER. This is used to make the redis-settings in src/argus/site/settings/base.py, setting Q_CLUSTER. Override in local settings if necessary.

Running django-q2

Start the brokers with

python manage.py qcluster

It does not run in the background and can be turned off with Ctrl-C. Note: if any changes are made to the code, qcluster must be restarted, it does not automatically discover changes.

Check if a task worked or failed in Django Admin, the "Django Q2 Queue"-section.

Test sending notifications

Test queuing notifications by

1. Disabling profiles sending to non-email destinations
2. Ensuring that EMAIL_BACKEND is 'django.core.mail.backends.console.EmailBackend', this pumps the contents of the email to the console/sys.stdout where qcluster is running.
3. python manage.py create_fake_incident

In the terminal where you do step 3, the last log-line will contain something like:
"Notification: will be sending notification for "'Incident start':..".

In the terminal where qcluster is running, you'll get a log line starting with "Notification: sending event "'Incident start':..". The header and body will be dumped as plaintext after this line. After the email there will be a log line starting with "Notification: sent event "'Incident start':..".

Docker

The postgres image has been updated to version 13, this means the volume should be deleted and remade. Clean everything away with:

docker compose down --volumes

[github-actions]

Test results

       7 files     574 suites   21m 39s ⏱️
   462 tests    461 ✔️ 1 💤 0 ❌
3 234 runs  3 227 ✔️ 7 💤 0 ❌

Results for commit 4fe2960.

♻️ This comment has been updated with latest results.

[hmpf]

Missing:

• docs (how to)
• work with bulk-fix
• update docker this-and-that

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 73.07692% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 84.70%. Comparing base (6488afa) to head (4fe2960).

Files	Patch %	Lines
src/argus/incident/signals.py	56.25%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #630      +/-   ##
==========================================
+ Coverage   84.61%   84.70%   +0.09%     
==========================================
  Files          75       75              
  Lines        3750     3754       +4     
==========================================
+ Hits         3173     3180       +7     
+ Misses        577      574       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lunkwill42]

This looks like you're on the right track. I got this to work with only slight modifications.

I added an extra qcluster service to docker-compose.yml, which basically is a copy of api, but which executes the manage.py qcluster command rather than daphne, and can get it to dispatch notifications according to my profile when I inject fake ones.

[lunkwill42]

I added an extra qcluster service to docker-compose.yml, which basically is a copy of api, but which executes the manage.py qcluster command rather than daphne, and can get it to dispatch notifications according to my profile when I inject fake ones.

i.e. it's usable, but I don't like the redundant environment definitions in docker-compose.yml - I'm sure there some way we can template those out, though.

[hmpf]

I've looked at bulk-changes (Incident.objects.create_events()).

Should we make one notification per change (easy, by putting the new events on the queue one at a time) or should we make a single notification for a "bulk-change"-event? The problem with the latter is matching it with filters: a filter might hit one of the events in the bulk, but not all of them.

[lunkwill42]

I've looked at bulk-changes (Incident.objects.create_events()).

Should we make one notification per change (easy, by putting the new events on the queue one at a time) or should we make a single notification for a "bulk-change"-event? The problem with the latter is matching it with filters: a filter might hit one of the events in the bulk, but not all of them.

I like the easy and less complicated route of one queued job per notification. It shouldn't become a bottleneck, but if it does, we can work it out later.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

No Coverage information
0.0% Duplication

[sonarcloud]

Quality Gate passed

Issues
3 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[sonarcloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud