All things filters

[hmpf]

Place to discuss, braindump, think about filters. Which to have, how to combine, how to make them pluginable etc.

• Filtering of incident list in API is done by django-filter, see argus.incident.filters
• Checking whether an incident should trigger a notification is done in the methods incident_fits and event_fits, they do NOT depend on django-filter and are in general messy.
• The current frontend uses both: the buttons trigger GET-requests, and "save" saves in the notificationprofile.Filter model

The pull request #812 decouples filter-magic from everywhere in the code. Not sure if it should be merged as is now just to have an easier time going forward or if it should also have pluginable filters.

[hmpf]

#812 moves (almost) everything relevant into the new module argus.filter.

[elfjes]

We discussed defining a contract for FilterBackend classes. FilterBackends will be what is used by argus when a user wants to use a specific filter to retrieve a list of incidents (and/or events) that match (fit) the filter. The interface should be agnostic of the filtering logic and filter definition. We have two types of filters

• Argus simple filters. These have a few specified fields that can be filtered on, such as acked and open
• Geant boolean filters: These are filters that can combine different selection criteria for an incident, such as looking at sub keys in the incident metadata. These selection criteria are combined using boolean logic (and/or)

A FilterBackend should have different methods (names tbd)

• "get me the WHERE clause (as a django Q()?) for a specific filter (these can then be used to combine filters into a single query, or enrich an existing query for incidents/events)
• given an incident (pk?) and one or more specific filters, does this match the filter(s)

I think with those methods we can already do a lot

Filters can then be defined as a json field in postgres and be given a name. It is up to the implementation of the FilterBackend to define the schema of this json field.

[hmpf]

Swap-points:

Our homemade spaghetti

FilterWrapper.(incident|event)_fits for matching incoming/new events with ONE Filter-model

ComplexWrapper.(incident|event)_fits for matching with multiple filters combined together, currently used to match NotificationProfiles to Incidents on incoming/new events. (we loop over all filters when sending notifications).

FilterBlobSerializer uses a drf serializer to validate the contents of the jsonblob in Filter.filter. This is the magic needed

validate_jsonfilter uses FilterBlobSerializer but is only ever used once in the whole codebase. Hm.

Complex filters

Currently, NotificationProfile has one or more filters that an incoming/new event and its incident is matched against. The filters are ANDed together, this is hardcoded. The actual lookup is done in ComplexWrapper.(incident|event)_fits.

Logic for combining Filters in other ways should not be added to NotificationProfile but be separate functionality.

Maybe there should be a ComplexFilter model to combine Filter models in other ways than AND, and this needs to be recursive? So a ComplexFilter could have m2m to ComplexFilter and Filter? Then NotificationProfile would have a foreign key to ONE ComplexFilter instead of a m2m to Filter.

The ComplexFilter for todays solution would then just be: operand is AND, no m2m to other ComplexFilters, but m2m to Filter.

Fine-dining via django-filter

argus.filter.filters has the IncidentFilter and SourceLockedIncidentFilter which is used to match GET query parameters in the API.

argus.incident.views has an OpenAPI parameters list that describes IncidentFilter and SourceLockedIncidentFilter.

Needed: a filter__name to select a filter by name in addition to pk. With this in place it might not be necessary to change IncidentFilter and SourceLockedIncidentFilter.

[hmpf]

Frontend-wise, one template for the entire "filter"-section in the incident-list page, which is backed by a view somehow and who knows how many other views and templates with htmx. But the top view and template fragment needs to be swappable.

[elfjes]

Yes. As for how I see the geant implementation to look like, I have created an issue our repo: GEANT/geant-argus#3

this would then need to be injected somewhere in the argus htmx frontend

[elfjes]

Concerning strategies for swapping out FilterWrappers

We could perhaps leverage customization of the AppConfig for the argus or argus filter app