Make rrule fast forwarding stable

[fosterseth]

SUMMARY

By stable, we mean future occurrences of the rrule should be the same before and after the fast forward operation.

The problem before was that we were fast forwarding to exactly 7 days ago. For some rrules, this does not retain the old occurrences (for example, freq=HOURLY and INTERVAL=23). Thus, jobs would launch at unexpected times any time the schedule would fast forward.

This change makes sure we fast forward in increments of the rrule INTERVAL (converted to seconds), thus the new dtstart should be in the occurrence list of the old rrule.

DETAIL

It is important that each rrule in the rruleset gets fast forwarded independently. For example consider this rruleset

DTSTART;TZID=America/New_York:20201118T200000
RRULE:FREQ=HOURLY;INTERVAL=5
RRULE:FREQ=HOURLY;INTERVAL=7

Each of those rrules will need its own dtstart, since the occurrences from each rrule do not perfectly overlap.

the API also support exclusion rules, which are basically just rrules. So we can use the same method to fast forward those too.

PERFORMANCE

We want to retain the performance gains from introducing fast forwarding to begin with

create a bunch of old schedules

for i in range(2000):
    Schedule.objects.create(name=f's{i}', rrule='DTSTART;TZID=America/New_York:20190517T000000 RRULE:FREQ=HOURLY;INTERVAL=7', unified_job_template_id=7)

runit will update_computed_fields for each schedule

def runit():
    for sch in Schedule.objects.all():
        sch.update_computed_fields()

benchmark it

cProfile.run("runit()", "bench.txt")

before:

after:

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

[PabloHiro]

Fast forward won't work for really large intervals. For example if you specify HOURLY and INTERVAL=1200 (that is 50 days worth of time), then we can't fast forward because one CHUNK of the interval doesn't fit in the period of time we are trying to fast forward (window is 30 days). In this case, we'll revert back to the old style of just updating dtstart to 7 days ago. We log a warning, so hopefully the user will update their rrule to be frequency DAYS instead of HOURS.

Can this be an HTTP 400 when a Schedule is created with an "invalid" rrule? Basically preventing this type of rrules to be created in the first place.

Also, I would specify the value of the fast forward amount in the logs, so the user does not need to guess the value by trial and error for existing "invalid" rrules

[pb82]

Ok, I think I understood what's going on:

1. To avoid computing a large number of events for an rrule with a dtstart way back in the past, we only compute from one week ago.
2. To not mess up the interval, we can't just subtract 7 days, we have to go back an amount of time that is dividable by the interval (that's what (fast_forward_seconds // interval) * interval does).

Did I get that right?

[pb82]

as @PabloHiro mentioned, this would lead to the exact problem that is being fixed. Can we just reject the rrule here?

[PabloHiro]

I wonder how we should handle the cases where a user has these "invalid" rules already. We can prevent further invalid rules, but this code has a solution to handle the existing ones.

[pb82]

@fosterseth can this be merged? The failures seem to be about code coverage.

[webknjaz]

got a few observations below

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud