one expiration task per provider type

[maskarb]

Jira Ticket

COST-####

Description

This change will:

• utilize cascade_delete to remove the expired OCPUsageReportPeriod instead of the Django delete method
• run one data expiration task per schema per provider_type, rather than 1 task per provider

Testing

1. load test data:

$ make create-test-customer; make load-test-customer-data test_source=AWS

2. check for report data when done ingesting:
   http://localhost:8000/api/cost-management/v1/reports/aws/costs/?filter[time_scope_value]=-30
   (see data for 2 months)
3. set RETAIN_NUM_MONTHS=1 and restart koku-worker
4. open a python shell:

$ REDIS_HOST=localhost make shell

(set the REDIS_HOST so that the shell can reach Redis)

5. use the Orchestrator to trigger a deletion:

>>> from masu.processor.orchestrator import Orchestrator
>>> orchestrator = Orchestrator()
>>> orchestrator.remove_expired_report_data(simulate=False)
[
    {"schema": "org1234567", "provider_type": "GCP-local", "async_id": "6f896ddc-7590-4d8e-8279-a4bd428c6651"},
    {"schema": "org1234567", "provider_type": "IBM-local", "async_id": "390a5303-91a2-4c96-94f4-0b7a8b65d99f"},
    {"schema": "org1234567", "provider_type": "AWS-local", "async_id": "efdd8a49-a13e-4592-846d-6f46ee197d1e"},
    {"schema": "org1234567", "provider_type": "Azure-local", "async_id": "0122ac56-020e-4fab-b624-77245c54a18b"},
    {"schema": "org1234567", "provider_type": "OCI-local", "async_id": "e0b624e6-7b3b-4383-8c93-f9342b084ddc"},
    {"schema": "org1234567", "provider_type": "OCP", "async_id": "b56d2cf2-6d2e-4988-a4ac-a0971c6b3c50"},
]

See that each provider-type only appears once in the result list AND OCP is called last.

6. clear redis cache:

make delete-redis-cache

7. Revisit the report endpoint and see data only exists for the current month:
   http://localhost:8000/api/cost-management/v1/reports/aws/costs/?filter[time_scope_value]=-30

Release Notes

• proposed release note

* [COST-####](https://issues.redhat.com/browse/COST-####) Fix some things

[codecov]

Codecov Report

Attention: Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.1%. Comparing base (78793e3) to head (50fcaf3).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5432     +/-   ##
=======================================
- Coverage   94.2%   94.1%   -0.0%     
=======================================
  Files        371     371             
  Lines      31546   31550      +4     
  Branches    3379    3382      +3     
=======================================
+ Hits       29703   29704      +1     
- Misses      1196    1199      +3     
  Partials     647     647