EOSC Accounting #17

enolfc · 2024-07-18T16:51:23Z

Summary

Implement changes needed to make Notebooks report accounting as needed by EOSC Lot 1/3
Modernise the code as well

Changes to the main accounting script:

it changes the configuration and splits the configuration into a [default] section and keeps there most of the stuff that was on [prometheus] which was a mix of everything
the DB is updated with a new field (flavor) which is populated from the egi.eu/flavor annotation
APEL records are not generated by default, needs to be explicitly enabled in the config
default VO is removed, not clear that it was used at all

In the EOSC accounting:

it does not query prometheus directly anymore but does queries to the existing database for VM that have
it can use a from-date and to-date for reporting, by default it will report the previous day
uses client credentials grant instead of refresh

Other changes

it moves to pyproject.toml, we may think on additional tooling for managing the project in the future

Related issue :

WARNING: this will break an upgrade if there is a running DB already, we should prepare some migration of the model to avoid issues.

With the arg in __init__ was making all queries fail

The code will report from the last succesfully completed execution

egi_notebooks_accounting/eosc.py

andrea-manzi · 2024-07-26T12:19:01Z

i'm testing manullay for now, but i see some strange aggregated values being published to Accounting:

DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'urn:geant:eosc-federation.eu:testing:group:eosc#testing.eosc-federation.eu', 'value': 57002.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 190 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-5e869742-24f1-4ae0-a21c-13a64cfe7115', 'value': 7751.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 189 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-d71db0cf-3d77-4f97-976d-8fa5a51fe3ff', 'value': 96.0} to accounting

i have generated the DB from prometheus on fullhub instance using the pods py script

andrea-manzi · 2024-07-26T12:56:04Z

i'm testing manullay for now, but i see some strange aggregated values being published to Accounting:

DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'urn:geant:eosc-federation.eu:testing:group:eosc#testing.eosc-federation.eu', 'value': 57002.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 190 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-5e869742-24f1-4ae0-a21c-13a64cfe7115', 'value': 7751.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 189 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-d71db0cf-3d77-4f97-976d-8fa5a51fe3ff', 'value': 96.0} to accounting

i have generated the DB from prometheus on fullhub instance using the pods py script

could be that this is related to this query https://github.com/EGI-Federation/egi-notebooks-accounting/blob/master/egi_notebooks_accounting/pods.py#L46 which is returning seconds? while we would like to report hours?

andrea-manzi · 2024-07-26T13:25:27Z

i'm testing manullay for now, but i see some strange aggregated values being published to Accounting:
DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'urn:geant:eosc-federation.eu:testing:group:eosc#testing.eosc-federation.eu', 'value': 57002.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 190 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-5e869742-24f1-4ae0-a21c-13a64cfe7115', 'value': 7751.0} to accounting DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443 DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 189 DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-d71db0cf-3d77-4f97-976d-8fa5a51fe3ff', 'value': 96.0} to accounting
i have generated the DB from prometheus on fullhub instance using the pods py script

could be that this is related to this query https://github.com/EGI-Federation/egi-notebooks-accounting/blob/master/egi_notebooks_accounting/pods.py#L46 which is returning seconds? while we would like to report hours?

mm no that value is coming from the pod.wall

enolfc · 2024-07-29T07:52:35Z

mm no that value is coming from the pod.wall

that's also in seconds, so we need to convert to hours

andrea-manzi · 2024-07-29T09:07:49Z

mm no that value is coming from the pod.wall

that's also in seconds, so we need to convert to hours

yes now numbers seems correct!

DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 200
DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-5e869742-24f1-4ae0-a21c-13a64cfe7115', 'value': 2.1530555555555555} to accounting
DEBUG:root:Pushing to accounting - 668bdf9c744c3c7ae7531916
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.acc.staging.eosc.grnet.gr:443
DEBUG:urllib3.connectionpool:https://api.acc.staging.eosc.grnet.gr:443 "POST /accounting-system/installations/668bdf9c744c3c7ae7531916/metrics HTTP/1.1" 201 201
DEBUG:root:Sending metric {'metric_definition_id': '668bdd5988e1d617b217ecb9', 'time_period_start': '2024-07-25T00:00:00Z', 'time_period_end': '2024-07-25T23:59:00Z', 'user': '[email protected]', 'group': 'pp-d71db0cf-3d77-4f97-976d-8fa5a51fe3ff', 'value': 0.02666666666666667} to accounting

@valtri do you think you can test it on the testing JHub instance ?

andrea-manzi

Tested successfully by CESNET

enolfc added 26 commits July 18, 2024 17:49

Remove shebang

f3e25f3

Bring this to more modern packaging

1b9cf89

Make version discoverable from git

20e52f8

Remove version

4d328d8

Move deps outside

9b41733

Better build

d97ee36

Add flavor to the model

bb5eb63

WARNING: this will break an upgrade if there is a running DB already, we should prepare some migration of the model to avoid issues.

Remove shebang

0092b11

No executables needed here

d4d935e

Minor linting fixes

fd98563

Removed value (apparently not used?)

ee537c2

Fix the model

9e3141f

With the arg in __init__ was making all queries fail

First attempt to get EOSC accounting

e0ef8fd

Fix EOSC accounting config

5cec580

Introduce dry-run for testing

21094ab

Remove shebang

b4699f4

Reorganise configuration

184931a

Adapt helm to new code

288bbe0

Allow for setting from/to dates

51dd95c

Add shared volume to the eosc job

cba1332

Self is not a thing here

96c9c85

Add debug

c675c56

Linting fixes

1fae951

Linting

295cec5

even more linting

609809a

Fix this again

5bdbe9b

enolfc marked this pull request as ready for review July 23, 2024 11:25

enolfc added 2 commits July 23, 2024 17:49

Add timestamp file for reporting

e96cbc0

The code will report from the last succesfully completed execution

Add timestamp to the helm chart

0efb1ff

andrea-manzi reviewed Jul 25, 2024

View reviewed changes

egi_notebooks_accounting/eosc.py Outdated Show resolved Hide resolved

andrea-manzi added 3 commits July 25, 2024 21:42

add missing method parameters

3f1e9af

running black

dcdc7a4

fix linter error

27c5614

Report hours, not seconds

482039b

enolfc requested a review from valtri August 19, 2024 11:49

enolfc force-pushed the eosc-accounting branch from cc0c16a to 482039b Compare August 21, 2024 11:35

andrea-manzi approved these changes Aug 21, 2024

View reviewed changes

andrea-manzi merged commit 5f048a6 into EGI-Federation:master Aug 21, 2024
2 checks passed

enolfc deleted the eosc-accounting branch August 21, 2024 11:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EOSC Accounting #17

EOSC Accounting #17

enolfc commented Jul 18, 2024 •

edited

Loading

andrea-manzi commented Jul 26, 2024

andrea-manzi commented Jul 26, 2024

andrea-manzi commented Jul 26, 2024

enolfc commented Jul 29, 2024

andrea-manzi commented Jul 29, 2024

andrea-manzi left a comment

EOSC Accounting #17

EOSC Accounting #17

Conversation

enolfc commented Jul 18, 2024 • edited Loading

Summary

andrea-manzi commented Jul 26, 2024

andrea-manzi commented Jul 26, 2024

andrea-manzi commented Jul 26, 2024

enolfc commented Jul 29, 2024

andrea-manzi commented Jul 29, 2024

andrea-manzi left a comment

Choose a reason for hiding this comment

enolfc commented Jul 18, 2024 •

edited

Loading