SWATCH-3177 Fix invalid remittance calculation when an amendment is applied #4033

mstead · 2024-12-10T18:44:03Z

Description

This bug happens when the incoming events are in conflict (2 events with the same instance, metric and timestamp) and the hardware measurement type (HMT) is different. In the case of the reproducer events, the rhelemeter event does not specify the HMT which will get defaulted to PHYSICAL, not AWS. Since remittance uses the HMT to determine the current total remittance, it returns an incorrect value for the total when the usage for the second event is received.

This only happens when a conflict occurs and the value changes, since the measurements for the hour are replaced when tallied.

Resolution

When determining the current remittance total, the billable usage
component no longer considers the hardware_measurement_type. As
such, when TallySummary messages are created during the hourly tally,
we must also exclude the measurement type when calculating the
current total for a metric, as this value is used when determining
remittance.

This PR also removes the hardware measurement type from billable usage
and remittance since it does not apply to remittance in general.

Affected Components

swatch-billable-usage

The currentTotal of TallySummary.TallyMeasurement are now calculated to not include the hardware_type.
The billable_usage_remittance table no longer has the hardware_measurement_type.

All Swatch Billing Producers

Incoming BillableUsage messages no longer have the hardwareMeasurementType included.

Testing

IQE Test MR: IQE TEST

Steps

Deployment Tips

When testing locally, I deployed the following services. Please read the test steps carefully, since the first deployment should be done on the MAIN branch.

swatch-contracts

QUARKUS_MANAGEMENT_ENABLED=false SERVER_PORT=8001 ./gradlew :swatch-contracts:quarkusDev

swatch-billable-usage

SERVER_PORT=8002 QUARKUS_MANAGEMENT_PORT=9004 DEV_MODE=true ./gradlew :swatch-billable-usage:quarkusDev

swatch-tally

SPRING_PROFILES_ACTIVE=api,worker,kafka-queue DEV_MODE=true ./gradlew clean :bootRun

Testing

First, reproduce the bug on the MAIN branch.

Create a cost-management event for a host instance that includes the hardware_type.

http :9080/topics/platform.rhsm-subscriptions.service-instance-ingress Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json --raw '{
        "records": [
          {
            "key": "3340851",
            "value": {
              "service_type": "RHEL System",
              "timestamp": "2024-06-03T14:00:00Z",
              "expiration": "2024-07-03T15:00:00Z",
              "display_name": "automation__cluster_testHostgge",
              "measurements": [
                {
                  "value": 4,
                  "uom": "vCPUs",
                  "metric_id": "vCPUs"
                }
              ],
              "product_ids": [
                "204"
              ],
              "cloud_provider": "AWS",
              "hardware_type": "Cloud",
              "sla": "Premium",
              "usage": "Production",
              "billing_provider": "aws",
              "billing_account_id": "testkfk",
              "product_tag": [
                "rhel-for-x86-els-payg-addon"
              ],
              "conversion": false,
              "event_source": "cost-management",
              "event_type": "vCPUs",
              "org_id": "3340851",
              "instance_id": "testHostgge"
            }
          }
        ]
      }
	  '

Perform an hourly tally for the org.

http POST ":8000/api/rhsm-subscriptions/v1/internal/tally/hourly?org=3340851" x-rh-swatch-psk:placeholder

Verify 4 AWS vCPUs were remitted.

select * from billable_usage_remittance where org_id='3340851';

Create a rhelemeter style event for the SAME HOST (no cloud_provider or hardware type).

http :9080/topics/platform.rhsm-subscriptions.service-instance-ingress Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json --raw '{
        "records": [
          {
            "key": "3340851",
            "value": {
              "service_type": "RHEL System",
              "timestamp": "2024-06-03T14:00:00Z",
              "expiration": "2024-07-03T15:00:00Z",
              "display_name": "automation__cluster_testHostgge",
              "measurements": [
                {
                  "value": 8,
                  "uom": "vCPUs",
                  "metric_id": "vCPUs"
                }
              ],
              "product_ids": [
                "204"
              ],
              "sla": "Premium",
              "usage": "Production",
              "billing_provider": "aws",
              "billing_account_id": "testkfk",
              "product_tag": [
                "rhel-for-x86-els-payg-addon"
              ],
              "conversion": false,
              "event_source": "rhelemeter",
              "event_type": "vCPUs",
              "org_id": "3340851",
              "instance_id": "testHostgge"
            }
          }
        ]
      }
	  '

Run an hourly tally for the org.

http POST ":8000/api/rhsm-subscriptions/v1/internal/tally/hourly?org=3340851" x-rh-swatch-psk:placeholder

Because the event should have been amended (4 - 4) + 8, the expected billable usage should have been and additional 4 vCPUs since we already billed for 4 (from the first event). HOWEVER, you'll notice that there were 8 additional vCPUs billed due to the PHYSICAL hardware measurement type. We have overbilled by 4 vCPUs.

select * from billable_usage_remittance where org_id='3340851';

Check out the branch with the bug fix and deploy. After deploying, check the billable_usage_remittance table to make sure that the hardware_measurement_type column was dropped.

Next verify that we will recover correctly from the 4 additional vCPUs that we previously billed for.

http :9080/topics/platform.rhsm-subscriptions.service-instance-ingress Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json --raw '{
        "records": [
          {
            "key": "3340851",
            "value": {
              "service_type": "RHEL System",
              "timestamp": "2024-06-03T15:00:00Z",
              "expiration": "2024-07-03T15:00:00Z",
              "display_name": "automation__cluster_testHostgge",
              "measurements": [
                {
                  "value": 10,
                  "uom": "vCPUs",
                  "metric_id": "vCPUs"
                }
              ],
              "product_ids": [
                "204"
              ],
              "cloud_provider": "AWS",
              "hardware_type": "Cloud",
              "sla": "Premium",
              "usage": "Production",
              "billing_provider": "aws",
              "billing_account_id": "testkfk",
              "product_tag": [
                "rhel-for-x86-els-payg-addon"
              ],
              "conversion": false,
              "event_source": "cost-management",
              "event_type": "vCPUs",
              "org_id": "3340851",
              "instance_id": "testHostgge"
            }
          }
        ]
      }
	  '

Perform an hourly tally for the org.

http POST ":8000/api/rhsm-subscriptions/v1/internal/tally/hourly?org=3340851" x-rh-swatch-psk:placeholder

Since an additional 10 vCPUs were reported in the event we just sent, and we've previously overbilled for 4 vCPUs due to the bug, after the tally we should see that we billed for 6 vCPUs.

Next, attempt to reproduce the bug scenario WITH A DIFFERENT MONTH to ensure the bug is addressed when starting fresh.

Create a cost-management event for a host instance that includes the hardware_type.

http :9080/topics/platform.rhsm-subscriptions.service-instance-ingress Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json --raw '{
        "records": [
          {
            "key": "3340851",
            "value": {
              "service_type": "RHEL System",
              "timestamp": "2024-07-03T14:00:00Z",
              "expiration": "2024-08-03T15:00:00Z",
              "display_name": "automation__cluster_testHostgge",
              "measurements": [
                {
                  "value": 4,
                  "uom": "vCPUs",
                  "metric_id": "vCPUs"
                }
              ],
              "product_ids": [
                "204"
              ],
              "cloud_provider": "AWS",
              "hardware_type": "Cloud",
              "sla": "Premium",
              "usage": "Production",
              "billing_provider": "aws",
              "billing_account_id": "testkfk",
              "product_tag": [
                "rhel-for-x86-els-payg-addon"
              ],
              "conversion": false,
              "event_source": "cost-management",
              "event_type": "vCPUs",
              "org_id": "3340851",
              "instance_id": "testHostgge"
            }
          }
        ]
      }
	  '

Perform an hourly tally for the org.

http POST ":8000/api/rhsm-subscriptions/v1/internal/tally/hourly?org=3340851" x-rh-swatch-psk:placeholder

Create a rhelemeter style event for the SAME HOST (no cloud_provider or hardware type).

http :9080/topics/platform.rhsm-subscriptions.service-instance-ingress Content-Type:application/vnd.kafka.json.v2+json Accept:application/vnd.kafka.v2+json --raw '{
        "records": [
          {
            "key": "3340851",
            "value": {
              "service_type": "RHEL System",
              "timestamp": "2024-07-03T14:00:00Z",
              "expiration": "2024-08-03T15:00:00Z",
              "display_name": "automation__cluster_testHostgge",
              "measurements": [
                {
                  "value": 8,
                  "uom": "vCPUs",
                  "metric_id": "vCPUs"
                }
              ],
              "product_ids": [
                "204"
              ],
              "sla": "Premium",
              "usage": "Production",
              "billing_provider": "aws",
              "billing_account_id": "testkfk",
              "product_tag": [
                "rhel-for-x86-els-payg-addon"
              ],
              "conversion": false,
              "event_source": "rhelemeter",
              "event_type": "vCPUs",
              "org_id": "3340851",
              "instance_id": "testHostgge"
            }
          }
        ]
      }
	  '

Run an hourly tally for the org.

http POST ":8000/api/rhsm-subscriptions/v1/internal/tally/hourly?org=3340851" x-rh-swatch-psk:placeholder

Check the remittance for 2024-07 to ensure that we did not overbill by 4 vCPUs. Since an amendment had occurred when the 2nd event was received, we should have only remitted an additional 4 vCPUs, for a total of 8 vCPUs for 2024-07.

liwalker-rh · 2024-12-11T20:44:22Z

/retest

Aurobinda55 · 2024-12-12T15:25:34Z

/retest

kflahert · 2024-12-12T20:33:07Z

/retest

Aurobinda55 · 2024-12-13T15:22:06Z

Verification steps :

On the main branch

Deployed EE,
Created a rhel-payg-addon event with event source cost management, value =2,instance-uuid='rhel_instance',billing_account_id='test-123'
sync hourly tally
Again, created another rhel-payg-addon event with event source=rhelemeter , value =7,instance-uuid='rhel_instance',billing_account_id='test-123', timestamp=same as above
sync hourly tally
conflicts happened(verified)
Checked the remittance record and it showed two records,

  Out[27]: 
[{'orgId': '3340851',
  'productId': 'rhel-for-x86-els-payg-addon',
  'metricId': 'vCPUs',
  'billingProvider': 'aws',
  'billingAccountId': 'test-123',
  'remittedValue': 2.0,
  'accumulationPeriod': '2024-12',
  'remittanceDate': '2024-12-13T13:23:55.771941Z',
  'remittanceStatus': 'succeeded',
  'remittanceErrorCode': 'null'},
 {'orgId': '3340851',
  'productId': 'rhel-for-x86-els-payg-addon',
  'metricId': 'vCPUs',
  'billingProvider': 'aws',
  'billingAccountId': 'test-123',
  'remittedValue': 7.0,
  'accumulationPeriod': '2024-12',
  'remittanceDate': '2024-12-13T13:25:43.53135Z',
  'remittanceStatus': 'pending',
  'remittanceErrorCode': 'null'}]

Also, I checked the billable usage message has hardware type=AWS/Cloud

Switched to the PR branch and deployed the PR image

Created one more events with event source =cost-management, timestampe=same as previous event, value=15,instance-id=same as previous
sync tally hourly
Conflict happened
Fetched the remittance records
Verified hardware measurement type removed from the existing record and new record
There are two records in the remittance table, 1st record with remitted value 6, status Pending
2nd record with value 9 status succeeded

[{'orgId': '3340851',
  'productId': 'rhel-for-x86-els-payg-addon',
  'metricId': 'vCPUs',
  'billingProvider': 'aws',
  'billingAccountId': 'test-123',
  'remittedValue': 6.0,
  'accumulationPeriod': '2024-12',
  'remittanceDate': '2024-12-13T13:49:05.750887Z',
  'remittanceStatus': 'pending',
  'remittanceErrorCode': 'null'},
 {'orgId': '3340851',
  'productId': 'rhel-for-x86-els-payg-addon',
  'metricId': 'vCPUs',
  'billingProvider': 'aws',
  'billingAccountId': 'test-123',
  'remittedValue': 9.0,
  'accumulationPeriod': '2024-12',
  'remittanceDate': '2024-12-13T13:25:43.53135Z',
  'remittanceStatus': 'succeeded',
  'remittanceErrorCode': 'null'}]

@mstead I think this is expected here because already 9 were sent(due to the bug in the main branch), and now 15, so 15-9=6 in the pending state? but need your confirmation here .

Again, Created one more event with event source=rhelemeter, timestampe=same as previous event, instanc-id=same as previous, value=2
sync tally hourly
Fetch remittance record
Now, I see one record status =succeeded, with a value of 15 and the latest value of 2 ignored

[{'orgId': '3340851',
  'productId': 'rhel-for-x86-els-payg-addon',
  'metricId': 'vCPUs',
  'billingProvider': 'aws',
  'billingAccountId': 'test-123',
  'remittedValue': 15.0,
  'accumulationPeriod': '2024-12',
  'remittanceDate': '2024-12-13T13:49:05.750887Z',
  'remittanceStatus': 'succeeded',
  'remittanceErrorCode': 'null'}]

Now checked for the previous month's scenario

Created two events having event source= cost-management, rhelmeter, same timestamp, same instance id, value 2 and 7 for each event
sync hourly tally
Now I see, two records in the remittance table 1st record value=5 pending 2nd record value=2 status=succeeded

{
       "accumulationPeriod": "2024-11",
       "billingAccountId": "test-123",
       "billingProvider": "aws",
       "metricId": "vCPUs",
       "orgId": "3340851",
       "productId": "rhel-for-x86-els-payg-addon",
       "remittanceDate": "2024-12-13T14:32:09.255147Z",
       "remittanceErrorCode": "null",
       "remittanceStatus": "pending",
       "remittedValue": 5.0
   },
   {
       "accumulationPeriod": "2024-11",
       "billingAccountId": "test-123",
       "billingProvider": "aws",
       "metricId": "vCPUs",
       "orgId": "3340851",
       "productId": "rhel-for-x86-els-payg-addon",
       "remittanceDate": "2024-12-13T14:28:10.887495Z",
       "remittanceErrorCode": "null",
       "remittanceStatus": "succeeded",
       "remittedValue": 2.0
   },

@mstead I think this is expected here, because the latest value 7 adjusted with the first event value which is 2 ?
Also checked, the remittance records have no HMT ,

account_number          | 
org_id                  | 3340851
product_id              | rhel-for-x86-els-payg-addon
metric_id               | vCPUs
accumulation_period     | 2024-11
sla                     | Premium
usage                   | Production
billing_provider        | aws
billing_account_id      | test-123
remitted_pending_value  | 2
remittance_pending_date | 2024-12-13 14:28:10.887495+00
retry_after             | 
tally_id                | 8adf1a76-ae29-4a46-817a-61d7bd034ff5
uuid                    | 3faadcbb-8789-4e64-8c80-1c7b64b8a4be
billed_on               | 2024-12-13 14:32:09.412855+00
error_code              | 
status                  | succeeded

Checked on the swatch-billable-usage-service and can't see the HMT in the message.

mstead · 2024-12-13T15:43:04Z

@mstead I think this is expected here because already 9 were sent(due to the bug in the main branch), and now 15, so 15-9=6 in the pending state? but need your confirmation here .

Yes, that is correct.

@mstead I think this is expected here, because the latest value 7 adjusted with the first event value which is 2 ?

Yes, this is also correct.

Aurobinda55 · 2024-12-16T15:00:18Z

/retest

Aurobinda55 · 2024-12-16T16:03:27Z

There are two failed tests in this PR due to the absence of the hardware_measurement_type column, which has been removed in this PR. I see that there is already a PR on the IQE side to handle the changes: commit link. However, due to an issue while releasing the tag in the iqe-rhsm plugin (v2024.12.16.0 tag), the CI job is unable to pull the latest image and fails with the same error. I am currently working on identifying the root cause of the failure.

wottop · 2024-12-16T16:35:24Z

lgtm, but I would like to have another person approve the logic.

liwalker-rh · 2024-12-17T21:00:23Z

/retest

liwalker-rh · 2024-12-18T16:41:35Z

/retest

liwalker-rh · 2024-12-18T19:44:44Z

These two tests have failed multiple times now in PR checks for this PR:

test_multiple_tally_summaries_aggregated: https://ci.ext.devshift.net/job/RedHatInsights-rhsm-subscriptions-pr-check/9344/testReport/junit/tests.component.swatch_billable_usage/test_swatch_billable_usage/test_multiple_tally_summaries_aggregated/
test_multiple_tally_summaries_not_aggregated_for_different_billing_account_ids: https://ci.ext.devshift.net/job/RedHatInsights-rhsm-subscriptions-pr-check/9344/testReport/junit/tests.component.swatch_billable_usage/test_swatch_billable_usage/test_multiple_tally_summaries_not_aggregated_for_different_billing_account_ids/

Is this an issue or is this expected behavior and the tests need to be updated?

liwalker-rh · 2024-12-18T20:21:21Z

/retest

kflahert · 2024-12-18T21:19:46Z

@liwalker-rh I just wrote these and I think its some timing issues I need to fix. I'll work on an IQE fix.

liwalker-rh · 2024-12-19T23:07:48Z

/retest

liwalker-rh · 2025-01-02T17:15:03Z

/retest

liwalker-rh · 2025-01-03T14:15:21Z

/retest

liwalker-rh

Looks good in testing and the related IQE tests are working as expected. Will merge those tests once this has been merged.

mstead · 2025-01-07T14:15:40Z

/retest

liwalker-rh · 2025-01-07T15:58:41Z

/retest

liwalker-rh · 2025-01-08T13:02:55Z

/retest

lindseyburnett

Unapproving pending failing test investigation

liwalker-rh · 2025-01-08T17:34:13Z

/retest

liwalker-rh · 2025-01-08T22:04:31Z

/retest

When determining the current remittance total, the billable usage component no longer considers the hardware_measurement_type. As such, when TallySummary messages are created during the hourly tally, we must also exclude the measurement type when calculating the current total for a metric, as this value is used when determining remittance.

This patch removes the hardware measurement type from billable usage and remittance since it does not apply to remittance in general.

mstead · 2025-01-10T18:42:28Z

/retest

liwalker-rh · 2025-01-13T16:19:52Z

/retest

mstead requested review from lindseyburnett and liwalker-rh December 10, 2024 18:44

Aurobinda55 self-requested a review December 13, 2024 14:07

lindseyburnett self-assigned this Dec 16, 2024

kahowell added Dev Pull requests that need developer review QE Pull request should be approved by QE before merge labels Dec 16, 2024

kahowell assigned lindseyburnett and liwalker-rh and unassigned lindseyburnett Dec 16, 2024

mstead force-pushed the mstead/SWATCH-3177-amendment-remittance-issue branch from 3a34bf2 to 34aedfc Compare December 16, 2024 20:10

liwalker-rh approved these changes Jan 6, 2025

View reviewed changes

liwalker-rh added QE/approved Pull requests that have been approved by all assigned QEs and removed QE Pull request should be approved by QE before merge labels Jan 6, 2025

lindseyburnett approved these changes Jan 7, 2025

View reviewed changes

lindseyburnett added Dev/approved Pull requests that have been approved by all assigned developers and removed Dev Pull requests that need developer review labels Jan 7, 2025

lindseyburnett added QE Pull request should be approved by QE before merge Dev Pull requests that need developer review and removed Dev/approved Pull requests that have been approved by all assigned developers QE/approved Pull requests that have been approved by all assigned QEs labels Jan 8, 2025

lindseyburnett requested changes Jan 8, 2025

View reviewed changes

liwalker-rh self-requested a review January 8, 2025 17:34

Sgitario force-pushed the mstead/SWATCH-3177-amendment-remittance-issue branch from 34aedfc to 4b114c9 Compare January 9, 2025 14:32

mstead added 2 commits January 10, 2025 11:30

Remove hardware measurement type from billable usage and remittance

85a18a3

This patch removes the hardware measurement type from billable usage and remittance since it does not apply to remittance in general.

mstead force-pushed the mstead/SWATCH-3177-amendment-remittance-issue branch from 4b114c9 to 85a18a3 Compare January 10, 2025 15:40

liwalker-rh approved these changes Jan 14, 2025

View reviewed changes

liwalker-rh added QE/approved Pull requests that have been approved by all assigned QEs and removed QE Pull request should be approved by QE before merge labels Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SWATCH-3177 Fix invalid remittance calculation when an amendment is applied #4033

SWATCH-3177 Fix invalid remittance calculation when an amendment is applied #4033

mstead commented Dec 10, 2024

liwalker-rh commented Dec 11, 2024

Aurobinda55 commented Dec 12, 2024

kflahert commented Dec 12, 2024

Aurobinda55 commented Dec 13, 2024

mstead commented Dec 13, 2024

Aurobinda55 commented Dec 16, 2024

Aurobinda55 commented Dec 16, 2024 •

edited

Loading

wottop commented Dec 16, 2024

liwalker-rh commented Dec 17, 2024

liwalker-rh commented Dec 18, 2024

liwalker-rh commented Dec 18, 2024

liwalker-rh commented Dec 18, 2024

kflahert commented Dec 18, 2024

liwalker-rh commented Dec 19, 2024

liwalker-rh commented Jan 2, 2025

liwalker-rh commented Jan 3, 2025

liwalker-rh left a comment

mstead commented Jan 7, 2025

liwalker-rh commented Jan 7, 2025

liwalker-rh commented Jan 8, 2025

lindseyburnett left a comment

liwalker-rh commented Jan 8, 2025

liwalker-rh commented Jan 8, 2025

mstead commented Jan 10, 2025

liwalker-rh commented Jan 13, 2025

SWATCH-3177 Fix invalid remittance calculation when an amendment is applied #4033

Are you sure you want to change the base?

SWATCH-3177 Fix invalid remittance calculation when an amendment is applied #4033

Conversation

mstead commented Dec 10, 2024

Description

Resolution

Affected Components

Testing

Steps

Deployment Tips

Testing

liwalker-rh commented Dec 11, 2024

Aurobinda55 commented Dec 12, 2024

kflahert commented Dec 12, 2024

Aurobinda55 commented Dec 13, 2024

mstead commented Dec 13, 2024

Aurobinda55 commented Dec 16, 2024

Aurobinda55 commented Dec 16, 2024 • edited Loading

wottop commented Dec 16, 2024

liwalker-rh commented Dec 17, 2024

liwalker-rh commented Dec 18, 2024

liwalker-rh commented Dec 18, 2024

liwalker-rh commented Dec 18, 2024

kflahert commented Dec 18, 2024

liwalker-rh commented Dec 19, 2024

liwalker-rh commented Jan 2, 2025

liwalker-rh commented Jan 3, 2025

liwalker-rh left a comment

Choose a reason for hiding this comment

mstead commented Jan 7, 2025

liwalker-rh commented Jan 7, 2025

liwalker-rh commented Jan 8, 2025

lindseyburnett left a comment

Choose a reason for hiding this comment

liwalker-rh commented Jan 8, 2025

liwalker-rh commented Jan 8, 2025

mstead commented Jan 10, 2025

liwalker-rh commented Jan 13, 2025

Aurobinda55 commented Dec 16, 2024 •

edited

Loading