SWATCH-2358: Only meter ELS 3rd party converted PAYG instances

[lindseyburnett]

Jira issue: SWATCH-2358

Description

We want to differentiate and count ELS PAYG instances separately - depending on if they were 3rd party migrated or not.

To support this, a new field was added to skus, "SPECIAL_PRICING_FLAG". When that field contains the flag "MIGRATION_OFFERING", instances associated with that offering will be considered 3rd party migrated.

This PR does a handful of things:

1. Add support for the field isMigrationProduct in the swatch-product-configuration.
2. When looking up a product tag during Offering sync, consider the new field.
3. When looking up a product tag during CostMgmt event ingestion, consider the new field.
4. When looking up a product tag during Rhelemeter metric ingestion, consider the new field.

This PR effectively disables support for non-migration ELS PAYG being metered in swatch. SWATCH-2401 re-enables ELS paygo non-migration support.

Testing

IQE MR: https://gitlab.cee.redhat.com/insights-qe/iqe-rhsm-subscriptions-plugin/-/merge_requests/666

For offering and capacity:

1. Create offering
2. Check for producttag by sku
3. Create subscription
4. Check for capacity

Setup

1. Run the core app:
   RHSM_SUBSCRIPTIONS_ENABLE_SYNCHRONOUS_OPERATIONS=true DEV_MODE=true PROM_URL=http://localhost:8082/api/metrics/v1/telemeter/api/v1 SWATCH_CONTRACTS_INTERNAL_SERVICE=http://localhost:8882 SERVER_PORT=8000 SPRING_PROFILES_ACTIVE=worker,kafka-queue,api,capacity-ingress,rh-marketplace,kafka-queue,rhsm-conduit ./gradlew clean :bootRun

Steps and Verification for Offering and Capacity

1. app.rhsm_subscriptions.sync_offering(sku="RH02781MO")
2. app.rhsm_subscriptions.get_product_tags_from_skus('RH02781MO')

Response Verification:
rhel-for-x86-els-payg

Offering table:
RH02781MO,RHEL Server,RHEL,,,,,,Premium,Production,"Red Hat Enterprise Linux for Third Party Linux Migration with Extended Life Cycle Support (On-Demand, Monthly Production Support)",false,,true,MIGRATION_OFFERING
sku_product_tag:
RH02781MO,rhel-for-x86-els-payg

3. from smqe_tools import internal_api_add_sku
4. internal_api_add_sku(sku="RH02781MO", org_id=app.user.identity.org_id, start_day="2024-01-01", end_day="2024-06-30", billing_provider="aws", billing_account_id="9458475893")
9. app.rhsm_subscriptions.get_skus_report(product_id="rhel-for-x86-els-payg")

Response Verification:

[{'billing_provider': 'aws',
  'capacity': 0,
  'has_infinite_quantity': False,
  'hypervisor_capacity': 0,
  'next_event_date': datetime.datetime(2024, 6, 30, 0, 17, 2, 462000, tzinfo=tzlocal()),
  'next_event_type': 'Subscription End',
  'product_name': 'Red Hat Enterprise Linux for Third Party Linux Migration '
                  'with Extended Life Cycle Support (On-Demand, Monthly '
                  'Production Support)',
  'quantity': 1,
  'service_level': 'Premium',
  'sku': 'RH02781MO',
  'subscriptions': [{'id': '300782', 'number': '5299628'}],
  'total_capacity': 0,
  'usage': 'Production'}]

For rhelemeter instances

Using the swatch-support-scripts repo from gitlab, start a token refresher for stage rhelemeter. You'll need to set VAULT_TOKEN first.

./fetch-vault-credentials
./rhelemeter-stage

Start swatch-metrics with rhelemeter as the event source

EVENT_SOURCE=rhelemeter PROM_URL="http://localhost:8082/api/v1" ./gradlew :swatch-metrics:quarkusDev

Kick off metering for rhel-for-x86-els-payg product tag with an orgId that has els metrics with conversions_success label set to true.

curl -X 'POST' \
  'http://localhost:8000/api/swatch-metrics/v1/internal/metering/rhel-for-x86-els-payg?orgId=16787820' \
  -H 'accept: */*' \
  -H 'x-rh-swatch-psk: placeholder' \
  -d ''

Verify that the Event message that got put onto the service-instance-ingress topic has the "conversion":true attribute and the product_tag field gets cleared out. This is to force that event ingestion looks up the proper product tag, taking the conversion status into consideration.

http://localhost:3030/#/cluster/default/topic/n/platform.rhsm-subscriptions.service-instance-ingress/

  {
    "topic": "platform.rhsm-subscriptions.service-instance-ingress",
    "key": 16787820,
    "value": {
      "service_type": "RHEL System",
      "timestamp": "2024-05-03T14:00:00Z",
      "expiration": "2024-05-03T15:00:00Z",
      "display_name": "aws1.host-metering.test",
      "measurements": [
        {
          "value": 1,
          "uom": "vCPUs",
          "metric_id": "vCPUs"
        }
      ],
      "product_ids": [
        "204",
        "69"
      ],
      "sla": "Premium",
      "usage": "Production",
      "billing_provider": "aws",
      "billing_account_id": "customerAccount16787820",
      "product_tag": [""],
      "conversion": true,
      "event_source": "rhelemeter",
      "event_type": "snapshot_rhel-for-x86-els-payg_vcpus",
      "org_id": "16787820",
      "instance_id": "4cfd68a6-a0b7-4590-b9c0-068a5936765b",
      "metering_batch_id": "c050e9f1-fc0b-44b0-bd3d-3e469fc4e0b8"""
    },
    "partition": 0,
    "offset": 7
  }

There isn't data in stage to test when the label is missing or set to false. You can manually modify the payload and put a message on the topic though. The key should match the org_id and the value is the json.

[ntkathole]

/retest

[lindseyburnett]

/retest

[Aurobinda55]

/retest

[ntkathole]

/retest

[ntkathole]

Two test failures on IQE are related to:

1. SKU in stage do not have special migration offering attribute. (raised a request to get it updated, I think meanwhile, we should skip this test ?)
2. conversion true support is not yet in IQE, added in https://gitlab.cee.redhat.com/insights-qe/iqe-rhsm-subscriptions-plugin/-/merge_requests/697

I am still verifying the functionality, should finish by tomorrow.

[ntkathole]

@lindseyburnett I am seeing if event has product_tag for rhel-for-x86-els-payg and even if conversion is False, host is considered in rhel-for-x86-els-payg. I expected host to not present in rhel-for-x86-els-payg.

Reproducer :

Create events as below, sync hourly tally and see instances api showing host even if conversion is False.

{
'event_source': 'cost-management',
  'event_type': 'snapshot_vCPUs',
  'org_id': '3340851',
  'instance_id': 'd00a9d66-3f7e-400d-b9bb-77b33fcd66b5',
  'event_id': '340fe3e7-814a-4c91-b6f3-a5ef3c9bf2ed',
  'service_type': 'RHEL System',
  'timestamp': '2024-05-17T10:00:00Z',
  'record_date': '2024-05-17T11:29:48.808034155Z',
  'expiration': '2024-05-17T11:00:00Z',
  'display_name': 'automation__cluster_d00a9d66-3f7e-400d-b9bb-77b33fcd66b5',
  'measurements': [{'value': 1.0, 'uom': 'vCPUs'}],
  'cloud_provider': 'AWS',
  'hardware_type': 'Cloud',
  'product_ids': ['204'],
  'sla': 'Premium',
  'usage': 'Production',
  'billing_provider': 'aws',
  'product_tag': ['rhel-for-x86-els-payg'],
  'conversion': False
}

[ntkathole]

/retest

[lindseyburnett]

Fixed this by adding product tag verification to our "event validation" check at the very beginning of ingestion. If data in the event doesn't align with a product tag, a warning message appears in the logs and validation fails. If the event has a product tag on it, but the other data doesn't match the configured product tag, the same error will happen.

Events that fail validation don't get saved to the events table in the database.

@lindseyburnett I am seeing if event has product_tag for rhel-for-x86-els-payg and even if conversion is False, host is considered in rhel-for-x86-els-payg. I expected host to not present in rhel-for-x86-els-payg.

Reproducer :

Create events as below, sync hourly tally and see instances api showing host even if conversion is False.

[ntkathole]

Looks good 👍
Related automation changes in https://gitlab.cee.redhat.com/insights-qe/iqe-rhsm-subscriptions-plugin/-/merge_requests/697