[Subscriptions Release] - pre tag commit: 'rhsm-subscriptions-1.0.83'.

README.md

@@ -1,6 +1,78 @@
-# Local Deployment
+# Subscription watch
 
-## Prerequisites
+Subscription watch tracks usage and capacity at an account-level.
+Account-level reporting means that subscriptions are not directly associated to machines,
+containers, or service instances.
+
+![Context diagram for Subscription watch](docs/context.svg)
+
+Subscription watch can be thought of as several services that provide related functionality:
+
+<details>
+<summary>system conduit</summary>
+Service that syncs system data from Hosted Candlepin into HBI.
+
+![Container diagram for system conduit](docs/container-system-conduit.svg)
+</details>
+
+<details>
+<summary>metrics ingress</summary>
+Services that sync system/instance telemetry data into Subscription watch.
+
+![Container diagram for Metrics Ingress](docs/container-metrics-ingress.svg)
+</details>
+
+<details>
+<summary>tally</summary>
+Service that tallies system usage based on telemetry data from various sources.
+
+![Container diagram for Tally](docs/container-tally.svg)
+</details>
+
+<details>
+<summary>subscription sync</summary>
+Service that syncs subscription/offering data from RH IT services.
+
+![Container diagram for Subscription Sync](docs/container-subscription-sync.svg)
+</details>
+
+<details>
+<summary>API/UI</summary>
+Customer facing views of the usage and capacity data.
+
+![Container diagram for API/UI](docs/container-ui.svg)
+</details>
+
+<details>
+<summary>billing usage notification</summary>
+Services that notify billing services of hourly usage.
+
+![Container diagram for Billing Producers](docs/container-billing.svg)
+</details>
+
+Networking diagrams show how requests are routed:
+
+<details>
+<summary>Customer-facing API</summary>
+
+![Networking diagram for customer API](docs/networking-public-api.svg)
+</details>
+
+<details>
+<summary>Admin/Internal API</summary>
+
+![Networking diagram for internal APIs](docs/networking-jolokia-api.svg)
+</details>
+
+## Deployment
+
+There are currently 3 different ways to deploy the components, with running them locally as the 
+preferred development workflow.
+
+<details>
+<summary>Local Development</summary>
+
+### Prerequisites
 
 First, ensure you have podman-compose, podman and java 11 installed:
 
@@ -16,7 +88,7 @@ Ensure the checkout has the HBI submodule initialized:
 git submodule update --init --recursive
 ```
 
-## Dependent services
+### Dependent services
 
 *NOTE*: To run any of the following commands using docker, 
 
@@ -54,10 +126,24 @@ manually start containers for the services you wish to deploy locally.
 
 If you prefer to use local postgresql service, you can use `init_dbs.sh`.
 
-## Build and Run rhsm-subscriptions
+### Kafka
+
+`podman-compose` deploys a kafka instance w/ a UI at http://localhost:3030
+
+Two environment variables can be used to manipulate the offsets of the kafka
+consumers:
+
+- `KAFKA_SEEK_OVERRIDE_END` when set to `true` seeks to the very end
+- `KAFKA_SEEK_OVERRIDE_TIMESTAMP` when set to an OffsetDateTime, seeks the
+  queue to this position.
+
+These changes are permanent, committed the next time the kafka consumer is detected
+as idle.
+
+### Build and Run rhsm-subscriptions
 
 ```
-./gradlew bootRun
+./gradlew :bootRun
 ```
 
 Spring Boot [defines many properties](https://docs.spring.io/spring-boot/docs/2.3.4.RELEASE/reference/htmlsingle/#common-application-properties)
@@ -70,10 +156,10 @@ We also define a number of service-specific properties (see [Environment Variabl
 For example, the `server.port` (or `SERVER_PORT` env var) property changes the listening port:
 
 ```
-SERVER_PORT=9090 ./gradlew bootRun
+SERVER_PORT=9090 ./gradlew :bootRun
 ```
 
-## Profiles
+### Profiles
 
 We have a number of profiles. Each profile activates a subset of components in the codebase.
 
@@ -101,7 +187,7 @@ Each profile has a `@Configuration` class that controls which components get act
 
 If no profiles are specified, the default profiles list in `application.yaml` is applied.
 
-## Deployment Notes
+### Deployment Notes
 
 RHSM Subscriptions is meant to be deployed under the context path "/". The
 location of app specific resources are then controlled by the
@@ -114,9 +200,14 @@ can vary based on an environment variable given to the pod.
 
 ### Static Endpoints
 
-* /actuator/health - A Spring Actuator that we use as OKD
+These are served on port 9000. When running locally, you can access them via
+http://localhost:9000.
+
+* /jolokia - REST access to JMX beans via Jolokia
+* /hawtio - Admin UI interface to JMX beans and more
+* /health - A Spring Actuator that we use as k8s
   liveness/readiness probe.
-* /actuator/info - An actuator that reads the information from
+* /info - An actuator that reads the information from
   `META-INF/build-info.properties` and reports it. The response includes
   things like the version number.
 
@@ -193,7 +284,10 @@ RHSM_RBAC_USE_STUB=true ./gradlew bootRun
 * `CLOUDIGRADE_PORT`: cloudigrade service port
 * `CLOUDIGRADE_MAX_CONNECTIONS`: max concurrent connections to cloudigrade service
 
-### Clowder
+</details>
+
+<details>
+<summary>Clowder</summary>
 
 Clowder exposes the services it provides in an Openshift config map.  This config map appears 
 in the container as a JSON file located by default at the path defined by `ACG_CONFIG` environment
@@ -246,7 +340,10 @@ E.g.
 $ ACG_CONFIG=$(pwd)/swatch-core/src/test/resources/test-clowder-config.json ./gradlew bootRun
 ```
 
-## Deploy to Openshift
+</details>
+
+<details>
+<summary>Deploy to Openshift via Templates</summary>
 
 Prerequisite secrets:
 
@@ -268,7 +365,9 @@ oc process -f templates/rhsm-subscriptions-scheduler.yml | oc create -f -
 oc process -f templates/rhsm-subscriptions-worker.yml | oc create -f -
 ```
 
-## Release Notes
+</details>
+
+## Release Process
 
 You can perform a release using `./gradlew release` **on the develop
 branch**. This command will invoke a
@@ -284,21 +383,7 @@ doing semantic versioning, simply accept the defaults.
 The plugin will create the tag and bump the version. You just need to
 push with `git push --follow-tags origin develop main`.
 
-## Kafka
-
-`podman-compose` deploys a kafka instance w/ a UI at http://localhost:3030
-
-Two environment variables can be used to manipulate the offsets of the kafka
-consumers:
-
-- `KAFKA_SEEK_OVERRIDE_END` when set to `true` seeks to the very end
-- `KAFKA_SEEK_OVERRIDE_TIMESTAMP` when set to an OffsetDateTime, seeks the
-  queue to this position.
-
-These changes are permanent, committed the next time the kafka consumer is detected
-as idle.
-
-## Dashboard
+## Grafana Dashboards
 
 See App-SRE documentation on updating dashboards for more info.
 
@@ -323,5 +408,9 @@ EOF
 Possibly useful, to extract the JSON from the k8s configmap file:
 
 ```
-oc convert -f dashboards/grafana-dashboard-subscription-watch.configmap.yaml -o go-template --template='{{ index .data "subscription-watch.json" }}' > subscription-watch.json
+oc extract -f dashboards/grafana-dashboard-subscription-watch.configmap.yaml --confirm
 ```
+
+## License
+
+Subscription watch components are licensed GPLv3 (see LICENSE for more details).

bin/insert-mock-hosts

@@ -1,10 +1,10 @@
 #!/usr/bin/env python3
 import argparse
-import uuid
-import os
 import json
+import os
 import subprocess
 import sys
+import uuid
 
 output = sys.stdout
 
@@ -16,13 +16,13 @@ def generate_host(inventory_id=None, insights_id=None, account_number=None, org_
                   skip_buckets=False, is_hbi=False):
 
     account = account_number or "account123"
-    _create_account_service(account, 'HBI_HOST')
-
     host_id = uuid.uuid4()
     inventory_id=inventory_id or uuid.uuid4()
     insights_id=insights_id or uuid.uuid4()
+
     if not is_guest:
         hypervisor_uuid = None
+
     if is_hbi:
         host_fields = {
             'id': inventory_id,
@@ -52,6 +52,7 @@ def generate_host(inventory_id=None, insights_id=None, account_number=None, org_
             'reporter': 'rhsm-conduit',
         }
     else:
+        _create_account_service(account, 'HBI_HOST')
         host_fields = {
             'id': host_id,
             'instance_id': inventory_id,

build.gradle

@@ -6,15 +6,15 @@ buildscript {
         mavenCentral()
     }
     dependencies {
-        classpath "org.openapitools:openapi-generator-gradle-plugin:5.3.1"
+        classpath "org.openapitools:openapi-generator-gradle-plugin:5.4.0"
         classpath "org.jsonschema2pojo:jsonschema2pojo-gradle-plugin:1.1.1"
-        classpath "de.undercouch:gradle-download-task:4.1.2"
+        classpath "de.undercouch:gradle-download-task:5.0.1"
     }
 }
 
 plugins {
     id "io.spring.dependency-management" version "1.0.11.RELEASE"
-    id "com.diffplug.spotless" version "6.2.0"
+    id "com.diffplug.spotless" version "6.2.1"
     id "jacoco"
     id "org.sonarqube" version "3.3"
     id "org.springframework.boot" version "2.6.3"
@@ -28,7 +28,7 @@ plugins {
 
 ext {
     swagger_annotations_version = "1.6.4"
-    swagger_ui_version = "4.2.1"
+    swagger_ui_version = "4.4.1"
     resteasy_version = "3.6.3.Final"
     spring_boot_version = "2.6.3"
     jboss_jaxrs_api_version = "2.0.2.Final"

docker-compose.yml

@@ -21,7 +21,12 @@ services:
       - "3030:3030"
       - "9092:9092"
       - "29092:29092"
+    expose:
+      - "9092"
   inventory:
     build:
       context: insights-host-inventory
       dockerfile: dev.dockerfile
+    environment:
+      - INVENTORY_DB_HOST=db
+      - KAFKA_BOOTSTRAP_SERVERS=kafka:9092

docs/Dockerfile.plantuml

@@ -0,0 +1,3 @@
+FROM docker.io/fedora:latest
+RUN yum install -y java-11-openjdk graphviz plantuml
+WORKDIR /mnt

docs/Makefile

@@ -0,0 +1,6 @@
+.PHONY:
+all: $(patsubst %.puml,%.svg,$(wildcard *.puml))
+
+%.svg: %.puml
+	test -n "$$(podman images -q -f reference=localhost/plantuml)" || podman build -f Dockerfile.plantuml . -t plantuml
+	podman run --rm -v $$(pwd):/mnt:z plantuml plantuml -tsvg $<

docs/README.md

@@ -0,0 +1,2 @@
+NOTE: the diagrams in this directory can be regenerated by running `make`
+from this directory.

docs/container-billing.puml

@@ -0,0 +1,33 @@
+@startuml
+!include <C4/C4_Container>
+
+' see https://forum.plantuml.net/8356/white-backgrounds-are-rendered-transparent-in-svg
+skinparam backgroundColor #FEFEFE
+
+LAYOUT_WITH_LEGEND()
+
+title Container diagram for Subscription watch Billing Producers
+
+Person(customer, "Red Hat Customer")
+
+Boundary(crc, console.redhat.com) {
+  Boundary(swatch, "Subscription watch") {
+    System(tally, "Tally", "System that tallies system usage based on telemetry.")
+    ContainerQueue(tally_summaries, "platform.rhsm-subscriptions.tally", "Kafka Topic")
+    System_Boundary(billing_producers, "Billing Producers") {
+      Container(marketplace_worker, "RH Marketplace Worker", "Java, Container", "Notifies RH Marketplace of hourly usage.")
+    }
+  }
+}
+
+System_Ext(rh_marketplace, "marketplace.redhat.com", "Service responsible for billing the customer for PAYG usage.")
+
+' tally process
+Rel(tally, tally_summaries, "Produces to")
+
+' marketplace notification
+Rel_Back(tally_summaries, marketplace_worker, "Consumes from")
+Rel(marketplace_worker, rh_marketplace, "Notifies", "JSON/HTTPS")
+
+Rel(customer, rh_marketplace, "Uses", "HTTPS")
+@enduml