docs(analytics): Add documentation for setting up data services and e…

…nabling data features in control center (#4741)

README.md

@@ -10,7 +10,7 @@ The single API to access payment ecosystems across 130+ countries</div>
 
 <p align="center">
   <a href="#%EF%B8%8F-quick-start-guide">Quick Start Guide</a> •
-  <a href="https://github.com/juspay/hyperswitch/blob/main/docs/try_local_system.md">Local Setup Guide</a> •
+  <a href="/docs/try_local_system.md">Local Setup Guide</a> •
   <a href="#-fast-integration-for-stripe-users">Fast Integration for Stripe Users</a> •
   <a href="https://api-reference.hyperswitch.io/introduction"> API Docs </a> •
   <a href="#-supported-features">Supported Features</a> •

config/dashboard.toml

@@ -0,0 +1,38 @@
+[default.theme]
+primary_color="#006DF9"
+primary_hover_color="#005ED6"
+sidebar_color="#242F48"
+
+[default.endpoints]
+api_url="http://localhost:8080" # The backend hyperswitch API server for making payments
+sdk_url="http://localhost:9050/HyperLoader.js" # SDK distribution url used for loading the SDK in control center
+logo_url=""
+favicon_url=""
+mixpanel_token=""
+
+[default.features]
+test_live_toggle=false
+is_live_mode=false
+email=false
+quick_start=false
+audit_trail=true
+system_metrics=false
+sample_data=false
+frm=false
+payout=true
+recon=false
+test_processors=true
+feedback=false
+mixpanel=false
+generate_report=false
+user_journey_analytics=false
+authentication_analytics=false
+surcharge=false
+dispute_evidence_upload=false
+paypal_automatic_flow=false
+threeds_authenticator=false
+global_search=false
+dispute_analytics=true
+configure_pmts=false
+branding=false
+totp=false

crates/analytics/docs/README.md

@@ -0,0 +1,104 @@
+# Running Kafka & Clickhouse with Analytics and Events Source Configuration
+
+This document provides instructions on how to run Kafka and Clickhouse using Docker Compose, and how to configure the analytics and events source.
+
+## Architecture
+     +------------------------+
+     |       Hyperswitch      |
+     +------------------------+
+                |
+                |
+                v
+     +------------------------+
+     |         Kafka          |
+     |  (Event Stream Broker) |
+     +------------------------+
+                |
+                |
+                v
+     +------------------------+
+     |  ClickHouse            |
+     |  +------------------+  |
+     |  | Kafka Engine     |  |
+     |  |    Table         |  |
+     |  +------------------+  |
+     |            |           |
+     |            v           |
+     |  +------------------+  |
+     |  | Materialized     |  |
+     |  |    View (MV)     |  |
+     |  +------------------+  |
+     |            |           |
+     |            v           |
+     |  +------------------+  |
+     |  | Storage Table    |  |
+     |  +------------------+  |
+     +------------------------+
+
+
+## Starting the Containers
+
+Docker Compose can be used to start all the components.
+
+Run the following command:
+
+```bash
+docker compose --profile olap up -d
+```
+This will spawn up the following services
+1. kafka
+2. clickhouse
+3. opensearch
+
+## Setting up Kafka
+
+Kafka-UI is a visual tool for inspecting Kafka and it can be accessed at `localhost:8090` to view topics, partitions, consumers & generated events.
+
+## Setting up Clickhouse
+
+Once Clickhouse is up and running, you can interact with it via web.
+
+You can either visit the URL (`http://localhost:8123/play`) where the Clickhouse server is running to get a playground, or you can bash into the Clickhouse container and execute commands manually.
+
+Run the following commands:
+
+```bash
+# On your local terminal
+docker compose exec clickhouse-server bash
+
+# Inside the clickhouse-server container shell
+clickhouse-client --user default
+
+# Inside the clickhouse-client shell
+SHOW TABLES;
+```
+
+## Configuring Analytics and Events Source
+
+To use Clickhouse and Kafka, you need to enable the `analytics.source` and update the `events.source` in the configuration file.
+
+You can do this in either the `config/development.toml` or `config/docker_compose.toml` file.
+
+Here's an example of how to do this:
+
+```toml
+[analytics]
+source = "clickhouse"
+
+[events]
+source = "kafka"
+```
+
+After making this change, save the file and restart your application for the changes to take effect.
+
+## Enabling Data Features in Dashboard
+
+To check the data features in the dashboard, you need to enable them in the `config/dashboard.toml` configuration file.
+
+Here's an example of how to do this:
+
+```toml
+[default.features]
+audit_trail=true
+system_metrics=true
+```

docker-compose.yml

@@ -1,5 +1,3 @@
-version: "3.8"
-
 volumes:
   pg_data:
   redisinsight_store:
@@ -137,8 +135,8 @@ services:
       context: ./docker
       dockerfile: web.Dockerfile
     environment:
-      - HYPERSWITCH_PUBLISHABLE_KEY=$HYPERSWITCH_PUBLISHABLE_KEY
-      - HYPERSWITCH_SECRET_KEY=$HYPERSWITCH_SECRET_KEY
+      - HYPERSWITCH_PUBLISHABLE_KEY=${HYPERSWITCH_PUBLISHABLE_KEY:PUBLISHABLE_KEY}
+      - HYPERSWITCH_SECRET_KEY=${HYPERSWITCH_SECRET_KEY:SECRET_KEY}
       - HYPERSWITCH_SERVER_URL=${HYPERSWITCH_SERVER_URL:-http://hyperswitch-server:8080}
       - HYPERSWITCH_CLIENT_URL=${HYPERSWITCH_CLIENT_URL:-http://localhost:9050}
       - SELF_SERVER_URL=${SELF_SERVER_URL:-http://localhost:5252}
@@ -156,8 +154,11 @@ services:
     ports:
       - "9000:9000"
     environment:
-      - apiBaseUrl=http://localhost:8080
-      - sdkBaseUrl=http://localhost:9050/HyperLoader.js
+      - configPath=/tmp/dashboard-config.toml
+    volumes:
+      - ./config/dashboard.toml:/tmp/dashboard-config.toml
+    depends_on:
+      - hyperswitch-web
     labels:  
       logs: "promtail" 
 
@@ -185,38 +186,23 @@ services:
       - redis-cluster
     networks:
       - router_net
-    command: "bash -c 'export COUNT=${REDIS_CLUSTER_COUNT:-3}
-
-      \ if [ $$COUNT -lt 3 ]
-
-      \ then
-
-      \ echo \"Minimum 3 nodes are needed for redis cluster\"
-
-      \ exit 1
-
-      \ fi
-
-      \ HOSTS=\"\"
-
-      \ for ((c=1; c<=$$COUNT;c++))
-
-      \ do
-
-      \ NODE=$COMPOSE_PROJECT_NAME-redis-cluster-$$c:6379
-
-      \ echo $$NODE
-
-      \ HOSTS=\"$$HOSTS $$NODE\"
-
-      \ done
-
-      \ echo Creating a cluster with $$HOSTS
-
-      \ redis-cli --cluster create $$HOSTS --cluster-yes
-
-      \ '"
-
+    command: |-
+      bash -c 'export COUNT=${REDIS_CLUSTER_COUNT:-3}
+      if [ $$COUNT -lt 3 ]
+      then
+      echo \"Minimum 3 nodes are needed for redis cluster\"
+      exit 1
+      fi
+      HOSTS=\"\"
+      for ((c=1; c<=$$COUNT;c++))
+      do
+      NODE=$COMPOSE_PROJECT_NAME-redis-cluster-$$c:6379
+      echo $$NODE
+      HOSTS=\"$$HOSTS $$NODE\"
+      done
+      echo Creating a cluster with $$HOSTS
+      redis-cli --cluster create $$HOSTS --cluster-yes
+      '
   ### Monitoring
   grafana:
     image: grafana/grafana:latest
@@ -304,7 +290,7 @@ services:
     networks:
       - router_net
     profiles:
-      - full_kv
+      - monitoring
     ports:
       - "8001:8001"
     volumes:
@@ -390,24 +376,20 @@ services:
     hostname: opensearch
     environment:
       - "discovery.type=single-node"
-    expose:
-      - "9200"
+    profiles:
+      - olap
     ports:
       - "9200:9200"
     networks:
       - router_net
-    profiles:
-      - olap
 
   opensearch-dashboards:
     image: opensearchproject/opensearch-dashboards:1.3.14
     ports:
       - 5601:5601
-    expose:
-      - "5601"
+    profiles:
+      - olap
     environment:
       OPENSEARCH_HOSTS: '["https://opensearch:9200"]'
     networks:
       - router_net
-    profiles:
-      - olap

docs/try_local_system.md

@@ -13,7 +13,7 @@ Check the Table Of Contents to jump to the relevant section.
 **Table Of Contents:**
 
 - [Run hyperswitch using Docker Compose](#run-hyperswitch-using-docker-compose)
-  - [Run the scheduler and monitoring services](#run-the-scheduler-and-monitoring-services)
+  - [Running additional services](#running-additional-services)
 - [Set up a development environment using Docker Compose](#set-up-a-development-environment-using-docker-compose)
 - [Set up a Rust environment and other dependencies](#set-up-a-rust-environment-and-other-dependencies)
   - [Set up dependencies on Ubuntu-based systems](#set-up-dependencies-on-ubuntu-based-systems)
@@ -68,9 +68,16 @@ Check the Table Of Contents to jump to the relevant section.
    If the command returned a `200 OK` status code, proceed with
    [trying out our APIs](#try-out-our-apis).
 
-### Run the scheduler and monitoring services
+### Running additional services
 
-You can run the scheduler and monitoring services by specifying suitable profile
+The default behaviour for docker compose only runs the following services:
+1. postgres
+2. redis (standalone)
+3. hyperswitch server
+4. hyperswitch control center
+5. hyperswitch web sdk
+
+You can run the scheduler, data and monitoring services by specifying suitable profile
 names to the above Docker Compose command.
 To understand more about the hyperswitch architecture and the components
 involved, check out the [architecture document][architecture].
@@ -93,6 +100,13 @@ involved, check out the [architecture document][architecture].
   logs using the "Explore" tab, select Loki as the data source, and select the
   container to query logs from.
 
+- To run the data services (Clickhouse, Kafka and Opensearch) you can specify the `olap` profile
+
+   ```shell
+   docker compose --profile olap up -d
+   ```
+   You can read more about using the data services [here][data-docs]
+
 - You can also specify multiple profile names by specifying the `--profile` flag
   multiple times.
   To run both the scheduler components and monitoring services, the Docker
@@ -109,6 +123,7 @@ Once the services have been confirmed to be up and running, you can proceed with
 [docker-compose-config]: /config/docker_compose.toml
 [docker-compose-yml]: /docker-compose.yml
 [architecture]: /docs/architecture.md
+[data-docs]: /crates/analytics/docs/README.md
 
 ## Set up a development environment using Docker Compose