MVP of Quesma using ELK stack with Clickhouse

Minimal Viable Product of Quesma. It provides Elasticsearch API compatibility for the observability use case with Clickhouse as a backend.

The main debugging interface is available at localhost:9999. You can follow the links.

Architecture

Architecture Decision Records

https://github.com/QuesmaOrg/poc-elk-mitmproxy/tree/main/adr

Quesma

L4 Proxy

Loading

flowchart LR;
    client-->l4_proxy;
    l4_proxy-->client;
    l4_proxy-->elasticsearch;
    elasticsearch-->l4_proxy;
    l4_proxy-.->http_server;
    http_server-.->statistics_processor;

Dual-Write

Loading

flowchart LR;
    client-->routing_proxy;
    routing_proxy-->elasticsearch;
    elasticsearch-->routing_proxy;
    routing_proxy-.->statistics_processor;
    routing_proxy-->clickhouse;
    clickhouse-->routing_proxy;
    routing_proxy-->response_matcher;
    routing_proxy-->response_matcher;
    routing_proxy-->client;
    response_matcher-.->UI((UI))

Docker Compose Setup

Loading

flowchart LR;
    kibana-->mitmproxy;
    mitmproxy-->kibana;
    device-log-generator-->mitmproxy;
    log-generator-->mitmproxy;
    mitmproxy-->quesma;
    quesma-->mitmproxy;
    quesma-->clickhouse[(clickhouse)];
    clickhouse-->quesma;
    quesma-->elasticsearch[(elasticsearch)];
    elasticsearch-->quesma;

Kibana

Kibana is available at localhost:5601.

MITM Proxy

Mitmweb is available at localhost:8081.

It is a man in the middle inspection tool, please consult the mitmproxy documentation.

You can enable the Python script by uncommenting docker-compose.yml in services.mitmproxy.run.

You can further edit it mitmproxy/request.py.

Very useful for quick dumps in mitmproxy/requests:

tail -f mitmproxy/requests/logs-X-X.txt

Some filters that you might find useful for filtering out noise requests (copy-paste into the Search box):

!/_doc & !security & !metrics & !.kibana_alerting & !_nodes &!kibana_task_manager & !_pit & !_monitoring & !_xpack & !.reporting & !.kibana & !heartbeat & !_aliases & !_field_caps & !_license & !.logs-endpoint & !.fleet- & !traces & !_cluster & !_resolve & !_mapping & !logs-cloud & !.monitoring & !.ds-risk

This will also filter out insert requests:

!/_doc & !security & !metrics & !.kibana_alerting & !_nodes &!kibana_task_manager & !_pit & !_monitoring & !_xpack & !.reporting & !.kibana & !heartbeat & !_aliases & !_field_caps & !_license & !.logs-endpoint & !.fleet- & !traces & !_cluster & !_resolve & !_mapping & !logs-cloud & !.monitoring & !.ds-risk & !_bulk

Memory Profiling

Container-friendly pprof endpoint is exposed at localhost:9999/debug/pprof/

Fetch a memory profile

curl http://localhost:9999/debug/pprof/heap > heap.out
go tool pprof -http=:8082 heap.out 

Now, head over to localhost:8082 and you can inspect the memory profile

Clickhouse-client

To connect to the client when clickhouse-server is running on localhost:8123/play

Alternatively, you can find the container name using docker ps and use the command line:

docker exec -it poc-elk-mitmproxy-clickhouse-1 clickhouse-client

Once you connected, you run typical SQL commands such as:

SHOW TABLES;
DESCRIBE logs;
SELECT * FROM logs LIMIT 10;

Running locally with debugger

It's possible to run Quesma from IDE with a debugger. For that, you need to start the auxiliary services with the local-debug.yml file, which provides minimal set of dependencies - Elasticsearch, Clickhouse, and Kibana with sample data sets.

1. Navigate to quesma/main.go and click Run application menu in your IDE. Optionally you can also run the main.go file from the command line.

2. Start auxiliary services with the following command:

   HOST_IP=$(ifconfig en0 | awk '/inet / {print $2}') docker-compose -f docker/local-debug.yml up

   This is minimalistic setup with Elasticsearch, ClickHouse, and Kibana populated with sample data sets. There's also MITM proxy to help you inspect the actual traffic.

   NOTE: Since we're all using Mac's, Docker deamon cannot use host network mode. The only option for processes running in containers to connect locally running Quesma process is to pass the IP address like this.

3. If you set proper breakpoints in your IDE, you should see the execution stopped at the breakpoint.

4. Profit!