From d63bfad08170ebb4f71f18be6128a59ef4401b51 Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Fri, 9 Feb 2024 01:30:22 +0100 Subject: [PATCH 1/4] Metrics: Add dedicated page about metrics/monitoring Absorb "Telegraf" section from "ETL" page. --- docs/index.md | 1 + docs/integrate/etl.md | 24 ------------------------ docs/integrate/metrics.md | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 24 deletions(-) create mode 100644 docs/integrate/metrics.md diff --git a/docs/index.md b/docs/index.md index 074e3e3..b89830a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -164,6 +164,7 @@ ORM Libraries :hidden: Load and Export +System Metrics Business Intelligence Machine Learning Data Visualization diff --git a/docs/integrate/etl.md b/docs/integrate/etl.md index d19de08..bb1c2ae 100644 --- a/docs/integrate/etl.md +++ b/docs/integrate/etl.md @@ -323,27 +323,6 @@ _Please note the adapters are a work in progress._ ``` -## Telegraf - -```{div} -:style: "float: right" -[![](https://raw.githubusercontent.com/influxdata/branding/master/docs/img/logo-usage/logo-symbol-black.svg){w=180px}](https://www.influxdata.com/time-series-platform/telegraf/) -``` - -[Telegraf] is a leading open source server agent to help you collect metrics -from your stacks, sensors, and systems. More than 200 adapters to connect -to other systems leaves nothing to be desired. - -- [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics] - -![](https://www.influxdata.com/wp-content/uploads/Main-Diagram_06.01.2022v1.png){h=200px} - -```{seealso} -[CrateDB and Telegraf] -``` - - - [2023 SIGMOD Systems Award]: https://sigmod.org/2023-sigmod-systems-award/ [Aiven]: https://aiven.io/ [Aiven for Apache Flink]: https://aiven.io/flink @@ -368,7 +347,6 @@ to other systems leaves nothing to be desired. [CrateDB and Apache Kafka]: https://crate.io/integrations/cratedb-and-kafka [CrateDB and Kestra]: https://crate.io/integrations/cratedb-and-kestra [CrateDB and Node-RED]: https://crate.io/integrations/cratedb-and-node-red -[CrateDB and Telegraf]: https://crate.io/integrations/cratedb-and-telegraf [Data Ingestion using Kafka and Kafka Connect]: https://crate.io/docs/crate/howtos/en/latest/integrations/kafka-connect.html [dbt]: https://www.getdbt.com/ [dbt Cloud]: https://www.getdbt.com/product/dbt-cloud/ @@ -392,9 +370,7 @@ to other systems leaves nothing to be desired. [Overview about more managed Kafka offerings]: https://keen.io/blog/managed-apache-kafka-vs-diy/ [Setting up data pipelines with CrateDB and Kestra]: https://community.crate.io/t/setting-up-data-pipelines-with-cratedb-and-kestra-io/1400 [Singer]: https://www.singer.io/ -[Telegraf]: https://www.influxdata.com/time-series-platform/telegraf/ [Tutorial: Replicating data to CrateDB with Debezium and Kafka]: https://community.crate.io/t/replicating-data-to-cratedb-with-debezium-and-kafka/1388 [Updating stock market data automatically with CrateDB and Apache Airflow]: https://community.crate.io/t/updating-stock-market-data-automatically-with-cratedb-and-apache-airflow/1304 [Using dbt with CrateDB]: https://community.crate.io/t/using-dbt-with-cratedb/1566 -[Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics]: https://crate.io/blog/use-cratedb-with-telegraf-an-agent-for-collecting-reporting-metrics [Webinar: How to replicate data from other databases to CrateDB with Debezium and Kafka]: https://crate.io/resources/webinars/lp-wb-debezium-kafka diff --git a/docs/integrate/metrics.md b/docs/integrate/metrics.md new file mode 100644 index 0000000..80f28a0 --- /dev/null +++ b/docs/integrate/metrics.md @@ -0,0 +1,35 @@ +(metrics)= +# Processing monitoring metrics with CrateDB + +Storing metrics data for the long term is a common need in systems monitoring +scenarios. CrateDB offers corresponding integration adapters. + + +(telegraf)= +## Telegraf + +```{div} +:style: "float: right" +[![](https://raw.githubusercontent.com/influxdata/branding/master/docs/img/logo-usage/logo-symbol-black.svg){w=180px}](https://www.influxdata.com/time-series-platform/telegraf/) +``` + +[Telegraf] is a leading open source server agent to help you collect metrics +from your stacks, sensors, and systems. More than 200 adapters to connect +to other systems leaves nothing to be desired. + +- [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics] + +![](https://www.influxdata.com/wp-content/uploads/Main-Diagram_06.01.2022v1.png){h=200px} + +```{seealso} +[CrateDB and Telegraf] +``` + +```{div} +:style: "clear: both" +``` + + +[CrateDB and Telegraf]: https://crate.io/integrations/cratedb-and-telegraf +[Telegraf]: https://www.influxdata.com/time-series-platform/telegraf/ +[Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics]: https://crate.io/blog/use-cratedb-with-telegraf-an-agent-for-collecting-reporting-metrics From 06405454a1427bd20b30312890b5d4f03e63d9ce Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Fri, 9 Feb 2024 01:54:09 +0100 Subject: [PATCH 2/4] Metrics: Improve section about Telegraf Add "What's inside" and "Scope" subsections. --- docs/integrate/metrics.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/integrate/metrics.md b/docs/integrate/metrics.md index 80f28a0..8148eee 100644 --- a/docs/integrate/metrics.md +++ b/docs/integrate/metrics.md @@ -17,6 +17,25 @@ scenarios. CrateDB offers corresponding integration adapters. from your stacks, sensors, and systems. More than 200 adapters to connect to other systems leaves nothing to be desired. +Telegraf is a server-based agent for collecting and sending all metrics and +events from databases, systems, and IoT sensors. Telegraf is written in Go +and compiles into a single binary with no external dependencies, and requires +a very minimal memory footprint. + +### Scope + +- **IoT sensors**: Collect critical stateful data (pressure levels, temperature + levels, etc.) with popular protocols like MQTT, ModBus, OPC-UA, and Kafka. + +- **DevOps Tools and frameworks**: Gather metrics from cloud platforms, + containers, and orchestrators like GitHub, Kubernetes, CloudWatch, Prometheus, + and more. + +- **System telemetry**: Metrics from system telemetry like iptables, Netstat, + NGINX, and HAProxy help provide a full stack view of your apps. + +### Resources + - [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics] ![](https://www.influxdata.com/wp-content/uploads/Main-Diagram_06.01.2022v1.png){h=200px} From 0747de4bacf7de5519e3594303a08943e1d2c6ea Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Fri, 9 Feb 2024 01:54:47 +0100 Subject: [PATCH 3/4] Metrics: Add section about Prometheus --- docs/integrate/metrics.md | 70 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) diff --git a/docs/integrate/metrics.md b/docs/integrate/metrics.md index 8148eee..eed3cad 100644 --- a/docs/integrate/metrics.md +++ b/docs/integrate/metrics.md @@ -1,10 +1,69 @@ (metrics)= -# Processing monitoring metrics with CrateDB +# Monitoring and Metrics with CrateDB Storing metrics data for the long term is a common need in systems monitoring scenarios. CrateDB offers corresponding integration adapters. +(prometheus)= +## Prometheus + +```{div} +:style: "float: right" +[![](https://github.com/crate/crate-clients-tools/assets/453543/8ddb109f-b45f-46b0-8103-30ba491f7142){w=180px}](https://prometheus.io/) +``` + +[Prometheus] is an open-source systems monitoring and alerting toolkit +for collecting metrics data from applications and infrastructures. + +Prometheus collects and stores its metrics as time series data, i.e. +metrics information is stored with the timestamp at which it was recorded, +alongside optional key-value pairs called labels. + +### Features + +Prometheus's main features are: + +- a multi-dimensional data model with time series data identified by metric name and key/value pairs +- PromQL, a flexible query language to leverage this dimensionality +- no reliance on distributed storage; single server nodes are autonomous +- time series collection happens via a pull model over HTTP +- pushing time series is supported via an intermediary gateway +- targets are discovered via service discovery or static configuration +- multiple modes of graphing and dashboarding support + + +### Remote Endpoints and Storage + +The [Prometheus remote endpoints and storage] subsystem, based on its +[remote write] and [remote read] features, allows to transparently +send and receive metric samples. It is primarily intended for long term +storage. + +This is where CrateDB comes into place. Using the [CrateDB Prometheus +Adapter], one can easily store the collected metrics data in CrateDB and +take advantage of its high ingestion and query speed and friendly UI to +massively scale-out Prometheus. + + +### Resources + +- [CrateDB Prometheus Adapter] +- [Getting Started With Prometheus and CrateDB for Long-Term Storage] +- [Storing long-term metrics with Prometheus in CrateDB] +- [Webinar: Using Prometheus and Grafana with CrateDB Cloud] + +![](https://github.com/crate/crate-clients-tools/assets/453543/26b47686-889a-4137-a87f-d6a6b38d56d2){h=200px} + +```{seealso} +[CrateDB and Prometheus] +``` + +```{div} +:style: "clear: both" +``` + + (telegraf)= ## Telegraf @@ -49,6 +108,15 @@ a very minimal memory footprint. ``` +[CrateDB and Prometheus]: https://cratedb.com/integrations/cratedb-and-prometheus [CrateDB and Telegraf]: https://crate.io/integrations/cratedb-and-telegraf +[CrateDB Prometheus Adapter]: https://github.com/crate/cratedb-prometheus-adapter +[Getting Started With Prometheus and CrateDB for Long-Term Storage]: https://cratedb.com/blog/getting-started-prometheus-cratedb-long-term-storage +[Prometheus]: https://prometheus.io/ +[Prometheus remote endpoints and storage]: https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage +[remote read]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read +[remote write]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write +[Storing long-term metrics with Prometheus in CrateDB]: https://community.cratedb.com/t/storing-long-term-metrics-with-prometheus-in-cratedb/1012 [Telegraf]: https://www.influxdata.com/time-series-platform/telegraf/ [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics]: https://crate.io/blog/use-cratedb-with-telegraf-an-agent-for-collecting-reporting-metrics +[Webinar: Using Prometheus and Grafana with CrateDB Cloud]: https://cratedb.com/resources/webinars/lp-wb-prometheus-grafana From d488964290753415b1c808a383e87fbc35da9ebf Mon Sep 17 00:00:00 2001 From: Andreas Motl Date: Fri, 9 Feb 2024 02:02:49 +0100 Subject: [PATCH 4/4] Metrics: Improve page layout / reading flow --- docs/integrate/metrics.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/integrate/metrics.md b/docs/integrate/metrics.md index eed3cad..3c04bf0 100644 --- a/docs/integrate/metrics.md +++ b/docs/integrate/metrics.md @@ -9,7 +9,7 @@ scenarios. CrateDB offers corresponding integration adapters. ## Prometheus ```{div} -:style: "float: right" +:style: "float: right; margin-left: 0.3em" [![](https://github.com/crate/crate-clients-tools/assets/453543/8ddb109f-b45f-46b0-8103-30ba491f7142){w=180px}](https://prometheus.io/) ``` @@ -20,7 +20,7 @@ Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. -### Features +**Features** Prometheus's main features are: @@ -33,7 +33,7 @@ Prometheus's main features are: - multiple modes of graphing and dashboarding support -### Remote Endpoints and Storage +**Remote Endpoints and Storage** The [Prometheus remote endpoints and storage] subsystem, based on its [remote write] and [remote read] features, allows to transparently @@ -46,7 +46,7 @@ take advantage of its high ingestion and query speed and friendly UI to massively scale-out Prometheus. -### Resources +**Resources** - [CrateDB Prometheus Adapter] - [Getting Started With Prometheus and CrateDB for Long-Term Storage] @@ -68,8 +68,8 @@ massively scale-out Prometheus. ## Telegraf ```{div} -:style: "float: right" -[![](https://raw.githubusercontent.com/influxdata/branding/master/docs/img/logo-usage/logo-symbol-black.svg){w=180px}](https://www.influxdata.com/time-series-platform/telegraf/) +:style: "float: right; margin-left: 0.3em" +[![](https://github.com/crate/crate-clients-tools/assets/453543/3f0b4525-5344-42fe-bae6-1b0970fa0540){w=180px}](https://www.influxdata.com/time-series-platform/telegraf/) ``` [Telegraf] is a leading open source server agent to help you collect metrics @@ -81,7 +81,7 @@ events from databases, systems, and IoT sensors. Telegraf is written in Go and compiles into a single binary with no external dependencies, and requires a very minimal memory footprint. -### Scope +**Overview** - **IoT sensors**: Collect critical stateful data (pressure levels, temperature levels, etc.) with popular protocols like MQTT, ModBus, OPC-UA, and Kafka. @@ -93,7 +93,7 @@ a very minimal memory footprint. - **System telemetry**: Metrics from system telemetry like iptables, Netstat, NGINX, and HAProxy help provide a full stack view of your apps. -### Resources +**Resources** - [Use CrateDB With Telegraf, an Agent for Collecting & Reporting Metrics]