v6.1.7

new features

• Metrics
  • Mark content_length in OTel as metrics, detailed field mapping reference document
  • Request Log increased session_length metric
• Tracing
  • Support parsing the sw8 field in the Dubbo protocol, extracting TraceID, SpanID
    -Event
  • Automatically generate cloud server, K8s Pod add, delete, and change events, and add Grafana Dashboard
• Management
  • Supports configuring the port range for application protocol parsing to reduce the false positive rate. For the configuration method, refer to [l7-protocol-ports configuration item](https://github.com/deepflowys/deepflow/blob/main/server/controller/model/ vtap_group_config_example.go#L816)
  • deepflow-agent supports requesting deepflow-server with a fixed IP address of the load balancer. For the configuration method, refer to [proxy_controller_ip and analyzer_ip fields](https://github.com/deepflowys/deepflow/blob/main/server/controller/model/ vtap_group_config_example.go#L215)
  • Support specifying tap-mode in agent-group-config, so that deepflow-agent can Run in special CNI environment such as macvlan
  • Support configuring kubernetes-cluster-name in deepflow-agent.yaml for specifying K8s cluster name

Optimize

• SQL API
  • When the automatic grouping resource type resource_glX_type is an IP address, directly reuse resource_glX_id to represent the subnet ID, resource_glX to represent the IP address
• Management
  • Reduced database permission requirements when using Managed RDS and ClickHouse
  • Automatically balance the number of deepflow-agents served by each deepflow-server
  • Use OpenTelemetry to monitor the call chain inside deepflow-server

v6.1.6

new features

• AutoLogging
  • Added attribute.http_user_agent and attribute.http_referer fields for HTTP protocol
  • Added attribute.rpc_service field, the value is ServiceName of Dubbo/gRPC
  • Added endpoint field, the value is ServiceName/MethodName of Dubbo/gRPC
  • Support for marking HTTP2 data conforming to the gRPC protocol specification as gRPC (instead of HTTP2) protocol
• AutoTagging
  • Supports synchronizing the resource information of AWS public cloud, and automatically injects it into the observation data as a tag
  • Supports simultaneous injection of cloud resources and container tags for observation data of public cloud-hosted K8s clusters
• manage
  • Support configuring whether deepflow-agent enables parsing of various application protocols
  • Support configuring the regular expression of Golang/openssl process name for deepflow-agent to collect data through eBPF uprobe
  • Supports deepflow-agent standalone mode, where Flow Log and Request Log are written to local log files
  • Support i18n, default display is English

Optimize

• AutoLogging
  • Match SQL keywords to reduce the false positive rate of MySQL and PostgreSQL protocol identification
• Grafana
  • Display the SQL query statement in the Query Editor of the Panel to help developers understand how to call the API
  • Optimized the display of empty field information in Distributed Tracing flame graph
• manage
  • Optimize the traffic between deepflow-server and clickhouse, and preferentially write to the clickhouse Pod on the same node
  • Support to compress OTel Span data received by deepflow-agent, the bandwidth consumption when sending to deepflow-server can be reduced by about 7 times

v6.1.5

new features

• AutoMetrics, AutoTracing, AutoLogging
  • Support collection of PostgreSQL performance indicators and access logs, and associate them with distributed tracing
  • Support to collect HTTPS performance indicators and access logs using openssl library, and correlate them to distributed tracing
• Integration
  • deepflow-server supports RemoteRead interface for Prometheus
  • deepflow-agent supports skipping otel-collector to receive OpenTelemetry data directly
  • The query statement of Grafana Variable supports the use of custom variables and built-in variables, [see documentation](https://deepflow.yunshan.net/docs/zh/server-integration/query/sql/#use-tag-self Name filtering), usage scenarios include:
    • filter the value range of the variable pod with the currently selected value of the variable pod_cluster
    • Use the input content of the variable ingress_wildcard to change the value range of the variable ingress
    • Use the current values ​​of the built-in variables $__from and $__to to improve the query speed of the variable value range
  • Add two zero-intrusive observability dashboards in Grafana: K8s Ingresss, SQL Monitoring
• SQL API
  • string_enum and int_enum types of Tag support using Enum() function to translate Value into Name for query filtering and result return
  • Support SELECT tags/attributes/labels to query all tag.X/attribute.X/label.X fields of each row of data without specifying specific field names
• manage
  • Support ClickHouse cold data to use disk (as an alternative to object storage)
  • Support using deepflow-ctl agent rebalance to balance deepflow-agent to new and restored deepflow-server

Optimize

-AutoLogging

• Sort out the application protocol analysis process and lower the threshold of Add support for more application protocols
• AutoTagging
  • Added deepflow-ctl cloud info command to debug the resource information synchronized from the cloud platform API
• SmartEncoding
  • Recycle the tag encoding value of deleted resources to improve compression ratio and query speed
• SQL API
  • Optimize the display_name of the enumeration value of the server_port Tag field, including the corresponding int value to avoid unclear meaning
• manage
  • deepflow-server is modified to use Deployment Controller to deploy, Please pay attention to update helm chart when upgrading
  • deepflow-agent supports running in unprivileged mode. For specific permission requirements, please refer to [Reference Documentation](https://deepflow.yunshan.net/docs/zh/install/overview/#Running permissions and kernel requirements)
  • Optimize the mapping relationship between Prometheus metrics and DeepFlow Table, each Metrics corresponds to a Table

v6.1.4

1.1. New Features

• AutoTagging
  • Supports synchronizing the resource information of Tencent's public cloud, and automatically injects it into the observation data as a label
• AutoTracing
  • Supports associated application spans and network spans in environments where eBPF cannot run, eliminating tracking blind spots
• SQL API
  • show tags adds fields that return map types, such as labels, attributes, tags
  • show tag values Added limit, offset, like parameters
• Production environment deployment
  • Supports using managed ClickHouse and MySQL
  • When deepflow-agent has not completed registration, it supports configuration issued by deepflow-server
• Grafana
  • Added DeepFlow self-monitoring Dashboard

1.2. Optimization

• SQL API
  • Metrics data in OTel Span can be returned via show metrics API
• system capability
  • Support deepflow-server master election without relying on sidecar
  • deepflow-server supports backward compatibility with deepflow-agent
  • By default, it is synchronized with the NTP server of the container node where deepflow-server is located
  • -v output of normalized process

v6.1.3

Application

AutoMetrics
New indicators: client waiting delay, number of SYN packets, number of SYN-ACK packets, number of SYN retransmission packets, number of SYN-ACK retransmission packets
AutoTracing
Support to associate eBPF uprobe Span with cBPF Span, OTel Span and display it in the trace flame graph
AutoLogging
Support using eBPF uprobe to collect Golang HTTP2, HTTP2_TLS calls
Support collecting Golang process uprobe data with standard symbol table trimmed (Golang >= 1.13 and < 1.18)
AutoTagging
For K8s nodes that are not associated with a cloud server, the cloud server label is automatically generated
Supports synchronizing resource information of Huawei public cloud
Querier SQL API
Fields after GROUP BY are returned automatically, no need to explicitly declare after SELECT
Grafana
Added thumbnail display for DeepFlow Topo and DeepFlow AppTracing
Optimized the Span Tip in the tracing flame graph, showing the time-consuming ratio of the Span itself

System

The time to wait for the agent to come online at the first deployment was optimized from 7 minutes to 4 minutes
Access to deepflow-server and clickhouse in the same K8s cluster no longer uses NodeIP
deepflow-server uses externalTrafficPolicy=Cluster by default to avoid the unavailability of the externalTrafficPolicy=Local function of kube-proxy in some environments and some CNI compatibility issues. It can be manually changed to Local to optimize cross-cluster traffic
deepflow-server adds ext-metrics-ttl, flow-metrics-ttl, flow-log-ttl configuration parameters to initialize data retention time
deepflow-agent supports writing l4_flow_log and l7_flow_log to local files
deepflow-agent removes dependencies on libbpf

v6.1.2

V6.1.2 is the second version after open source, we officially started the rhythm of a small version every two weeks.

Application

• AutoMetrics, AutoLogging
  • MySQL adds COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_FETCH, COM_STMT_CLOSE command parsing capability
  • Support collecting MQTT 3.1 call logs and performance metrics
• SQL API
  • Support SELECT labels to get all custom label columns

System

• deepflow-agent

  • Added the Inactive IP Indicator Data configuration item. After this configuration is turned off, IPs that do not reply to traffic will be aggregated
  • Use BPF pre-filtered acquisition interface to improve acquisition performance
  • Provide deb installation package

• deepflow-ctl

  • agent-group-config update no longer needs to specify the agent-group-id parameter

• hotfix: #784

v6.1.1

V6.1.1 took three months to develop. It is a major iterative version of the V6 series and the first community version officially released by DeepFlow. Since then, the enterprise version and the community version will be iterated simultaneously, and the development rhythm of release a small version every two weeks will be promoted, and the last small version of V6.1 (LTS) will be released after about four small versions. Version).

In order to create a highly automated open source observability platform, we have made major adjustments to the software architecture in this version, including merging too many microservices, eliminating some components for DaemonSet, HostNet and other deployment modes dependencies, use Golang to refactor some Python modules, etc.

Most importantly, in this release we have greatly enhanced application performance monitoring capabilities:

• Support the integration and association of OpenTelemetry and SkyWalking tracking data, unlock full-stack, full-link distributed tracking capabilities without blind spots, and improve communication efficiency between different development teams
• Support the integration and association of Prometheus and Telegraf indicator data, and accumulate system, application, and business full-stack indicators on one platform, so that operation and maintenance, development, and operation can work together
• Enhanced AutoTagging and SmartEncoding capabilities to support injecting more K8s custom Labels
• Enhanced eBPF AutoTracing capability to support the collection of Golang HTTPS calls

Note: The Release Note below does not include Enterprise Edition features.

application function

• Indicator data enhancement
  • Support for integrating Prometheus metrics data (implementing remote_write interface)
  • Support integration of Telegraf metrics data
  • Supports automatic injection of cloud resources, container resources, and K8s custom Labels (AutoTagging) into the integrated indicator data
• Distributed tracing enhancement
  • Support integrated OpenTelemetry tracking data
  • Support integrated SkyWalking tracking data
  • Support associated applications, systems, and network spans to achieve full-stack, full-link tracking capabilities without blind spots
  • Supports automatic injection of cloud resources, container resources, and K8s custom Labels into the integrated tracking data (AutoTagging)
• Enhanced application call collection capabilities
  • Support eBPF to collect Golang's HTTPS call logs and performance metrics
  • Support eBPF/BPF to collect MQTT 3.1 call logs and performance indicators (Beta)
  • Application protocol analysis to remove port number restrictions
• Grafana support enhancements
  • Added interactive DeepFlow Query Editor, a new zero-threshold query condition editor
  • Added DeepFlow AppTracing Panel for drawing call chain tracing flame graphs
  • Added DeepFlow Topo Panel for drawing call topology
• Search ability improved
  • Support query historical data of deleted resources
  • Support to use K8s custom Label to search all data
  • Support grouping all data based on K8s custom Label
  • Support for creating template variables based on all searchable fields
• Flow log aggregation logic adjustment
  • The flow log only records the TCP Flags that appear in the current cycle
  • Lag timeout after captured FIN-FIN-ACK sequence, waiting for last ACK
  • VLAN field in storage traffic
• Added Math operator for index quantity

System functions

• Provides the deepflow-ctl command line tool for CLI management
• Agent (collector)
  • Supports parsing multiple specifications of TraceID and SpanID in HTTP and Dubbo protocols at the same time
  • Support to collect network and application performance data of VIP on loopback interface
  • The default configuration of the acquisition network port supports Calico, Flannel, Qemu, Cilium, Kube-OVN, localhost, physical network card
  • Move all configuration items that do not prevent the collector from starting to be issued by the controller, and provide a declarative group configuration API
  • Support remote limit CPU usage of deepflow-agent (Rust)
  • Support self-healing after collector allocation controller or data node failure
• Server (controller, data node)
  • Support synchronizing K8s custom Label
  • Support for lag deletion of deleted resource information
  • Provides unified Schema API and SQL query API for all data tables
  • The main module of the merge controller and data node is deepflow-server
  • Optimized the deployment method of controllers and data nodes, and reduced the dependence on DaemonSet and HostNet
  • Optimize the main controller election logic and remove the dependency on Zookeeper
  • Eliminate plaintext keys stored in the database