Add SRV resolver to loadbalancer exporter to use hostnames and track IPs

.chloggen/srv-resolver-for-loadbalancing-exporter.yaml

@@ -0,0 +1,27 @@
+# Use this changelog template to create an entry for release notes.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: 'enhancement'
+
+# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
+component: loadbalancingexporter
+
+# A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: New SRV resolver for loadbalancing exporter for static hostnames with changing IPs
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+issues: ["18412"]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:
+
+# If your change doesn't affect end users or the exported elements of any package,
+# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
+# Optional: The change log or logs in which this entry should be included.
+# e.g. '[user]' or '[user, api]'
+# Include 'user' if the change is relevant to end users.
+# Include 'api' if there is a change to a library API.
+# Default: '[user]'
+change_logs: []

exporter/loadbalancingexporter/README.md

@@ -54,7 +54,7 @@ This also supports service name based exporting for traces. If you have two or m
 Refer to [config.yaml](./testdata/config.yaml) for detailed examples on using the processor.
 
 * The `otlp` property configures the template used for building the OTLP exporter. Refer to the OTLP Exporter documentation for information on which options are available. Note that the `endpoint` property should not be set and will be overridden by this exporter with the backend endpoint.
-* The `resolver` accepts a `static` node, a `dns` or a `k8s` service. If all three are specified, `k8s` takes precedence.
+* The `resolver` accepts a `static` node, a `dns`, `srv`, or a `k8s` service. If all four are specified, `k8s` takes precedence.
 * The `hostname` property inside a `dns` node specifies the hostname to query in order to obtain the list of IP addresses.
 * The `dns` node also accepts the following optional properties:
   * `hostname` DNS hostname to resolve.
@@ -65,9 +65,14 @@ Refer to [config.yaml](./testdata/config.yaml) for detailed examples on using th
   * `service` Kubernetes service to resolve, e.g. `lb-svc.lb-ns`. If no namespace is specified, an attempt will be made to infer the namespace for this collector, and if this fails it will fall back to the `default` namespace.
   * `ports` port to be used for exporting the traces to the addresses resolved from `service`. If `ports` is not specified, the default port 4317 is used. When multiple ports are specified, two backends are added to the load balancer as if they were at different pods.
 * The `routing_key` property is used to route spans to exporters based on different parameters. This functionality is currently enabled only for `trace` pipeline types. It supports one of the following values:
-    * `service`: exports spans based on their service name. This is useful when using processors like the span metrics, so all spans for each service are sent to consistent collector instances for metric collection. Otherwise, metrics for the same services are sent to different collectors, making aggregations inaccurate. 
-    * `traceID` (default): exports spans based on their `traceID`.
-    * If not configured, defaults to `traceID` based routing.
+  * `service`: exports spans based on their service name. This is useful when using processors like the span metrics, so all spans for each service are sent to consistent collector instances for metric collection. Otherwise, metrics for the same services are sent to different collectors, making aggregations inaccurate. 
+  * `traceID` (default): exports spans based on their `traceID`.
+  * If not configured, defaults to `traceID` based routing.
+* The `srv` node accepts the following optional properties:
+  * `hostname` DNS SRV hostname to resolve.
+  * `port` port to be used for exporting the traces to the IP addresses resolved from `hostname`. If `port` is not specified, the default port 4317 is used.
+  * `interval` resolver interval in go-Duration format, e.g. `5s`, `1d`, `30m`. If not specified, `5s` will be used.
+  * `timeout` resolver timeout in go-Duration format, e.g. `5s`, `1d`, `30m`. If not specified, `1s` will be used.
 
 Simple example
 ```yaml
@@ -156,6 +161,36 @@ service:
         - loadbalancing
 ```
 
+The SRV Resolver is useful in situations when you want to return hostnames instead of IPs for endpoints. An example would be a `StatefulSet`-backed headless kubernetes `Service` with istio. Example:
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: localhost:4317
+
+processors:
+
+exporters:
+  loadbalancing:
+    protocol:
+      otlp: {}
+    resolver:
+      srv:
+        hostname: _<svc-port-name>._<svc-port-protocol>.<svc-name>.<svc-namespace>.svc.cluster.local
+    routing_key: traceID
+
+service:
+  pipelines:
+    traces:
+      receivers:
+        - otlp
+      processors: []
+      exporters:
+        - loadbalancing
+```
+
 For testing purposes, the following configuration can be used, where both the load balancer and all backends are running locally:
 ```yaml
 receivers:

exporter/loadbalancingexporter/config.go

@@ -35,6 +35,7 @@ type ResolverSettings struct {
 	Static *StaticResolver `mapstructure:"static"`
 	DNS    *DNSResolver    `mapstructure:"dns"`
 	K8sSvc *K8sSvcResolver `mapstructure:"k8s"`
+	SRV    *SRVResolver    `mapstructure:"srv"`
 }
 
 // StaticResolver defines the configuration for the resolver providing a fixed list of backends
@@ -50,8 +51,17 @@ type DNSResolver struct {
 	Timeout  time.Duration `mapstructure:"timeout"`
 }
 
-// K8sSvcResolver defines the configuration for the DNS resolver
+// K8sSvcResolver defines the configuration for the kubernetes Service resolver
 type K8sSvcResolver struct {
 	Service string  `mapstructure:"service"`
 	Ports   []int32 `mapstructure:"ports"`
 }
+
+// TODO: Should a common struct be used for dns-based resolvers?
+// SRVResolver defines the configuration for the DNS resolver of SRV records for headless Services
+type SRVResolver struct {
+	Hostname string        `mapstructure:"hostname"`
+	Port     string        `mapstructure:"port"`
+	Interval time.Duration `mapstructure:"interval"`
+	Timeout  time.Duration `mapstructure:"timeout"`
+}

exporter/loadbalancingexporter/go.mod

@@ -19,6 +19,7 @@ require (
 	k8s.io/api v0.28.4
 	k8s.io/apimachinery v0.28.4
 	k8s.io/client-go v0.28.4
+	golang.org/x/exp v0.0.0-20230321023759-10a507213a29
 	k8s.io/utils v0.0.0-20230406110748-d93618cff8a2
 	sigs.k8s.io/controller-runtime v0.16.3
 )

exporter/loadbalancingexporter/loadbalancer.go

@@ -85,6 +85,15 @@ func newLoadBalancer(params exporter.CreateSettings, cfg component.Config, facto
 			return nil, err
 		}
 	}
+	if oCfg.Resolver.SRV != nil {
+		srvLogger := params.Logger.With(zap.String("resolver", "DNS SRV"))
+
+		var err error
+		res, err = newSRVResolver(srvLogger, oCfg.Resolver.SRV.Hostname, oCfg.Resolver.SRV.Port, oCfg.Resolver.SRV.Interval, oCfg.Resolver.SRV.Timeout)
+		if err != nil {
+			return nil, err
+		}
+	}
 
 	if res == nil {
 		return nil, errNoResolver