diff --git a/docs/dotnet/dotnet-connection-traces.md b/docs/dotnet/dotnet-connection-traces.md index de725aa6e8..47b84ecff6 100644 --- a/docs/dotnet/dotnet-connection-traces.md +++ b/docs/dotnet/dotnet-connection-traces.md @@ -2,50 +2,62 @@ linkTitle: HTTP client and server ---> -# Semantic Conventions for HTTP client connection spans emitted by .NET +# Semantic Conventions for HTTP client spans emitted by .NET **Status**: [Experimental][DocumentStatus] -This article defines semantic conventions for HTTP client connections, DNS and TLS spans emitted by .NET. +This article defines semantic conventions for HTTP client, DNS and TLS spans emitted by .NET. -* [HTTP client connection](#http-connection) -* [HTTP connection setup](#http-connection-setup) -* [Socket connection setup](#socket-connection-setup) -* [DNS](#dns) -* [TLS](#tls) -* [Examples](#examples) +- [HTTP Request: wait for connection](#http-request-wait-for-connection) +- [HTTP connection setup](#http-connection-setup) +- [Socket connection setup](#socket-connection-setup) +- [DNS](#dns) +- [TLS](#tls) +- [Examples](#examples) + - [HTTP request was performed on a connection that was immediately available](#http-request-was-performed-on-a-connection-that-was-immediately-available) + - [HTTP request that has to wait for existing connection to become available](#http-request-that-has-to-wait-for-existing-connection-to-become-available) + - [HTTP request has to wait for connection setup](#http-request-has-to-wait-for-connection-setup) + - [HTTP request has to wait for connection setup and other requests on that connection to complete](#http-request-has-to-wait-for-connection-setup-and-other-requests-on-that-connection-to-complete) -## Overview +.NET `HttpClient` reports HTTP client spans according to [HTTP Semantic Conventions](/docs/http/http-spans.md#http-client). -.NET reports spans related to HTTP connection establishment and its stages starting from .NET 9. -Application developers are encouraged to enable corresponding instrumentation in development and test environments. +TODO: we might need to describe some .NET specifics. Leaving it out for now. -Enabling connection-level details in production may result in increased telemetry volume and reduced performance. +In addition to HTTP request spans, `HttpClient` reports spans describing HTTP connection establishment and its stages. While such spans represent low-level details, connection lifetime is usually measured in minutes, therefore in common case -when application is under the load and connections are not established for every request, the rate of connection-related spans -is expected to be much lower than the rate of HTTP client spans. +when application is under the load, the rate of connection-related spans is expected to be much lower than the rate of HTTP client spans. -## HTTP client connection +Application developers are encouraged to enable corresponding instrumentation in development or test environments. Using connection-level +instrumentation in production should be done with caution as it increases the volume of reported telemetry and has performance overhead that +depends on the application. -HTTP connection span describes the lifetime of the HTTP connection. TODO - after it's created or from the initiation time? -HTTP connection duration is also reported as [`http.client.connection.duration` metric](/dotnet/dotnet-http-metrics.md#metric-httpclientconnectionduration). +## HTTP Request: wait for connection -Client HTTP request spans SHOULD have links to this connection span (when it is reported). +The span describes the time it takes for the HTTP request to obtain a connection from the connection pool. +The span is reported only if there is no connection that's readily available. It's reported as a child of HTTP client span. -Span name SHOULD be `http connection {address}:{server.port}`. -The `{address}` SHOULD be `server.address` when it's available and `network.peer.address` otherwise. +The span ends when the connection is obtained - it could happen when an existing connection becomes available or once +a new connection is established, so the time wait-for-connection span tracks is different than one describing [*HTTP connection setup*](#http-connection-setup). -Span kind SHOULD be `CLIENT` +If a new connection was created to serve the request and the corresponding [*HTTP connection setup*](#http-connection-setup) was reported, the instrumentation adds the +link to the *HTTP connection setup* on the *Wait for connection* span. + +Span name SHOULD be `wait_for_connection {server.address}:{server.port}`. -Corresponding `Activity.OperationName` is `System.Net.Http.Connections.Connection`, `ActivitySource` name - `System.Net.Http.Connections`. +Span kind SHOULD be `INTERNAL` + +Corresponding `Activity.OperationName` is `System.Net.Http.TODO.`, `ActivitySource` name - `System.Net.Http.TODO`. Span added in .NET Core 9. - +The time it takes to get a connection from the pool is also reported by the +[`http.client.request.time_in_queue` metric](/docs/dotnet/dotnet-http-metrics.md#metric-httpclientrequesttime_in_queue). + + @@ -54,11 +66,11 @@ Span added in .NET Core 9. | Attribute | Type | Description | Examples | [Requirement Level](https://opentelemetry.io/docs/specs/semconv/general/attribute-requirement-level/) | Stability | |---|---|---|---|---|---| -| [`error.type`](/docs/attributes-registry/error.md) | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | -| [`network.peer.address`](/docs/attributes-registry/network.md) | string | Peer address of the network connection - IP address or Unix domain socket name. | `10.1.2.80`; `/tmp/my.sock` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | -| [`network.protocol.version`](/docs/attributes-registry/network.md) | string | The actual version of the protocol used for network communication. [2] | `1.1`; `2` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | -| [`server.address`](/docs/attributes-registry/server.md) | string | Server domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name. [3] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | -| [`server.port`](/docs/attributes-registry/server.md) | int | Server port number. [4] | `80`; `8080`; `443` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| [`error.type`](/docs/attributes-registry/error.md) | string | Describes a class of error the operation ended with. [1] | `System.OperationCanceledException` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| [`http.request.method`](/docs/attributes-registry/http.md) | string | HTTP request method. [2] | `GET`; `POST`; `HEAD` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| [`network.protocol.version`](/docs/attributes-registry/network.md) | string | The actual version of the protocol used for network communication. [3] | `1.1`; `2` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| [`server.address`](/docs/attributes-registry/server.md) | string | Server domain name if available without reverse DNS lookup; otherwise, IP address or Unix domain socket name. [4] | `example.com`; `10.1.2.80`; `/tmp/my.sock` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| [`server.port`](/docs/attributes-registry/server.md) | int | Server port number. [5] | `80`; `8080`; `443` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | | [`url.scheme`](/docs/attributes-registry/url.md) | string | The [URI scheme](https://www.rfc-editor.org/rfc/rfc3986#section-3.1) component identifying the used protocol. | `https`; `ftp`; `telnet` | `Recommended` | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | **[1]:** The `error.type` SHOULD be predictable, and SHOULD have low cardinality. @@ -81,11 +93,26 @@ it's RECOMMENDED to: * Use a domain-specific attribute * Set `error.type` to capture all errors, regardless of whether they are defined within the domain-specific set or not. -**[2]:** If protocol version is subject to negotiation (for example using [ALPN](https://www.rfc-editor.org/rfc/rfc7301.html)), this attribute SHOULD be set to the negotiated version. If the actual protocol version is not known, this attribute SHOULD NOT be set. +**[2]:** HTTP request method value SHOULD be "known" to the instrumentation. +By default, this convention defines "known" methods as the ones listed in [RFC9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-methods) +and the PATCH method defined in [RFC5789](https://www.rfc-editor.org/rfc/rfc5789.html). -**[3]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent the server address behind any intermediaries, for example proxies, if it's available. +If the HTTP request method is not known to instrumentation, it MUST set the `http.request.method` attribute to `_OTHER`. -**[4]:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. +If the HTTP instrumentation could end up converting valid HTTP request methods to `_OTHER`, then it MUST provide a way to override +the list of known HTTP methods. If this override is done via environment variable, then the environment variable MUST be named +OTEL_INSTRUMENTATION_HTTP_KNOWN_METHODS and support a comma-separated list of case-sensitive known HTTP methods +(this list MUST be a full override of the default known method, it is not a list of known methods in addition to the defaults). + +HTTP method names are case-sensitive and `http.request.method` attribute value MUST match a known HTTP method name exactly. +Instrumentations for specific web frameworks that consider HTTP methods to be case insensitive, SHOULD populate a canonical equivalent. +Tracing instrumentations that do so, MUST also set `http.request.method_original` to the original value. + +**[3]:** If protocol version is subject to negotiation (for example using [ALPN](https://www.rfc-editor.org/rfc/rfc7301.html)), this attribute SHOULD be set to the negotiated version. If the actual protocol version is not known, this attribute SHOULD NOT be set. + +**[4]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent the server address behind any intermediaries, for example proxies, if it's available. + +**[5]:** When observed from the client side, and when communicating through an intermediary, `server.port` SHOULD represent the server port behind any intermediaries, for example proxies, if it's available. @@ -96,6 +123,22 @@ it's RECOMMENDED to: | `_OTHER` | A fallback error value to be used when the instrumentation doesn't define a custom value. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +`http.request.method` has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used. + +| Value | Description | Stability | +|---|---|---| +| `_OTHER` | Any HTTP method that the instrumentation has no prior knowledge of. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `CONNECT` | CONNECT method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `DELETE` | DELETE method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `GET` | GET method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `HEAD` | HEAD method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `OPTIONS` | OPTIONS method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `PATCH` | PATCH method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `POST` | POST method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `PUT` | PUT method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | +| `TRACE` | TRACE method. | ![Stable](https://img.shields.io/badge/-stable-lightgreen) | + + @@ -107,11 +150,8 @@ it's RECOMMENDED to: The span describes the establishment of the HTTP connection. It includes the time it takes to resolve the DNS, establish the socket connection, and perform the TLS handshake. -It's different from `dotnet.http.connection` span, which describes the lifetime of the HTTP connection. -Client HTTP request spans SHOULD have links to this connection span (when it is reported). - -Span name SHOULD be `http connection_setup {address}:{server.port}`. +Span name SHOULD be `HTTP connection_setup {address}:{server.port}`. The `{address}` SHOULD be `server.address` when it's available and `network.peer.address` otherwise. Span kind SHOULD be `CLIENT`. @@ -179,7 +219,9 @@ it's RECOMMENDED to: ## Socket connection setup The span describes the establishment of the socket connection. -It's different from `dotnet.http.connection_setup` span, which covers the DNS and TLS handshake duration in addition to socket connection setup. +It's different from [*HTTP connection setup*](#http-connection-setup) span, which covers the DNS and TLS handshake duration in addition to socket connection setup. + +When *Socket connection setup* span is reported along with *HTTP connection setup* span, the socket span becomes a child of HTTP connection setup. Span name SHOULD be `socket connection_setup {network.peer.address}:{network.peer.port}`. Span kind SHOULD be `CLIENT`. @@ -271,6 +313,9 @@ The span describes DNS lookup duration and outcome. DNS lookup duration is also reported by [`dns.lookup.duration` metric](/docs/dotnet/dotnet-dns-metrics.md#metric-dnslookupduration). +When *DNS* span is reported along with *HTTP connection setup* and *Socket connection setup* span, the *DNS* span becomes a child of *HTTP connection setup* +and a sibling of *Socket connection setup*. + Span name SHOULD be `DNS {dns.question.name}`. Span kind SHOULD be `CLIENT` @@ -334,6 +379,9 @@ The span describes TLS handshake. Span name SHOULD be `TLS {dns.question.name}`. // TODO - decide on TLS! Span kind SHOULD be `CLIENT` +When *TLS* span is reported along with *HTTP connection setup* and *Socket connection setup* span, the *DNS* span becomes a child of *HTTP connection setup* +and a sibling of *Socket connection setup*. + Corresponding `Activity.OperationName` is `System.Net.Security.TlsHandshake`, `ActivitySource` name - `System.Net.Security`. Added in .NET Core 9. @@ -396,4 +444,61 @@ it's RECOMMENDED to: ## Examples +### HTTP request was performed on a connection that was immediately available + +If connection is immediately available for the request, `HttpClient` creates one span for HTTP request. + +``` +<------------------ GET / (CLIENT, trace=t1, span=s1) ---------------------------> +``` + +### HTTP request that has to wait for existing connection to become available + +If connection was not immediately available for the request, `HttpClient` creates span for HTTP request and +*Wait for connection* span. In this example, the existing connection became available, so no additional spans +related to connection setup were reported. + +``` +<------------------ GET / (CLIENT, trace=t1, span=s1) ---------------------------> +<-- wait_for_connection host:port (INTERNAL, trace=t1, span=s2) --> +``` + +### HTTP request has to wait for connection setup + +If connection was not immediately available for the request, `HttpClient` creates span for HTTP request and +*Wait for connection* span. In this example, a new connection was created and the request was executed on it immediately after +connection was created. There was no cached DNS record for the host. + +``` +<----------------------- GET / (trace=t1, span=s1) --------------------------------> +<-- wait_for_connection (trace=t1, span=s2, link_to=t2,s3) --> + + +<--------- HTTP connection_setup (trace=t2, span=s3) --------> +<--- DNS ---> + <---- socket connection_setup ----> + <--- TLS ---> +``` + +### HTTP request has to wait for connection setup and other requests on that connection to complete + +If connection was not immediately available for the request, `HttpClient` creates span for HTTP request and +*Wait for connection* span. In this example, request was performed on a new connection, but before this connection +became available for this request, other requests were performed on it. + +``` + <--------------------- GET / (trace=t1, span=s1) ------------------------------> + <---- wait_for_connection (trace=t1, span=s2, link_to=t2,s3) ----> + + +<--- HTTP connection_setup - (trace=t2, span=s3) ---> +<-- DNS --> + <-- socket connection_setup --> + <-- TLS --> +``` + +The *HTTP connection_setup* has started before this request, it also ended much earlier than +*Wait for connection* span, indicating that there is a queue of requests and high demand for +connections in the pool. + [DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status diff --git a/model/trace/dotnet/dotnet-connections.yaml b/model/trace/dotnet/dotnet-connections.yaml index cb2eca5a2f..2baca88cca 100644 --- a/model/trace/dotnet/dotnet-connections.yaml +++ b/model/trace/dotnet/dotnet-connections.yaml @@ -1,22 +1,27 @@ groups: - - id: dotnet.http.connection + - id: dotnet.http.request.wait_for_connection type: span - span_kind: client - brief: 'The span is emitted by .NET runtime and describes the lifetime of the HTTP connection.' + span_kind: internal + brief: 'The span is emitted by .NET runtime and describes the time it takes for HTTP request to obtain a connection to run on.' note: | - HTTP connection duration is also reported by - [`http.client.connection.duration` metric](/dotnet/dotnet-http-metrics.md#metric-httpclientconnectionduration). + The span is reported only if there are no connections that are readily available to perform the request. + The span ends when the connection is obtained - it could happen if an existing connection became available or + a new connection was established, so the time wait-for-connection span tracks is different than `dotnet.http.connection_setup`. + + The time it takes to get a connection from the pool is also reported by the + [`http.client.request.time_in_queue` metric](/docs/dotnet/dotnet-http-metrics.md#metric-httpclientrequesttime_in_queue). - `Activity.OperationName`: `System.Net.Http.Connections.Connection` - `ActivitySource` name: `System.Net.Http.Connections` + `Activity.OperationName`: `System.Net.Http.WaitForConnection` + `ActivitySource` name: `"System.Net.Http` # TODO - should it be different from HTTP requests so that users can opt-in independently? Added in: .NET Core 9 attributes: + - ref: http.request.method - ref: network.protocol.version - - ref: network.peer.address - ref: server.address - ref: server.port - - ref: error.type - ref: url.scheme + - ref: error.type + examples: ["System.OperationCanceledException"] - id: dotnet.http.connection_setup type: span