Table of Contents
Sampling is a mechanism to control the noise and overhead introduced by OpenTelemetry by reducing the number of samples of traces collected and sent to the backend.
Sampling may be implemented on different stages of a trace collection. The earliest sampling could happen before the trace is actually created, and the latest sampling could happen on the Collector which is out of process.
The OpenTelemetry API has two properties responsible for the data collection:
IsRecording
field of aSpan
. Iffalse
the currentSpan
discards all tracing data (attributes, events, status, etc.). Users can use this property to determine if collecting expensive trace data can be avoided. Span Processor MUST receive only those spans which have this field set totrue
. However, Span Exporter SHOULD NOT receive them unless theSampled
flag was also set.Sampled
flag inTraceFlags
onSpanContext
. This flag is propagated via theSpanContext
to child Spans. For more details see the W3C Trace Context specification. This flag indicates that theSpan
has beensampled
and will be exported. Span Exporters MUST receive those spans which haveSampled
flag set to true and they SHOULD NOT receive the ones that do not.
The flag combination SampledFlag == false
and IsRecording == true
means that the current Span
does record information, but most likely the child
Span
will not.
The flag combination SampledFlag == true
and IsRecording == false
could cause gaps in the distributed trace, and because of this OpenTelemetry API
MUST NOT allow this combination.
The following table summarizes the expected behavior for each combination of
IsRecording
and SampledFlag
.
IsRecording |
Sampled Flag |
Span Processor receives Span? | Span Exporter receives Span? |
---|---|---|---|
true | true | true | true |
true | false | true | false |
false | true | Not allowed | Not allowed |
false | false | false | false |
The SDK defines the interface Sampler
as well as a set of
built-in samplers and associates a Sampler
with each [TracerProvider
].
When asked to create a Span, the SDK MUST query the Sampler
's ShouldSample
method before actually creating the span, and act accordingly:
see description of ShouldSample
's return value below for how to set IsRecording
and Sampled
on the Span,
and the table above on whether to pass the Span
to SpanProcessor
s.
A non-recording span MAY be implemented using the same mechanism as when a Span
is created with no API-implementation installed
(sometimes called a NoOpSpan
or DefaultSpan
).
Sampler
interface allows users to create custom samplers which will return a
sampling SamplingResult
based on information that is typically available just
before the Span
was created.
Returns the sampling Decision for a Span
to be created.
Required arguments:
- Parent
SpanContext
. May be invalid to indicate a root span. TraceId
of theSpan
to be created. If the parentSpanContext
contains a validTraceId
, they MUST always match.- Name of the
Span
to be created. SpanKind
of theSpan
to be created.- Initial set of
Attributes
of theSpan
to be created. - Collection of links that will be associated with the
Span
to be created. Typically useful for batch operations, see Links Between Spans.
Return value:
It produces an output called SamplingResult
which contains:
- A sampling
Decision
. One of the following enum values:DROP
-IsRecording() == false
, span will not be recorded and all events and attributes will be dropped.RECORD_ONLY
-IsRecording() == true
, butSampled
flag MUST NOT be set.RECORD_AND_SAMPLE
-IsRecording() == true
ANDSampled
flag` MUST be set.
- A set of span Attributes that will also be added to the
Span
. The returned object must be immutable (multiple calls may return different immutable objects). - A
Tracestate
that will be associated with theSpan
through the newSpanContext
. Note: If the sampler returns an emptyTracestate
here, theTracestate
will be cleared, so samplers should normally return the passed-inTracestate
if they do not intend to change it.
Returns the sampler name or short description with the configuration. This may
be displayed on debug pages or in the logs. Example:
"TraceIdRatioBased{0.000100}"
.
Description MUST NOT change over time and caller can cache the returned value.
OpenTelemetry supports a number of built-in samplers to choose from.
The default sampler is ParentBased(root=AlwaysOn)
.
- Returns
RECORD_AND_SAMPLE
always. - Description MUST be
AlwaysOnSampler
.
- Returns
DROP
always. - Description MUST be
AlwaysOffSampler
.
- The
TraceIdRatioBased
MUST ignore the parentSampledFlag
. To respect the parentSampledFlag
, theTraceIdRatioBased
should be used as a delegate of theParentBased
sampler specified below. - Description MUST be
TraceIdRatioBased{0.000100}
.
TODO: Add details about how the TraceIdRatioBased
is implemented as a function
of the TraceID
.
- The sampling algorithm MUST be deterministic. A trace identified by a given
TraceId
is sampled or not independent of language, time, etc. To achieve this, implementations MUST use a deterministic hash of theTraceId
when computing the sampling decision. By ensuring this, running the sampler on any childSpan
will produce the same decision. - A
TraceIdRatioBased
sampler with a given sampling rate MUST also sample all traces that anyTraceIdRatioBased
sampler with a lower sampling rate would sample. This is important when a backend system may want to run with a higher sampling rate than the frontend system, this way all frontend traces will still be sampled and extra traces will be sampled on the backend only.
- This is a composite sampler.
ParentBased
helps distinguished between the following cases:- No parent (root span).
- Remote parent (
SpanContext.IsRemote() == true
) withSampledFlag
equalstrue
- Remote parent (
SpanContext.IsRemote() == true
) withSampledFlag
equalsfalse
- Local parent (
SpanContext.IsRemote() == false
) withSampledFlag
equalstrue
- Local parent (
SpanContext.IsRemote() == false
) withSampledFlag
equalsfalse
Required parameters:
root(Sampler)
- Sampler called for spans with no parent (root spans)
Optional parameters:
remoteParentSampled(Sampler)
(default: AlwaysOn)remoteParentNotSampled(Sampler)
(default: AlwaysOff)localParentSampled(Sampler)
(default: AlwaysOn)localParentNotSampled(Sampler)
(default: AlwaysOff)
Parent | parent.isRemote() | parent.IsSampled() | Invoke sampler |
---|---|---|---|
absent | n/a | n/a | root() |
present | true | true | remoteParentSampled() |
present | true | false | remoteParentNotSampled() |
present | false | true | localParentSampled() |
present | false | false | localParentNotSampled() |
New Tracer
instances are always created through a TracerProvider
(see
API). The name
and version
arguments
supplied to the TracerProvider
must be used to create an
InstrumentationLibrary
instance which is stored on the created
Tracer
.
Configuration (i.e., Span processors and Sampler
)
MUST be managed solely by the TracerProvider
and it MUST provide some way to
configure them, at least when creating or initializing it.
The TracerProvider MAY provide methods to update the configuration. If
configuration is updated (e.g., adding a SpanProcessor
),
the updated configuration MUST also apply to all already returned Tracers
(i.e. it MUST NOT matter whether a Tracer
was obtained from the
TracerProvider
before or after the configuration change).
Note: Implementation-wise, this could mean that Tracer
instances have a
reference to their TracerProvider
and access configuration only via this
reference.
The API-level definition for Span's interface
only defines write-only access to the span.
This is good because instrumentations and applications are not meant to use the data
stored in a span for application logic.
However, the SDK needs to eventually read back the data in some locations.
Thus, the SDK specification defines sets of possible requirements for
Span
-like parameters:
-
Readable span: A function receiving this as argument MUST be able to access all information that was added to the span, as listed in the API spec. In particular, it MUST also be able to access the
InstrumentationLibrary
andResource
information (implicitly) associated with the span. It must also be able to reliably determine whether the Span has ended (some languages might implement this by having an end timestamp ofnull
, others might have an explicithasEnded
boolean).A function receiving this as argument might not be able to modify the Span.
Note: Typically this will be implemented with a new interface or (immutable) value type. In some languages SpanProcessors may have a different readable span type than exporters (e.g. a
SpanData
type might contain an immutable snapshot and aReadableSpan
interface might read information directly from the same underlying data structure that theSpan
interface manipulates). -
Read/write span: A function receiving this as argument must have access to both the full span API as defined in the API-level definition for span's interface and additionally must be able to retrieve all information that was added to the span (as with readable span).
It MUST be possible for functions being called with this to somehow obtain the same
Span
instance and type that the span creation API returned (or will return) to the user (for example, theSpan
could be one of the parameters passed to such a function, or a getter could be provided).
Span processor is an interface which allows hooks for span start and end method
invocations. The span processors are invoked only when
IsRecording
is true.
Built-in span processors are responsible for batching and conversion of spans to exportable representation and passing batches to exporters.
Span processors can be registered directly on SDK TracerProvider
and they are
invoked in the same order as they were registered.
Each processor registered on TracerProvider
is a start of pipeline that consist
of span processor and optional exporter. SDK MUST allow to end each pipeline with
individual exporter.
SDK MUST allow users to implement and configure custom processors and decorate built-in processors for advanced scenarios such as tagging or filtering.
The following diagram shows SpanProcessor
's relationship to other components
in the SDK:
+-----+--------------+ +-------------------------+ +-------------------+
| | | | | | |
| | | | Batching Span Processor | | SpanExporter |
| | +---> Simple Span Processor +---> (JaegerExporter) |
| | | | | | |
| SDK | Span.start() | +-------------------------+ +-------------------+
| | Span.end() |
| | |
| | |
| | |
| | |
+-----+--------------+
OnStart
is called when a span is started. This method is called synchronously
on the thread that started the span, therefore it should not block or throw
exceptions.
Parameters:
span
- a read/write span object for the started span. It SHOULD be possible to keep a reference to this span object and updates to the span SHOULD be reflected in it. For example, this is useful for creating a SpanProcessor that periodically evaluates/prints information about all active span from a background thread.parentContext
- the parentContext
of the span that the SDK determined (the explicitly passedContext
, the currentContext
or an emptyContext
if that was explicitly requested).
Returns: Void
OnEnd
is called after a span is ended (i.e., the end timestamp is already set).
This method MUST be called synchronously within the Span.End()
API,
therefore it should not block or throw an exception.
Parameters:
Span
- a readable span object for the ended span. Note: Even if the passed Span may be technically writable, since it's already ended at this point, modifying it is not allowed.
Returns: Void
Shuts down the processor. Called when SDK is shut down. This is an opportunity for processor to do any cleanup required.
Shutdown
SHOULD be called only once for each SpanProcessor
instance. After
the call to Shutdown
, subsequent calls to OnStart
, OnEnd
, or ForceFlush
are not allowed. SDKs SHOULD ignore these calls gracefully, if possible.
Shutdown
SHOULD provide a way to let the caller know whether it succeeded,
failed or timed out.
Shutdown
MUST include the effects of ForceFlush
.
Shutdown
SHOULD complete or abort within some timeout. Shutdown
can be
implemented as a blocking API or an asynchronous API which notifies the caller
via a callback or an event. Language library authors can decide if they want to
make the shutdown timeout configurable.
Exports all spans that have not yet been exported to the configured Exporter
.
ForceFlush
SHOULD provide a way to let the caller know whether it succeeded,
failed or timed out.
ForceFlush
SHOULD only be called in cases where it is absolutely necessary,
such as when using some FaaS providers that may suspend the process after an
invocation, but before the Processor
exports the completed spans.
ForceFlush
SHOULD complete or abort within some timeout. ForceFlush
can be
implemented as a blocking API or an asynchronous API which notifies the caller
via a callback or an event. Language library authors can decide if they want to
make the flush timeout configurable.
The standard OpenTelemetry SDK MUST implement both simple and batch processors, as described below. Other common processing scenarios should be first considered for implementation out-of-process in OpenTelemetry Collector
This is an implementation of SpanProcessor
which passes finished spans
and passes the export-friendly span data representation to the configured
SpanExporter
, as soon as they are finished.
Configurable parameters:
exporter
- the exporter where the spans are pushed.
This is an implementation of the SpanProcessor
which create batches of finished
spans and passes the export-friendly span data representations to the
configured SpanExporter
.
Configurable parameters:
exporter
- the exporter where the spans are pushed.maxQueueSize
- the maximum queue size. After the size is reached spans are dropped. The default value is2048
.scheduledDelayMillis
- the delay interval in milliseconds between two consecutive exports. The default value is5000
.exportTimeoutMillis
- how long the export can run before it is cancelled. The default value is30000
.maxExportBatchSize
- the maximum batch size of every export. It must be smaller or equal tomaxQueueSize
. The default value is512
.
Span Exporter
defines the interface that protocol-specific exporters must
implement so that they can be plugged into OpenTelemetry SDK and support sending
of telemetry data.
The goal of the interface is to minimize burden of implementation for protocol-dependent telemetry exporters. The protocol exporter is expected to be primarily a simple telemetry data encoder and transmitter.
The exporter must support two functions: Export and Shutdown. In
strongly typed languages typically there will be 2 separate Exporter
interfaces, one that accepts spans (SpanExporter) and one that accepts metrics
(MetricsExporter).
Exports a batch of readable spans. Protocol exporters that will implement this function are typically expected to serialize and transmit the data to the destination.
Export() will never be called concurrently for the same exporter instance. Export() can be called again only after the current call returns.
Export() must not block indefinitely, there must be a reasonable upper limit
after which the call must time out with an error result (Failure
).
Any retry logic that is required by the exporter is the responsibility of the exporter. The default SDK SHOULD NOT implement retry logic, as the required logic is likely to depend heavily on the specific protocol and backend the spans are being sent to.
Parameters:
batch - a batch of readable spans. The exact data type of the batch is language
specific, typically it is some kind of list,
e.g. for spans in Java it will be typically Collection<SpanData>
.
Returns: ExportResult:
ExportResult is one of:
Success
- The batch has been successfully exported. For protocol exporters this typically means that the data is sent over the wire and delivered to the destination server.Failure
- exporting failed. The batch must be dropped. For example, this can happen when the batch contains bad data and cannot be serialized.
Note: this result may be returned via an async mechanism or a callback, if that is idiomatic for the language implementation.
Shuts down the exporter. Called when SDK is shut down. This is an opportunity for exporter to do any cleanup required.
Shutdown
should be called only once for each Exporter
instance. After the
call to Shutdown
subsequent calls to Export
are not allowed and should
return a Failure
result.
Shutdown
should not block indefinitely (e.g. if it attempts to flush the data
and the destination is unavailable). Language library authors can decide if they
want to make the shutdown timeout configurable.
Based on the generic interface definition laid out above library authors must define the exact interface for the particular language.
Authors are encouraged to use efficient data structures on the interface boundary that are well suited for fast serialization to wire formats by protocol exporters and minimize the pressure on memory managers. The latter typically requires understanding of how to optimize the rapidly-generated, short-lived telemetry data structures to make life easier for the memory manager of the specific language. General recommendation is to minimize the number of allocations and use allocation arenas where possible, thus avoiding explosion of allocation/deallocation/collection operations in the presence of high rate of telemetry data generation.
These are examples on what the Exporter
interface can look like in specific
languages. Examples are for illustration purposes only. Language library authors
are free to deviate from these provided that their design remain true to the
spirit of Exporter
concept.
type SpanExporter interface {
Export(batch []ExportableSpan) ExportResult
Shutdown()
}
type ExportResult struct {
Code ExportResultCode
WrappedError error
}
type ExportResultCode int
const (
Success ExportResultCode = iota
Failure
)
public interface SpanExporter {
public enum ResultCode {
Success, Failure
}
ResultCode export(Collection<ExportableSpan> batch);
void shutdown();
}