All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- #117 Switch to golangci-lint and fix all the lint (lot of changes, some just formatting, some might have impact on functionality)
- #118 Upgrade Go to 1.23
- #119 Upgrade all deps
- #116 Add
make test-release
to test the release process and run it on the default branch.
- #115 Add
qemu
before release to fix the multi-platform release issue.
Big thanks to @mohammedalics who did all the work on this release!
- #113 Remove docker remote version to use the default circleci version
- #101 prometheusIngester add
offset
option to query, allowing to query bit older data to ensure consistency on prometheus compatible systems using remote write.
- #95 prometheusIngester sends user-agent header
- #96 prometheusIngester headers from environment value now works
- #94 prometheusIngester now can send custom headers to prometheus api server
- #88 slo-rules-generator does not output empty
escalate
label - #87 slo-exporter's SLO recording rule for uptime success events
- slo-rules-generator now properly truncates files when overwriting them
- #77 Experimental slo-rules-generator tool
- #78 Upgrade dependencies mainly to avoid reported CVEs in grafana/loki
- #80 prometheus-ingester: Add staleness support and possibility to have shorter query interval
- #81 Upgrade dependencies to avoid CVE in containerd and image-spec
- #69 Experimental Exemplars support for the
prometheus_exporter
module.
- #71 Fix corner cases in StringMap.Merge(), StringMap.Without()
- #60 New module eventMetadataRenamer
- #48 New ingester module
kafkaIngester
- #44 Install missing ca-certificates to docker base image
- #43 slo-event-producer:
slo_matcher
values are now regular expressions.
- #37 New ingester module
envoyAccessLogServer
to receive remote access logs from an Envoy proxy over the gRPC.
- #39 Prometheus-exporter: Additional SLO event metadata does not get overwritten.
- #40 Upgrade to Go 1.15
- #28 Abstracted statistical classifier history to generic interface in new storage package.
- #34 Prometheus recording rules: Unified computation for SLO classes uptime, uptime-alt (slo-expoter's SLO)
- #34 Prometheus alerts: missing data alert now explicitly matches only enabled SLO domains
- #32 Prometheus alerts, recording rules: now fully reflects our internal setup
- #31 Grafana dashboard: SLO Drilldown shows current increase of events on background of error budget change
- #33 prometheus-ingester now always expose query_fails_total
- #32 Grafana dashboard: SLO Effective Burn-rate
- #27 Empty configuration is now evaluated as invalid
- #25
--version
command-line flag showing only the build version
- #24 DynamicClassifier: Panic on CSV with unexpected number of fields.
- #18 Tailer: Upgraded hpcloud/tail package to latest commit.
- #20 CI:
- Release additional Docker image tags:
latest
Latest released version.vX
Latest released version for the major version.vX.Y
Latest released version for the minor version.
- Prebuilt binaries for:
slo-exporter_darwin_386
slo-exporter_darwin_amd64
slo-exporter_linux_386
slo-exporter_linux_amd64
slo-exporter_windows_386
slo-exporter_windows_amd64
- Release additional Docker image tags:
- CI pipeline
- build
- on release, publish docker image and github release with binaries
- Dockerfile labels
- Dockerfile src image
- SLO computation recording rules, alerts
- slo-exporter grafana dashboard
- PrometheusIngester: Fixed unit of the
slo_exporter_prometheus_ingester_query_duration_seconds_bucket
metric.
- BREAKING Dropped the
normalizer
module in favour of therelabel
andeventKeyGenerator
modules. Those can be used to sanitize metadata values and compose the event key from any metadata keys. - BREAKING Dropped the
eventFilter
module in favour of therelabel
module. - BREAKING sloEventProducer: dropped
honor_slo_result
configuration option, same behavior can be now achieved using failure conditions and metadata filters. - BREAKING sloEventProducer: all operators renamed
- Dynamic classifier no longer exposes status_code label to unclassified events metric, you have to explicitly specify it in the
unclassifiedEventMetadataKeys
configuration option. - BREAKING Tailer: no longer sets the event key, use the
eventKeyGenerator
module. - BREAKING Tailer: no longer sets the SLO classification of the event, use the
metadataClassifier
module.
- New module
relabel
allowing to modify event metadata using Prometheus relabel config. - New flag
--check-config
to verify if configuration is ok. - Enabled mutex and block profiling.
- Upgraded to Go 1.14
- PrometheusIngester: added
query_type
label forslo_exporter_prometheus_ingester_query_fails_total
metric
- PrometheusIngester: fixed isolation of
histogram_increase
query type causing invalid computation of increase.
- sloEventProducer: notEqualTo, notMatchesRegexp, numberNotEqualTo operators
- PrometheusIngester: last defined query shadows the previous ones
- DynamicClassifier: Log entire event on unsuccessful classification.
- sloEventProducer: Expose metrics based on slo_rules configuration (
exposeRulesAsMetrics
slo_event_producer module option).
- New Prometheus-ingester's query configuration option 'resultAsQuantity'
- New module
metadataClassifier
to classify event based on it's metadata. See the docs for more info.
- prometheus_exporter's LabelNames.sloApp default value now matches documented one (slo_app)
- sloEventProducer: Added new operators
equalTo
,numberEqualTo
,numberEqualOrHigherThan
andnumberEqualOrLessThan
.
- BREAKING prometheusIngester: Renamed
increase
query type tocounter_increase
. - BREAKING sloEventProducer: Dropped unused configuration key
event_type
in rules file.
- Prometheus ingester: New query type
histogram_increase
to generate events with for each bucket.
- Prometheus ingester 'increase' query type
- New module
eventKeyGenerator
to generate event key from its metadata. See the docs. - Generic metadata matcher for slo_event_producer
- Prometheus ingester query types. (Existing named as 'simple')
- Generalize eventKey access/usage for all ingesters by putting it to event.Metadata
- Possibility to dynamically set the log level using HTTP endpoint, see the docs.
- StatisticalClassifier module now allows to set default weights.
- Fixed logging in reporting metrics from prometheus exporter module.
- Roll back mutex and blocking goroutine profiling in
pprof
beacause of an issue in go 1.14.1.
- New module
statisticalClassifier
to classify events based on previous events classification statistical distribution. Read more in the module documentation. - Enabled mutex and blocking goroutine profiling in
pprof
.
- Fixed CPU usage burst caused by leaking timers.
- Added Go debugging handlers to web interface on
/debug/pprof/
. For usage see https://blog.golang.org/pprof.
- Removed redundant namespacing for dynamic classifier metric
events_processed_total
.
- Kubernetes manifests fixed to use
/liveness
probe endpoint.
- Kubernetes manifests fixed to match the breaking changes in 4.0.0.
- BREAKING Pipeline structure is now defined using the
pipeline
configuration option. For more information see the architecture documentation. - BREAKING The
log_level
configuration option was removed and replaced with the--log-level
command line flag. Also it can be still configured with the ENV variableSLO_EXPORTER_LOGLEVEL
. - BREAKING The
--disable-timescale-exporter
and--disable-prometheus-exporter
flags were dropped in favor of dynamic pipeline structure configuration. - BREAKING The timescale exporter was dropped.
- BREAKING The
minimumGracefulShutdownDuration
configuration option was replaced withafterPipelineShutdownDelay
to be more intuitive.
- HttpRequest.Headers, HttpRequest.Metadata is now filled only with data not matching conf.tailer.emptyGroupRE.
- Drop frpcStatus as a dedicated attribute for HttpRequest.
- If eventKey matching group in tailer RE is nonempty, its value is propagated to HttpRequest.EventKey.
- Inconsistencies in aggregated SLO metrics exposed to Prometheus.
- Normalizer now does not drop event if eventKey already set.
- Refactored prometheus exporter to implement Collector interface and to not require known labels beforehand.
- BREAKING Event filter module now filters using metadata using
metadataFilter
. OldfilteredHttpStatusCodeMatchers
andfilteredHttpHeaderMatchers
were dropped. - BREAKING Failure criteria configuration synatx of sloEventProducer module has changed.
failure_criteria
is nowfailure_conditions
criterium
is now calledoperator
- Operators are evaluated on event metadata.
key
field was added in order to specify on which metadata is the given operator to be evaluated. - Old criteria were dropped and newly available operators are
matchesRegexp
,numberHigherThan
anddurationHigherThan
. - Example of new failure conditions syntax:
failure_conditions: - operator: matchesRegexp key: "metadataKey" value: ".*"
- Possibility to add additional metadata labels to
events_processed_total
metric of dynamic classifier usingunclassifiedEventMetadataKeys
.
- Slo_rules now support honor_slo_result.
- All unknown named groups parsed by tailer are set as HTTP headers.
- Delayed graceful shutdown.
- BREAKING Graceful shutdown timeout conf options.
- gracefulShutdownTimeout replaced with maximumGracefulShutdownDuration.
- afterGracefulShutdownDelay replaced with minimumGracefulShutdownDuration.
- app_build_info metric
- Fixed hanging shutdown when all modules ended without explicit termination request.
- Tailer line matching regular expression is now part of configuration.
- Tailer is able to initialize event with SloClassification, if provided within log line.
- Update dynamic classifier cache with data from already classified event.
- E2e-tests' run_tests.sh is now checking whether slo_exporter is running, before test will proceed.
- Implemented prometheus ingester.
- Optional gracefulShutdownTimeout configuration.
- BREAKING Request event normalizer now uses regular expressions for filtering.
- Config option
filteredHttpStatusCodes
is nowfilteredHttpStatusCodeMatchers
and is list of regular expressions instead of integers. - Config option
filteredHttpHeaders
is nowfilteredHttpHeaderMatchers
and is map of regular expression matching header name to regular expression matching header value.
- Config option
- Refactored loading of event normalizer configuration.
- StringMap.Without fixed logic and added tests.
- StringMap.Merge do not return empty map when merged with nil.
- Implemented graceful shutdown for each pipeline processor.
- Process multiple domains from single data source.
- Typo in label failedToClassify.
- Normalize also
.ico
files as:image
.
- Frontend API GraphQL endpoints without
operationName
are now classified asno_slo
.