v3.3.0

3.3.0 (2024-11-19)

⚠ BREAKING CHANGES

• blooms: Introduce a new block schema (V3) (#14038)
• blooms: Index structured metadata into blooms (#14061)

Features

• ability to log stream selectors before service name detection (#14154) (d7ff426)
• add app_name as a service label (#13660) (f2a16f4)
• add backoff mechanism to the retention process (#14182) (3136880)
• add functions to common.libsonnet for warpstream (#14123) (2bde071)
• add gauge loki_ingest_storage_reader_phase (#14679) (f5b0fb6)
• add gauge to track the partition_id (#14713) (a142b3d)
• add missing cluster label to mixins (#12870) (547ca70)
• add query user and query source to "executing query" log lines (#14320) (4d69929)
• add retries for s3 ObjectExists calls (#14062) (73cbbb0)
• add structured metadata to the promtail push API (#14153) (66cffcb)
• Add support for partition ingester in dashboards (#14498) (70deebf)
• Allows to configure client_max_body_size (#12924) (809a024)
• Apply patterns line length limit to json message key (#14296) (41fafd8)
• blooms: Add bloom planner and bloom builder to backend target (#13997) (bf60455)
• blooms: disk-backed queue for the bloom-planner (backport k227) (#14927) (1f6828b)
• blooms: Index structured metadata into blooms (#14061) (a2fbaa8)
• blooms: Only write key and key=value to blooms (#14686) (3af0004)
• Configurable list of json fields to mine patterns (#14528) (7050897)
• detected field values (#14350) (7983f94)
• distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a)
• Do not add empty blooms to offsets (#14577) (51c42e8)
• Extract task computing into a strategy interface (#13690) (ab5e6ea)
• fluentd-plugin-datadog-loki: support custom http headers (#14299) (e59035e)
• helm: ✨ add additional service annotations for components in distributed mode (#14131) (5978f13)
• helm: add configurable extraEnvFrom to admin-api and enterprisegw (#14533) (5d78a3a)
• helm: Add kubeVersionOverride for Helm chart (#14434) (0935d77)
• helm: Add persistence option to memcached on Helm chart (#13619) (ef1df0e)
• helm: add tolerations to pattern-ingester statefulset (#13605) (09530c0)
• helm: Allow setting node attributes to tokengen and provisioner (#14311) (c708ae6)
• helm: Replace bloom compactor with bloom planner and builder (#14003) (08e61ca)
• helm: update chart with loki version 3.2.0 (#14281) (11b92ee)
• Helm: Update Loki Helm chart for restricted environments (#14440) (adc7538)
• implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63)
• Implement owned streams calculation using Partition Ring (#14282) (3c36ba9)
• Implement WAL segment ingestion via Kafka with partition ring (#14043) (d178f4c)
• Improve pattern ingester tracing (#14707) (80aec25)
• ingester: implement partition shuffle sharding for ingester (#14304) (1a4436c)
• Introduce new ObjectExistsWithSize API to (#14268) (ac422b3)
• Introduce shardable probabilistic topk for instant queries. (backport k227) (#14765) (02eb024)
• jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177)
• kafka: Add Ingestion from Kafka in Ingesters (#14192) (b6e9945)
• kafka: Add support for SASL auth to Kafka (#14487) (e2a209c)
• kafka: Enable querier to optionally query partition ingesters (#14418) (633bb5e)
• kafka: enqueue commit offset only once per batch process (#14278) (beca6f3)
• kafka: Implement limiter using partition ring for Kafka (#14359) (5cbb239)
• kafka: Remove rate limits for kafka ingestion (#14460) (83a8893)
• kafka: Replay kafka from last commit before allowing ingesters to become ready (#14330) (39b57ec)
• kafka: Start ingester flush loop before trying to catch up from Kafka (#14505) (524ed81)
• logcli: add gzip compression option (#14598) (4d3f9f5)
• loki: include structured_metadata size while asserting rate limit (#14571) (a962edb)
• max-allowed-line-length: add config to set max-allowed-line-length in pattern ingester (#14070) (0780456)
• mixin / add loki compaction not successfull alert (#14239) (da04f50)
• mixin, allow overriding of some labels by parameterizing mixin recording/alert rules (#11495) (f1425b6)
• mixins / allow bloom dashboards disabling (#14177) (ce2e6d5)
• mixins: Allow hiding useless rows in loki-operational (#13646) (3aa4f22)
• mixins: merge resources dashboards for ssd into one (#13471) (45b8719)
• move detected field logic to query frontend (#14212) (36ace66)
• move metric aggregation to a per-tenant config (#14709) (c1fde26)
• New bloom planning using chunk size TSDB stats (#14547) (673ede1)
• Optionally require writes to kafka on Push requests (#14186) (7c78232)
• revert "feat: add functions to common.libsonnet for warpstream" (#14129) (18c27f9)
• ruler: enables ruler store that uses clients from thanos-io/objstore pkg (#11713) (8bca2e7)
• storage: AWS backend using thanos.io/objstore (#11221) (b872246)
• storage: Azure backend using thanos.io/objstore (#11315) (5824e3d)
• storage: GCS backend using thanos.io/objstore (#11132) (c059ace)
• support ruler sidecar in singleBinary mode (#13572) (684baf7)
• track discarded data by usageTracker (#14081) (c65721e)

Bug Fixes

• detected_fields: return parsed labels when parsers are passed (#14047) (aa1ac99)
• Add additional validation for timeout while retrieving headers (#14217) (8322e51)
• Add s3 principal to iam policy attached to sqs in lambda-promtail terraform code (#14619) (db0889e)
• Add tenant limits as dependency to pattern ingester (#14665) (31eea90)
• aggregated-metrics: correctly create logfmt string (#14124) (63e84b4)
• allow any level for aggregated metrics (#14255) (c001a1d)
• allow rename of structuremetadata labels (#13955) (2d4792a)
• always write detected_level when enabled, even if unknown (#14464) (41c6b6c)
• blooms: Check length of tasks before accessing first element in slice (#14634) (601f549)
• blooms: Copy chunks from ForSeries (backport k227) (#14864) (d10f79c)
• blooms: Do not restart builders when planner disconnects (backport k227) (#14922) (213e8ee)
• blooms: Exclude label filters where label name is part of the series labels. (#14661) (d1668f6)
• blooms: Fix panic in initialisation of the bloom planner and builder (#14110) (8307c42)
• blooms: Fix strategy logger and add task test (backport k227) (#14921) (dc36a1e)
• blooms: Fix tenants slice on loadTenantTables (backport k227) (#14901) (540dd5a)
• blooms: Skip multi-tenant TSDBs during bloom planning (backport k227) (#14888) (631cff3)
• build: Use Debian Bullseye base image for build image (#14368) (3beb8ff)
• canary: Reconnect immediately upon tail max duration (#14287) (9267ee3)
• ci: fixed Publish Rendered Helm Chart Diff workflow (#14365) (6de6420)
• ci: updated helm diff rendering workflow (#14424) (916e511)
• config: Copy Alibaba and IBM object storage configuration from common (#14297) (59ff1ec)
• config: migrate renovate config (#14646) (a67d8ef)
• correct _extracted logic in detected fields (#14064) (1b3ba53)
• correct OTLP documentation typo (#14602) (063c590)
• deps: update aws-sdk-go-v2 monorepo (#13986) (6f49123)
• deps: update aws-sdk-go-v2 monorepo (#14742) (53a1ab7)
• deps: update github.com/grafana/dskit digest to 687ec48 (#14395) (c2f38e1)
• deps: update github.com/grafana/dskit digest to 7c41a40 (#14277) (f39cdbd)
• deps: update github.com/grafana/dskit digest to 931a021 (#14032) (7c18642)
• deps: update github.com/grafana/dskit digest to b69ac1b (#14355) (9d7a6ea)
• deps: update github.com/grafana/dskit digest to f52de24 (#14319) (a4f3edf)
• deps: update github.com/twmb/franz-go/pkg/kfake digest to cea7aa5 (#14590) (688c42a)
• deps: update k8s.io/utils digest to 702e33f (#14033) (b7eecc7)
• deps: update module cloud.google.com/go/bigtable to v1.33.0 (#14580) (a0920ed)
• deps: update module cloud.google.com/go/pubsub to v1.45.0 (#14361) (4351238)
• deps: update module cloud.google.com/go/pubsub to v1.45.1 (#14650) (f173708)
• deps: update module cloud.google.com/go/storage to v1.46.0 (#14744) (8e45116)
• deps: update module github.com/alicebob/miniredis/v2 to v2.33.0 (#14721) (7bfda25)
• deps: update module github.com/aws/aws-sdk-go to v1.55.5 (#14715) (03f0f5a)
• deps: update module github.com/axiomhq/hyperloglog to v0.2.0 (#14722) (0167b22)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.189 (#14044) (7fb34b4)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.192 (#14337) (6f7cae2)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.196 (#14651) (478085a)
• deps: update module github.com/baidubce/bce-sdk-go to v0.9.197 (#14682) (b898294)
• deps: update module github.com/coder/quartz to v0.1.2 (#14652) (7459e07)
• deps: update module github.com/felixge/fgprof to v0.9.5 (#14338) (a2ad3aa)
• deps: update module github.com/fsouza/fake-gcs-server to v1.50.2 (#14313) (275c97c)
• deps: update module github.com/hashicorp/raft to v1.7.1 (#14005) (e9cec1d)
• deps: update module github.com/ibm/go-sdk-core/v5 to v5.17.5 (#14045) (677d217)
• deps: update module github.com/ibm/go-sdk-core/v5 to v5.18.1 (#14716) (8395acd)
• deps: update module github.com/ibm/ibm-cos-sdk-go to v1.11.1 (#14342) (aa82a7c)
• deps: update module github.com/klauspost/compress to v1.17.10 (#14352) (e23c5ed)
• deps: update module github.com/minio/minio-go/v7 to v7.0.76 (#14006) (51f9376)
• deps: update module github.com/minio/minio-go/v7 to v7.0.77 (#14353) (d0e3ef7)
• deps: update module github.com/minio/minio-go/v7 to v7.0.80 (#14654) (eec2513)
• deps: update module github.com/ncw/swift/v2 to v2.0.3 (#14356) (c843288)
• deps: update module github.com/prometheus/client_golang to v1.20.5 (#14655) (e12f843)
• deps: update module github.com/schollz/progressbar/v3 to v3.17.0 (#14720) (4419d0f)
• deps: update module github.com/shirou/gopsutil/v4 to v4.24.10 (#14719) (3280376)
• deps: update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357) (c8e6a9d)
• deps: update module github.com/shopify/sarama to v1.43.3 (#14059) (1cf4813)
• deps: update module github.com/spf13/afero to v1.11.0 (#14060) (bbbd82b)
• deps: update module go.etcd.io/bbolt to v1.3.11 (#14358) (b7bccfc)
• deps: update module golang.org/x/net to v0.29.0 (#14341) (1b6b9da)
• detected_fields: always return empty array as null (#14112) (93009d4)
• distributor: validate partition ring is kafka is enabled (#14303) (8438d41)
• do not retain span logger created with index set initialized at query time (#14027) (4e41744)
• downgrade grpc to fix regression (#14065) (8c38d46)
• enable service detection for otlp endoint (#14036) (4f962ef)
• Expand matching for additional variations (#14221) (71d7291)
• fix bug in query result marshaling for invalid utf8 characters (#14585) (f411a07)
• helm: add missing loki.storage.azure.chunkDelimiter parameter to Helm chart (#14011) (08c70cc)
• helm: Check for rbac.namespaced condition before creating roles (#14201) (3f47f09)
• helm: Fix persistence configuration for Memcached (#14049) (ee6e1cf)
• helm: Fix wrong port name referenced for ingress NetworkPolicy (#12907) (963a25b)
• helm: Various fixes and enhancements for bloom components (#14128) (dc0cbd4)
• Improve docs for min and max table offsets (backport k227) (#14929) (3161fdc)
• kafka: Fixes partition selection in distributors (#14242) (3f47233)
• kafka: Fixes writer initialization for arm32 (#14115) (4da035b)
• kafka: Set namespace for Loki kafka metrics (#14426) (8aa8a2b)
• label_format: renamed label should use ParsedLabel category (#14515) (82fb2f0)
• level detection for warning level (#14444) (242a852)
• lint errors (#14574) (99ef900)
• log-to-span: timestamp.Time should be called with milliseconds (#14196) (f8d9143)
• logcli: Check for errors before checking for exists when fetching data (backport k227) (#14906) (31b2a63)
• logcli: create new tail response for every line (#14525) (bcfd0d1)
• logql: Fix panic in json parsing when using empty array index (#14393) (833bf0d)
• logql: updated JSONExpressionParser not to unescape extracted values if it is JSON object. (#14499) (08b1a90)
• missing dep PartitionRing for Ingester (#14292) (6354ded)
• mixin: Remove pod label from disk usage aggregation (#14180) (5d45c96)
• mixins / loki-resources-overview panel layout (#14178) (8f54ec6)
• mixins: add backend path section in loki-operational for single scalable deployment (#13023) (16881ab)
• mixins: disk space utilization panels with latest KSM versions (#13486) (0ea7431)
• mixins: retention dashboards fix metric name (#14617) (c762b9b)
• More correctly report starting phase during kafka-reader startup (#14632) (ea798e0)
• move partition_id into label to make PromQL easier (#14714) (e6cf423)
• nix build, downgrade toolchain to go1.23.1 (#14442) (26dfd62)
• pattern: Fixes latency metric namespace for tee to pattern (#14241) (ae955ed)
• promtail config unmarshalling (#14408) (a05431f)
• promtail parser for azureeventhubs message without time field (#14218) (2e62abb)
• promtail: validate scrape_config job name, do not allow duplicate job names (#13719) (f2d3499)
• Propagate query stats from quantile & topk queries (#13831) (78b275b)
• remove usage of unsafe string in label adapter unmarshal (#14216) (758364c)
• Rename mispelled filename (#14237) (cf1d4a3)
• report correct status code for metric and log queries in metrics.go (#12102) (900751c)
• Report PSRL error message correctly (#14187) (a475153)
• Revert "fix(deps): update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357)" (#14437) (d53955b)
• s3: disable client retries when congestion control is enabled (#14588) (cff9f43)
• sharding: apply offset to both from and through in shard request (#14256) (17c472d)
• skipping label if it contains special symbol (#14068) (55e374e)
• storage/chunk/client/aws: have GetObject check for canceled context (#14420) (5f325aa)
• Transform ObjectExistsWithSize into GetAttributes (#14329) (2f56f50)
• Update AWS storage timeout error for Go 1.23 behavior (#14226) (a4642b5)
• Update renovate ignore for operator API with new module path (#14581) (c9b2907)
• Wait for OwnedStreams service in Ingester startup (#14208) (a4aee4f)

Performance Improvements

• blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f)
• logql: Micro-optimizations for IP filter (#14072) (c5083c7)

Miscellaneous Chores

• blooms: Introduce a new block schema (V3) (#14038) (5395daf)