Skip to content

Latest commit

 

History

History
685 lines (528 loc) · 55.8 KB

CHANGELOG.md

File metadata and controls

685 lines (528 loc) · 55.8 KB
Raw

master / unreleased

Features

  • 1677 owen-d: Introduces numeric literals in LogQL
  • 1686 owen-d: Introduces the distributor.max-line-size flag and associated yaml config. When enabled, lines longer than this config will not be accepted.
  • 1662 owen-d: Introduces binary operators in LogQL
  • 1572 owen-d: Introduces the querier.query-ingesters-within flag and associated yaml config. When enabled, queries for a time range that do not overlap this lookback interval will not be sent to the ingesters.
  • 1558 owen-d: Introduces ingester.max-chunk-age which specifies the maximum chunk age before it's cut.
  • 1565 owen-d: The query frontend's split_queries_by_interval can now be specified as an override

1.3.0 (2020-01-16)

What's New??

With 1.3.0 we are excited to announce several improvements focusing on performance!

First and most significant is the Query Frontend:

  • 1442 cyriltovena: Loki Query Frontend

The query frontend allows for sharding queries by time and dispatching them in parallel to multiple queriers, giving true horizontal scaling ability for queries. Take a look at the jsonnet changes to see how we are deploying this in our production setup. Keep an eye out for a blog post with more information on how the frontend works and more information on this exciting new feature.

In our quest to improve query performance, we discovered that gzip, while good for compression ratio, is not the best for speed. So we introduced the ability to select from several different compression algorithms:

  • 1411 cyriltovena: Adds configurable compression algorithms for chunks

We are currently testing out LZ4 and snappy, LZ4 seemed like a good fit however we found that it didn't always compress the same data to the same output which was causing some troubles for another important improvement:

  • 1438 pstibrany: pkg/ingester: added sync period flags

Extending on the work done by @bboreham on Cortex, @pstibrany added a few new flags and code to synchronize chunks between ingesters, which reduces the number of chunks persisted to object stores and therefore also reduces the number of chunks loaded on queries and the amount of de-duplication work which needs to be done.

As mentioned above, LZ4 was in some cases compressing the same data with a different result which was interfering with this change, we are still investigating the cause of this issue (It may be in how we implemented something, or may be in the compression code itself). For now we have switched to snappy which has seen a reduction in data written to the object store from almost 3x the source data (with a replication factor of 3) to about 1.5x, saving a lot of duplicated log storage!

Another valuable change related to chunks:

  • 1406 slim-bean: allow configuring a target chunk size in compressed bytes

With this change you can set a chunk_target_size and Loki will attempt to fill a chunk to approx that size before flushing (previously a chunk size was a hard coded 10 blocks where the default block size is 262144 bytes). Larger chunks are beneficial for a few reasons, mainly on reducing API calls to your object store when performing queries, but also in reducing overhead in a few places, especially when processing very high volume log streams.

Another big improvement is the introduction of accurate rate limiting when running microservices:

  • 1486 pracucci: Add ingestion rate global limit support

Previously the rate limit was applied at each distributor, however with traffic split over many distributors the limit would need to be adjusted accordingly. This meant that scaling up distributors required changing the limit. Now this information is communicated between distributors such that the limit should be applied accurately regardless of the number of distributors.

And last but not least on the notable changes list is a new feature for Promtail:

  • 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

With this change Promtail can receive syslogs via TCP! Thanks to @bastjan for all the hard work on this submission!

Important things to note:

  • 1519 Changes a core behavior in Loki regarding logs with duplicate content AND duplicate timestamps, previously Loki would store logs with duplicate timestamps and content, moving forward logs with duplicate content AND timestamps will be silently ignored. Mainly this change is to prevent duplicates that appear when a batch is retried (the first entry in the list would be inserted again, now it will be ignored). Logs with the same timestamp and different content will still be accepted.
  • 1486 Deprecated -distributor.limiter-reload-period flag / distributor's limiter_reload_period config option.

All Changes

Once again we can't thank our community and contributors enough for the significant work that everyone is adding to Loki, the entire list of changes is long!!

Loki

  • 1526 codesome: Support for aggregation
  • 1522 cyriltovena: Adds support for the old query string regexp in the frontend.
  • 1519 rfratto: pkg/chunkenc: ignore duplicate lines pushed to a stream
  • 1511 sandlis: querier: fix panic in tailer when max tail duration exceeds
  • 1499 slim-bean: Fix a panic in chunk prefetch
  • 1495 slim-bean: Prefetch chunks while processing
  • 1496 cyriltovena: Add duplicates info and remove timing informations.
  • 1490 owen-d: Fix/deadlock frontend queue
  • 1489 owen-d: unifies reverse iterators
  • 1488 cyriltovena: Fixes response json encoding and add regression tests.
  • 1486 pracucci: Add ingestion rate global limit support* 1493 pracucci: Added max streams per user global limit
  • 1480 cyriltovena: Close iterator properly and check nil before releasing buffers.
  • 1473 rfratto: pkg/querier: don't query all ingesters
  • 1470 cyriltovena: Validates limit parameter.
  • 1448 cyriltovena: Improving storage benchmark
  • 1445 cyriltovena: Add decompression tracing instrumentation.
  • 1442 cyriltovena: Loki Query Frontend
  • 1438 pstibrany: pkg/ingester: added sync period flags
  • 1433 zendern: Using strict parsing for yaml configs
  • 1425 pstibrany: pkg/ingester: Added possibility to disable transfers.
  • 1423 pstibrany: pkg/chunkenc: Fix BenchmarkRead to focus on reading chunks, not converting bytes to string
  • 1421 pstibrany: pkg/chunkenc: change default LZ4 buffer size to 64k.
  • 1420 cyriltovena: Sets the chunk encoding correctly when creating chunk from bytes.
  • 1419 owen-d: Enables Series API in loki
  • 1413 pstibrany: RangeQuery benchmark optimizations
  • 1411 cyriltovena: Adds configurable compression algorithms for chunks
  • 1409 slim-bean: change the chunk size histogram to allow for bigger buckets
  • 1408 slim-bean: forgot to register the new metric for counting blocks per chunk
  • 1406 slim-bean: allow configuring a target chunk size in compressed bytes
  • 1405 pstibrany: Convert string to bytes once only when doing string filtering.
  • 1396 pstibrany: pkg/cfg: print help only when requested, and print it on stdout
  • 1383 beornf: Read websocket close in tail handler
  • 1071 rfratto: pkg/ingester: limit total number of errors a stream can return on push
  • 1545 joe-elliott: Critical n => m conversions
  • 1541 owen-d: legacy endpoint 400s metric queries

Promtail

  • 1515 slim-bean: Promtail: Improve position and size metrics
  • 1485 p37ruh4: Fileglob parsing fixes
  • 1472 owen-d: positions.ignore-corruptions
  • 1453 chancez: pkg/promtail: Initialize counters to 0 when creating client
  • 1436 rfratto: promtail: add support for passing through journal entries as JSON
  • 1426 wphan: Support microsecond timestamp format
  • 1416 pstibrany: pkg/promtail/client: missing URL in client returns error
  • 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

Fluent Bit

  • 1455 JensErat: fluent-bit-plugin: re-enable failing JSON marshaller tests; pass error instead of logging and ignoring
  • 1294 JensErat: fluent-bit: multi-instance support
  • 1514 shane-axiom: fluent-plugin-grafana-loki: Add fluentd_thread label when flush_thread_count > 1

FluentD

  • 1500 cyriltovena: Bump fluentd plugin to 1.2.6.
  • 1475 Horkyze: fluentd-plugin: call gsub for strings only

Docker Driver

  • 1414 cyriltovena: Adds tenant-id for docker driver.

Logcli

  • 1492 sandlis: logcli: replaced GRAFANA_* with LOKI_* in logcli env vars, set default server url for logcli to localhost

Helm

  • 1534 olivierboudet: helm : fix fluent-bit parser configuration syntax
  • 1506 terjesannum: helm: add podsecuritypolicy for fluent-bit
  • 1431 eugene100: Helm: fix issue with config.clients
  • 1430 olivierboudet: helm : allow to define custom parsers to use with fluentbit.io/parser annotation
  • 1418 evalsocket: Helm chart url added in helm.md
  • 1336 terjesannum: helm: support adding init containers to the loki pod
  • 1530 WeiBanjo: Allow extra command line args for external labels like hostname

Jsonnet

  • 1518 benjaminhuo: Fix error 'Field does not exist: jaeger_mixin' in tk show
  • 1501 anarcher: jsonnet: fix common/defaultPorts parameters
  • 1497 cyriltovena: Update Loki mixin to include frontend QPS and latency.
  • 1478 cyriltovena: Fixes the typo in the result cache config of the Loki ksonnet lib.
  • 1543 sh0rez: fix(ksonnet): use apps/v1

Docs

  • 1531 fitzoh: Documentation: Add note on using Loki with Amazon ECS
  • 1521 rfratto: docs: Document timestamp ordering rules
  • 1516 rfratto: Link to release docs in README.md, not master docs
  • 1508 cyriltovena: Fixes bad json in Loki API documentation.
  • 1505 sandlis: doc: fix sample yaml in docs for installing promtail to k8s
  • 1481 terjesannum: docs: fix broken promtail link
  • 1474 Eraac: : information about max_look_back_period
  • 1471 cyriltovena: Update README.md
  • 1466 Eraac: : Update IAM requirement
  • 1441 vtereso: : README spelling fix
  • 1437 daixiang0: fix all misspell
  • 1432 joe-elliott: Removed unsupported encodings from docs
  • 1399 vishesh92: Docs: Add configuration docs for redis
  • 1394 chancez: Documentation: Fix example AWS storage configuration
  • 1227 daixiang0: Add docker install doc
  • 1560 robshep: Promtail Docs: Update output.md
  • 1546 mattmendick: Removing third-party link
  • 1539 j18e: docs: fix syntax error in pipeline example

Build

  • 1494 pracucci: Fixed TOUCH_PROTOS in all DroneCI pipelines
  • 1479 owen-d: TOUCH_PROTOS build arg for dockerfile
  • 1476 owen-d: initiates docker daemon for circle windows builds
  • 1469 rfratto: Makefile: re-enable journal scraping on ARM

New Members!

  • 1415 cyriltovena: Add Joe as member of the team.

1.2.0 (2019-12-09)

One week has passed since the last Loki release, and it's time for a new one!

Notable Changes

We have continued our work making our API Prometheus-compatible. The key changes centered around API compatibility are:

  • 1370 slim-bean: Change /loki/api/v1/label to loki/api/v1/labels
  • 1381 owen-d: application/x-www-form-urlencoded support

Meanwhile, @pstibrany has done great work ensuring that Loki handles hash collisions properly:

  • 1247 pstibrany: pkg/ingester: handle labels mapping to the same fast fingerprint.

Other Changes

❤️ All PR's are important to us, thanks everyone for continuing to help support and improve Loki! ❤️

Features

  • 1372 cyriltovena: Let Loki start when using the debug image.
  • 1300 pstibrany: pkg/ingester: check that ingester is in LEAVING state when transferring chunks and claiming tokens. Required when using memberlist client.

Bug Fixes/Improvements

  • 1376 jstaffans: Fluentd: guard against nil values when sanitizing labels
  • 1371 cyriltovena: Logql benchmark and performance improvement.
  • 1363 cyriltovena: Fixes fluentd new push path API.
  • 1353 pstibrany: docs: Fix grpc_listen_host and http_listen_host.
  • 1350 Eraac: documentation: iam requirement for autoscaling

1.1.0 (2019-12-04)

It's been a busy 2 weeks since the 1.0.0 release and quite a few important PR's have been merged to Loki.

The most significant:

  • 1322 rfratto: Fix v1 label API to be Prometheus-compatible

Some might call this a breaking change, we are instead calling it a bug fix as our goal was to be prometheus compatible and we were not 😃

But please be aware if you are using the /loki/api/v1/label or /loki/api/v1/label/<name>/values the JSON result will be different in 1.1.0

Old result:

{
  "values": [
    "label1",
    "label2",
    "labeln"
  ]
}

New result:

{
  "status": "success",
  "data": [
    "label1",
    "label2",
    "labeln"
  ]
}

ALSO IMPORTANT

  • 1160 daixiang0: replace gzip with zip

Binaries will now be zipped instead of gzipped as many people voiced their opinion that zip is likely to be installed on more systems by default.

If you had existing automation to download and install binaries this will have to be updated to use zip instead of gzip

Notable Fixes and Improvements

  • Broken version info in startup log message:

    1095 pstibrany: Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases.

  • The hashing algorithm used to calculate the hash for a stream was creating hash collisions in some instances. Please Note this is just one part of the fix and is only in Promtail, the second part for Loki can be tracked in PR1247 which didn't quite make the cut for 1.1.0 and will be in 1.2.0:

    1254 pstibrany: pkg/promtail/client: Handle fingerprint hash collisions

  • Thank you @putrasattvika for finding and fixing an important bug where logs were some logs were missed in a query shortly after a flush!

    1299 putrasattvika: storage: fix missing logs with batched chunk iterator

  • Thank you @danieldabate for helping to again improve our API to be more Prometheus compatible:

    1355 danieldabate: HTTP API: Support duration and float formats for step parameter

  • LogQL will support duration formats that are not typically handled by Go like [1d] or [1w]

    1357 cyriltovena: Supports same duration format in LogQL as Prometheus

Everything Else

❤️ All PR's are important to us, thanks everyone for continuing to help support and improve Loki! ❤️

  • 1349 Eraac: documentation: using parsable value in example
  • 1343 dgzlopes: doc(configuration): Fix duration format.
  • 1342 whothey: Makefile: add debug symbols to loki and promtail debug builds
  • 1341 adamjohnson01: Update loki helm chart to support service account annotations
  • 1340 adamjohnson01: Pull in cortex changes to support IAM roles for EKS
  • 1339 cyriltovena: Update gem version.
  • 1333 daixiang0: fix broken link
  • 1328 cyriltovena: Fixes linter warning from the yacc file.
  • 1326 dawidmalina: Wrong api endpoint in fluent-plugin-grafana-loki
  • 1320 roidelapluie: Metrics: use Namespace everywhere when declaring metrics
  • 1318 roidelapluie: Use tenant as label name for discarded_samples metrics
  • 1317 roidelapluie: Expose discarded bytes metric
  • 1316 slim-bean: Removing old file needed for dep (no longer needed)
  • 1312 ekeih: Docs: Add missing ) in LogQL example
  • 1311 pstibrany: Include positions filename in the error when YAML unmarshal fails.
  • 1310 JensErat: fluent-bit: sorted JSON and properly convert []byte to string
  • 1304 pstibrany: promtail: write positions to new file first, move to target location afterwards
  • 1303 zhangjianweibj: grafana#1302
  • 1298 rfratto: pkg/promtail: remove journal target forced path
  • 1279 rfratto: Fix loki_discarded_samples_total metric
  • 1278 rfratto: docs: update limits_config to new structure from #948
  • 1276 roidelapluie: Update fluentbit README.md based on my experience
  • 1274 sh0rez: chore(ci): drone-cli
  • 1273 JensErat: fluent-bit: tenant ID configuration
  • 1266 polar3130: add description about tenant stage
  • 1262 Eraac: documentation: iam requirement for autoscaling
  • 1261 rfratto: Document systemd journal scraping
  • 1249 cyriltovena: Move to jsoniter instead of default json package
  • 1223 jgehrcke: authentication.md: replace "user" with "tenant"
  • 1204 allanhung: fluent-bit-plugin: Auto add Kubernetes labels to Loki labels

1.0.0 (2019-11-19)

🎉 Nearly a year since Loki was announced at KubeCon in Seattle 2018 we are very excited to announce the 1.0.0 release of Loki! 🎉

A lot has happened since the announcement, the project just recently passed 1000 commits by 138 contributors over 700+ PR's accumulating over 7700 GitHub stars!

Internally at Grafana Labs we have been using Loki to monitor all of our infrastructure and ingest around 1.5TB/10 billion log lines a day. Since the v0.2.0 release we have found Loki to be reliable and stable in our environments.

We are comfortable with the state of the project in our production environments and think it's time to promote Loki to a non-beta release to communicate to everyone that they should feel comfortable using Loki in their production environments too.

API Stability

With the 1.0.0 release our intent is to try to follow Semver rules regarding stability with some aspects of Loki, focusing mainly on the operating experience of Loki as an application. That is to say we are not planning any major changes to the HTTP API, and anything breaking would likely be accompanied by a major release with backwards compatibility support.

We are currently NOT planning on maintaining Go API stability with this release, if you are importing Loki as a library you should be prepared for any kind of change, including breaking, even in minor or bugfix releases.

Loki is still a young and active project and there might be some breaking config changes in non-major releases, rest assured this will be clearly communicated and backwards or overlapping compatibility will be provided if possible.

Changes

There were not as many changes in this release as the last, mainly we wanted to make sure Loki was mostly stable before 1.0.0. The most notable change is the inclusion of the V11 schema in PR's 1201 and 1280. The V11 schema adds some more data to the index to improve label queries over large amounts of time and series. Currently we have not updated the Helm or Ksonnet to use the new schema, this will come soon with more details on how it works.

The full list of changes:

  • 1280 owen-d: Fix duplicate labels (update cortex)
  • 1260 rfratto: pkg/loki: unmarshal module name from YAML
  • 1257 rfratto: helm: update default terminationGracePeriodSeconds to 4800
  • 1251 obitech: docs: Fix promtail releases download link
  • 1248 rfratto: docs: slightly modify language in community Loki packages section
  • 1242 tarokkk: fluentd: Suppress unread configuration warning
  • 1239 pracucci: Move ReservedLabelTenantID out from a dedicated file
  • 1238 oke-py: helm: loki-stack supports k8s 1.16
  • 1237 joe-elliott: Rollback google.golang.org/api to 0.8.0
  • 1235 woodsaj: ci: update triggers to use new deployment_tools location
  • 1234 rfratto: Standardize schema used in match stage
  • 1233 wapmorgan: Update docker-driver Dockerfile: add tzdb
  • 1232 rfratto: Fix drone deploy job
  • 1231 joe-elliott: Removed references to Loki free tier
  • 1226 clickyotomy: Update dependencies to use weaveworks/common upstream
  • 1221 slim-bean: use regex label matcher to not alert on any tail route latencies
  • 1219 MightySCollins: docs: Updated Kubernetes docs links in Helm charts
  • 1218 slim-bean: update dashboards to include the new /loki/api/v1/* endpoints
  • 1217 slim-bean: sum the bad words by name and level
  • 1216 joe-elliott: Remove rules that reference no longer existing metrics
  • 1215 Eraac: typo url
  • 1214 takanabe: Correct wrong document paths about querying
  • 1213 slim-bean: Fix docker latest and master tags
  • 1212 joe-elliott: Update loki operational
  • 1206 sandlis: ksonnet: fix replication always set to 3 in ksonnet
  • 1203 joe-elliott: Chunk iterator performance improvement
  • 1202 beorn7: Simplify regexp's
  • 1201 cyriltovena: Update cortex to bring v11 schema
  • 1189 putrasattvika: fluent-plugin: Add client certificate verification
  • 1186 tarokkk: fluentd: Refactor label_keys and and add extract_kubernetes_labels configuration

0.4.0 (2019-10-24)

A huge thanks to the 36 contributors who submitted 148 PR's since 0.3.0!

Notable Changes

  • With PR 654 @cyriltovena added a really exciting new capability to Loki, a Prometheus compatible API with support for running metric style queries against your logs! Take a look at how to write metric queries for logs

    PLEASE NOTE: To use metric style queries in the current Grafana release 6.4.x you will need to add Loki as a Prometheus datasource in addition to having it as a Log datasource and you will have to select the correct source for querying logs vs metrics, coming soon Grafana will support both logs and metric queries directly to the Loki datasource!

  • PR 1022 (and a few others) @joe-elliott added a new set of HTTP endpoints in conjunction with the work @cyriltovena to create a Prometheus compatible API as well as improve how labels/timestamps are handled

    IMPORTANT: The new /api/v1/* endpoints contain breaking changes on the query paths (push path is unchanged) Eventually the /api/prom/* endpoints will be removed

  • PR 847 owes a big thanks to @cosmo0920 for contributing his Fluent Bit go plugin, now loki has Fluent Bit plugin support!!

  • PR 982 was a couple weeks of painstaking work by @rfratto for a much needed improvement to Loki's docs! Check them out!

  • PR 980 by @sh0rez improved how flags and config file's are loaded to honor a more traditional order of precedence:

    1. Defaults
    2. Config file
    3. User-supplied flag values (command line arguments)

    PLEASE NOTE: This is potentially a breaking change if you were passing command line arguments that also existed in a config file in which case the order they are given priority now has changed!

  • PR 1062 and 1089 have moved Loki from Dep to Go Modules and to Go 1.13

Loki

Features/Improvements/Changes

  • Loki 1171 cyriltovena: Moves request parsing into the loghttp package
  • Loki 1145 joe-elliott: Update /loki/api/v1/push to use the v1 json format
  • Loki 1128 sandlis: bigtable-backup: list backups just before starting deletion of wanted backups
  • Loki 1100 sandlis: logging: removed some noise in logs from live-tailing
  • Loki/build 1089 joe-elliott: Go 1.13
  • Loki 1088 pstibrany: Updated cortex to latest master.
  • Loki 1085 pracucci: Do not retry chunks transferring on shutdown in the local dev env
  • Loki 1084 pracucci: Skip ingester tailer filtering if no filter is set
  • Loki/build1062 joe-elliott: dep => go mod
  • Loki 1049 joe-elliott: Update loki push path
  • Loki 1044 joe-elliott: Fixed broken logql request filtering
  • Loki/tools 1043 sandlis: bigtable-backup: use latest bigtable backup docker image with fix for list backups
  • Loki 1030 polar3130: fix typo in error messages
  • Loki/tools 1028 sandlis: bigtable-backup: verify backups to work on latest list of backups
  • Loki 1022 joe-elliott: Loki HTTP/JSON Model Layer
  • Loki 1016 slim-bean: Revert "Updated stream json objects to be more parse friendly (#1010)"
  • Loki 1010 joe-elliott: Updated stream json objects to be more parse friendly
  • Loki 1009 cyriltovena: Make Loki HTTP API more compatible with Prometheus
  • Loki 1008 wardbekker: Improved Ingester out-of-order error for faster troubleshooting
  • Loki 1001 slim-bean: Update new API paths
  • Loki 998 sandlis: Change unit of duration params to hours to align it with duration config at other places in Loki
  • Loki 980 sh0rez: feat: configuration source precedence
  • Loki 948 sandlis: limits: limits implementation for loki
  • Loki 947 sandlis: added a variable for storing periodic table duration as an int to be …
  • Loki 938 sandlis: vendoring: update cortex to latest master
  • Loki/tools 930 sandlis: fix incrementing of bigtable_backup_job_backups_created metric
  • Loki/tools 920 sandlis: bigtable-backup tool fix
  • Loki/tools 895 sandlis: bigtable-backup-tool: Improvements
  • Loki 755 sandlis: Use grpc client config from cortex for Ingester to get more control
  • Loki 654 cyriltovena: LogQL: Vector and Range Vector Aggregation.

Bug Fixes

  • Loki 1114 rfratto: pkg/ingester: prevent shutdowns from processing during joining handoff
  • Loki 1097 joe-elliott: Reverted cloud.google.com/go to 0.44.1
  • Loki 986 pracucci: Fix panic in tailer due to race condition between send() and close()
  • Loki 975 sh0rez: fix(distributor): parseError BadRequest
  • Loki 944 rfratto: pkg/querier: fix concurrent access to querier tail clients

Promtail

Features/Improvements/Changes

  • Promtail/pipeline 1179 pracucci: promtail: fix handling of JMESPath expression returning nil while parsing JSON
  • Promtail/pipeline 1123 pracucci: promtail: added action_on_failure support to timestamp stage
  • Promtail/pipeline 1122 pracucci: promtail: initialize extracted map with initial labels
  • Promtail/pipeline 1112 cyriltovena: Add logql filter to match stages and drop capability
  • Promtail/journal 1109 rfratto: Clarify journal warning
  • Promtail 1083 pracucci: Increased promtail's backoff settings in prod and improved doc
  • Promtail 1026 erwinvaneyk: promtail: fix externalURL and path prefix issues
  • Promtail 976 slim-bean: Wrap debug log statements in conditionals to save allocations
  • Promtail 973 ctrox: tests: Set default value for BatchWait as ticker does not accept 0
  • Promtail 969 ctrox: promtail: Use ticker instead of timer for batch wait
  • Promtail 952 pracucci: promtail: add metrics on sent and dropped log entries
  • Promtail 934 pracucci: promtail: do not send the last batch - to ingester - if empty
  • Promtail 921 rfratto: promtail: add "max_age" field to configure cutoff for journal reading
  • Promtail 883 adityacs: Add pipeline unit testing to promtail

Bugfixes

  • Promtail 1194 slim-bean: Improve how we record file size metric to avoid a race in our file lagging alert
  • Promtail/journal 1072 rfratto: build: enable journal in promtail linux release build

Docs

  • Docs 1176 rfratto: docs: add example and documentation about using JMESPath literals
  • Docs 1139 joe-elliott: Moved client docs and add serilog example
  • Docs 1132 kailwallin: FixedTypo.Update README.md
  • Docs 1130 pracucci: docs: fix Promtail / Loki capitalization
  • Docs 1129 pracucci: docs: clarified the relation between retention period and table period
  • Docs 1124 geowa4: Client recommendations documentation tweaks
  • Docs 1106 cyriltovena: Add fluent-bit missing link in the main documentation page.
  • Docs 1099 pracucci: docs: improve table manager documentation
  • Docs 1094 rfratto: docs: update stages README with the docker and cri stages
  • Docs 1091 daixiang0: docs(stage): add docker and cri
  • Docs 1077 daixiang0: doc(fluent-bit): add missing namespace
  • Docs 1073 flouthoc: Re Fix Docs: PR grafana#1053 got erased due to force push.
  • Docs 1069 daixiang0: doc: unify GOPATH
  • Docs 1068 daixiang0: doc: skip jb init when using Tanka
  • Docs 1067 rfratto: Fix broken links to docs in README.md
  • Docs 1064 jonaskello: Fix spelling of HTTP header
  • Docs 1063 rfratto: docs: fix deprecated warning in api.md
  • Docs 1060 rfratto: Add Drone CI badge to README.md
  • Docs 1053 flouthoc: Fix Docs: Change Imagepull policy to IfNotpresent / Add loki-canary b…
  • Docs 1048 wassan128: Loki: Fix README link
  • Docs 1042 daixiang0: doc(ksonnet): include ksonnet-lib
  • Docs 1039 sh0rez: doc(production): replace ksonnet with Tanka
  • Docs 1036 sh0rez: feat: -version flag
  • Docs 1025 oddlittlebird: Update CONTRIBUTING.md
  • Docs 1024 oddlittlebird: Update README.md
  • Docs 1014 polar3130: Fix a link to correct doc and fix a typo
  • Docs 1006 slim-bean: fixing lots of broken links and a few typos
  • Docs 1005 SmilingNavern: Fix links to correct doc
  • Docs 1004 rfratto: docs: fix example with pulling systemd logs
  • Docs 1003 oddlittlebird: Loki: Update README.md
  • Docs 984 tomgs: Changing "Usage" link in main readme after docs change
  • Docs 983 daixiang0: update positions.yaml location reference
  • Docs 982 rfratto: Documentation Rewrite
  • Docs 961 worr: doc: Add permissions that IAM roles for Loki need
  • Docs 933 pracucci: doc: move promtail doc into dedicated subfolder
  • Docs 924 pracucci: doc: promtail known failure modes
  • Docs 910 slim-bean: docs(build): Update docs around releasing and fix bug in version updating script
  • Docs 850 sh0rez: docs: general documentation rework

Build

  • Build 1157 daixiang0: Update golint
  • Build 1133 daixiang0: bump up golangci to 1.20
  • Build 1121 pracucci: Publish loki-canary binaries on release
  • Build 1054 pstibrany: Fix dep check warnings by running dep ensure
  • Build/release 1018 slim-bean: updating the image version for loki-canary and adding the version increment to the release_prepare script
  • Build/CI 997 slim-bean: full cirlce
  • Build/CI 996 rfratto: ci/drone: fix deploy command by escaping double quotes in JSON body
  • Build/CI 995 slim-bean: use the loki-build-image for calling circle
  • Build/CI 994 slim-bean: Also need bash for the deploy step from drone
  • Build/CI 993 slim-bean: Add make to the alpine image used for calling the circle deploy task from drone.
  • Build/CI 992 sh0rez: chore(packaging): fix GOPATH being overwritten
  • Build/CI 991 sh0rez: chore(packaging): deploy from drone
  • Build/CI 990 sh0rez: chore(ci/cd): breaking the circle
  • Build 989 sh0rez: chore(packaging): simplify tagging
  • Build 981 sh0rez: chore(packaging): loki windows/amd64
  • Build 958 daixiang0: sync release pkgs name with release note
  • Build/CI 914 rfratto: ci: update apt-get before installing deps for rootless step
  • Build 911 daixiang0: optimize image tag script

Deployment

  • Ksonnet 1023 slim-bean: make promtail daemonset name configurable
  • Ksonnet 1021 rfratto: ksonnet: update memcached and memcached-exporter images
  • Ksonnet 1020 rfratto: ksonnet: use consistent hashing in memcached client configs
  • Ksonnet 1017 slim-bean: make promtail configmap name configurable
  • Ksonnet 946 rfratto: ksonnet: remove prefix from kvstore.consul settings in loki config
  • Ksonnet 926 slim-bean: feat(promtail): Make cluster role configurable
  • Helm 1174 rally25rs: loki-stack: Add release name to prometheus service name.
  • Helm 1152 nicr9: docs(helm): fix broken link to grafana datasource
  • Helm 1134 minhdanh: Helm chart: Allow additional scrape_configs to be added
  • Helm 1111 ekarlso: helm: Add support for passing arbitrary secrets
  • Helm 1110 marcosnils: Bump grafana image in loki helm chart
  • Helm 1104 marcosnils: : Deploy prometheus from helm chart
  • Helm 1058 polar3130: Helm: Remove default value of storageClassName in loki/loki helm chart
  • Helm 1056 polar3130: Helm: Fix the reference error of loki/loki helm chart
  • Helm 967 makocchi-git: helm chart: Add missing operator to promtail
  • Helm 937 minhdanh: helm chart: Add support for additional labels and scrapeTimeout for serviceMonitors
  • Helm 909 angelbarrera92: Feature: Add extra containers to loki helm chart
  • Helm 855 ikeeip: set helm chart appVersion while release
  • Helm 675 cyriltovena: Helm default ingester config

Loki Canary

  • Loki-canary 1137 slim-bean: Add some additional logging to the canary on queries
  • Loki-canary 1131 rfratto: pkg/canary: use default HTTP client when reading from Loki

Logcli

  • Logcli 1168 sh0rez: feat(cli): order flags by categories
  • Logcli 1115 pracucci: logcli: introduced QueryStringBuilder utility to clean up query string encoding
  • Logcli 1103 pracucci: logcli: added --step support to query command
  • Logcli 987 joe-elliott: Logcli: Add Support for New Query Path

Tooling

  • Dashboards 1188 joe-elliott: Adding Operational dashboards
  • Dashboards 1143 joe-elliott: Improved compression ratio histogram
  • Dashboards 1126 joe-elliott: Fix Loki Chunks Dashboard
  • Tools 1108 joe-elliott: Updated push path to current prod

Plugins

  • DockerDriver 972 cyriltovena: Add stream label to docker driver
  • DockerDriver 971 cyriltovena: Allow to pass max-size and max-file to the docker driver
  • DockerDriver 970 mindfl: docker-driver compose labels support
  • Fluentd 928 candlerb: fluent-plugin-grafana-loki: Escape double-quotes in labels, and suppress labels with value nil
  • Fluent Bit 1155 cyriltovena: rollback fluent-bit push path until we release 0.4
  • Fluent Bit 1096 JensErat: fluent-bit: edge case tests
  • Fluent Bit 847 cosmo0920: fluent-bit shared object go plugin

Misc

Loki is now using a Bot to help keep issues and PR's pruned based on age/relevancy. Please don't hesitate to comment on an issue or PR that you think was closed by the stale-bot which you think should remain open!!

  • Github 965 rfratto: Change label used to keep issues from being marked as stale to keepalive
  • Github 964 rfratto: Add probot-stale configuration to close stale issues.

0.3.0 (2019-08-16)

Features/Enhancements

  • Loki 877 pracucci: loki: Improve Tailer loop
  • Loki 870 sandlis: bigtable-backup: update docker image for bigtable-backup tool
  • Loki 862 sandlis: live-tailing: preload all the historic entries before query context is cancelled
  • Loki 858 pracucci: loki: removed unused TestGZIPCompression
  • Loki 854 adityacs: Readiness probe for querier
  • Loki 851 cyriltovena: Add readiness probe to distributor deployment.
  • Loki 894 rfratto: ksonnet: update ingester config to transfer chunks on rollout
  • Build 901 sh0rez: chore(packaging): set tag length to 7
  • Build 900 sh0rez: chore(ci/cd): fix grafanasaur credentials and CircleCI image build
  • Build 891 sh0rez: chore(ci/cd): build containers using drone.io
  • Build 888 rfratto: Makefile: disable building promtail with systemd support on non-amd64 platforms
  • Build 887 slim-bean: chore(packaging): Dockerfile make avoid containers
  • Build 886 sh0rez: chore(packaging): wrong executable format
  • Build 855 ikeeip: set helm chart appVersion while release
  • Promtail 856 martinbaillie: promtail: Add ServiceMonitor and headless Service
  • Promtail 809 rfratto: Makefile: build promtail with CGO_ENABLED if GOHOSTOS=GOOS=linux
  • Promtail 730 rfratto: promtail: Add systemd journal support

809, 730 NOTE: Systemd journal support is currently limited to amd64 images, arm support should come in the future when the transition to building the arm image and binaries is done natively via an arm container

  • Docs 896 dalance: docs: fix link format
  • Docs 876 BouchaaraAdil: update Docs: update Retention section on Operations doc file
  • Docs 864 temal-: docs: Replace old values in operations.md
  • Docs 853 cyriltovena: Add governance documentation
  • Deployment 874 slim-bean: make our ksonnet a little more modular by parameterizing the chunk and index stores
  • Deployment 857 slim-bean: Reorder relabeling rules to prevent pod label from overwriting config define labels

857 POSSIBLY BREAKING: If you relied on a custom pod label to overwrite one of the labels configured by the other sections of the scrape config: job, namespace, instance, container_name and/or __path__, this will no longer happen, the custom pod labels are now loaded first and will be overwritten by any of these listed labels.

Fixes

  • Loki 897 pracucci: Fix panic in tailer when an ingester is removed from the ring while tailing
  • Loki 880 cyriltovena: fix a bug where nil line buffer would be put back
  • Loki 859 pracucci: loki: Fixed out of order entries allowed in a chunk on edge case
  • Promtail 893 rfratto: pkg/promtail/positions: remove executable bit from positions file
  • Deployment 867 slim-bean: Update read dashboard to include only query and label query routes
  • Deployment 865 sandlis: fix broken jsonnet for querier
  • Canary 889 slim-bean: fix(canary): Fix Flaky Tests
  • Pipeline 869 jojohappy: Pipeline: Fixed labels process test with same objects
  • Logcli 863 adityacs: Fix Nolabels parse metrics

0.2.0 (2019-08-02)

There were over 100 PR's merged since 0.1.0 was released, here's a highlight:

Features / Enhancements

  • Loki: 521 Query label values and names are now fetched from the store.
  • Loki: 541 Improvements in live tailing of logs.
  • Loki: 713 Storage memory improvement.
  • Loki: 764 Tailing can fetch previous logs for context.
  • Loki: 782 Performance improvement: Query storage by iterating through chunks in batches.
  • Loki: 788 Querier timeouts.
  • Loki: 794 Support ingester chunk transfer on shutdown.
  • Loki: 729 Bigtable backup tool support.
  • Pipeline: 738 Added a template stage for manipulating label values.
  • Pipeline: 732 Support for Unix timestamps.
  • Pipeline: 760 Support timestamps without year.
  • Helm: 641 Helm integration testing.
  • Helm: 824 Add service monitor.
  • Helm: 830 Customize namespace.
  • Docker-Plugin: 663 Created a Docker logging driver plugin.
  • Fluent-Plugin: 669 Ability to specify keys to remove.
  • Fluent-Plugin: 709 Multi-worker support.
  • Fluent-Plugin: 792 Add prometheus for metrics and update gems.
  • Build: 668,762 Build multiple architecture containers.
  • Loki-Canary: 772 Moved into Loki project.

Bugfixes

There were many fixes, here are a few of the most important:

  • Promtail: 650 Build on windows.
  • Fluent-Plugin: 667 Rename fluent plugin.
  • Docker-Plugin: 813 Fix panic for newer docker version (18.09.7+).

0.1.0 (2019-06-03)

First (beta) Release!