docker_events input plugin does not split events

[paulmey]

Bug Report

Describe the bug

The docker_events plugin reads from the socket, assuming that it only received one event. However, when there are many events, it might read 2 or more events at the same time, leading to a string that contains two or more JSON blobs with a newline in between.

To Reproduce
Run fluent-bit with docker_events input and json parser in one terminal:

# docker run --rm --name fb -v /var/run:/var/run fluent/fluent-bit:latest -R /fluent-bit/etc/parsers.conf -i docker_events -p 'Parser=json' -vvv

Then generate a high rate of docker events by running 10 docker exec's in parallel:

# for i in `seq 1 10`; do docker exec fb /fluent-bit/bin/fluent-bit --help & done

The first terminal will show the log with some parser errors:

Fluent Bit v3.2.4
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2025/01/06 14:51:48] [ info] Configuration:
[2025/01/06 14:51:48] [ info]  flush time     | 1.000000 seconds
[2025/01/06 14:51:48] [ info]  grace          | 5 seconds
[2025/01/06 14:51:48] [ info]  daemon         | 0
[2025/01/06 14:51:48] [ info] ___________
[2025/01/06 14:51:48] [ info]  inputs:
[2025/01/06 14:51:48] [ info]      docker_events
[2025/01/06 14:51:48] [ info] ___________
[2025/01/06 14:51:48] [ info]  filters:
[2025/01/06 14:51:48] [ info] ___________
[2025/01/06 14:51:48] [ info]  outputs:
[2025/01/06 14:51:48] [ info] ___________
[2025/01/06 14:51:48] [ info]  collectors:
[2025/01/06 14:51:48] [ info] [fluent bit] version=3.2.4, commit=5b0ff04120, pid=1
[2025/01/06 14:51:48] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/01/06 14:51:48] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/06 14:51:48] [ info] [simd    ] disabled
[2025/01/06 14:51:48] [ info] [cmetrics] version=0.9.9
[2025/01/06 14:51:48] [ info] [ctraces ] version=0.5.7
[2025/01/06 14:51:48] [ info] [input:docker_events:docker_events.0] initializing
[2025/01/06 14:51:48] [ info] [input:docker_events:docker_events.0] storage_strategy='memory' (memory only)
[2025/01/06 14:51:48] [debug] [docker_events:docker_events.0] created event channels: read=25 write=26
[2025/01/06 14:51:48] [trace] [input:docker_events:docker_events.0 at /src/fluent-bit/plugins/in_docker_events/docker_events.c:63] writing to socket GET /events HTTP/1.0


[2025/01/06 14:51:48] [debug] [input:docker_events:docker_events.0] read 181 bytes from socket
[2025/01/06 14:51:48] [ info] [input:docker_events:docker_events.0] listening for events on /var/run/docker.sock
[2025/01/06 14:51:48] [ info] [sp] stream processor started
[2025/01/06 14:51:54] [trace] [input:docker_events:docker_events.0 at /src/fluent-bit/plugins/in_docker_events/docker_events.c:322] tried to parse: {"status":"exec_create: /fluent-bit/bin/fluent-bit --help","id":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","from":"fluent/fluent-bit:latest","Type":"container","Action":"exec_create: /fluent-bit/bin/fluent-bit --help","Actor":{"ID":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","Attributes":{"author":"Eduardo Silva \[email protected]\u003e","description":"Fluent Bit multi-architecture container image","execID":"21aad945b78d7cc357ba1ce0c541129c72fa13b22bc13a6346d6c5b8517d94d0","image":"fluent/fluent-bit:latest","name":"fb","org.opencontainers.image.authors":"Eduardo Silva \[email protected]\u003e","org.opencontainers.image.created":"2024-12-30T21:35:24.097Z","org.opencontainers.image.description":"Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows","org.opencontainers.image.documentation":"https://docs.fluentbit.io/","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"5b0ff0412017a1e31535d863f9f5343f39b08f90","org.opencontainers.image.source":"https://github.com/fluent/fluent-bit","org.opencontainers.image.title":"fluent-bit","org.opencontainers.image.url":"https://github.com/fluent/fluent-bit","org.opencontainers.image.vendor":"Fluent Organization","org.opencontainers.image.version":"3.2.4","vendor":"Fluent Organization","version":"3.2.4"}},"scope":"local","time":1736175114,"timeNano":1736175114737436421}
{"status":"exec_start: /fluent-bit/bin/fluent-bit --help","id":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","from":"fluent/fluent-bit:latest","Type":"container","Action":"exec_start: /fluent-bit/bin/fluent-bit --help","Actor":{"ID":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","Attributes":{"author":"Eduardo Silva \[email protected]\u003e","description":"Fluent Bit multi-architecture container image","execID":"98e683851c298b9894ced26d30bb3462e30c3fb86fadcb7f701ce3e887fb96fd","image":"fluent/fluent-bit:latest","name":"fb","org.opencontainers.image.authors":"Eduardo Silva \[email protected]\u003e","org.opencontainers.image.created":"2024-12-30T21:35:24.097Z","org.opencontainers.image.description":"Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows","org.opencontainers.image.documentation":"https://docs.fluentbit.io/","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"5b0ff0412017a1e31535d863f9f5343f39b08f90","org.opencontainers.image.source":"https://github.com/fluent/fluent-bit","org.opencontainers.image.title":"fluent-bit","org.opencontainers.image.url":"https://github.com/fluent/fluent-bit","org.opencontainers.image.vendor":"Fluent Organization","org.opencontainers.image.version":"3.2.4","vendor":"Fluent Organization","version":"3.2.4"}},"scope":"local","time":1736175114,"timeNano":1736175114738159532}

[2025/01/06 14:51:54] [trace] [input:docker_events:docker_events.0 at /src/fluent-bit/plugins/in_docker_events/docker_events.c:323] buf_size 8000
[2025/01/06 14:51:54] [error] [input:docker_events:docker_events.0] parser returned an error: -1
[2025/01/06 14:51:54] [debug] [task] created task=0x74735e6365a0 id=0 without routes, dropping.
[2025/01/06 14:51:54] [debug] [task] destroy task=0x74735e6365a0 (task_id=0)
[2025/01/06 14:51:55] [trace] [input:docker_events:docker_events.0 at /src/fluent-bit/plugins/in_docker_events/docker_events.c:322] tried to parse: {"status":"exec_die","id":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","from":"fluent/fluent-bit:latest","Type":"container","Action":"exec_die","Actor":{"ID":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","Attributes":{"author":"Eduardo Silva \[email protected]\u003e","description":"Fluent Bit multi-architecture container image","execID":"21aad945b78d7cc357ba1ce0c541129c72fa13b22bc13a6346d6c5b8517d94d0","exitCode":"0","image":"fluent/fluent-bit:latest","name":"fb","org.opencontainers.image.authors":"Eduardo Silva \[email protected]\u003e","org.opencontainers.image.created":"2024-12-30T21:35:24.097Z","org.opencontainers.image.description":"Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows","org.opencontainers.image.documentation":"https://docs.fluentbit.io/","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"5b0ff0412017a1e31535d863f9f5343f39b08f90","org.opencontainers.image.source":"https://github.com/fluent/fluent-bit","org.opencontainers.image.title":"fluent-bit","org.opencontainers.image.url":"https://github.com/fluent/fluent-bit","org.opencontainers.image.vendor":"Fluent Organization","org.opencontainers.image.version":"3.2.4","vendor":"Fluent Organization","version":"3.2.4"}},"scope":"local","time":1736175115,"timeNano":1736175115025496173}
{"status":"exec_die","id":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","from":"fluent/fluent-bit:latest","Type":"container","Action":"exec_die","Actor":{"ID":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","Attributes":{"author":"Eduardo Silva \[email protected]\u003e","description":"Fluent Bit multi-architecture container image","execID":"8fd40c55004524aa9e3f65f79c953769108d9831d9c8d99a7fc0f31ab11227f9","exitCode":"0","image":"fluent/fluent-bit:latest","name":"fb","org.opencontainers.image.authors":"Eduardo Silva \[email protected]\u003e","org.opencontainers.image.created":"2024-12-30T21:35:24.097Z","org.opencontainers.image.description":"Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows","org.opencontainers.image.documentation":"https://docs.fluentbit.io/","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"5b0ff0412017a1e31535d863f9f5343f39b08f90","org.opencontainers.image.source":"https://github.com/fluent/fluent-bit","org.opencontainers.image.title":"fluent-bit","org.opencontainers.image.url":"https://github.com/fluent/fluent-bit","org.opencontainers.image.vendor":"Fluent Organization","org.opencontainers.image.version":"3.2.4","vendor":"Fluent Organization","version":"3.2.4"}},"scope":"local","time":1736175115,"timeNano":1736175115027400105}
{"status":"exec_die","id":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","from":"fluent/fluent-bit:latest","Type":"container","Action":"exec_die","Actor":{"ID":"14f8180ceaf4a098b697c2931883dd02d55eec22928e5c59d67d45ae607ca126","Attributes":{"author":"Eduardo Silva \[email protected]\u003e","description":"Fluent Bit multi-architecture container image","execID":"84ac1665ccddf29363f068da41b9ea474edcf3e37564252e2978b3cdb4d84ff7","exitCode":"0","image":"fluent/fluent-bit:latest","name":"fb","org.opencontainers.image.authors":"Eduardo Silva \[email protected]\u003e","org.opencontainers.image.created":"2024-12-30T21:35:24.097Z","org.opencontainers.image.description":"Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows","org.opencontainers.image.documentation":"https://docs.fluentbit.io/","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"5b0ff0412017a1e31535d863f9f5343f39b08f90","org.opencontainers.image.source":"https://github.com/fluent/fluent-bit","org.opencontainers.image.title":"fluent-bit","o
[2025/01/06 14:51:55] [trace] [input:docker_events:docker_events.0 at /src/fluent-bit/plugins/in_docker_events/docker_events.c:323] buf_size 8000
[2025/01/06 14:51:55] [error] [input:docker_events:docker_events.0] parser returned an error: -1
[2025/01/06 14:51:55] [debug] [task] created task=0x74735e636500 id=0 without routes, dropping.
[2025/01/06 14:51:55] [debug] [task] destroy task=0x74735e636500 (task_id=0)
[2025/01/06 14:52:07] [debug] [task] created task=0x74735e6365a0 id=0 without routes, dropping.
[2025/01/06 14:52:07] [debug] [task] destroy task=0x74735e6365a0 (task_id=0)

Expected behavior
Each docker event should result in one event in fluent-bit.

Your Environment

Repro on Ubuntu 22.04 with Fluent Bit v3.2.4 from the official docker image and Docker package version 5:27.4.1-1ubuntu.22.04jammy