fluentd-kubernetes-daemonset:v1.2-debian-graylog image issue #215
Comments
We need feedback for this problem: #213
BTW, the failure file, systemd.conf, is not updated recently. I checked systemd.conf with local fluentd and parsing file itself is succeeded. Could you paste your cluster's conf file? |
|
I've compared complete /fluentd/etc directory, previous (working) image against the new one and systemd.conf is the same. The only thing that's different is line 9 in fluent.conf. Luckily one of the old K8S instances had original (working) version of image on it so I pushed that to private docker hub repo which solved the problem. |
|
I'm seeing the same issue - Easily reproducible via: ✔︎ ~ docker run --rm -it fluent/fluentd-kubernetes-daemonset:v1.2.6-debian-graylog
2018-10-25 10:10:24 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2018-10-25 10:10:25 +0000 [info]: using configuration file: <ROOT>
<source>
@type systemd
@id in_systemd_kubelet
matches [{"_SYSTEMD_UNIT":"kubelet.service"}]
read_from_head true
tag "kubelet"
<storage>
@type "local"
persistent true
path "/var/log/fluentd-journald-kubelet-cursor.json"
</storage>
</source>
<source>
@type systemd
@id in_systemd_docker
matches [{"_SYSTEMD_UNIT":"docker.service"}]
read_from_head true
tag "docker.systemd"
<storage>
@type "local"
persistent true
path "/var/log/fluentd-journald-docker-cursor.json"
</storage>
</source>
<source>
@type systemd
@id in_systemd_bootkube
matches [{"_SYSTEMD_UNIT":"bootkube.service"}]
read_from_head true
tag "bootkube"
<storage>
@type "local"
persistent true
path "/var/log/fluentd-journald-bootkube-cursor.json"
</storage>
</source>
<match fluent.**>
@type null
</match>
<source>
@type tail
@id in_tail_container_logs
path "/var/log/containers/*.log"
pos_file "/var/log/fluentd-containers.log.pos"
tag "kubernetes.*"
read_from_head true
<parse>
@type "json"
time_format "%Y-%m-%dT%H:%M:%S.%NZ"
time_type string
</parse>
</source>
<source>
@type tail
@id in_tail_minion
path "/var/log/salt/minion"
pos_file "/var/log/fluentd-salt.pos"
tag "salt"
<parse>
@type "regexp"
expression /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
time_format "%Y-%m-%d %H:%M:%S"
</parse>
</source>
<source>
@type tail
@id in_tail_startupscript
path "/var/log/startupscript.log"
pos_file "/var/log/fluentd-startupscript.log.pos"
tag "startupscript"
<parse>
@type "syslog"
</parse>
</source>
<source>
@type tail
@id in_tail_docker
path "/var/log/docker.log"
pos_file "/var/log/fluentd-docker.log.pos"
tag "docker"
<parse>
@type "regexp"
expression /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
</parse>
</source>
<source>
@type tail
@id in_tail_etcd
path "/var/log/etcd.log"
pos_file "/var/log/fluentd-etcd.log.pos"
tag "etcd"
<parse>
@type "none"
</parse>
</source>
<source>
@type tail
@id in_tail_kubelet
multiline_flush_interval 5s
path "/var/log/kubelet.log"
pos_file "/var/log/fluentd-kubelet.log.pos"
tag "kubelet"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_kube_proxy
multiline_flush_interval 5s
path "/var/log/kube-proxy.log"
pos_file "/var/log/fluentd-kube-proxy.log.pos"
tag "kube-proxy"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_kube_apiserver
multiline_flush_interval 5s
path "/var/log/kube-apiserver.log"
pos_file "/var/log/fluentd-kube-apiserver.log.pos"
tag "kube-apiserver"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_kube_controller_manager
multiline_flush_interval 5s
path "/var/log/kube-controller-manager.log"
pos_file "/var/log/fluentd-kube-controller-manager.log.pos"
tag "kube-controller-manager"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_kube_scheduler
multiline_flush_interval 5s
path "/var/log/kube-scheduler.log"
pos_file "/var/log/fluentd-kube-scheduler.log.pos"
tag "kube-scheduler"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_rescheduler
multiline_flush_interval 5s
path "/var/log/rescheduler.log"
pos_file "/var/log/fluentd-rescheduler.log.pos"
tag "rescheduler"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_glbc
multiline_flush_interval 5s
path "/var/log/glbc.log"
pos_file "/var/log/fluentd-glbc.log.pos"
tag "glbc"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_cluster_autoscaler
multiline_flush_interval 5s
path "/var/log/cluster-autoscaler.log"
pos_file "/var/log/fluentd-cluster-autoscaler.log.pos"
tag "cluster-autoscaler"
<parse>
@type "kubernetes"
expression /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/m
time_format "%m%d %H:%M:%S.%N"
</parse>
</source>
<source>
@type tail
@id in_tail_kube_apiserver_audit
multiline_flush_interval 5s
path "/var/log/kubernetes/kube-apiserver-audit.log"
pos_file "/var/log/kube-apiserver-audit.log.pos"
tag "kube-apiserver-audit"
<parse>
@type "multiline"
format_firstline "/^\\S+\\s+AUDIT:/"
format1 /^(?<time>\S+) AUDIT:(?: (?:id="(?<id>(?:[^"\\]|\\.)*)"|ip="(?<ip>(?:[^"\\]|\\.)*)"|method="(?<method>(?:[^"\\]|\\.)*)"|user="(?<user>(?:[^"\\]|\\.)*)"|groups="(?<groups>(?:[^"\\]|\\.)*)"|as="(?<as>(?:[^"\\]|\\.)*)"|asgroups="(?<asgroups>(?:[^"\\]|\\.)*)"|namespace="(?<namespace>(?:[^"\\]|\\.)*)"|uri="(?<uri>(?:[^"\\]|\\.)*)"|response="(?<response>(?:[^"\\]|\\.)*)"|\w+="(?:[^"\\]|\\.)*"))*/
time_format "%Y-%m-%dT%T.%L%Z"
</parse>
</source>
<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<match **>
@type gelf
@id out_graylog
@log_level "info"
include_tag_key true
host ""
port
buffer_chunk_limit 4096K
buffer_queue_limit 512
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 8
<buffer>
flush_mode interval
retry_type exponential_backoff
flush_thread_count 8
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 4096K
queue_limit_length 512
</buffer>
<inject>
tag_key tag
</inject>
</match>
</ROOT>
2018-10-25 10:10:25 +0000 [info]: starting fluentd-1.2.6 pid=8 ruby="2.3.3"
2018-10-25 10:10:25 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
/fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': expected end of line at systemd.conf line 9,12 (Fluent::ConfigParseError)
8: @id in_systemd_kubelet
9: matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
------------^
10: <storage>
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:132:in `parse_element'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:in `parse_element'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:166:in `block in eval_include'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:160:in `each'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:160:in `eval_include'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:145:in `parse_include'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:104:in `parse_element'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:43:in `parse!'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:33:in `parse'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config.rb:39:in `parse'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:259:in `load_config'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:630:in `block in supervise'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/config_loader.rb:43:in `reload_config'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/config_loader.rb:36:in `initialize'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/daemon.rb:32:in `initialize'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine.rb:33:in `new'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine.rb:33:in `create'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:629:in `supervise'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:502:in `run_supervisor'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/bin/fluentd:8:in `<top (required)>'
from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `load'
from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in `<main>' |
|
I'm having the same issue. Tried to replace "matches" with "filters" but received the same error message. I thought problem introduced by commit 7f90e65 but it must be something else. |
|
I confirmed only debian-graylog images have this problem. |
|
Ahhhhhh... I found the cause. JSON module is more strict and fluentd uses JSON module for validating json parameters. So loading latest |
|
one user has already reported this problem before graylog-labs/gelf-rb#63 |
|
Just saw that you already merged a PR and pushed the new images to Dockerhub: #219 Can confirm that this starts up fine now with the most recent image published ( Thanks a lot for the quick fix! |
|
Thanks for testing latest image. I close this issue. BTW, the problem is not fixed properly. Please ping gelf-rb author for correct fix. |
|
Hello @repeatedly 👋 , Is there a way we can take over the gem since it is abandoned and not being upgraded since 2018. Is there a way to work with rubygems and make a fork just to fix these config issues for now? EDIT: I do not mind working on this and fixing it for the community. We don't want our config-reloading to break fluentd because a gem is forcing using the wrong yajl-ruby package. You have repeatedly (😃) made comments about the right solution to the gem maintainers: graylog-labs/gelf-rb#73 (comment) |
|
Hello, Excuse my ignorance, all this is not my experty. this is happening on following images: either its related to the above bug or not, what can be done in order to use this package? is there a bypass? |
It looks like the this image was re-uploaded to Docker Hub repo with the issue in /fluentd/etc/fluent.conf which caused error starting all new K8S instances.
Offending line "@log_level info"
Error:
2018-10-24 14:38:55 +0000 [info]: starting fluentd-1.2.6 pid=8 ruby="2.3.3" 2018-10-24 14:38:55 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"] /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/basic_parser.rb:92:inparse_error!': expected end of line at systemd.conf line 9,12 (Fluent::ConfigParseError)8: @id in_systemd_kubelet
9: matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
10:
from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:132:in
parse_element' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:inparse_element'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:166:in
block in eval_include' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:160:ineach'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:160:in
eval_include' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:145:inparse_include'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:104:in
parse_element' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:43:inparse!'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:33:in
parse' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/config.rb:39:inparse'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:259:in
load_config' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:630:inblock in supervise'from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/config_loader.rb:43:in
reload_config' from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/config_loader.rb:36:ininitialize'from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine/daemon.rb:32:in
initialize' from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine.rb:33:innew'from /fluentd/vendor/bundle/ruby/2.3.0/gems/serverengine-2.0.7/lib/serverengine.rb:33:in
create' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:629:insupervise'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:502:in
run_supervisor' from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/lib/fluent/command/fluentd.rb:310:in<top (required)>'from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in
require' from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:inrequire'from /fluentd/vendor/bundle/ruby/2.3.0/gems/fluentd-1.2.6/bin/fluentd:8:in
<top (required)>' from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:inload'from /fluentd/vendor/bundle/ruby/2.3.0/bin/fluentd:22:in
<main>'Is that a common practice to overwrite already published images with tags? Can we expect in the future things to break randomly because something was updated?
The text was updated successfully, but these errors were encountered: