Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ctre: Excessive logging of readiness probes #26

Closed
wdoekes opened this issue Jul 11, 2023 · 1 comment
Closed

ctre: Excessive logging of readiness probes #26

wdoekes opened this issue Jul 11, 2023 · 1 comment

Comments

@wdoekes
Copy link
Member

wdoekes commented Jul 11, 2023

For instance kubectl -n elastic-system get sts elastic-es-data-nodes -o yaml:

        image: docker.elastic.co/elasticsearch/elasticsearch:8.6.0
...
        readinessProbe:
          exec:
            command:
            - bash
            - -c
            - /mnt/elastic-internal/scripts/readiness-probe-script.sh

This reports the following in ctr events:

2023-07-11 09:00:14.701377274 +0000 UTC k8s.io /tasks/exec-added {"container_id":"2183b9078407a6ec6b8c1d32dedc7edb50fb4c65201fa18b80883dd3ca8d98a2","exec_id":"6fc34d1a3468fc27ea7c79e5bb5687a34e66a5e8103ae1531f7c6257c26d374f"}

2023-07-11 09:00:14.760695774 +0000 UTC k8s.io /tasks/exec-started {"container_id":"2183b9078407a6ec6b8c1d32dedc7edb50fb4c65201fa18b80883dd3ca8d98a2","exec_id":"6fc34d1a3468fc27ea7c79e5bb5687a34e66a5e8103ae1531f7c6257c26d374f","pid":2025219}

2023-07-11 09:00:14.786905943 +0000 UTC k8s.io /tasks/exit {"container_id":"2183b9078407a6ec6b8c1d32dedc7edb50fb4c65201fa18b80883dd3ca8d98a2","id":"6fc34d1a3468fc27ea7c79e5bb5687a34e66a5e8103ae1531f7c6257c26d374f","pid":2025219,"exited_at":"2023-07-11T09:00:14.786887503Z"}

While it runs bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%-10u %-5d ", elapsed / 1e6, pid); join(args->argv); }':

11286      2025219 bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh
11288      2025219 /mnt/elastic-internal/scripts/readiness-probe-script.sh
11289      2025219 bash /mnt/elastic-internal/scripts/readiness-probe-script.sh
11289      2025219 bash /mnt/elastic-internal/scripts/readiness-probe-script.sh
11289      2025219 bash /mnt/elastic-internal/scripts/readiness-probe-script.sh
11289      2025219 bash /mnt/elastic-internal/scripts/readiness-probe-script.sh
11289      2025219 bash /mnt/elastic-internal/scripts/readiness-probe-script.sh

(Not sure about those dupes?)

ctre logs these in journald:

# journalctl -u ctre -o short-iso -S '-1 minutes' --grep elastic | cut -d' ' -f1,3-
2023-07-11T11:09:04+0200 ctre[2554]: - 1689066544 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:04+0200 ctre[2554]: - 1689066544 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:06+0200 ctre[2554]: - 1689066546 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:09+0200 ctre[2554]: - 1689066549 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:09+0200 ctre[2554]: - 1689066549 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:11+0200 ctre[2554]: - 1689066551 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:14+0200 ctre[2554]: - 1689066554 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0
2023-07-11T11:09:14+0200 ctre[2554]: - 1689066554 k8s.io docker.elastic.co/elasticsearch/elasticsearch:8.6.0

Which is a bit excessive.
@wdoekes
Copy link
Member Author

wdoekes commented Oct 18, 2023

Fixed by d85407b

@wdoekes wdoekes closed this as completed Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wdoekes