Option to not log docker metrics

[pierreozoux]

It would be nice to be able to not log all the docker metrics.
Currently it is logging all docker metrics like:
app.disk.requests 91236
But this is already taken care by our monitoring solution, so we don't need these logs.

[czerwingithub]

Thanks for reaching out. Just to be clear, you want to turn reporting the container metrics for all containers, correct? (We've had feature requests to turn on for just a specific list of containers).

There is a way to turn them off currently, but it's a little non-intuitive. We'll have in something to make it easier to do. If you do want to turn them off, you just have to remove the definition for the scalyr_agent.builtin_monitors.docker_monitor from the /etc/scalyr-agent-2/agent.d/docker.json file configuration file. I know, it seems weird to remove that when you are running in docker... but when you are using the Scalyr Docker Agent in "syslog" mode, the docker monitor's only purpose is to collect the metrics.

You can use any method you would like to to edit that file in the running scalyr-docker-agent container, but we do have instructions on how to do it from the commandline of the underlying host: https://www.scalyr.com/help/install-agent-docker#modify-config

[pierreozoux]

Yes, I want to turn them off.

But I currently collect logs from docker api, it is just easier, so I guess I'm out of luck?

[czerwingithub]

Using the docker api -- just be sure you are on a recent version of Docker... there were bugs pre 1.13 that caused our customers a lot of issues and why we had to go to the syslog approach. I don't disagree with you it's easier.. once we confident the bugs are gone from Docker and most folks are not using pre 1.13 docker, we'll change the default back.

Unfortunately, yes, you are a little out of luck right now. We can include a feature in the next release that would allow you to turn off metric collection. Realistically, it wouldn't be out until two weeks from now. (We have other things going into the release, and we need some time to qualify it.)

Is two weeks good enough, or is this a more urgent issue?

[pierreozoux]

we are using kubernetes, installed with kops 1.6-beta2, and it looks like the docker version is 1.12.6

I can totally wait 2 weeks!

Thanks!

By the way: helm/charts#1066

[czerwingithub]

Hey, that's great about your kubernetes contribution. Thanks for that. Doing better kubernete support is #2 right now in terms of major new features we want to add to the agent. I'm hoping the work begins in the next week or so.

As for the docker version -- they did fix some of the bugs in 1.12, so you are probably ok. If you have any issue where the a container's log stops flowing from Docker to the scalyr-agent, then it might be a symptom of the bug.

Ok, I'll follow up when the new agent is released. I'll close of this request when that occurs.

[czerwingithub]

I just wanted to confirm that we released the 2.0.27 version of the Scalyr Agent and it now includes an option to disable collecting metrics for Docker containers. You just have to edit the add the following option to the configuration for scalyr_agent.builtin_monitors.docker_monitor

report_container_metrics: false,

If you have trouble doing that, let me know and I can help with your specific configuration file. I'm going to close this issue out, but please reopen if you find it does not solve your problem.

[pierreozoux]

Thanks for this!

I tried this way: https://github.com/kubernetes/charts/pull/1066/files#diff-25fadf37c06bbe21c5b4983a56cc9f6dR15

But didn't seem to work :/ Could you help please?

[czerwingithub]

Hello... sorry, maybe I didn't advertise this well enough, that line must be in the scalyr_agent.builtin_monitors.docker_monitor stanza. Please move line 15 to right after line 19, and it should work. Let me know if it doesn't.

[pierreozoux]

Yes, it did the trick! Thanks a lot for your support!