logspout

FORK INFO

This is a fork of logspout, with the following changed merged in:

• Prevent duplicate logs with BACKLOG=false: gliderlabs#201
• Filter containers by label: gliderlabs#236
• HTTP adapter: https://github.com/raychaser/logspout-http
• HTTP adapter can forward container labels

Docker Hub automated builds for gliderlabs/logspout:latest and progrium/logspout:latest are now pointing to the release branch. For master, use gliderlabs/logspout:master. Individual versions are also available as saved images in releases.

Logspout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system.

It's a mostly stateless log appliance. It's not meant for managing log files or looking at history. It is just a means to get your logs out to live somewhere else, where they belong.

For now it only captures stdout and stderr, but a module to collect container syslog is planned.

Getting logspout

Logspout is a very small Docker container (15.2MB virtual, based on Alpine). Pull the latest release from the index:

$ docker pull gliderlabs/logspout:latest

You can also download and load a specific version:

$ curl -s dl.gliderlabs.com/logspout/v2.tgz | docker load

Using logspout

Route all container output to remote syslog

The simplest way to use logspout is to just take all logs and ship to a remote syslog. Just pass a syslog URI (or several comma separated URIs) as the command. Here we show use of the tls encrypted transport option in the URI. Also, we always mount the Docker Unix socket with -v to /var/run/docker.sock:

$ docker run --name="logspout" \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	syslog+tls://logs.papertrailapp.com:55555

logspout will gather logs from other containers that are started without the -t option and are configured with a logging driver that works with docker logs (journald and json-file).

To see what data is used for syslog messages, see the syslog adapter docs.

Ignoring specific containers

You can tell logspout to ignore specific containers by setting an environment variable when starting your container, like so:-

$ docker run -d -e 'LOGSPOUT=ignore' image

Or, by adding a label which you define by setting an environment variable when running logspout:

$ docker run --name="logspout" \
    -e EXCLUDE_LABEL=logspout.exclude \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    gliderlabs/logspout
$ docker run -d --label logspout.exclude=true image

Including specific containers

You can tell logspout to only include certain containers by setting filter parameters on the URI:

$ docker run \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	raw://192.168.10.10:5000?filter.name=*_db
	
$ docker run \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	raw://192.168.10.10:5000?filter.id=3b6ba57db54a
	
$ docker run \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	raw://192.168.10.10:5000?filter.sources=stdout%2Cstderr

# Forward logs from containers with both label 'a' starting with 'x', and label 'b' ending in 'y'.
$ docker run \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	raw://192.168.10.10:5000?filter.labels=a:x*%2Cb:*y

Note that you must URL-encode parameter values such as the comma in filter.sources and filter.labels.

Multiple logging destinations

You can route to multiple destinations by comma-separating the URIs:

$ docker run \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout \
	raw://192.168.10.10:5000?filter.name=*_db,syslog+tls://logs.papertrailapp.com:55555?filter.name=*_app

Inspect log streams using curl

Using the httpstream module, you can connect with curl to see your local aggregated logs in realtime. You can do this without setting up a route URI.

$ docker run -d --name="logspout" \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	--publish=127.0.0.1:8000:80 \
	gliderlabs/logspout
$ curl http://127.0.0.1:8000/logs

You should see a nicely colored stream of all your container logs. You can filter by container name and more. You can also get JSON objects, or you can upgrade to WebSocket and get JSON logs in your browser.

See httpstream module for all options.

Create custom routes via HTTP

Using the routesapi module logspout can also expose a /routes resource to create and manage routes.

$ curl $(docker port `docker ps -lq` 8000)/routes \
	-X POST \
	-d '{"source": {"filter": "db", "types": ["stderr"]}, "target": {"type": "syslog", "addr": "logs.papertrailapp.com:55555"}}'

That example creates a new syslog route to Papertrail of only stderr for containers with db in their name.

Routes are stored on disk, so by default routes are ephemeral. You can mount a volume to /mnt/routes to persist them.

See routesapi module for all options.

Suppressing backlog tail

You can tell logspout to only display log entries since container "start" or "restart" event by setting a BACKLOG=false environment variable (equivalent to docker logs --tail=0):

$ docker run -d --name="logspout" \
	-e 'BACKLOG=false' \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	gliderlabs/logspout

The default behaviour is to output all logs since creation of the container (equivalent to docker logs --tail=all or simply docker logs).

NOTE: Use of this option may cause the first few lines of log output to be missed following a container being started, if the container starts outputting logs before logspout has a chance to see them. If consistent capture of every line of logs is critical to your application, you might want to test thorougly and/or avoid this option (at the expense of getting the entire backlog for every restarting container). This does not affect containers that are removed and recreated.

HTTP Adapter

This again assumes that the unique token for the Sumo Logic HTTP collector endpoint is in the environment as $SUMO_HTTP_TOKEN.

$ docker run -e DEBUG=1 \
    -v /var/run/docker.sock:/tmp/docker.sock \
    -e LOGSPOUT=ignore \
    gliderlabs/logspout:latest \
    "https://collectors.sumologic.com?http.path=/receiver/v1/http/$SUMO_HTTP_TOKEN&http.gzip=true"

HTTP Adapter A Note On The Form Of The Route Parameter

The route URI for an HTTP(S) endpoint should just include the hostname. The HTTP path currently has to specified as a parameter. For example, for Sumo Logic the endpoint URI with for an HTTP collector endpoint would look like this:

https://collectors.sumologic.com/receiver/v1/http/SUMO_HTTP_TOKEN

But for Logspout, it needs to be written like this:

https://collectors.sumologic.com?http.path=/receiver/v1/http/SUMO_HTTP_TOKEN

HTTP Adapter - Additional Parameters

In addition to the http.path parameter discussed above, the following parameters are available:

http.buffer.capacity controls the size of a buffer used to accumulate logs. The default capacity of the buffer is 100 logs.

http.buffer.timeout indicates after how much time the adapter will send the logs accumulated in the buffer if the buffer capacity hasn't been reached. The default timeout is 1000ms (1s).

http.labels allows a comma-separated (see note on URL encoding) list of container labels to be included under the labels JSON key. e.g., to include the labels foo and io.rancher.stack_service.name:

$ docker run -e DEBUG=1 \
    -v /var/run/docker.sock:/tmp/docker.sock \
    -e LOGSPOUT=ignore \
    gliderlabs/logspout:latest \
    "https://collectors.sumologic.com?http.path=/receiver/v1/http/$SUMO_HTTP_TOKEN&http.gzip=true&http.labels=foo%2Cio.rancher.stack_service.name"

If http.gzip is set to true, the logs will be compressed with GZIP. This is off by default, but for example supported by Sumo Logic.

Modules

The standard distribution of logspout comes with all modules defined in this repository. You can remove or add new modules with custom builds of logspout. Just edit the modules.go file and do a docker build.

Builtin modules

• adapters/raw
• adapters/syslog
• transports/tcp
• transports/tls
• transports/udp
• httpstream
• routesapi

Third-party modules

• logspout-kafka
• logspout-redis...
• logspout-logstash
• logspout-redis-logstash

Loggly support

Use logspout to stream your docker logs to Loggly via the Loggly syslog endpoint.

$ docker run --name logspout -d --volume=/var/run/docker.sock:/var/run/docker.sock \
    -e SYSLOG_STRUCTURED_DATA="<Loggly API Key>@41058 tag=\"some tag name\"" \
    gliderlabs/logspout \
    syslog+tcp://logs-01.loggly.com:514

Test container

This is a nice container to use for testing your logspout config, it will print out a log line every 3 seconds.

$ docker run --rm --name echo3 alpine:3.4 sh -c 'while true; do echo Hello, the time is $(date)!; sleep 3; done'

Contributing

As usual, pull requests are welcome.

Discuss logspout development with us on Freenode in #gliderlabs.

Releasing

Push a tag in the format v#.#.# - e.g., v3.1.2. A docker image will be built on Circle CI and tag pushed to Docker Hub.

Sponsor

This project was made possible by DigitalOcean and Deis.

License

BSD