From 3368e96e360e1313e5196cc4e6f2b10a83c11bf6 Mon Sep 17 00:00:00 2001 From: Tiago Rodrigo Lampert Date: Thu, 4 Oct 2018 16:41:11 -0300 Subject: [PATCH 01/11] Fix typos filesytem -> filesystem seperated -> separated --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c827d6c2..a16c8619 100644 --- a/README.md +++ b/README.md @@ -273,9 +273,9 @@ logspout supports modification of the client TLS settings via environment variab | Environment Variable | Description | | :--- | :--- | | `LOGSPOUT_TLS_DISABLE_SYSTEM_ROOTS` | when set to `true` it disables loading the system trust store into the trust store of logspout | -| `LOGSPOUT_TLS_CA_CERTS` | a comma seperated list of filesystem paths to pem encoded CA certificates that should be added to logsput's TLS trust store. Each pem file can contain more than one certificate | -| `LOGSPOUT_TLS_CLIENT_CERT` | filesytem path to pem encoded x509 client certificate to load when TLS mutual authentication is desired | -| `LOGSPOUT_TLS_CLIENT_KEY` | filesytem path to pem encoded client private key to load when TLS mutual authentication is desired | +| `LOGSPOUT_TLS_CA_CERTS` | a comma separated list of filesystem paths to pem encoded CA certificates that should be added to logsput's TLS trust store. Each pem file can contain more than one certificate | +| `LOGSPOUT_TLS_CLIENT_CERT` | filesystem path to pem encoded x509 client certificate to load when TLS mutual authentication is desired | +| `LOGSPOUT_TLS_CLIENT_KEY` | filesystem path to pem encoded client private key to load when TLS mutual authentication is desired | | `LOGSPOUT_TLS_HARDENING` | when set to `true` it enables stricter client TLS settings designed to mitigate some known TLS vulnerabilities | #### Example TLS settings From 89c2a08735ecf2cc87d1831ff545dc2a01a2d837 Mon Sep 17 00:00:00 2001 From: Michael Hobbs Date: Thu, 4 Oct 2018 11:04:51 -0700 Subject: [PATCH 02/11] fix CHANGLELOG formatting --- CHANGELOG.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e84aa19e..a2c3003f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,13 +28,12 @@ All notable changes to this project will be documented in this file. - @gbolo enforced the use of `go 1.8+` in order to accommodate some TLS settings ## [v3.2.5] - 2018-06-05 +### Fixed +- @michaelshobbs fix working_directory so we don't duplicate test runs - @gmelika panic if reconnect fails - @masterada Added multiline adapter - @billimek sleeping and syncing to fix issues with docker hub builds -### Fixed -- @michaelshobbs fix working_directory so we don't duplicate test runs - ### Added - @chris7444 take the hostname from /etc/host_hostname if the file is there - @chris7444 update README.md for swarm deployments PR #329 From 23663f2cd9f91cefe9b92146270aedd597a2820d Mon Sep 17 00:00:00 2001 From: Tom Lankhorst Date: Mon, 29 Oct 2018 11:03:59 +0100 Subject: [PATCH 03/11] Suggest to disable userns-remap for logspout yyyy/mm/dd hh:MM:ss pump ended: Get http://unix.sock/containers/json: dial unix /var/run/docker.sock: permission denied --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index c827d6c2..8fd82874 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ logspout will gather logs from other containers that are started **without the ` To see what data is used for syslog messages, see the [syslog adapter](http://github.com/gliderlabs/logspout/blob/master/adapters) docs. +The container must be able to access the Docker Unix socket to mount it. This is typically a problem when [namespace remapping](https://docs.docker.com/engine/security/userns-remap/) is enabled. To disable remapping for the logspout container, pass the `--userns=host` flag to `docker run`, `.. create`, etc. + #### Ignoring specific containers You can tell logspout to ignore specific containers by setting an environment variable when starting your container, like so:- From 29d77641f9664a74fafea2ce655f610d796a63d4 Mon Sep 17 00:00:00 2001 From: CodeLingo Bot Date: Sat, 16 Mar 2019 15:39:27 +1300 Subject: [PATCH 04/11] Fix function comments based on best practices from Effective Go (#432) Signed-off-by: CodeLingo Bot --- transports/tls/tls_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/transports/tls/tls_test.go b/transports/tls/tls_test.go index 2f5fef8b..275faf98 100644 --- a/transports/tls/tls_test.go +++ b/transports/tls/tls_test.go @@ -101,7 +101,7 @@ func TestSystemRootCAsAndCustomCAs(t *testing.T) { } } -// TestLoadingClientCert should test the behaviour of loading +// TestLoadingClientCertAndKey: should test the behaviour of loading // a pem encoded client x509 certificate and private key func TestLoadingClientCertAndKey(t *testing.T) { os.Unsetenv(envDisableSystemRoots) From 8d3a40e592f0f12b8ad6282faa0cbb8b76654dfe Mon Sep 17 00:00:00 2001 From: William Ward Date: Mon, 10 Jun 2019 18:07:15 -0500 Subject: [PATCH 05/11] update alpine image tag --- Dockerfile | 2 +- Dockerfile.dev | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 3cbd22ec..9c33cace 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:3.8 +FROM alpine:3.9 ENTRYPOINT ["/bin/logspout"] VOLUME /mnt/routes EXPOSE 80 diff --git a/Dockerfile.dev b/Dockerfile.dev index df1918a2..113ec3b2 100644 --- a/Dockerfile.dev +++ b/Dockerfile.dev @@ -1,4 +1,4 @@ -FROM alpine:3.8 +FROM alpine:3.9 VOLUME /mnt/routes EXPOSE 80 From bf715464f7b232ea9eaa19cbace2abafc4db744f Mon Sep 17 00:00:00 2001 From: Teri Hornych Date: Mon, 9 Dec 2019 15:02:14 +0100 Subject: [PATCH 06/11] Document accessible data in RAW_FORMAT template --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index 8fd82874..e8eac97c 100644 --- a/README.md +++ b/README.md @@ -194,6 +194,14 @@ If you use multiline logging with raw, it's recommended to json encode the Data The raw adapter has a function `toJSON` that can be used to format the message/fields to generate JSON-like output in a simple way, or full JSON output. +The RAW_FORMAT env variable is used as a [Go template](https://golang.org/pkg/text/template/) with a [`Message` struct](https://github.com/gliderlabs/logspout/blob/master/router/types.go#L52) passed as data. You can access the following fields + +* `Source` - source stream name ("stdout", "stderr", ...) +* `Data` - original log message +* `Time` - a Go [`Time` struct](https://golang.org/pkg/time/#Time) +* `Container` - a [go-dockerclient](https://github.com/fsouza/go-dockerclient) `Container` struct (see [container.go](https://github.com/fsouza/go-dockerclient/blob/master/container.go#L443) source file for accessible fields) + + Use examples: ##### Mixed JSON + generic: From 621524edba170d90147845de6d48bbdec8a2d858 Mon Sep 17 00:00:00 2001 From: Michael Hobbs Date: Mon, 9 Dec 2019 12:08:06 -0700 Subject: [PATCH 07/11] fix linting --- .golangci.yml | 77 ++++++++++++++++++++++++++++ Dockerfile | 2 +- Dockerfile.dev | 15 +++--- Makefile | 29 +++++++---- adapters/multiline/multiline.go | 14 ++--- adapters/multiline/multiline_test.go | 3 +- adapters/raw/raw.go | 2 +- adapters/syslog/syslog.go | 27 ++++------ adapters/syslog/syslog_test.go | 20 +++----- build.sh | 8 +-- cfg/cfg.go | 11 ++++ circle.yml | 6 +-- custom/build.sh | 8 +-- custom/modules.go | 3 +- glide.lock | 48 ----------------- glide.yaml | 9 ---- go.mod | 19 +++++++ go.sum | 35 +++++++++++++ healthcheck/healthcheck.go | 5 +- httpstream/httpstream.go | 15 +++--- logspout.go | 40 ++++++--------- modules.go | 6 +-- router/extpoints.go | 28 ++++------ router/http.go | 10 ++-- router/persist.go | 2 +- router/pump.go | 72 +++++++++++++------------- router/pump_test.go | 2 +- router/routes.go | 18 ++++--- router/types.go | 18 +++---- routesapi/routesapi.go | 6 +-- transports/tls/tls.go | 7 +-- transports/tls/tls_test.go | 12 ++--- 32 files changed, 322 insertions(+), 255 deletions(-) create mode 100644 .golangci.yml create mode 100644 cfg/cfg.go delete mode 100644 glide.lock delete mode 100644 glide.yaml create mode 100644 go.mod create mode 100644 go.sum diff --git a/.golangci.yml b/.golangci.yml new file mode 100644 index 00000000..e97b1d9f --- /dev/null +++ b/.golangci.yml @@ -0,0 +1,77 @@ +linters-settings: + errcheck: + # ignore cases where we truly don't care about the returned error + ignore: net/http:^Write$,io:^WriteString$ + govet: + check-shadowing: true + settings: + printf: + funcs: + - (github.com/golangci/golangci-lint/pkg/logutils.Log).Infof + - (github.com/golangci/golangci-lint/pkg/logutils.Log).Warnf + - (github.com/golangci/golangci-lint/pkg/logutils.Log).Errorf + - (github.com/golangci/golangci-lint/pkg/logutils.Log).Fatalf + golint: + min-confidence: 0 + gocyclo: + min-complexity: 12 + maligned: + suggest-new: true + dupl: + threshold: 100 + goconst: + min-len: 2 + min-occurrences: 2 + depguard: + list-type: blacklist + misspell: + locale: US + lll: + line-length: 160 + goimports: + local-prefixes: github.com/gliderlabs/logspout + gocritic: + settings: + hugeParam: + sizeThreshold: 160 + enabled-tags: + - performance + nakedret: + max-func-lines: 65 + +linters: + enable-all: true + disable: + - maligned + - prealloc + - gochecknoglobals + - funlen + - gochecknoinits + +issues: + # Excluding configuration per-path, per-linter, per-text and per-source + exclude-rules: + # Exclude some linters from running on tests files. + - path: _test\.go + linters: + - dupl + - errcheck + - goconst + - gocyclo + - gosec + - lll + - nakedret + - unparam + - funlen + +run: + deadline: 2m + issues-exit-code: 1 + +# golangci.com configuration +# https://github.com/golangci/golangci/wiki/Configuration +service: + golangci-lint-version: 1.18.x # use the fixed version to not introduce new linters unexpectedly + prepare: + - echo "here I can run custom commands, but no preparation needed for this repo" + diff --git a/Dockerfile b/Dockerfile index 9c33cace..c55f9b32 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:3.9 +FROM alpine:3.10 ENTRYPOINT ["/bin/logspout"] VOLUME /mnt/routes EXPOSE 80 diff --git a/Dockerfile.dev b/Dockerfile.dev index 113ec3b2..c0247fa2 100644 --- a/Dockerfile.dev +++ b/Dockerfile.dev @@ -1,12 +1,9 @@ -FROM alpine:3.9 +FROM alpine:3.10 VOLUME /mnt/routes EXPOSE 80 -ENV GOPATH /go -RUN apk --no-cache add go build-base git mercurial ca-certificates -COPY . /go/src/github.com/gliderlabs/logspout -WORKDIR /go/src/github.com/gliderlabs/logspout -RUN go get -CMD go get \ - && go build -ldflags "-X main.Version=dev" -o /bin/logspout \ - && exec /bin/logspout +RUN apk --no-cache add go build-base git mercurial ca-certificates curl +COPY . /src +WORKDIR /src +CMD go build -ldflags "-X main.Version=dev" -o /bin/logspout \ + && exec /bin/logspout diff --git a/Makefile b/Makefile index 645b9ef7..2bf17370 100644 --- a/Makefile +++ b/Makefile @@ -5,11 +5,13 @@ VERSION=$(shell cat VERSION) # max image size of 40MB MAX_IMAGE_SIZE := 40000000 +GOBIN := $(shell go env GOPATH | awk -F ":" '{ print $$1 }')/bin +GOLANGCI_LINT_VERSION := v1.18.0 + ifeq ($(shell uname), Darwin) XARGS_ARG="-L1" endif GOPACKAGES ?= $(shell go list ./... | egrep -v 'custom|vendor') -GOLINT := go list ./... | egrep -v '/custom/|/vendor/' | xargs $(XARGS_ARG) golint | egrep -v 'extpoints.go|types.go' TEST_ARGS ?= -race ifdef TEST_RUN @@ -37,12 +39,21 @@ build-custom: docker tag $(NAME):$(VERSION) gliderlabs/$(NAME):master cd custom && docker build -t $(NAME):custom . -lint: - test -x $(GOPATH)/bin/golint || go get github.com/golang/lint/golint - go get \ - && go install $(GOPACKAGES) \ - && go tool vet -v $(shell ls -d */ | egrep -v 'custom|vendor/' | xargs $(XARGS_ARG)) - @if [ -n "$(shell $(GOLINT) | cut -d ':' -f 1)" ]; then $(GOLINT) && exit 1 ; fi +lint-requirements: +ifeq ($(shell which golangci-lint), ) + curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(GOBIN) $(GOLANGCI_LINT_VERSION) +endif + +lint: lint-requirements + $(GOBIN)/golangci-lint run + +lint-ci-direct: lint-requirements + $(GOBIN)/golangci-lint --verbose run + +lint-ci: build-dev + docker run \ + -v $(PWD):/go/src/github.com/gliderlabs/logspout \ + $(NAME):dev make -e lint-ci-direct test: build-dev docker run \ @@ -76,7 +87,7 @@ test-healthcheck: -p 8000:80 \ -v /var/run/docker.sock:/var/run/docker.sock \ $(NAME):$(VERSION) - sleep 2 + sleep 5 docker logs $(NAME)-healthcheck docker inspect --format='{{ .State.Running }}' $(NAME)-healthcheck | grep true curl --head --silent localhost:8000/health | grep "200 OK" @@ -85,7 +96,7 @@ test-healthcheck: test-custom: docker run --name $(NAME)-custom $(NAME):custom || true - docker logs $(NAME)-custom | grep -q logstash + docker logs $(NAME)-custom 2>&1 | grep -q logstash docker rmi gliderlabs/$(NAME):master || true docker rm $(NAME)-custom || true diff --git a/adapters/multiline/multiline.go b/adapters/multiline/multiline.go index 2b0ade06..d2c1d3e3 100644 --- a/adapters/multiline/multiline.go +++ b/adapters/multiline/multiline.go @@ -9,7 +9,8 @@ import ( "sync" "time" - "github.com/fsouza/go-dockerclient" + docker "github.com/fsouza/go-dockerclient" + "github.com/gliderlabs/logspout/router" ) @@ -40,11 +41,10 @@ type Adapter struct { } // NewMultilineAdapter returns a configured multiline.Adapter -func NewMultilineAdapter(route *router.Route) (a router.LogAdapter, err error) { +func NewMultilineAdapter(route *router.Route) (a router.LogAdapter, err error) { //nolint:gocyclo enableByDefault := true enableStr := os.Getenv("MULTILINE_ENABLE_DEFAULT") if enableStr != "" { - var err error enableByDefault, err = strconv.ParseBool(enableStr) if err != nil { return nil, errors.New("multiline: invalid value for MULTILINE_ENABLE_DEFAULT (must be true|false): " + enableStr) @@ -92,8 +92,8 @@ func NewMultilineAdapter(route *router.Route) (a router.LogAdapter, err error) { flushAfter := 500 * time.Millisecond flushAfterStr := os.Getenv("MULTILINE_FLUSH_AFTER") if flushAfterStr != "" { - timeoutMS, err := strconv.Atoi(flushAfterStr) - if err != nil { + timeoutMS, errConv := strconv.Atoi(flushAfterStr) + if errConv != nil { return nil, errors.New("multiline: invalid value for multiline_timeout (must be number): " + flushAfterStr) } flushAfter = time.Duration(timeoutMS) * time.Millisecond @@ -135,7 +135,7 @@ func NewMultilineAdapter(route *router.Route) (a router.LogAdapter, err error) { } // Stream sends log data to the next adapter -func (a *Adapter) Stream(logstream chan *router.Message) { +func (a *Adapter) Stream(logstream chan *router.Message) { //nolint:gocyclo wg := sync.WaitGroup{} wg.Add(1) go func() { @@ -173,7 +173,7 @@ func (a *Adapter) Stream(logstream chan *router.Message) { a.buffers[cID] = message } else { isLastLine := a.isLastLine(message) - + if oldExists { old.Data += a.separator + message.Data message = old diff --git a/adapters/multiline/multiline_test.go b/adapters/multiline/multiline_test.go index d1f05ee0..2f32ac30 100644 --- a/adapters/multiline/multiline_test.go +++ b/adapters/multiline/multiline_test.go @@ -7,7 +7,8 @@ import ( "testing" "time" - "github.com/fsouza/go-dockerclient" + docker "github.com/fsouza/go-dockerclient" + "github.com/gliderlabs/logspout/router" ) diff --git a/adapters/raw/raw.go b/adapters/raw/raw.go index 3b93d152..bb483a50 100644 --- a/adapters/raw/raw.go +++ b/adapters/raw/raw.go @@ -21,7 +21,7 @@ var funcs = template.FuncMap{ "toJSON": func(value interface{}) string { bytes, err := json.Marshal(value) if err != nil { - log.Println("error marshalling to JSON: ", err) + log.Println("error marshaling to JSON: ", err) return "null" } return string(bytes) diff --git a/adapters/syslog/syslog.go b/adapters/syslog/syslog.go index 72fd4b4b..77d75296 100644 --- a/adapters/syslog/syslog.go +++ b/adapters/syslog/syslog.go @@ -15,6 +15,7 @@ import ( "text/template" "time" + "github.com/gliderlabs/logspout/cfg" "github.com/gliderlabs/logspout/router" ) @@ -34,7 +35,7 @@ func init() { } func setRetryCount() { - if count, err := strconv.Atoi(getopt("RETRY_COUNT", strconv.Itoa(defaultRetryCount))); err != nil { + if count, err := strconv.Atoi(cfg.GetEnvDefault("RETRY_COUNT", strconv.Itoa(defaultRetryCount))); err != nil { retryCount = uint(defaultRetryCount) } else { retryCount = uint(count) @@ -42,14 +43,6 @@ func setRetryCount() { debug("setting retryCount to:", retryCount) } -func getopt(name, dfault string) string { - value := os.Getenv(name) - if value == "" { - value = dfault - } - return value -} - func debug(v ...interface{}) { if os.Getenv("DEBUG") != "" { log.Println(v...) @@ -61,7 +54,7 @@ func getHostname() string { if err == nil && len(content) > 0 { hostname = strings.TrimRight(string(content), "\r\n") } else { - hostname = getopt("SYSLOG_HOSTNAME", "{{.Container.Config.Hostname}}") + hostname = cfg.GetEnvDefault("SYSLOG_HOSTNAME", "{{.Container.Config.Hostname}}") } return hostname } @@ -77,18 +70,18 @@ func NewSyslogAdapter(route *router.Route) (router.LogAdapter, error) { return nil, err } - format := getopt("SYSLOG_FORMAT", "rfc5424") - priority := getopt("SYSLOG_PRIORITY", "{{.Priority}}") - pid := getopt("SYSLOG_PID", "{{.Container.State.Pid}}") + format := cfg.GetEnvDefault("SYSLOG_FORMAT", "rfc5424") + priority := cfg.GetEnvDefault("SYSLOG_PRIORITY", "{{.Priority}}") + pid := cfg.GetEnvDefault("SYSLOG_PID", "{{.Container.State.Pid}}") hostname = getHostname() - tag := getopt("SYSLOG_TAG", "{{.ContainerName}}"+route.Options["append_tag"]) - structuredData := getopt("SYSLOG_STRUCTURED_DATA", "") + tag := cfg.GetEnvDefault("SYSLOG_TAG", "{{.ContainerName}}"+route.Options["append_tag"]) + structuredData := cfg.GetEnvDefault("SYSLOG_STRUCTURED_DATA", "") if route.Options["structured_data"] != "" { structuredData = route.Options["structured_data"] } - data := getopt("SYSLOG_DATA", "{{.Data}}") - timestamp := getopt("SYSLOG_TIMESTAMP", "{{.Timestamp}}") + data := cfg.GetEnvDefault("SYSLOG_DATA", "{{.Data}}") + timestamp := cfg.GetEnvDefault("SYSLOG_TIMESTAMP", "{{.Timestamp}}") if structuredData == "" { structuredData = "-" diff --git a/adapters/syslog/syslog_test.go b/adapters/syslog/syslog_test.go index d6dcca40..1584266b 100644 --- a/adapters/syslog/syslog_test.go +++ b/adapters/syslog/syslog_test.go @@ -2,7 +2,6 @@ package syslog import ( "bufio" - "fmt" "io" "io/ioutil" "log" @@ -15,22 +14,17 @@ import ( "text/template" "time" - docker "github.com/fsouza/go-dockerclient" - "github.com/gliderlabs/logspout/router" - _ "github.com/gliderlabs/logspout/transports/tcp" _ "github.com/gliderlabs/logspout/transports/tls" _ "github.com/gliderlabs/logspout/transports/udp" + + docker "github.com/fsouza/go-dockerclient" + + "github.com/gliderlabs/logspout/router" ) const ( - testPriority = "{{.Priority}}" - testTimestamp = "{{.Timestamp}}" - testHostname = "{{.Container.Config.Hostname}}" - testTag = "{{.ContainerName}}" - testPid = "{{.Container.State.Pid}}" - testData = "{{.Data}}" - connCloseIdx = 5 + connCloseIdx = 5 ) var ( @@ -41,8 +35,6 @@ var ( Hostname: "8dfafdbc3a40", }, } - testTmplStr = fmt.Sprintf("<%s>%s %s %s[%s]: %s\n", - testPriority, testTimestamp, testHostname, testTag, testPid, testData) hostHostnameFilename = "/etc/host_hostname" hostnameContent = "hostname" badHostnameContent = "hostname\r\n" @@ -89,7 +81,7 @@ func TestSyslogReconnectOnClose(t *testing.T) { case msg := <-done: // Don't check a message that we know was dropped if msgnum%connCloseIdx == 0 { - _ = <-messages + <-messages msgnum++ } check(t, adapter.(*Adapter).tmpl, <-messages, msg) diff --git a/build.sh b/build.sh index 7aed13cb..92bcc8c0 100755 --- a/build.sh +++ b/build.sh @@ -1,14 +1,10 @@ #!/bin/sh set -e apk add --update go build-base git mercurial ca-certificates -mkdir -p /go/src/github.com/gliderlabs -cp -r /src /go/src/github.com/gliderlabs/logspout -cd /go/src/github.com/gliderlabs/logspout -export GOPATH=/go -go get github.com/Masterminds/glide && $GOPATH/bin/glide install +cd /src go build -ldflags "-X main.Version=$1" -o /bin/logspout apk del go git mercurial build-base -rm -rf /go /var/cache/apk/* /root/.glide +rm -rf /root/go /var/cache/apk/* # backwards compatibility ln -fs /tmp/docker.sock /var/run/docker.sock diff --git a/cfg/cfg.go b/cfg/cfg.go new file mode 100644 index 00000000..a101342e --- /dev/null +++ b/cfg/cfg.go @@ -0,0 +1,11 @@ +package cfg + +import "os" + +// GetEnvDefault is a helper function to retrieve an env variable value OR return a default value +func GetEnvDefault(name, dfault string) string { + if val := os.Getenv(name); val != "" { + return val + } + return dfault +} diff --git a/circle.yml b/circle.yml index 9626e440..306484ba 100644 --- a/circle.yml +++ b/circle.yml @@ -2,17 +2,15 @@ version: 2 jobs: build: machine: true - working_directory: ~/.go_workspace/src/github.com/gliderlabs/logspout + working_directory: /home/circleci/logspout environment: DEBUG: true steps: - checkout - - run: | - go get github.com/golang/lint/golint - run: | make circleci - run: | - make lint + make lint-ci - run: | make build - run: | diff --git a/custom/build.sh b/custom/build.sh index 017c113e..92bcc8c0 100755 --- a/custom/build.sh +++ b/custom/build.sh @@ -1,14 +1,10 @@ #!/bin/sh set -e apk add --update go build-base git mercurial ca-certificates -mkdir -p /go/src/github.com/gliderlabs -cp -r /src /go/src/github.com/gliderlabs/logspout -cd /go/src/github.com/gliderlabs/logspout -export GOPATH=/go -go get +cd /src go build -ldflags "-X main.Version=$1" -o /bin/logspout apk del go git mercurial build-base -rm -rf /go /var/cache/apk/* /root/.glide +rm -rf /root/go /var/cache/apk/* # backwards compatibility ln -fs /tmp/docker.sock /var/run/docker.sock diff --git a/custom/modules.go b/custom/modules.go index ec4500d7..0904f966 100644 --- a/custom/modules.go +++ b/custom/modules.go @@ -1,9 +1,10 @@ package main import ( + _ "github.com/looplab/logspout-logstash" + _ "github.com/gliderlabs/logspout/adapters/syslog" _ "github.com/gliderlabs/logspout/transports/tcp" _ "github.com/gliderlabs/logspout/transports/tls" _ "github.com/gliderlabs/logspout/transports/udp" - _ "github.com/looplab/logspout-logstash" ) diff --git a/glide.lock b/glide.lock deleted file mode 100644 index 97ee4fa9..00000000 --- a/glide.lock +++ /dev/null @@ -1,48 +0,0 @@ -hash: 83a577e65396190336bd5117580f29dbf983a3e2fdc5732a111ae56f229ed978 -updated: 2017-11-07T22:39:32.638917215-06:00 -imports: -- name: github.com/docker/docker - version: ad969f1aa782478725a7f338cf963fa82f484609 - subpackages: - - opts - - pkg/archive - - pkg/fileutils - - pkg/homedir - - pkg/idtools - - pkg/ioutils - - pkg/longpath - - pkg/pools - - pkg/promise - - pkg/stdcopy - - pkg/system -- name: github.com/docker/engine-api - version: 98348ad6f9c89bb10f31ac32cd1b12cbadd292b6 - subpackages: - - types/filters - - types/versions -- name: github.com/docker/go-units - version: f2d77a61e3c169b43402a0a1e84f06daf29b8190 -- name: github.com/fsouza/go-dockerclient - version: 1a3d0cfd7814bbfe44ada7617654948c99891749 -- name: github.com/gorilla/context - version: aed02d124ae4a0e94fea4541c8effd05bf0c8296 -- name: github.com/gorilla/mux - version: 9fa818a44c2bf1396a17f9d5a3c0f6dd39d2ff8e -- name: github.com/hashicorp/go-cleanhttp - version: ad28ea4487f05916463e2423a55166280e8254b5 -- name: github.com/opencontainers/runc - version: 9d7831e41d3ef428b67685eeb27f2b4a22a92391 - subpackages: - - libcontainer/user -- name: github.com/Sirupsen/logrus - version: f3cfb454f4c209e6668c95216c4744b8fddb2356 -- name: golang.org/x/net - version: f841c39de738b1d0df95b5a7187744f0e03d8112 - subpackages: - - context - - websocket -- name: golang.org/x/sys - version: a408501be4d17ee978c04a618e7a1b22af058c0e - subpackages: - - unix -testImports: [] diff --git a/glide.yaml b/glide.yaml deleted file mode 100644 index d5370a9c..00000000 --- a/glide.yaml +++ /dev/null @@ -1,9 +0,0 @@ -package: github.com/gliderlabs/logspout -excludeDirs: -- custom -import: -- package: github.com/fsouza/go-dockerclient -- package: github.com/gorilla/mux -- package: golang.org/x/net - subpackages: - - websocket diff --git a/go.mod b/go.mod new file mode 100644 index 00000000..61201246 --- /dev/null +++ b/go.mod @@ -0,0 +1,19 @@ +module github.com/gliderlabs/logspout + +go 1.13 + +require ( + github.com/Sirupsen/logrus v0.10.1-0.20160601113210-f3cfb454f4c2 // indirect + github.com/docker/docker v1.4.2-0.20160708193732-ad969f1aa782 // indirect + github.com/docker/engine-api v0.3.2-0.20160708123604-98348ad6f9c8 // indirect + github.com/docker/go-units v0.3.1 // indirect + github.com/fsouza/go-dockerclient v0.0.0-20160624230725-1a3d0cfd7814 + github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4 // indirect + github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b + github.com/hashicorp/go-cleanhttp v0.0.0-20160407174126-ad28ea4487f0 // indirect + github.com/looplab/logspout-logstash v0.0.0-20171130125839-68a4e47e757d + github.com/opencontainers/runc v1.0.0-rc1.0.20160706165155-9d7831e41d3e // indirect + github.com/stretchr/testify v1.4.0 // indirect + golang.org/x/net v0.0.0-20160707223729-f841c39de738 + golang.org/x/sys v0.0.0-20160704031755-a408501be4d1 // indirect +) diff --git a/go.sum b/go.sum new file mode 100644 index 00000000..facf3a67 --- /dev/null +++ b/go.sum @@ -0,0 +1,35 @@ +github.com/Sirupsen/logrus v0.10.1-0.20160601113210-f3cfb454f4c2 h1:3BYvDlSNPyoYk6lr17s9IueNAabOBur3f3uVULjbhTA= +github.com/Sirupsen/logrus v0.10.1-0.20160601113210-f3cfb454f4c2/go.mod h1:rmk17hk6i8ZSAJkSDa7nOxamrG+SP4P0mm+DAvExv4U= +github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/docker/docker v1.4.2-0.20160708193732-ad969f1aa782 h1:akNYo0V5gmaDl8LUnd9G1/3Y8ohMl3s2b8gfvBHnURo= +github.com/docker/docker v1.4.2-0.20160708193732-ad969f1aa782/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/engine-api v0.3.2-0.20160708123604-98348ad6f9c8 h1:H443uS3liJKNFX8++m61ng44AnC+HKV1MU9Uhmq3hUk= +github.com/docker/engine-api v0.3.2-0.20160708123604-98348ad6f9c8/go.mod h1:xtQCpzf4YysNZCVFfIGIm7qfLvYbxtLkEVVfKhTVOvw= +github.com/docker/go-units v0.3.1 h1:QAFdsA6jLCnglbqE6mUsHuPcJlntY94DkxHf4deHKIU= +github.com/docker/go-units v0.3.1/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/fsouza/go-dockerclient v0.0.0-20160624230725-1a3d0cfd7814 h1:FSgKZZ2VFfEZkvrRfZ5LmUEgVapd9pnam49bjbV6a7M= +github.com/fsouza/go-dockerclient v0.0.0-20160624230725-1a3d0cfd7814/go.mod h1:KpcjM623fQYE9MZiTGzKhjfxXAV9wbyX2C1cyRHfhl0= +github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4 h1:3nOfQt8sRPYbXORD5tJ8YyQ3HlL2Jt3LJ2U17CbNh6I= +github.com/gorilla/context v0.0.0-20160525203319-aed02d124ae4/go.mod h1:kBGZzfjB9CEq2AlWe17Uuf7NDRt0dE0s8S51q0aT7Yg= +github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b h1:OFvZV3a+25cGJH9dETHw0nk0wV6hLZI7IJijOkXEFS0= +github.com/gorilla/mux v0.0.0-20160605233521-9fa818a44c2b/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2zaAs= +github.com/hashicorp/go-cleanhttp v0.0.0-20160407174126-ad28ea4487f0 h1:2l0haPDqCzZEO160UR5DSrrl8RWptFCoxFsSbRLJBaI= +github.com/hashicorp/go-cleanhttp v0.0.0-20160407174126-ad28ea4487f0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= +github.com/looplab/logspout-logstash v0.0.0-20171130125839-68a4e47e757d h1:XHk6tU5oWCIrflIeYkS7/6nKMvqo3q//bJud+L4BguI= +github.com/looplab/logspout-logstash v0.0.0-20171130125839-68a4e47e757d/go.mod h1:JGtIU22PbW89UiPrH/M1qcGZ9WkWlibloyUHGQ3Tgok= +github.com/opencontainers/runc v1.0.0-rc1.0.20160706165155-9d7831e41d3e h1:SO9iqX0giNVXkTwPdEENwl1wK+RqviyAnCEbJS5azMU= +github.com/opencontainers/runc v1.0.0-rc1.0.20160706165155-9d7831e41d3e/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +golang.org/x/net v0.0.0-20160707223729-f841c39de738 h1:N5K0l3yYkhlC0RSRoAhtspo2WgRvBMwZYoyB2ji+gkg= +golang.org/x/net v0.0.0-20160707223729-f841c39de738/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/sys v0.0.0-20160704031755-a408501be4d1 h1:QtO5ZFD1u7KisZhuRLL2GaZAZDdJqUcTdjFJj8OEePA= +golang.org/x/sys v0.0.0-20160704031755-a408501be4d1/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/healthcheck/healthcheck.go b/healthcheck/healthcheck.go index b552b4ff..bbd5bcd1 100644 --- a/healthcheck/healthcheck.go +++ b/healthcheck/healthcheck.go @@ -3,12 +3,13 @@ package healthcheck import ( "net/http" - "github.com/gliderlabs/logspout/router" "github.com/gorilla/mux" + + "github.com/gliderlabs/logspout/router" ) func init() { - router.HttpHandlers.Register(HealthCheck, "health") + router.HTTPHandlers.Register(HealthCheck, "health") } // HealthCheck returns a http.Handler for the health check diff --git a/httpstream/httpstream.go b/httpstream/httpstream.go index e1db6ad0..dc2fb894 100644 --- a/httpstream/httpstream.go +++ b/httpstream/httpstream.go @@ -8,13 +8,14 @@ import ( "os" "strconv" - "github.com/gliderlabs/logspout/router" "github.com/gorilla/mux" "golang.org/x/net/websocket" + + "github.com/gliderlabs/logspout/router" ) func init() { - router.HttpHandlers.Register(LogStreamer, "logs") + router.HTTPHandlers.Register(LogStreamer, "logs") } func debug(v ...interface{}) { @@ -51,16 +52,16 @@ func LogStreamer() http.Handler { logstream := make(chan *router.Message) defer close(logstream) - var closer <-chan bool + var closer <-chan struct{} if req.Header.Get("Upgrade") == "websocket" { debug("http: logs streamer connected [websocket]") - closerBi := make(chan bool) + closerBi := make(chan struct{}) defer websocketStreamer(w, req, logstream, closerBi) closer = closerBi } else { debug("http: logs streamer connected [http]") defer httpStreamer(w, req, logstream, route.MultiContainer()) - closer = w.(http.CloseNotifier).CloseNotify() + closer = req.Context().Done() } route.OverrideCloser(closer) @@ -100,7 +101,7 @@ func normalName(name string) string { return name[1:] } -func websocketStreamer(w http.ResponseWriter, req *http.Request, logstream chan *router.Message, closer chan bool) { +func websocketStreamer(w http.ResponseWriter, req *http.Request, logstream chan *router.Message, closer chan struct{}) { websocket.Handler(func(conn *websocket.Conn) { for logline := range logstream { if req.URL.Query().Get("source") != "" && logline.Source != req.URL.Query().Get("source") { @@ -108,7 +109,7 @@ func websocketStreamer(w http.ResponseWriter, req *http.Request, logstream chan } _, err := conn.Write(append(marshal(logline), '\n')) if err != nil { - closer <- true + closer <- struct{}{} return } } diff --git a/logspout.go b/logspout.go index ac29d878..2ec9e4a4 100644 --- a/logspout.go +++ b/logspout.go @@ -7,56 +7,48 @@ import ( "strings" "text/tabwriter" + "github.com/gliderlabs/logspout/cfg" "github.com/gliderlabs/logspout/router" ) // Version is the running version of logspout var Version string -func getopt(name, dfault string) string { - value := os.Getenv(name) - if value == "" { - value = dfault - } - return value -} - func main() { if len(os.Args) == 2 && os.Args[1] == "--version" { - fmt.Println(Version) + fmt.Printf("%s\n", Version) os.Exit(0) } - fmt.Printf("# logspout %s by gliderlabs\n", Version) - fmt.Printf("# adapters: %s\n", strings.Join(router.AdapterFactories.Names(), " ")) - fmt.Printf("# options : ") - if getopt("DEBUG", "") != "" { - fmt.Printf("debug:%s ", getopt("DEBUG", "")) + log.Printf("# logspout %s by gliderlabs\n", Version) + log.Printf("# adapters: %s\n", strings.Join(router.AdapterFactories.Names(), " ")) + log.Printf("# options : ") + if d := cfg.GetEnvDefault("DEBUG", ""); d != "" { + log.Printf("debug:%s\n", d) } - if getopt("BACKLOG", "") != "" { - fmt.Printf("backlog:%s ", getopt("BACKLOG", "")) + if b := cfg.GetEnvDefault("BACKLOG", ""); b != "" { + log.Printf("backlog:%s\n", b) } - fmt.Printf("persist:%s\n", getopt("ROUTESPATH", "/mnt/routes")) + log.Printf("persist:%s\n", cfg.GetEnvDefault("ROUTESPATH", "/mnt/routes")) var jobs []string for _, job := range router.Jobs.All() { - err := job.Setup() - if err != nil { - fmt.Println("!!", err) + if err := job.Setup(); err != nil { + log.Printf("!! %v\n", err) os.Exit(1) } if job.Name() != "" { jobs = append(jobs, job.Name()) } } - fmt.Printf("# jobs : %s\n", strings.Join(jobs, " ")) + log.Printf("# jobs : %s\n", strings.Join(jobs, " ")) routes, _ := router.Routes.GetAll() if len(routes) > 0 { - fmt.Println("# routes :") + log.Println("# routes :") w := new(tabwriter.Writer) w.Init(os.Stdout, 0, 8, 0, '\t', 0) - fmt.Fprintln(w, "# ADAPTER\tADDRESS\tCONTAINERS\tSOURCES\tOPTIONS") + fmt.Fprintln(w, "# ADAPTER\tADDRESS\tCONTAINERS\tSOURCES\tOPTIONS") //nolint:errcheck for _, route := range routes { fmt.Fprintf(w, "# %s\t%s\t%s\t%s\t%s\n", route.Adapter, @@ -67,7 +59,7 @@ func main() { } w.Flush() } else { - fmt.Println("# routes : none") + log.Println("# routes : none") } for _, job := range router.Jobs.All() { diff --git a/modules.go b/modules.go index 8ed81e09..1c2681e2 100644 --- a/modules.go +++ b/modules.go @@ -1,13 +1,13 @@ package main import ( - _ "github.com/gliderlabs/logspout/healthcheck" + _ "github.com/gliderlabs/logspout/adapters/multiline" _ "github.com/gliderlabs/logspout/adapters/raw" _ "github.com/gliderlabs/logspout/adapters/syslog" - _ "github.com/gliderlabs/logspout/adapters/multiline" + _ "github.com/gliderlabs/logspout/healthcheck" _ "github.com/gliderlabs/logspout/httpstream" _ "github.com/gliderlabs/logspout/routesapi" _ "github.com/gliderlabs/logspout/transports/tcp" - _ "github.com/gliderlabs/logspout/transports/udp" _ "github.com/gliderlabs/logspout/transports/tls" + _ "github.com/gliderlabs/logspout/transports/udp" ) diff --git a/router/extpoints.go b/router/extpoints.go index 422c71e0..be4a3fd6 100644 --- a/router/extpoints.go +++ b/router/extpoints.go @@ -3,9 +3,9 @@ package router import ( "reflect" - "sync" - "strings" "runtime" + "strings" + "sync" ) var registry = struct { @@ -119,10 +119,10 @@ func Unregister(name string) []string { return ifaces } -// HttpHandler +// HTTPHandler -var HttpHandlers = &httpHandlerExt{ - newExtensionPoint(new(HttpHandler)), +var HTTPHandlers = &httpHandlerExt{ + newExtensionPoint(new(HTTPHandler)), } type httpHandlerExt struct { @@ -133,22 +133,22 @@ func (ep *httpHandlerExt) Unregister(name string) bool { return ep.unregister(name) } -func (ep *httpHandlerExt) Register(component HttpHandler, name string) bool { +func (ep *httpHandlerExt) Register(component HTTPHandler, name string) bool { return ep.register(component, name) } -func (ep *httpHandlerExt) Lookup(name string) (HttpHandler, bool) { +func (ep *httpHandlerExt) Lookup(name string) (HTTPHandler, bool) { ext, ok := ep.lookup(name) if !ok { return nil, ok } - return ext.(HttpHandler), ok + return ext.(HTTPHandler), ok } -func (ep *httpHandlerExt) All() map[string]HttpHandler { - all := make(map[string]HttpHandler) +func (ep *httpHandlerExt) All() map[string]HTTPHandler { + all := make(map[string]HTTPHandler) for k, v := range ep.all() { - all[k] = v.(HttpHandler) + all[k] = v.(HTTPHandler) } return all } @@ -161,7 +161,6 @@ func (ep *httpHandlerExt) Names() []string { return names } - // AdapterFactory var AdapterFactories = &adapterFactoryExt{ @@ -204,7 +203,6 @@ func (ep *adapterFactoryExt) Names() []string { return names } - // AdapterTransport var AdapterTransports = &adapterTransportExt{ @@ -247,7 +245,6 @@ func (ep *adapterTransportExt) Names() []string { return names } - // Job var Jobs = &jobExt{ @@ -290,7 +287,6 @@ func (ep *jobExt) Names() []string { return names } - // LogRouter var LogRouters = &logRouterExt{ @@ -332,5 +328,3 @@ func (ep *logRouterExt) Names() []string { } return names } - - diff --git a/router/http.go b/router/http.go index 9fc17b83..09243ef8 100644 --- a/router/http.go +++ b/router/http.go @@ -4,11 +4,13 @@ import ( "fmt" "net/http" "strings" + + "github.com/gliderlabs/logspout/cfg" ) func init() { - bindAddress := getopt("HTTP_BIND_ADDRESS", "0.0.0.0") - port := getopt("PORT", getopt("HTTP_PORT", "80")) + bindAddress := cfg.GetEnvDefault("HTTP_BIND_ADDRESS", "0.0.0.0") + port := cfg.GetEnvDefault("PORT", cfg.GetEnvDefault("HTTP_PORT", "80")) Jobs.Register(&httpService{bindAddress, port}, "http") } @@ -19,11 +21,11 @@ type httpService struct { func (s *httpService) Name() string { return fmt.Sprintf("http[%s]:%s", - strings.Join(HttpHandlers.Names(), ","), s.port) + strings.Join(HTTPHandlers.Names(), ","), s.port) } func (s *httpService) Setup() error { - for name, handler := range HttpHandlers.All() { + for name, handler := range HTTPHandlers.All() { h := handler() http.Handle("/"+name, h) http.Handle("/"+name+"/", h) diff --git a/router/persist.go b/router/persist.go index 7e36767c..c86d06e2 100644 --- a/router/persist.go +++ b/router/persist.go @@ -49,7 +49,7 @@ func (fs RouteFileStore) GetAll() ([]*Route, error) { return routes, nil } -// Add writes a marshalled *Route to the RouteFileStore +// Add writes a marshaled *Route to the RouteFileStore func (fs RouteFileStore) Add(route *Route) error { return ioutil.WriteFile(fs.Filename(route.ID), marshal(route), 0644) } diff --git a/router/pump.go b/router/pump.go index 94e34786..5df27f54 100644 --- a/router/pump.go +++ b/router/pump.go @@ -10,10 +10,23 @@ import ( "sync" "time" - "github.com/fsouza/go-dockerclient" + docker "github.com/fsouza/go-dockerclient" + + "github.com/gliderlabs/logspout/cfg" ) -var allowTTY bool +const ( + defaultPumpName = "pump" + pumpEventStatusStartName = "start" + pumpEventStatusRestartName = "restart" + pumpEventStatusRenameName = "rename" + pumpEventStatusDieName = "die" + trueString = "true" +) + +var ( + allowTTY bool +) func init() { pump := &LogsPump{ @@ -21,16 +34,8 @@ func init() { routes: make(map[chan *update]struct{}), } setAllowTTY() - LogRouters.Register(pump, "pump") - Jobs.Register(pump, "pump") -} - -func getopt(name, dfault string) string { - value := os.Getenv(name) - if value == "" { - value = dfault - } - return value + LogRouters.Register(pump, defaultPumpName) + Jobs.Register(pump, defaultPumpName) } func debug(v ...interface{}) { @@ -40,14 +45,11 @@ func debug(v ...interface{}) { } func backlog() bool { - if os.Getenv("BACKLOG") == "false" { - return false - } - return true + return os.Getenv("BACKLOG") == "false" } func setAllowTTY() { - if t := getopt("ALLOW_TTY", ""); t == "true" { + if t := cfg.GetEnvDefault("ALLOW_TTY", ""); t == trueString { allowTTY = true } debug("setting allowTTY to:", allowTTY) @@ -82,12 +84,12 @@ func logDriverSupported(container *docker.Container) bool { func ignoreContainer(container *docker.Container) bool { for _, kv := range container.Config.Env { kvp := strings.SplitN(kv, "=", 2) - if len(kvp) == 2 && kvp[0] == "LOGSPOUT" && strings.ToLower(kvp[1]) == "ignore" { + if len(kvp) == 2 && kvp[0] == "LOGSPOUT" && strings.EqualFold(kvp[1], "ignore") { return true } } - excludeLabel := getopt("EXCLUDE_LABEL", "") + excludeLabel := cfg.GetEnvDefault("EXCLUDE_LABEL", "") excludeValue := "true" // support EXCLUDE_LABEL having a custom label value excludeLabelArr := strings.Split(excludeLabel, ":") @@ -97,7 +99,7 @@ func ignoreContainer(container *docker.Container) bool { } if value, ok := container.Config.Labels[excludeLabel]; ok { - return len(excludeLabel) > 0 && strings.ToLower(value) == strings.ToLower(excludeValue) + return len(excludeLabel) > 0 && strings.EqualFold(value, strings.ToLower(excludeValue)) } return false } @@ -110,7 +112,7 @@ func ignoreContainerTTY(container *docker.Container) bool { } func getInactivityTimeoutFromEnv() time.Duration { - inactivityTimeout, err := time.ParseDuration(getopt("INACTIVITY_TIMEOUT", "0")) + inactivityTimeout, err := time.ParseDuration(cfg.GetEnvDefault("INACTIVITY_TIMEOUT", "0")) assert(err, "Couldn't parse env var INACTIVITY_TIMEOUT. See https://golang.org/pkg/time/#ParseDuration for valid format.") return inactivityTimeout } @@ -130,7 +132,7 @@ type LogsPump struct { // Name returns the name of the pump func (p *LogsPump) Name() string { - return "pump" + return defaultPumpName } // Setup configures the pump @@ -144,7 +146,7 @@ func (p *LogsPump) rename(event *docker.APIEvents) { p.mu.Lock() defer p.mu.Unlock() container, err := p.client.InspectContainer(event.ID) - assert(err, "pump") + assert(err, defaultPumpName) pump, ok := p.pumps[normalID(event.ID)] if !ok { debug("pump.rename(): ignore: pump not found, state:", container.State.StateString()) @@ -162,10 +164,10 @@ func (p *LogsPump) Run() error { if err != nil { return err } - for _, listing := range containers { + for idx := range containers { p.pumpLogs(&docker.APIEvents{ - ID: normalID(listing.ID), - Status: "start", + ID: normalID(containers[idx].ID), + Status: pumpEventStatusStartName, }, false, inactivityTimeout) } events := make(chan *docker.APIEvents) @@ -176,21 +178,21 @@ func (p *LogsPump) Run() error { for event := range events { debug("pump.Run() event:", normalID(event.ID), event.Status) switch event.Status { - case "start", "restart": + case pumpEventStatusStartName, pumpEventStatusRestartName: go p.pumpLogs(event, backlog(), inactivityTimeout) - case "rename": + case pumpEventStatusRenameName: go p.rename(event) - case "die": + case pumpEventStatusDieName: go p.update(event) } } return errors.New("docker event stream closed") } -func (p *LogsPump) pumpLogs(event *docker.APIEvents, backlog bool, inactivityTimeout time.Duration) { +func (p *LogsPump) pumpLogs(event *docker.APIEvents, backlog bool, inactivityTimeout time.Duration) { //nolint:gocyclo id := normalID(event.ID) container, err := p.client.InspectContainer(id) - assert(err, "pump") + assert(err, defaultPumpName) if ignoreContainerTTY(container) { debug("pump.pumpLogs():", id, "ignored: tty enabled") return @@ -204,7 +206,7 @@ func (p *LogsPump) pumpLogs(event *docker.APIEvents, backlog bool, inactivityTim return } - var tail = getopt("TAIL", "all") + var tail = cfg.GetEnvDefault("TAIL", "all") var sinceTime time.Time if backlog { sinceTime = time.Unix(0, 0) @@ -260,7 +262,7 @@ func (p *LogsPump) pumpLogs(event *docker.APIEvents, backlog bool, inactivityTim if err != nil { _, four04 := err.(*docker.NoSuchContainer) if !four04 { - assert(err, "pump") + assert(err, defaultPumpName) } } else if container.State.Running { continue @@ -327,7 +329,7 @@ func (p *LogsPump) Route(route *Route, logstream chan *Message) { select { case event := <-updates: switch event.Status { - case "start", "restart": + case pumpEventStatusStartName, pumpEventStatusRestartName: if route.MatchContainer( normalID(event.pump.container.ID), normalName(event.pump.container.Name), @@ -336,7 +338,7 @@ func (p *LogsPump) Route(route *Route, logstream chan *Message) { event.pump.add(logstream, route) defer event.pump.remove(logstream) } - case "die": + case pumpEventStatusDieName: if strings.HasPrefix(route.FilterID, event.ID) { // If the route is just about a single container, // we can stop routing when it dies. diff --git a/router/pump_test.go b/router/pump_test.go index e05f9f99..62780c5b 100644 --- a/router/pump_test.go +++ b/router/pump_test.go @@ -43,7 +43,7 @@ func (rt *FakeRoundTripper) Reset() { rt.requests = nil } -func newTestClient(rt *FakeRoundTripper) docker.Client { +func newTestClient(rt http.RoundTripper) docker.Client { endpoint := "http://localhost:4243" client, _ := docker.NewClient(endpoint) client.HTTPClient = &http.Client{Transport: rt} diff --git a/router/routes.go b/router/routes.go index 6b23df6a..8812243b 100644 --- a/router/routes.go +++ b/router/routes.go @@ -1,7 +1,7 @@ package router import ( - "crypto/sha1" + "crypto/sha1" //nolint:gosec "errors" "fmt" "io" @@ -12,6 +12,8 @@ import ( "strings" "sync" "time" + + "github.com/gliderlabs/logspout/cfg" ) // Routes is all the configured routes @@ -38,7 +40,9 @@ func (rm *RouteManager) Load(persistor RouteStore) error { return err } for _, route := range routes { - rm.Add(route) + if err = rm.Add(route); err != nil { + return err + } } rm.persistor = persistor return nil @@ -72,7 +76,7 @@ func (rm *RouteManager) Remove(id string) bool { defer rm.Unlock() route, ok := rm.routes[id] if ok && route.closer != nil { - route.closer <- true + route.closer <- struct{}{} } delete(rm.routes, id) if rm.persistor != nil { @@ -130,15 +134,15 @@ func (rm *RouteManager) Add(route *Route) error { return err } if route.ID == "" { - h := sha1.New() + h := sha1.New() //nolint:gosec io.WriteString(h, strconv.Itoa(int(time.Now().UnixNano()))) route.ID = fmt.Sprintf("%x", h.Sum(nil))[:12] } - route.closer = make(chan bool) + route.closer = make(chan struct{}) route.adapter = adapter //Stop any existing route with this ID: if rm.routes[route.ID] != nil { - rm.routes[route.ID].closer <- true + rm.routes[route.ID].closer <- struct{}{} } rm.routes[route.ID] = route @@ -220,7 +224,7 @@ func (rm *RouteManager) Setup() error { } } - persistPath := getopt("ROUTESPATH", "/mnt/routes") + persistPath := cfg.GetEnvDefault("ROUTESPATH", "/mnt/routes") if _, err := os.Stat(persistPath); err == nil { return rm.Load(RouteFileStore(persistPath)) } diff --git a/router/types.go b/router/types.go index b17ec4c1..084c67c4 100644 --- a/router/types.go +++ b/router/types.go @@ -8,11 +8,11 @@ import ( "strings" "time" - "github.com/fsouza/go-dockerclient" + docker "github.com/fsouza/go-dockerclient" ) -// HttpHandler is an extension type for adding HTTP endpoints -type HttpHandler func() http.Handler +// HTTPHandler is an extension type for adding HTTP endpoints +type HTTPHandler func() http.Handler // AdapterFactory is an extension type for adding new log adapters type AdapterFactory func(route *Route) (LogAdapter, error) @@ -67,9 +67,9 @@ type Route struct { Address string `json:"address"` Options map[string]string `json:"options,omitempty"` adapter LogAdapter - closed bool - closer chan bool - closerRcv <-chan bool // used instead of closer when set + closed bool + closer chan struct{} + closerRcv <-chan struct{} // used instead of closer when set } // AdapterType returns a route's adapter type string @@ -87,7 +87,7 @@ func (r *Route) AdapterTransport(dfault string) string { } // Closer returns a route's closerRcv -func (r *Route) Closer() <-chan bool { +func (r *Route) Closer() <-chan struct{} { if r.closerRcv != nil { return r.closerRcv } @@ -95,13 +95,13 @@ func (r *Route) Closer() <-chan bool { } // OverrideCloser sets a Route.closer to closer -func (r *Route) OverrideCloser(closer <-chan bool) { +func (r *Route) OverrideCloser(closer <-chan struct{}) { r.closerRcv = closer } // Close sends true to a Route.closer func (r *Route) Close() { - r.closer <- true + r.closer <- struct{}{} } func (r *Route) matchAll() bool { diff --git a/routesapi/routesapi.go b/routesapi/routesapi.go index 9b679662..0928c8d8 100644 --- a/routesapi/routesapi.go +++ b/routesapi/routesapi.go @@ -6,12 +6,13 @@ import ( "log" "net/http" - "github.com/gliderlabs/logspout/router" "github.com/gorilla/mux" + + "github.com/gliderlabs/logspout/router" ) func init() { - router.HttpHandlers.Register(RoutesAPI, "routes") + router.HTTPHandlers.Register(RoutesAPI, "routes") } // RoutesAPI returns a handler for the routes API @@ -40,7 +41,6 @@ func RoutesAPI() http.Handler { w.Header().Add("Content-Type", "application/json") rts, _ := routes.GetAll() w.Write(append(marshal(rts), '\n')) - return }).Methods("GET") r.HandleFunc("/routes", func(w http.ResponseWriter, req *http.Request) { diff --git a/transports/tls/tls.go b/transports/tls/tls.go index 5c6697c5..297dfdc6 100644 --- a/transports/tls/tls.go +++ b/transports/tls/tls.go @@ -23,6 +23,7 @@ const ( envClientCert = "LOGSPOUT_TLS_CLIENT_CERT" envClientKey = "LOGSPOUT_TLS_CLIENT_KEY" envTLSHardening = "LOGSPOUT_TLS_HARDENING" + trueString = "true" ) var ( @@ -97,7 +98,7 @@ func createTLSConfig() (tlsConfig *tls.Config, err error) { // use stronger TLS settings if enabled // TODO: perhaps this should be default setting - if os.Getenv(envTLSHardening) == "true" { + if os.Getenv(envTLSHardening) == trueString { tlsConfig.InsecureSkipVerify = false tlsConfig.MinVersion = hardenedMinVersion tlsConfig.CipherSuites = hardenedCiphers @@ -112,7 +113,7 @@ func createTLSConfig() (tlsConfig *tls.Config, err error) { // if we cannot, then it's fatal. // NOTE that we ONLY fail if SystemCertPool returns an error, // not if our system trust store is empty or doesn't exist! - if os.Getenv(envDisableSystemRoots) != "true" { + if os.Getenv(envDisableSystemRoots) != trueString { tlsConfig.RootCAs, err = x509.SystemCertPool() if err != nil { return @@ -157,5 +158,5 @@ func createTLSConfig() (tlsConfig *tls.Config, err error) { // We will make this optional; the client cert pem file can contain more than one certificate tlsConfig.Certificates = []tls.Certificate{clientCert} } - return + return //nolint:nakedret } diff --git a/transports/tls/tls_test.go b/transports/tls/tls_test.go index 275faf98..08da2c2f 100644 --- a/transports/tls/tls_test.go +++ b/transports/tls/tls_test.go @@ -27,7 +27,7 @@ func createTestTLSConfig(t *testing.T) *tls.Config { return testTLSConfig } -// TestEmptyTrustStore should test the behaviour of having +// TestEmptyTrustStore should test the behavior of having // an empty TLS CA trust store. func TestEmptyTrustStore(t *testing.T) { os.Setenv(envDisableSystemRoots, "true") @@ -41,7 +41,7 @@ func TestEmptyTrustStore(t *testing.T) { } -// TestSingleCustomCA should test the behaviour of loading +// TestSingleCustomCA should test the behavior of loading // a single custom CA certificate in to the trust store. func TestSingleCustomCA(t *testing.T) { os.Setenv(envDisableSystemRoots, "true") @@ -55,7 +55,7 @@ func TestSingleCustomCA(t *testing.T) { } -// TestMultipleCustomCAs should test the behaviour of loading +// TestMultipleCustomCAs should test the behavior of loading // multiple custom CA certificates in to the trust store. func TestMultipleCustomCAs(t *testing.T) { os.Setenv(envDisableSystemRoots, "true") @@ -73,7 +73,7 @@ func TestMultipleCustomCAs(t *testing.T) { // TestSystemRootCAs should test that by default we load the system trust store func TestSystemRootCAs(t *testing.T) { - // default behaviour is none of these environment variables are set + // default behavior is none of these environment variables are set os.Unsetenv(envDisableSystemRoots) os.Unsetenv(envCaCerts) testTLSConfig := createTestTLSConfig(t) @@ -101,7 +101,7 @@ func TestSystemRootCAsAndCustomCAs(t *testing.T) { } } -// TestLoadingClientCertAndKey: should test the behaviour of loading +// TestLoadingClientCertAndKey: should test the behavior of loading // a pem encoded client x509 certificate and private key func TestLoadingClientCertAndKey(t *testing.T) { os.Unsetenv(envDisableSystemRoots) @@ -115,7 +115,7 @@ func TestLoadingClientCertAndKey(t *testing.T) { } } -// TestTLSHardening should test the behaviour of enabling TLS hardening +// TestTLSHardening should test the behavior of enabling TLS hardening func TestTLSHardening(t *testing.T) { os.Unsetenv(envDisableSystemRoots) os.Unsetenv(envCaCerts) From dc4a5875f687055872b04e0a6403007f0af1a1bf Mon Sep 17 00:00:00 2001 From: StudioEtrange Date: Sun, 2 Feb 2020 14:57:49 +0100 Subject: [PATCH 08/11] add link to logspout-fluentd detail of how logspout-fluentd was designed : https://medium.com/magine-engineering/using-fluentd-log-driver-but-still-want-to-debug-your-container-logs-locally-try-logspout-c5b9f6ca9d07 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 27d8b21a..eb1a8341 100644 --- a/README.md +++ b/README.md @@ -331,6 +331,8 @@ The standard distribution of logspout comes with all modules defined in this rep * [logspout-logstash](https://github.com/looplab/logspout-logstash) * [logspout-redis-logstash](https://github.com/rtoma/logspout-redis-logstash) * [logspout-gelf](https://github.com/micahhausler/logspout-gelf) for Graylog + * [logspout-fluentd](https://github.com/dsouzajude/logspout-fluentd) for fluentd - instead of using fluentd log driver + ### Loggly support From 952b41981a78714deafa0ece8a861290b8f6ec6f Mon Sep 17 00:00:00 2001 From: StudioEtrange Date: Sun, 2 Feb 2020 15:44:12 +0100 Subject: [PATCH 09/11] logspout-fluentd can forward to fluent-bit too --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index eb1a8341..88ea9bc6 100644 --- a/README.md +++ b/README.md @@ -331,7 +331,7 @@ The standard distribution of logspout comes with all modules defined in this rep * [logspout-logstash](https://github.com/looplab/logspout-logstash) * [logspout-redis-logstash](https://github.com/rtoma/logspout-redis-logstash) * [logspout-gelf](https://github.com/micahhausler/logspout-gelf) for Graylog - * [logspout-fluentd](https://github.com/dsouzajude/logspout-fluentd) for fluentd - instead of using fluentd log driver + * [logspout-fluentd](https://github.com/dsouzajude/logspout-fluentd) for fluentd or fluent-bit - instead of using fluentd log driver ### Loggly support From 35ece5e6a5ded6749c2b14c0f835cf13f7d15ae6 Mon Sep 17 00:00:00 2001 From: Michael Hobbs Date: Fri, 3 Apr 2020 11:37:11 -0700 Subject: [PATCH 10/11] release 3.2.7 --- CHANGELOG.md | 15 ++++++++++++++- VERSION | 2 +- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a2c3003f..4a5c894d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,18 @@ All notable changes to this project will be documented in this file. ### Changed +## [v3.2.7] - 2020-04-03 +### Fixed +@CodeLingoBot @gbolo Fix function comments based on best practices from Effective Go + +### Changed +@michaelshobbs update alpine to 3.10/go 1.12.12-r0 and fix linting +@whoisteri DOC Document accessible data in RAW_FORMAT template +@tiagorlampert DOC typos +@michaelshobbs DOC CHANGLELOG formatting +@tomlankhorst DOC Suggest to disable userns-remap for logspout +@StudioEtrange DOC add link to logspout-fluentd + ## [v3.2.6] - 2018-10-04 ### Fixed - @jdgiotta Spelling corrections and fixed stack compose formatting in example @@ -198,7 +210,8 @@ All notable changes to this project will be documented in this file. - Base container is now Alpine - Moved to gliderlabs organization -[unreleased]: https://github.com/gliderlabs/logspout/compare/v3.2.6...HEAD +[unreleased]: https://github.com/gliderlabs/logspout/compare/v3.2.7...HEAD +[v3.2.7]: https://github.com/gliderlabs/logspout/compare/v3.2.6...v3.2.7 [v3.2.6]: https://github.com/gliderlabs/logspout/compare/v3.2.5...v3.2.6 [v3.2.5]: https://github.com/gliderlabs/logspout/compare/v3.2.4...v3.2.5 [v3.2.4]: https://github.com/gliderlabs/logspout/compare/v3.2.3...v3.2.4 diff --git a/VERSION b/VERSION index ddd44db7..4671105d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v3.2.6 +v3.2.7 From 1d6cc28615227ea39c9ea252c70bd96f8cdb786b Mon Sep 17 00:00:00 2001 From: Michael Hobbs Date: Fri, 3 Apr 2020 11:39:37 -0700 Subject: [PATCH 11/11] fix changelog typo --- CHANGELOG.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a5c894d..eb1e1557 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,15 +12,15 @@ All notable changes to this project will be documented in this file. ## [v3.2.7] - 2020-04-03 ### Fixed -@CodeLingoBot @gbolo Fix function comments based on best practices from Effective Go +- @CodeLingoBot @gbolo Fix function comments based on best practices from Effective Go ### Changed -@michaelshobbs update alpine to 3.10/go 1.12.12-r0 and fix linting -@whoisteri DOC Document accessible data in RAW_FORMAT template -@tiagorlampert DOC typos -@michaelshobbs DOC CHANGLELOG formatting -@tomlankhorst DOC Suggest to disable userns-remap for logspout -@StudioEtrange DOC add link to logspout-fluentd +- @michaelshobbs update alpine to 3.10/go 1.12.12-r0 and fix linting +- @whoisteri DOC Document accessible data in RAW_FORMAT template +- @tiagorlampert DOC typos +- @michaelshobbs DOC CHANGLELOG formatting +- @tomlankhorst DOC Suggest to disable userns-remap for logspout +- @StudioEtrange DOC add link to logspout-fluentd ## [v3.2.6] - 2018-10-04 ### Fixed