Timed out

[mweel1]

I had to bring down the log server, and my go app stopped responding.

Any idea why this would happen? Should my go log calls be wrapped in a go routine?

Thank you

[nblumhardt]

Hi @mweel1!

I am not up to speed on the internals of this package, so apologies if this isn't useful info, but just in case you're stuck, Seq also accepts events in the GELF format via its Seq.Input.Gelf plug-in, and I think GELF transports are just about always non-blocking.

There are a couple of GELF hooks/formatters for Logrus out there.

HTH!

[mweel1]

@nblumhardt Hi,

Thanks for getting back to me, yeah it looked like it was blocking/sync on the http requests.

I implemented this, if you ever need it.

It basically uses a message queue and worker processes, and drops messages if the queue size reaches 5000.

We saw a huge performance boost in doing this.

import (
	"bytes"
	"fmt"
	"io/ioutil"
	"net/http"
	"time"

	"github.com/LyricTian/queue"
	"github.com/sirupsen/logrus"
)

// SeqHook sends logs to Seq via HTTP.
type SeqHook struct {
	endpoint string
	apiKey   string
	levels   []logrus.Level
	q        *queue.Queue
}

// NewSeqHook creates a Seq hook for logrus which can send log events to the
// host specified, for example:
//     logruseq.NewSeqHook("http://localhost:5341")
// Optionally, the hook can be used with an API key, for example:
//     logruseq.NewSeqHook("http://localhost:5341",
//         logruseq.OptionAPIKey("N1ncujiT5pYGD6m4CF0"))
// Optionally, which levels to log can be specified:
//     logruseq.NewSeqHook("http://localhost:5341",
//         logruseq.OptionLevels([]logrus.Level{
//             logrus.WarnLevel,
//             logrus.ErrorLevel,
//             logrus.FatalLevel,
//             logrus.PanicLevel,
//         }))
func NewSeqHook(host string, queue *queue.Queue, opts ...func(*SeqHookOptions)) *SeqHook {
	sho := &SeqHookOptions{
		levels: []logrus.Level{
			logrus.TraceLevel,
			logrus.DebugLevel,
			logrus.InfoLevel,
			logrus.WarnLevel,
			logrus.ErrorLevel,
			logrus.FatalLevel,
			logrus.PanicLevel,
		},
	}

	for _, opt := range opts {
		opt(sho)
	}

	endpoint := fmt.Sprintf("%v/api/events/raw", host)

	
	

	return &SeqHook{
		endpoint: endpoint,
		apiKey:   sho.apiKey,
		levels:   sho.levels,
		q: queue,
	}
}

func (h *SeqHook) copyEntry(e *logrus.Entry) *logrus.Entry {
	entry := logrus.NewEntry(e.Logger)
	entry.Data = make(logrus.Fields)
	entry.Time = e.Time
	entry.Level = e.Level
	entry.Message = e.Message
	for k, v := range e.Data {
		entry.Data[k] = v
	}
	return entry
}

// Fire sends a log entry to Seq.
func (hook *SeqHook) Fire(entry *logrus.Entry) error {	
	
	entry = hook.copyEntry(entry)

	// if our queue has 5000 items were overloading, lets start dropping logs		
	if (hook.q.GetJobCount() > 5000) {
		return nil;
	}

	hook.q.Push(queue.NewJob(entry, func(v interface{}) {
		hook.postSeq(v.(*logrus.Entry))
	}))

	return nil
}

func (h *SeqHook) postSeq(entry *logrus.Entry) {

	formatter := logrus.JSONFormatter{
		TimestampFormat: time.RFC3339Nano,
		FieldMap: logrus.FieldMap{
			logrus.FieldKeyMsg:   "@mt",
			logrus.FieldKeyLevel: "@l",
			logrus.FieldKeyTime:  "@t",
		},
	}

	data, _ := formatter.Format(entry)

	http.DefaultClient.Timeout = 10 * time.Second

	req, err := http.NewRequest("POST", h.endpoint, bytes.NewReader(data))

	if (err != nil) {
		fmt.Println("Error sending log to seq: ", err)		
		return;
	}

	req.Header.Set("Content-Type", "application/vnd.serilog.clef")

	if h.apiKey != "" {
		req.Header.Add("X-Seq-ApiKey", h.apiKey)
	}

	resp, err := http.DefaultClient.Do(req)
	
	if (err != nil) {
		fmt.Println("Error sending log to seq: ", err)

		return;
	}

	defer resp.Body.Close()

	if resp.StatusCode != http.StatusCreated {
		data, _ = ioutil.ReadAll(resp.Body)	
		fmt.Errorf("error creating seq event: %v", string(data))
	}

}

// Levels returns the levels for which Fire will be called.
func (hook *SeqHook) Levels() []logrus.Level {
	return hook.levels
}

// SeqHookOptions collects non-default Seq hook options.
type SeqHookOptions struct {
	apiKey string
	levels []logrus.Level

}

// OptionAPIKey sets the Seq API key option.
func OptionAPIKey(apiKey string) func(opts *SeqHookOptions) {
	return func(opts *SeqHookOptions) {
		opts.apiKey = apiKey
	}
}

// OptionLevels sets the levels for which Fire will be called.
func OptionLevels(levels []logrus.Level) func(opts *SeqHookOptions) {
	return func(opts *SeqHookOptions) {
		opts.levels = levels
	}
}```

[nblumhardt]

Awesome!

Unless you plan to use message templates with named holes in your Logrus messages, you might get slightly better handling on the Seq side by using the @m rather than @mt property for the message.

Thanks for sharing your solution! 😎

[alxyng]

Hey @mweel1 thanks for raising this. It looks like we need an option for a timeout on the http client, it's probably a good idea to use a dedicated client instead of the built in one too.