Detecting server crashes

[faragher]

So, for better sysadmins than myself: what is the "best" way to detect and report a NomadNet instance going down. Is there a way to check from NN itself, or is this a logwatch type of scenario?

I know we're rich with options for reporting these things: setting up an email client for your local user and sending to yourself, setting up an MX and sending it to your base email, SMS (which I think my provider no longer supports M2M), LXMF, etc. but is there a simple way to simply get a script that says "Okay, this has died, I'll send you a message?"

[markqvist]

Maybe you could wrap launching nomadnet in a bash script that simply sends some kind of event as soon as nomadnet terminates? Pretty rudimentary, but would work well, unless the entire server goes down. In that case you'd need something external monitoring it anyway.

[faragher]

That's simple and reliable. I like it.

[The-One-Law]

More credible than any string of dependencies.

[Erethon]

As you said, there are a ton of options available and what works best for you depends on what you already have at your disposable.

For dead simple things, I would suggest monit, even though it's somewhat dated, it still works as expected and it's pretty easy to setup. Example configs to see its simplicity https://mmonit.com/wiki/Monit/ConfigurationExamples.

Personally, I have node_exporter running in all of my hosts and I get metrics and alerts via Prometheus. Since you can run NN via a systemd unit, I get automatic metrics for it and alerts.

[faragher]

I do like Prometheus and Grafana.

[kc1awv]

I'm using netdata (not just for rnsd/nomadnet) and I believe I have proper process monitoring for rnsd and related services now...

[markqvist]

That's looking very nice