Add more prometheus metrics

[bonomat]

• maker-coordinator status: is the coordinator online: feat(maker): Monitor and expose status of services connected #1180
• orderbook status: is the orderbook online: feat(maker): Monitor and expose status of services connected #1180

outside the scope of this ticket, but related:

• Add maker prometheus metrics for position #1222

[holzeis]

Do we really need those business metrics in prometheus?

position metrics (PnL, size, direction, duration)

Isn't it sufficient to have this data in the database?

[bonomat]

Yes please, it's helpful to have an overview (e.g. for warnings) at any point of time.

[holzeis]

Well you can get an overview at any point in time also from the database, but I guess you want some alerting functionality?

[bonomat]

Yes, I want to have the alerting functionality to warn us if things are in the red zone.

[klochowicz]

is having a Postgresql data source in grafana still something we're considering?
it seems that having a periodic query on that database and including it as time series would be a better way to gather position data in the long run:
https://grafana.com/docs/grafana/latest/datasources/postgres/

(in the meantime, I'm doing the position data as prometheus metrics).

[bonomat]

is having a Postgresql data source in grafana still something we're considering?

Potentially, our data is now a better structured than it was with itchysats.

it seems that having a periodic query on that database and including it as time series would be a better way to gather position data in the long run:

I did some research a while ago and to my understanding grafana's sql connection expects the metrics in a timestamp format in the db to query them, i.e. a table somthing like this

timestamp	value
2023-08-28 21:05	1
2023-08-28 21:10	2
2023-08-28 21:15	3

My understanding is that you cannot query something periodically and use the timestamp from when you queried it. Just think about it, Grafana would need to store this data again in a database and add a timestamp to it.
The data in the db needs to have the timestamp added to make suitable graphs

Most of our tables have this structure already so we could reconsider it but Grafana is not meant to be used for business metrics.

There are other tools which could do this job better: I looked into a few and metabase, or redash was most promising which can be run on our machine.

But I guess that's something for a different day :D

[klochowicz]

I've extracted #1222 outside the scope of this ticket as my PR didn't address that part of the metrics.
cc: @holzeis who was interested in taking over some of the metrics.