[Observability] Open-source systems monitoring and alerting

[tomsmith8]

Ticket: Improve Monitoring for Jarvis Endpoint Timeouts

Overview

We are experiencing timeouts on certain Jarvis endpoints. The root cause has not been identified yet. To address this issue, we propose enhancing our monitoring capabilities using additional OSS tools.

---

Proposed Actions

1. Investigate Prometheus and Grafana

   • Prometheus: A powerful open-source monitoring solution.
   • Grafana: A visualization and dashboarding tool.

2. Integrate Node Exporter for Hardware Monitoring

   • Add Prometheus Node Exporter to swarm instances.
   • This will enable detailed hardware monitoring and provide metrics for Prometheus to collect.

3. Consider Enhancing Superadmin

   • Evaluate the possibility of adding monitoring functionality directly into the Superadmin tool.

---

Next Steps

• Install and configure Prometheus and Grafana on swarm instances.
• Integrate Node Exporter for hardware metrics.
• Set up dashboards in Grafana for endpoint performance and hardware monitoring.
• Determine feasibility and effort required for extending Superadmin with monitoring features.

---

Hardware Monitoring (Details)

• Node Exporter Setup: Install on all swarm instances.
• Metrics: CPU usage, memory, disk I/O, and network performance.
• Visualization: Use Grafana to create dashboards for real-time and historical hardware insights.

[tomsmith8]

@gonzaloaune has suggested some open source solutions to better observe what is going on in swarm. Especially as we keep seeing timeout issues for whinx on Jarvis endpoints but also for general monitoring and debugging.

Tagging for thoughts: @Evanfeenstra @kevkevinpal @tobi-bams

[tomsmith8]

@tobi-bams next project I think this would help