apt_info.py, security updates, rules and dashboards

[copolycube]

Hello,

with the script apt_info.py replacing apt.sh I'm trying to create the following :

• grafana dashboard
• alertmanager PromQL rules
  that would allow me to list the
• total packages to upgrade "regular" (0 for each host if none)
• number of "security" packages to upgrade (0 for each host if none), (and send alerts if some are to be upgraded)

I found a suggested PromQL by @dswarbrick (here)
but with sum by(job) (apt_upgrades_pending{origin=~".*Security.*"}) or vector(0) the result is {} 0 (only once, as a global result) when there are no match / no security upgrades on any instance:

• Is it possible to still have a vector with 0 for each original value of apt_upgrades_pending instead without modifying apt_info.py to export it explicitly ?

• side question : are there a set of alertmanager rules and grafana dashboards available somewhere to be used along the textfile-collector scripts in this repo ? Or is it possible to generate something just from the HELP and TYPE messages ?

Thanks in advance for your help !

[dswarbrick]

The crux of the issue is that Prometheus only stores "non-null" samples. So if the exporter or textfile collector doesn't expose a value (even a zero value) for a particular label set, then nothing gets stored by Prometheus.

You could add multiple queries to your dashboard, with explicit origin label matchers, combining that with or vector(0), and explicitly specify the legend label in Grafana. However this approach is essentially incompatible with the regex origin=~".*Security.*" matcher, where the exact origin labels are not known in advance.

[copolycube]

Ok, thanks for the explanation. I was looking at the PromQL functions and hadn't find anything relevant indeed :-(

Ideally I'm hoping to make the dashboards and alerts only rely on labels and variables, avoiding anything too specific so that it can be shared and replicate easily.

So far I've defined my grafana dashboard like this (if it can help anyone, and any suggestion welcome!) :

grafana dashboard JSON for apt_info (Click me)

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "type": "datasource",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__elements": {},
  "__requires": [
    {
      "type": "grafana",
      "id": "grafana",
      "name": "Grafana",
      "version": "9.3.1"
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    },
    {
      "type": "panel",
      "id": "stat",
      "name": "Stat",
      "version": ""
    },
    {
      "type": "panel",
      "id": "timeseries",
      "name": "Time series",
      "version": ""
    }
  ],
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "mappings": [
            {
              "options": {
                "0": {
                  "color": "dark-green",
                  "index": 0,
                  "text": "ok"
                },
                "1": {
                  "color": "dark-orange",
                  "index": 1,
                  "text": "reboot"
                }
              },
              "type": "value"
            }
          ],
          "noValue": "N/A",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 1
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 8,
      "options": {
        "colorMode": "background",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "text": {
          "titleSize": 15
        },
        "textMode": "value_and_name"
      },
      "pluginVersion": "9.3.1",
      "targets": [
        {
          "$$hashKey": "object:24",
          "aggregation": "Last",
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "decimals": 2,
          "displayAliasType": "Warning / Critical",
          "displayType": "Regular",
          "displayValueWithAlias": "Never",
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_reboot_required",
          "format": "time_series",
          "instant": true,
          "legendFormat": "{{instance}}",
          "range": false,
          "refId": "A",
          "units": "none",
          "valueHandler": "Number Threshold"
        }
      ],
      "title": "Reboot required ?",
      "transformations": [
        {
          "id": "renameByRegex",
          "options": {
            "regex": "nodeexporter.(.*):80",
            "renamePattern": "$1"
          }
        }
      ],
      "type": "stat"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 9,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "#EAB839",
                "value": 5
              },
              {
                "color": "red",
                "value": 20
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 4
      },
      "id": 6,
      "options": {
        "legend": {
          "calcs": [
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true,
          "sortBy": "Last",
          "sortDesc": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "asc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "builder",
          "expr": "sum by(instance) (apt_upgrades_pending)",
          "legendFormat": "{{instance}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "apt_upgrades_pending",
      "transformations": [
        {
          "id": "renameByRegex",
          "options": {
            "regex": "nodeexporter.(.*):80",
            "renamePattern": "$1"
          }
        }
      ],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 5
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 24,
        "x": 0,
        "y": 13
      },
      "id": 9,
      "options": {
        "legend": {
          "calcs": [
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true,
          "sortBy": "Last",
          "sortDesc": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "code",
          "expr": "sum by(instance) (apt_upgrades_pending{origin=~\".*security.*\"}) or vector(0)",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "security apt_upgrades_pending",
      "transformations": [
        {
          "id": "renameByRegex",
          "options": {
            "regex": "nodeexporter.(.*):80",
            "renamePattern": "$1"
          }
        }
      ],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 22
      },
      "id": 4,
      "options": {
        "legend": {
          "calcs": [
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "builder",
          "expr": "sum by(instance) (apt_upgrades_held)",
          "legendFormat": "{{instance}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "apt_upgrades_held",
      "transformations": [
        {
          "id": "renameByRegex",
          "options": {
            "regex": "nodeexporter.(.*):80",
            "renamePattern": "$1"
          }
        }
      ],
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${DS_PROMETHEUS}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 10,
        "w": 24,
        "x": 0,
        "y": 30
      },
      "id": 2,
      "options": {
        "legend": {
          "calcs": [
            "last"
          ],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${DS_PROMETHEUS}"
          },
          "editorMode": "builder",
          "expr": "sum by(instance) (apt_autoremove_pending)",
          "legendFormat": "{{instance}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "apt_autoremove_pending",
      "transformations": [
        {
          "id": "renameByRegex",
          "options": {
            "regex": "nodeexporter.(.*):80",
            "renamePattern": "$1"
          }
        }
      ],
      "type": "timeseries"
    }
  ],
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {},
        "datasource": {
          "type": "prometheus",
          "uid": "${DS_PROMETHEUS}"
        },
        "definition": "label_values(node_reboot_required, instance)",
        "hide": 0,
        "includeAll": true,
        "multi": true,
        "name": "instance",
        "options": [],
        "query": {
          "query": "label_values(node_reboot_required, instance)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "/nodeexporter.(.*):80/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "APT-ugrades",
  "uid": "gTSdTxtVk",
  "version": 21,
  "weekStart": ""
}

Edit : I shared my current dashboard here, but I'm not sure where would be the best place to do so to have it linked to the apt_info.py script

[dswarbrick]

Grafana has a community site for sharing dashboards at https://grafana.com/grafana/dashboards/

[copolycube]

@dswarbrick you need a paid subscription to be able to publish there

[dswarbrick]

@copolycube Yes, you need a Grafana account - but AFAIK, the "free forever cloud" account type should be sufficient.