getting stats from GPU issue

[Mrdindon]

Description

I run Proxmox on a AMD Ryzen 7 pro 4750G (iGpu).
I've installed rocm-smi package that gets installed at /usr/bin/rocm-smi
not sure if it was needed or not but I did a symlink to /usr/local/bin/rocm-smi
(ln -s /usr/bin/rocm-smi /usr/local/bin/rocm-smi)

Now when I start the agent I see :

DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"

OS / Architecture

Proxmox 8.3.2 / Debian 12

Beszel version

0.9.1

Installation method

Binary

Configuration

[Unit]
Description=Beszel Agent Service
Wants=network-online.target
After=network-online.target

[Service]
Environment="PORT=45876"
Environment="KEY=..."
Environment="EXTRA_FILESYSTEMS=/dev/dm-2"
Environment="GPU=true"
Environment="LOG_LEVEL=debug"
ExecStart=/opt/beszel-agent/beszel-agent
User=beszel
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Hub Logs

No response

Agent Logs

DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"

[henrygd]

Interesting, /usr/bin/rocm-smi should be fine.

Can you please tell me what the following commands output?

rocm-smi --showtemp --showuse --showpower --showmeminfo vram --json

su -s /bin/sh beszel -c "rocm-smi --showtemp --showuse --showpower --showmeminfo vram --json"

[Mrdindon]

root@pve:/opt/beszel# rocm-smi --showtemp --showuse --showpower --showmeminfo vram --json
{"card0": {"Temperature (Sensor edge) (C)": "33.0", "Temperature (Sensor junction) (C)": "N/A", "Temperature (Sensor memory) (C)": "N/A", "Temperature (Sensor HBM 0) (C)": "N/A", "Temperature (Sensor HBM 1) (C)": "N/A", "Temperature (Sensor HBM 2) (C)": "N/A", "Temperature (Sensor HBM 3) (C)": "N/A", "Average Graphics Package Power (W)": "N/A", "GPU use (%)": "0", "GFX Activity": "N/A", "VRAM Total Memory (B)": "536870912", "VRAM Total Used Memory (B)": "17502208"}}
root@pve:/opt/beszel#
root@pve:/opt/beszel#
root@pve:/opt/beszel# su -s /bin/sh beszel -c "rocm-smi --showtemp --showuse --showpower --showmeminfo vram --json"
{"card0": {"Temperature (Sensor edge) (C)": "33.0", "Temperature (Sensor junction) (C)": "N/A", "Temperature (Sensor memory) (C)": "N/A", "Temperature (Sensor HBM 0) (C)": "N/A", "Temperature (Sensor HBM 1) (C)": "N/A", "Temperature (Sensor HBM 2) (C)": "N/A", "Temperature (Sensor HBM 3) (C)": "N/A", "Average Graphics Package Power (W)": "N/A", "GPU use (%)": "0", "GFX Activity": "N/A", "VRAM Total Memory (B)": "536870912", "VRAM Total Used Memory (B)": "17502208"}}
root@pve:/opt/beszel#

Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected root device name=dm-1
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=enp8s0 sent=747967977497 recv=2299019839025
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=vmbr0 sent=3567042772 recv=1455203669
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=tap1004i0 sent=718326651 recv=248373
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=tap1005i0 sent=13389578019 recv=5736384949
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwpr1005p0 sent=13155500433 recv=5736384089
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwln1005i0 sent=5736384089 recv=13155500433
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwpr104p0 sent=172451186 recv=5325328
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwln104i0 sent=5325328 recv=172451186
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=tap1001i0 sent=2071816514923 recv=213822897425
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=tap116i0 sent=4637382686 recv=2087790388
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwpr116p0 sent=4637157371 recv=2087790388
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Detected network interface name=fwln116i0 sent=2087790388 recv=4637157371
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG GPU err="no GPU found - install nvidia-smi or rocm-smi"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG Getting stats
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG Temperature sensors="[{"sensorKey":"nvme_composite","temperature":29.85,"sensorHigh":79.85,"sensorCritical":84.85} {"sensorKey":"k10temp_tctl","temperature":44.625,"sensorHigh":0,"sensorCritical":0} {"sensorKey":"amdgpu_edge","temperature":33,"sensorHigh":0,"sensorCritical":0}]"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG sysinfo data="{Hostname:pve KernelVersion:6.8.12-5-pve Cores:8 Threads:16 CpuModel:AMD Ryzen 7 PRO 4750G with Radeon Graphics Uptime:1193241 Cpu:13.85 MemPct:40.51 DiskPct:6.4 Bandwidth:4.63 AgentVersion:0.9.1 Podman:false}"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG System stats data="{Stats:{Cpu:13.85 MaxCpu:0 Mem:62.1 MemUsed:25.16 MemPct:40.51 MemBuffCache:2.19 MemZfsArc:31.05 Swap:8 SwapUsed:7.86 DiskTotal:907.95 DiskUsed:55.71 DiskPct:6.4 DiskReadPs:4.57 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:2.33 NetworkRecv:2.3 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[amdgpu_edge:33 k10temp_tctl:44.63 nvme_composite:29.85] ExtraFs:map[] GPUData:map[]} Info:{Hostname:pve KernelVersion:6.8.12-5-pve Cores:8 Threads:16 CpuModel:AMD Ryzen 7 PRO 4750G with Radeon Graphics Uptime:1193241 Cpu:13.85 MemPct:40.51 DiskPct:6.4 Bandwidth:4.63 AgentVersion:0.9.1 Podman:false} Containers:[]}"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG Error getting docker stats err="Get "http://localhost/containers/json\": dial unix /var/run/docker.sock: connect: no such file or directory"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG Extra filesystems data=map[dm-2:0xc0002167e0]
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 DEBUG Stats data="{Stats:{Cpu:13.85 MaxCpu:0 Mem:62.1 MemUsed:25.16 MemPct:40.51 MemBuffCache:2.19 MemZfsArc:31.05 Swap:8 SwapUsed:7.86 DiskTotal:907.95 DiskUsed:55.71 DiskPct:6.4 DiskReadPs:4.57 DiskWritePs:0 MaxDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:2.33 NetworkRecv:2.3 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[amdgpu_edge:33 k10temp_tctl:44.63 nvme_composite:29.85] ExtraFs:map[dm-2:0xc0002167e0] GPUData:map[]} Info:{Hostname:pve KernelVersion:6.8.12-5-pve Cores:8 Threads:16 CpuModel:AMD Ryzen 7 PRO 4750G with Radeon Graphics Uptime:1193241 Cpu:13.85 MemPct:40.51 DiskPct:6.4 Bandwidth:4.63 AgentVersion:0.9.1 Podman:false} Containers:[]}"
Jan 30 10:14:06 pve beszel-agent[82277]: 2025/01/30 10:14:06 INFO Starting SSH server address=:45876
Jan 30 10:15:05 pve beszel-agent[82277]: 2025/01/30 10:15:05 DEBUG Getting stats

[henrygd]

Thanks, this is strange because it should show 'no gpu found' only if there's an error running rocm-smi.

Looks like it's giving you N/A for power, so my best guess is that it's returning a non zero exit code due to an issue with the power package.

Can you give me your rocm-smi version?

su -s /bin/sh beszel -c "rocm-smi -V"

And also what the exit code is?

# runs rocm-smi
su -s /bin/sh beszel -c "rocm-smi --showtemp --showuse --showpower --showproductname --showmeminfo vram --json"

# checks exit code
echo "exit code: $?"

I'll try to add better debug logs for this in the next release.

[Mrdindon]

Here you go :

root@pve:/opt/beszel# su -s /bin/sh beszel -c "rocm-smi -v"


======================= ROCm System Management Interface =======================
==================================== VBIOS =====================================
GPU[0]		: VBIOS version: 113-RENOIR-035
================================================================================
============================= End of ROCm SMI Log ==============================
root@pve:/opt/beszel#
root@pve:/opt/beszel# su -s /bin/sh beszel -c "rocm-smi --showtemp --showuse --showpower --showproductname --showmeminfo vram --json"
{"card0": {"Temperature (Sensor edge) (C)": "33.0", "Temperature (Sensor junction) (C)": "N/A", "Temperature (Sensor memory) (C)": "N/A", "Temperature (Sensor HBM 0) (C)": "N/A", "Temperature (Sensor HBM 1) (C)": "N/A", "Temperature (Sensor HBM 2) (C)": "N/A", "Temperature (Sensor HBM 3) (C)": "N/A", "Average Graphics Package Power (W)": "N/A", "GPU use (%)": "0", "GFX Activity": "N/A", "VRAM Total Memory (B)": "536870912", "VRAM Total Used Memory (B)": "17502208", "Card series": "Renoir", "Card model": "0x87e1", "Card vendor": "Advanced Micro Devices, Inc. [AMD/ATI]", "Card SKU": "RENOIR"}}
root@pve:/opt/beszel#
root@pve:/opt/beszel# echo "exit code: $?"
exit code: 2

[henrygd]

Thanks, I committed a fix for this a few minutes ago.

If you want to test you can compile the agent with these instructions. Remove --branch v0.9.1 to get the latest changes. You can build on a different machine if you want and use the same binary.

Otherwise the next release will likely be out next week. 👍