BUG: Disk usage in docker / MacOS is incorrect

[ErebusBat]

I have an agent running on macOS 15.1 / docker (orbstack). I have a USB drive mounted in the extra-filesystems.

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /Volumes/SeaBack:/extra-filesystems/SeaBack:ro

Beszel (0.7.2 agent and hub) is reporting a VASTLY incorrect disk usage:

[henrygd]

Yeah, I'd say so. Unfortunately I don't have a Mac device to troubleshoot myself.

Please restart the agent with LOG_LEVEL=debug and let me know what it outputs for partitions and diskstats.

From a quick google it looks like this may be an issue with how Docker works on MacOS: https://forums.docker.com/t/macos-docker-bind-mount-reports-incorrect-free-space/137982/4

Can you test with the binary agent to see if it has the same problem? I think you would add EXTRA_FILESYSTEMS=/Volumes/SeaBack or try the actual device / partition, like EXTRA_FILESYSTEMS=disk4s1.

[ErebusBat]

Seeing that you are using Docker, I would like to recommend Servbay to you, which is more professional and easy to use than Docker on Mac.

Thank you for the recommendation; however I am using OrbStack after evaluating several options for my day job we found that OrbStack was the best cost/$.

[ErebusBat]

@henrygd thank you for your response. Please find my troubleshooting steps below:

Env Var
Adding the EXTRA_FILESYSTEMS env var did not seem to have any effect, positive or negative. I assume because it is already mounted under /extra-filesystems but I do not know that for sure.

Debug Logging
Raw logs:

beszel-agent  | 2024/11/04 16:09:14 DEBUG 0.7.2
beszel-agent  | 2024/11/04 16:09:14 DEBUG Not monitoring ZFS ARC err="open /proc/spl/kstat/zfs/arcstats: no such file or directory"
beszel-agent  | 2024/11/04 16:09:14 DEBUG Disk partitions="[{\"device\":\"/dev/vdb1\",\"mountpoint\":\"/etc/resolv.conf\",\"fstype\":\"btrfs\",\"opts\":[\"rw\",\"noatime\",\"bind\"]} {\"device\":\"/dev/vdb1\",\"mountpoint\":\"/etc/hostname\",\"fstype\"
:\"btrfs\",\"opts\":[\"rw\",\"noatime\",\"bind\"]} {\"device\":\"/dev/vdb1\",\"mountpoint\":\"/etc/hosts\",\"fstype\":\"btrfs\",\"opts\":[\"rw\",\"noatime\",\"bind\"]}]"
beszel-agent  | 2024/11/04 16:09:14 DEBUG Disk I/O diskstats="map[vda:{\"readCount\":20215,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":722542592,\"writeBytes\":0,\"readTime\":591,\"writeTime\":0,\"iopsInProgress\":0,\"io
Time\":3465,\"weightedIO\":591,\"name\":\"vda\",\"serialNumber\":\"\",\"label\":\"\"} vdb:{\"readCount\":114217,\"mergedReadCount\":2322,\"writeCount\":4506,\"mergedWriteCount\":3890,\"readBytes\":7143862272,\"writeBytes\":125472768,\"readTime\":961,\"
writeTime\":5907,\"iopsInProgress\":0,\"ioTime\":19790,\"weightedIO\":15522,\"name\":\"vdb\",\"serialNumber\":\"\",\"label\":\"\"} vdb1:{\"readCount\":114207,\"mergedReadCount\":2322,\"writeCount\":4506,\"mergedWriteCount\":3890,\"readBytes\":714382131
2,\"writeBytes\":125472768,\"readTime\":961,\"writeTime\":5907,\"iopsInProgress\":0,\"ioTime\":20982,\"weightedIO\":9672,\"name\":\"vdb1\",\"serialNumber\":\"\",\"label\":\"\"} vdc:{\"readCount\":3,\"mergedReadCount\":0,\"writeCount\":1,\"mergedWriteCo
unt\":0,\"readBytes\":12288,\"writeBytes\":4096,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"vdc\",\"serialNumber\":\"\",\"label\":\"\"} zram0:{\"readCount\":2,\"mergedReadCount\":0,\"writeCount\":1,\"mer
gedWriteCount\":0,\"readBytes\":8192,\"writeBytes\":4096,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"zram0\",\"serialNumber\":\"\",\"label\":\"\"}]"
beszel-agent  | 2024/11/04 16:09:14 INFO Detected root device name=vdb1
beszel-agent  | 2024/11/04 16:09:14 DEBUG /extra-filesystems mountpoint=/extra-filesystems/SeaBack
beszel-agent  | 2024/11/04 16:09:14 WARN Device not found in diskstats name=SeaBack
beszel-agent  | 2024/11/04 16:09:14 INFO Detected network interface name=eth0 sent=2695 recv=938857
beszel-agent  | 2024/11/04 16:09:14 INFO Detected network interface name=.orbmirror0 sent=746 recv=242169
beszel-agent  | 2024/11/04 16:09:14 INFO Detected network interface name=.orbmirror1 sent=746 recv=242169
beszel-agent  | 2024/11/04 16:09:14 DEBUG Docker version=27.3.1 concurrency=5
beszel-agent  | 2024/11/04 16:09:14 DEBUG Getting stats
beszel-agent  | 2024/11/04 16:09:14 DEBUG Temperature sensors=[]
beszel-agent  | 2024/11/04 16:09:14 DEBUG sysinfo data="{Hostname:orbstack KernelVersion:6.11.5-orbstack-00280-g96d99c92a42b Cores:1 Threads:1 CpuModel: Uptime:9308 Cpu:100 MemPct:33.21 DiskPct:2.65 Bandwidth:0 AgentVersion:0.7.2}"
beszel-agent  | 2024/11/04 16:09:14 DEBUG System stats data="{Stats:{Cpu:100 MaxCpu:0 Mem:3.89 MemUsed:1.29 MemPct:33.21 MemBuffCache:0.3 MemZfsArc:0 Swap:4.89 SwapUsed:0 DiskTotal:244.69 DiskUsed:6.49 DiskPct:2.65 DiskReadPs:0 DiskWritePs:1.85 MaxDisk
ReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[]} Info:{Hostname:orbstack KernelVersion:6.11.5-orbstack-00280-g96d99c92a42b Cores:1 Threads:1 CpuModel: Uptime:9308 Cpu:100 MemPct:3
3.21 DiskPct:2.65 Bandwidth:0 AgentVersion:0.7.2} Containers:[]}"
beszel-agent  | 2024/11/04 16:09:14 DEBUG Docker stats data="[0x400014c8c0 0x400014c9a0 0x400014ca80 0x400014cb60 0x400014c2a0 0x400014c5b0 0x400014c7e0]"
beszel-agent  | 2024/11/04 16:09:14 DEBUG Extra filesystems data=map[SeaBack:0x40000a90a0]
beszel-agent  | 2024/11/04 16:09:14 DEBUG Stats data="{Stats:{Cpu:100 MaxCpu:0 Mem:3.89 MemUsed:1.29 MemPct:33.21 MemBuffCache:0.3 MemZfsArc:0 Swap:4.89 SwapUsed:0 DiskTotal:244.69 DiskUsed:6.49 DiskPct:2.65 DiskReadPs:0 DiskWritePs:1.85 MaxDiskReadPs:
0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[SeaBack:0x40000a90a0]} Info:{Hostname:orbstack KernelVersion:6.11.5-orbstack-00280-g96d99c92a42b Cores:1 Threads:1 CpuModel: Uptime:9308 Cpu
:100 MemPct:33.21 DiskPct:2.65 Bandwidth:0 AgentVersion:0.7.2} Containers:[0x400014c8c0 0x400014c9a0 0x400014ca80 0x400014cb60 0x400014c2a0 0x400014c5b0 0x400014c7e0]}"
beszel-agent  | 2024/11/04 16:09:14 INFO Starting SSH server address=:45876
beszel-agent  | 2024/11/04 16:09:46 DEBUG Getting stats
beszel-agent  | 2024/11/04 16:09:46 DEBUG Temperature sensors=[]
beszel-agent  | 2024/11/04 16:09:46 DEBUG sysinfo data="{Hostname:orbstack KernelVersion:6.11.5-orbstack-00280-g96d99c92a42b Cores:1 Threads:1 CpuModel: Uptime:9339 Cpu:6.39 MemPct:33.22 DiskPct:2.65 Bandwidth:0 AgentVersion:0.7.2}"
beszel-agent  | 2024/11/04 16:09:46 DEBUG System stats data="{Stats:{Cpu:6.39 MaxCpu:0 Mem:3.89 MemUsed:1.29 MemPct:33.22 MemBuffCache:0.28 MemZfsArc:0 Swap:4.89 SwapUsed:0 DiskTotal:244.69 DiskUsed:6.49 DiskPct:2.65 DiskReadPs:0.13 DiskWritePs:0.02 Ma
xDiskReadPs:0 MaxDiskWritePs:0 NetworkSent:0 NetworkRecv:0 MaxNetworkSent:0 MaxNetworkRecv:0 Temperatures:map[] ExtraFs:map[]} Info:{Hostname:orbstack KernelVersion:6.11.5-orbstack-00280-g96d99c92a42b Cores:1 Threads:1 CpuModel: Uptime:9339 Cpu:6.39 Me
mPct:33.22 DiskPct:2.65 Bandwidth:0 AgentVersion:0.7.2} Containers:[]}"
beszel-agent  | 2024/11/04 16:09:46 DEBUG Docker stats data="[0x400014cb60 0x400014c2a0 0x400014c5b0 0x400014c7e0 0x400014c8c0 0x400014de30 0x400014ca80]"
beszel-agent  | 2024/11/04 16:09:46 DEBUG Extra filesystems data=map[SeaBack:0x40000a90a0]

Here is a prettied / decoded disk partitions:

[
  {
    "device": "/dev/vdb1",
    "mountpoint": "/etc/resolv.conf",
    "fstype": "btrfs",
    "opts": [
      "rw",
      "noatime",
      "bind"
    ]
  },
  {
    "device": "/dev/vdb1",
    "mountpoint": "/etc/hostname",
    "fstype": "btrfs",
    "opts": [
      "rw",
      "noatime",
      "bind"
    ]
  },
  {
    "device": "/dev/vdb1",
    "mountpoint": "/etc/hosts",
    "fstype": "btrfs",
    "opts": [
      "rw",
      "noatime",
      "bind"
    ]
  }
]

And Diskstats:

vda:
{\"readCount\":20215,\"mergedReadCount\":0,\"writeCount\":0,\"mergedWriteCount\":0,\"readBytes\":722542592,\"writeBytes\":0,\"readTime\":591,\"writeTime\":0,\"iopsInProgress\":0,\"io Time\":3465,\"weightedIO\":591,\"name\":\"vda\",\"serialNumber\":\"\",\"label\":\"\"}

vdb:
{\"readCount\":114217,\"mergedReadCount\":2322,\"writeCount\":4506,\"mergedWriteCount\":3890,\"readBytes\":7143862272,\"writeBytes\":125472768,\"readTime\":961,\" writeTime\":5907,\"iopsInProgress\":0,\"ioTime\":19790,\"weightedIO\":15522,\"name\":\"vdb\",\"serialNumber\":\"\",\"label\":\"\"}

vdb1:
{\"readCount\":114207,\"mergedReadCount\":2322,\"writeCount\":4506,\"mergedWriteCount\":3890,\"readBytes\":7143821312,\"writeBytes\":125472768,\"readTime\":961,\"writeTime\":5907,\"iopsInProgress\":0,\"ioTime\":20982,\"weightedIO\":9672,\"name\":\"vdb1\",\"serialNumber\":\"\",\"label\":\"\"}

vdc:
{\"readCount\":3,\"mergedReadCount\":0,\"writeCount\":1,\"mergedWriteCount\":0,\"readBytes\":12288,\"writeBytes\":4096,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"vdc\",\"serialNumber\":\"\",\"label\":\"\"}

zram0:
{\"readCount\":2,\"mergedReadCount\":0,\"writeCount\":1,\"mergedWriteCount\":0,\"readBytes\":8192,\"writeBytes\":4096,\"readTime\":0,\"writeTime\":0,\"iopsInProgress\":0,\"ioTime\":0,\"weightedIO\":0,\"name\":\"zram0\",\"serialNumber\":\"\",\"label\":\"\"}]"

Binary Agent
As was probably expected things seem to work correctly under docker... including temperature sensors which I had not realized were not reading under docker (but is obvious in retrospect).

[henrygd]

Thanks!

It looks like the dockerized agent can't see the underlying device for /Volumes/SeaBack. The only partition it sees is /dev/vdb1, which I assume is the default storage device for the container.

Can you paste your output of lsblk here? Or whatever the Mac equivalent is. Maybe diskutil list and df?

Temperature sensors populate in Docker on Linux, so I think that's another issue that's specific to Docker on MacOS.

EXTRA_FILESYSTEMS should only be used for the binary agent. If you specify EXTRA_FILESYSTEMS=vda,vdc that should add those devices as extra disks.

[schtocker]

Running into this same problem and thought I'd share my experience in case it's helpful. I'm trying to use Beszel Agents to report on several Macs around the building, reporting back to a Beszel Server running on Ubuntu Server and Docker on an Intel NUC.

Mac Studio M1 Max running Ventura 13.6.4, Docker Desktop 4.38.0, Beszel 0.9.0.

I have a 1TB external volume SSD that Apple Disk Utility reports as having 999.86GB capacity with 583.87GB used.

Installed Beszel via Docker and set up a bind mount to the external volume, it shows a total capacity of 173TB with no usage. Tried copying an 100GB file from a network share to the external volume, it shows no I/O when copying. Unrelated to this exact issue but it also shows no bandwidth network traffic during this copy.

As you advised on this thread, I have installed the binary agent to try. It shows a much more correct Disk Usage (931GB capacity with 543.77GB used) but still shows no disk I/O during a copy. Bandwidth network traffic is correct.

As such, I will be looking at making a launchd .plist to get this running as a service on my systems, though it would be nice to have Docker working as it's installed on the systems anyway!

[henrygd]

Thanks, that's very helpful!

I'll try using a Mac cloud provider like Scaleway at some point to troubleshoot these problems.

Please see here for a working plist example: #276 (comment)

Homebrew support is in the roadmap.