Question about viogpu profiling capabilities

[sharkautarch]

It seems like there is a tracing.h file for WPP tracing in viogpudo.
Is WPP tracing locked behind a build-time config, or is it possible to enable it at runtime?
Lastly, do you have any other advice for how to best profile the viogpu driver & services?
For context, I've been experimenting with using virtio-gpu-pci w/ nvidia gpu passthru, and there appears to be some performance degradation w.r.t. viogpu that is not necessarily due to being bound by memory bandwidth limitations.

Reason why I know it isn't a memory bandwith problem is because I've been setting the guest display's resolution to 1280x800 or smaller (upscaled on the host to 1920x1200 using gamescope.) And testing w/ running heaven benchmark shows that I'm able to, on the host side, get 99-100fps.

However, outside of that, (including w/ testing running testufo on firefox and playing a game that I don't think uses fullscreen exclusive mode called ss13) I seem to be getting 50-60fps, and even sometimes 30-40fps (seems to suddenly switch between 50+fps and 30-40fps every few minutes or so)

And on firefox testufo, disabling dwm using a special tool seems to bring the fps back up to around 60fps:

Oddly enough, disabling dwm doesn't seem to help w/ fps as much for ss13

Reason why I'm posting a discussion and not an issue about this fps stuff is because I want to first check if somehow the recently-merged PR #1128 changes any of this

[YanVugenfirer]

Hello @sharkautarch ,

Are you testing the experimental PR #943, or just a regular driver?
A regular virtio-gpu driver in Windows doesn't support acceleration, so it is not expected to have any performance comparable to the actual physical GPU.

[sharkautarch]

@YanVugenfirer
I'm testing using nvidia gpu passthrough and virtio-gpu-pci on the same vm, where basically the nvidia-gpu does the rendering, and virtio-gpu-pci forwards the rendered stuff to the host.

Are you testing the experimental PR #943, or just a regular driver?

using the regular driver btw

Reason for why I'm using this setup is that I'm on a hybrid muxless laptop, and although the intel igpu hardware supports sr-iov, I'm still waiting for sr-iov support in the kernel w/ the Xe driver to mature.
I can tell that the nvidia-gpu in this setup is indeed increasing fps since w/o the nvidia gpu driver installed, the vm is stuck at less than 40fps, whereas with the nvidia gpu driver installed, it can technically go all the way to 100fps from the host perspective (via mangoapp HUD measuring frame times on the actual host machine) when running heaven benchmark in the vm (and heaven benchmark reports higher fps at around 300fps, tho ofc the fps drops down to 100fps due to bandwidth limitations.)

[YanVugenfirer]

OK. But what kind of virtio-gpu driver are you using?

[sharkautarch]

OK. But what kind of virtio-gpu driver are you using?

I believe I'm using the viogpudo driver

If you're asking what version, it's the one from: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/virtio-win-pkg-scripts-input/latest-build/virtio-win-prewhql-0.1.zip

[YanVugenfirer]

Then, it is expected to have diminished performance compared to pass-through.

[sharkautarch]

Yes, but it seems like there's some level of performance degradation that isn't due to bandwidth limitations because again:

• I'm able to get 100fps when running heaven benchmark
• I forgot to mention this, but I also was able to get ~120fps when running a linux vm (an old gparted live cd of all things) with the same libvirt/qemu config
  
• the windows desktop sometimes is able to run at 50-60 fps, where it remains like that for a while, but then will randomly go down to ~40fps, and then again after some time will go back up to 50-60fps and so on...
• Disabling dwm seems to bring the fps back to 60fps on firefox, no longer going below 50fps for as long as dwm is disabled, tho it doesn't seem to help as much for other programs/apps

I know that there is a certain amount of performance degradation that is unavoidable w/ this setup, but I just want to figure out how best to profile the system & driver to find anything where performance can be improved somewhat

I'm open to compiling the driver myself, if needed, to get better profiling details

[YanVugenfirer]

xperf can be used to profile where OS spends its time: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/windows-performance-toolkit-technical-reference

[sharkautarch]

@YanVugenfirer
I'm aware of xperf, wpr and wpa
Is it possible to enable WPP tracing for viogpudo without recompiling?

[vrozenfe]

WPP works in general.

But in the current version it is not possible to adjust the verbosity level. It is all or nothing. I will fry to fix it in the next public build

[vrozenfe]

For the testing purpose only you can try to activate
"UsePhysicalMemory" and "UsePresentProgress" flags (they are disabled in the downstream build)
https://github.com/virtio-win/kvm-guest-drivers-windows/blob/master/viogpu/viogpudo/viogpudo.inx#L72C7-L72C24

[sharkautarch]

@vrozenfe WRT the "UsePhysicalMemory" flag
From what I can tell, I don't think virtio-win viogpu driver will end up using physical memory when this "UsePhysicalMemory" register flag is set, due to the following:
(Due to how complicated the virtio-gpu technology stack is, it is always possible that I misunderstood something, so let me know if you find anything I missed or misread)

kvm-guest-drivers-windows/viogpu/viogpudo/viogpudo.cpp

Line 2316 in e6546dc

UINT fb_size = (UINT)m_PciResources.GetPciBar(0)->GetSize();

• currently the viogpu driver, when "UsePhysicalMemory" is set, uses the vga bar 0 from virtio-gpu (well technically virtio-vga) for the physical memory

• If the size of bar 0 is less than the maximum of all possible presets of resolutions, then the driver disables the direct use of physical memory.

  • The biggest preset resolution is 5120x2160 -> 42.1875 MiB
  • Tho even if the above point weren't the case, it also unconditionally forces the required size to be at least 16MiB (req_size = max(req_size, 0x1000000);). I have absolutely no clue why that is the case, would be interested to learn why that line exists...

• Obviously it definitely can't use physmem if there's not enough for the current resolution, and currently, the viogpu driver only does VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING & pass the number of (4KiB) pages at driver init time

• And the problem is that it turns out that, for the virtio-vga device, qemu currently hard-codes the vga bar 0 size to 8MB . This is a bit surprising, since the plain stdvga, and the qxl vga devices have options for setting the bar 0 size, but not for virtio-vga for some reason :(
  So, for now, unless you manually patch & recompile qemu, this viogpu driver will never use physical memory

I'm guessing that the easiest workaround is to patch qemu to allow for a bigger vga bar 0 size, and prob also doing an MR to add a config for the vga bar 0 size for virtio-vga

Only annoying issue w/ that is that I would imagine that no one would want the extra code complexity of vga bar 0 size negotiation between driver and virtio-vga device, so that wouldn't allow physmem use for viogpu driver to work out of the box.

The second possible 'solution' would be to make the viogpu driver do VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING and VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING each time the current resolution changes, thus allowing the driver's req_size to simply be the size of the current resolution. Tho ofc that would mean that physmem could only be used with relatively smaller resolutions: even 1080p (1920x1080x4 = 8.2944 MB) would be too big for use w/ physmem

The one other thought I have, is: would it be possible to use a hostmem or guest-only blob memory as driver physmem, since I think those may be likely to be backed by hugepages (I'm not entirely sure if that's actually the case tho)?

[vrozenfe]

The second possible 'solution' would be to make the viogpu driver do VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING and VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING each time the current resolution changes, thus allowing the driver's req_size to simply be the size of the current resolution. Tho ofc that would mean that physmem could only be used with relatively smaller resolutions: even 1080p (1920x1080x4 = 8.2944 MB) would be too big for use w/ physmem

Yes, that would be an ideal solution, And I was going to try implementing it.
Unfortunately, there are some problems with this approach. We allocate memory from non-paged pool which is always a scarce resource.
Technically, if we are trying to switch to a bigger resolution and failed to allocate a bigger memory buffer, at least we need to keep a current resolution, but it is still possible that we will fail to re-allocate this smaller buffer either, So, pre-allocating a big buffer sufficient for working with the biggest resolution seems to be the most robust solution even though it is not optimal in terms of memory usage,

Best,
Vadim.

The one other thought I have, is: would it be possible to use a hostmem or guest-only blob memory as driver physmem, since I think those may be likely to be backed by hugepages (I'm not entirely sure if that's actually the case tho)?

[sharkautarch]

Oh BTW, I think I made a mistake when I claimed here: #1133 (reply in thread)
that the linux guest was able to consistently acheive 120fps:
I now believe that the 120fps that mangoapp was reporting was over-inflated due to cursor repaints

[sharkautarch]

@vrozenfe
Another idea just occurred to me:
Instead of (only) deciding whether to use bar 0 physmem at driver init, viogpu still preallocates the maximum required memory at init, but it would additionally re-run the check for whether it should use physmem every time the current resolution changes.
Basically, the driver still makes sure it preallocates enough memory for the biggest resolution it can support, but it will then only use the current resolution size to determine whether to use bar 0-based physmem or the preallocated guest memory.
I guess you would still have to do VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING and VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING when resolution changes, but only when a resolution switch causes the driver to go from previously using physmem to currently using virtmem or vice versa.
This avoids any issues about re-allocation failing after a resolution change
This would mean that the driver would still be wasting a small amount of RAM sometimes, but for me personally I don’t care that much about RAM waste that’s less than 100MB in size, when it seems like Windows VMs require at least ~4GB of RAM anyways.
I mostly just care about viogpu performance