Rendering frame buffer at high frame rates with an overlay

[tralph3]

Hello. I just found this library, but before I invest into learning how to use it, I need to ask if my use-case would be possible.

I'm writing a libretro front-end (like RetroArch). I want to use the library for the GUI, but it needs to work sort of like an overlay.

You initially have a "blank" state, where no game is loaded, like a main menu. After a game is loaded, I get an integer array each frame which is the frame buffer I have to display.

I would need a way to display this as a widget, and be able to overlay stuff on top of it. For example, FPS counters, or more importantly, a pause menu that would let you change settings.

I also need the Graphics API context, be it Vulkan, OpenGL, or whatever. If this is not possible then I think I could use SDL as a rendering backend and just pass the pixels to Slint.

One more thing. Since this is for gaming, the UI should respond well to controller input. I don't care if it doesn't have good controller support, I'll be using SDL for that, but I need it to have a comfortable way to select widgets through code, so I can easily move to the "next widget" or jump to an arbitrary one, which is what I would be doing with a controller.

Is this doable? If it is, is it a hack? Or is this use-case well supported? Thanks.

[tronical]

I think this is generally doable. We only expose the OpenGL context right now, but if you look at the opengl_underlay example in our repo, you can see that it's possible to render with OpenGL as underlay and then Slint controls on top. The main caveat is that you need to make 100% sure that the OpenGL state is saved and restored.

[tralph3]

Thanks!

[SK83RJOSH]

Just wanted to chime in and say that I have implemented this for a similar project just this week.

Some issues you may run into:

• currently the easiest way to access the back buffer is take_screenshot
  • this is pretty slow, and unavoidable, especially without GL extensions for buffer sharing between different APIs
  • glutin doesn't support off screen buffers on Windows, but I opened a PR adding that yesterday, in CPP you'll probably have more mature libraries so this is a non-issue
  • all of the above would be much easier to handle via skia, but unfortunately there's no public renderer interface for it atm
• if you aren't in control of creating the window, there's a fair bit of glue you need
  • the embedded renderer examples are a great starting point
  • translating keycodes to event.text isn't well documented, and relies on magic values defined internally, take a look at the internal platforms for this
  • you need to do this with gamepad input, and translate input to these special values (arrow keys, enter, etc) – gilrs or SDL make this easier
• it's currently not very easy to know when slint wants keyboard/cursor input (eg/ hit testing, blocking input passthrough)
  • for this is best to just do a global input lock when the UI is displayed
  • there might be some ways to do it better but I'm currently unaware of them

It took a few days for me to do this, but it's very doable albeit cumbersome, and potentially a bit inefficient if not done with care. If you're using rust, feel free to ping me and we can share notes. 🙂

@tronical pinging you for visibility, since some of the above may be action points for y'all – I think my main takeaway from doing this, is that it'd be really nice to stabilize some kind of public interface for backends/renderers.

The user story for this is rather complicated right now, but it likely doesn't need to be. Even just exposing skia backend would go a very long way here, but in some cases, even that won't work particularly well if it doesn't support your target gfx api natively. I'm interfacing with a DX10 app for example, so skia would only help a little bit, as juggling two different directx contexts isn't much better. 😅

[tralph3]

Thanks for the write up! I ended up going for Iced instead of Slint, but if for some reason it doesn't prosper, I may take a second look at Slint.