Requirements for low-latency hosts

[JBetz]

Now that CLAP is enabling hosts and plugins to be written in languages other than C and C++, I'm curious what the viability of using higher level garbage-collected languages for plugin hosts is. Specifically, I'm interested in using languages like Haskell, Clojure, and Smalltalk to make a DAW.

Has anyone ventured down this path, and if so, were there any hard obstacles to achieving the real time requirements of music software? Assuming for now that all the plugins being used are still implemented in C-like languages.

[robbert-vdh]

See also the discussion in #129. You can make hosts that only process offline (so just applying effects to audio, no realtime playback) in any language, but if you want to do realtime playback at low latencies you really need to start caring about realtime safety. Which means that the audio threads may not be stalled for an indeterminate amount of time. If you don't do this, then you may end up with sporadic xruns and inconsistent DSP loads. Any language (runtime) that uses a garbage collector that has to periodically stop the world to do its thing thus immediately fails this requirement.

[baconpaul]

I started work on a Python host for testing but not for making a daw. I agree with Robert’s assessment that the audio thread running a gc is a non starter for real-time which is why I think rust c++ and Delphi are the daw languages with clap to date.

but if you want to do offline render then yeah this would work just fine. More of a sketch book than a daw.

i bet you could do quite well with Haskell and clojure also if your render thread was using ffi. Clojure tossing and event list over a thread boundary to an audio thread should work and require a relatively small amount of c-like code

all just random thoughts. Other than a proto python binding none of this is code

[robbert-vdh]

i bet you could do quite well with Haskell and clojure also if your render thread was using ffi

Not sure about Clojure, but Haskell's garbage collector still requires a lock at the FFI boundary no matter what. Which means that the GC will stall until all threads doing FFI calls have returned, and threads periodically doing FFI calls still need to wait for the GC. You could write a completely separate audio thread process written in C++ or Rust and interact using shared memory or sockets (like Bitwig does with Java), but that sort of defeats the exercise of writing a Haskell CLAP host. This behavior has also been causing performance bottlenecks for us in Accelerate (or in the accelerate-llvm-* backends to be specific).

[baconpaul]

Yes that’s exactly I was thinking Robert. You write code which spins a thread to do audio then use a lock free queue to message onto it in c++. Then the gc language just does ffi to the non audio thread to pack messages to audio.

Is that a Haskell program if you do that? Well … kinda. But it’s what you would do if you wanted reliable rt audio right?

[baconpaul]

And also I’m assuming the Haskell works at control rate more than audio rate so the messages are “less frequent” and known ahead of the render. For instance an automation curve

[JBetz]

Perhaps a better solution would be something like FluidSynth's sequencer, where you could schedule events from the host, routed though however many CLAP plugins, and then not have to deal with audio rendering at all. And maybe even fake real-time by only batching events one measure at a time.

The sequencer would of course still need to be written in a non-GC language but would provide a general solution for any GC language to use.

I've been using FluidSynth from Smalltalk in this way and so far haven't heard any defects in the audio output.

[JBetz]

The sequencer would of course still need to be written in a non-GC language but would provide a general solution for any GC language to use.

Okay, it seems I'm wrong about this.

If I play the FluidSynth sequencer in Smalltalk and add note animation during playback, the audio starts to stutter. Which makes me think that its audio process is somehow getting interrupted by the UI. I double checked that the FluidSynth threads are higher priority, and indeed they are (15 for FluidSynth vs 8 for Smalltalk), so it seems to be an issue with context switching between threads. And I'm almost certainly going to have the same problem with CLAP plugins.

How do DAWs generally handle this? I did read that FL Studio's UI refresh rate is synchronized with the audio buffer size. But I'm curious about other methods.