Device Nesting Slots for CLAP Plugins

[Trinitou]

I like that feature of Bitwig Studio very much where I can nest devices into others. E.g. useful for

• putting plugins into a delay's feedback path.
• putting plugins into some plugin-internal pre- or post distortion chain.
• putting plugins in between some automatic pre-/post gain compensation algorithm
• putting plugins in parallel device chains (like Bitwig's Layer, Selector or Split devices)
• some surround scenarios: if a plugin might want to have a nesting slot for different downmix/upmix plugins (slots with different number of input/output audio channels)

In theory it should be possible that a CLAP plugin offers nesting slots to a host that supports that kind of routing.
Some rough ideas here:

• The plugin doesn't need to know about the plugins that are dropped into the slot. The host should take care of that and can put plugins into that chain.
• Maybe the plugin wants to know if a nesting slot is used at all by the host to skip processing it
• Maybe the plugin wants to know the accumulated latency of all plugins nested inside a slot. This latency changes if nested plugins are inserted/removed
• Maybe a plugin's nesting slot has a maximum latency the nested plugins can take (e.g. if the slot is in a feedback loop of a delay), so the plugin would need to communicate that. But if there are only feed-forward slots in a plugin, this might not be necessary
• Each slot needs an individual id and name (like "Pre", "Post", "FB FX")
• It might be desirable to add/remove slots dynamically (like when adding/removing/duplicating layers in Bitwigs layer devices)
• as slots might process audio or note data, each slot
  • would need own note/audio in and out port definitions
  • would need some kind of own clap_process_t

Questions/difficulties:

• Who should call the process method of a nesting slot?
  • So the plugin at least needs to tell the host when the slot's input buffer/events are ready
• If a plugin has multiple slots, there might be a certain order in which the slots have to be processed
• Which aspects did I miss?

Maybe all that is way to complicated and maybe it would be an extension only the Bitwig guys could design with their experience in that domain. (?) Also as Bitwig so far is the only software supporting such kind of routings, it might be better suited in a Bitwig-specific extension.

[madah81pnz1]

One way to do this is for the plugin to expose additional audio ports.
For example, for a distortion effect, it could expose two more audio ports: feedback_in and feedback_out.
(feedback_out might not be needed in case it is the same as main audio out, but we can imagine the effect might have some additional processing after the distortion step.)

---> main audio in -+--> [ distortion ] --+-----> main audio out
                    |                     |
                    ^                     v
                    |                     |
                 feedback_in          feedback_out

In order to handle the case when the feedback ports are not connected by the host, a dry/wet mix parameter could be exposed by the plugin. Or maybe better is to use clap_audio_ports_config, the plugin can then provide two such configs; one with only the main audio ports, and one including the feedback audio ports.

Otherwise I think there is no way for the plugin to query the host if an audio port is connected though? clap_process_t states that "they must have the same count as specified by clap_plugin_audio_ports->count()."

struct clap_audio_buffer does have a latency field included, which could be set by the host to the nested plugin's latency. I don't know the correct way to handle this though, the plugin might need to set its own latency to a minimum of the one reported by the audio buffer, but during process() call it is too late to change the latency.

The plugin should be prepared to handle if the host connects feedback_out to feedback_in and reports the plugin's own latency, so it doesn't create a recursive loop of ever increasing latency.

[jaxter184]

This issue has some comments on it. #101
@baconpaul did you ever end up taking a swing?

[baconpaul]

not yet no

[abique]

For doing this, maybe the easiest is to host the plugin yourself: you scan the clap plugins (and maybe others), you build your own plugin cache and plugin chooser.
If you do so it'll work the same regardless of the host.

If you rely on the host it'll work half of the time. So basically you'd sell a non-working product to the user.

[Trinitou]

• It should be possible to forward the host thread pool to the nested plugins, shouldn't it?

[Trinitou]

And Bitwig should implement a Bitwig CLAP plugin which can host Bitwig devices/device chains. And if I drag Bitwig devices/a Bitwig device preset into that nesting CLAP plugin, it should automatically instanciate that Bitwig CLAP plugin with the devices/preset loaded. 😉

[elanhickler]

my use case would be swapping out a subtractive synth's filters

[abique]

Nested fx slots aren't polyphonic, you'd need custom logic for that.

[baconpaul]

a polyphonic audio port would be an interesting extension by the way. but not an easy one since it would have to grapple with the process signature in some ways which are tricky. but it would, i guess, mean you could write devices for polygrid in clap.

[LibrarySeph]

This makes a lot of sense to me. Do CLAPs know if their audio streams are being read? If not, the host would have to say when it was taking on one of these injection points because one would want the audio to pass right by if nothing was connected.