tempo changes

[pauldavisthefirst]

I'd like to start a discussion about handling tempo changes in CLAP. This is not handled adequately by any plugin API, not necessarily because of lack of mechanism, but more through lack of policy.

Prior to CLAP, most plugin APIs could arrange for the host to deliver tempo information to the plugin, and this typically is provided as a single value or tuple corresponding to the tempo at the start of the process() call.

CLAP allows tempo changes to be delivered as events in the queue, which is nice, but only works in a defined way for instantaneous tempo changes. It doesn't really cover the case of tempo ramping adequately.

We've already seen with Ardour that plugins that care about tempo frequently have their own ideas of what tempo ramps look like. Ardour provides them with the value at the start of the process() callback, and the plugins start interpolating over time based on successive values. Note that in Ardour, tempo ramps are continuously valued functions; we do not quantize the changes the way that (e.g.) Live does.

The problems arise from the fact that Ardour honors (apparent) human musical performance behavior, in which tempo ramps are generally exponential, whereas most plugins that do this (that we've seen so far) tend to assume linear ramps. This leads to the host and plugin getting out of sync, which can cause glitches at each process() call as the plugin receives a new (unexpectedly high or low) value from the host.

Since CLAP allows the host to provide multiple tempo events in the queue, one aspect of this problem - quantizating to tempo changes to process() cycle boundaries - is solved.

However, this by itself does not dictate:

• the interval the host should use between tempo events
• whether the plugin should interpolate between events, and if so, how

I would prefer a solution that (a) does not dictate the shape of tempo ramps (because we believe in what Ardour does, and do not want to fall back to the discrete behavior of DAWs such as Live) (b) somehow allows plugins to function correctly with DAWs with different ideas about ramping (accelerando).

Thoughts?

[Bremmers]

My host's tempo editor generates exponential ramps too. They're quantized to MIDI file tempo resolution. For CLAP plugins I send all these events, so the plugin sees exactly what's in the MIDI file.

And similar to MIDI files, I assume the plugin doesn't do any interpolation.

I agree the comments could tell a bit more about when to send tempo events.

[pauldavisthefirst]

MIDI file tempo resolution, as I understand it, is "24ths of a microsecond per quarter note". That quantizes the values; I was referring to quantizing when the changes occur. Live uses a remarkably coarse value, apparently. What interval between tempo events do you use when writing SMF?

[Bremmers]

I use 8 steps per quarter note.

[pauldavisthefirst]

So 32nd notes. That's even coarser than Live uses internally (as far as can be determined by using the application.

But it's probably enough. We just need guidance for plugins to not interpolate, and some agreed upon interval for ramps.

[Bremmers]

If a plugin interpolates the tempo events it could easily go out of sync with the real tempo I think. Is that what you're saying?

[pauldavisthefirst]

Yes, if the plugin sees the changes coming in and for any reason thinks it can "guess ahead", then the accuracy of the guess will be impacted by whether its ramp model is the same as the hosts. I will try to recall the names of the plugins that we've seen doing this, since to me it seems like an odd thing to do. The rule should be "no interpolation by the plugin", along with some specified min/max/recommended interval for the events.

[baconpaul]

@pauldavisthefirst thanks for the thoughtful idea. Can I ask a couple of questions

1. It is of course impractical and silly but if you sent a tempo event on every sample, there would be no problem right? The host could unambiguously move tempo through a change and the plugin would always know the instantaneous tempo. I am not suggesting this is a solution but am just using physics brain to understand if the limit works.

2. The event mechanism is great but you are right since you don't know when future events are coming (because its the future) you don't know a smoothing distance. For instance Surge smooths modulators over its internal 32 size block but bitwig sends them to us every 64 so external modulation has a slightly different shape than internal modulation. I wonder if a 'frequency hint' extension might be interesting for 1.1, where a host could advertise optionally that 'if i am sending you events approximating something roughly continuous, you will get them every n samples'. This would allow a ramping algorithm.

3. In 1.0.* this could indeed be solved by spec. I would be all in favor of modifying the spec to say "on a tempo change event a plugin can choose to apply the tempo change immediately or do ___" myself.

4. The relationship between tempo and song position changes also seems like another opportunity for drift. Is there a constraint between tempo changes and expected changes in ppqpos over a set of samples which could perhaps help?

Sorry to have questions not answers, but hope those are some constructive thoughts.

[pauldavisthefirst]

I think the correct rule is likely that the last tempo event sent by the host for a given process() cycle must be accepted by the plugin as the tempo for the final sample of that cycle. Plugins should not try to predict the future.

Re: 1 - I don't think it's actually impractical or silly in terms of data transfer - it's essentially one more channel of audio data. However, it would impose a somewhat absurd workload on some plugins (though note: no more than it does on the host)

Re: 2 - You don't need to smooth tempo

re: 3 - I don't believe that plugins should have this choice. If there's a tempo event at sample N, then that's the tempo at sample N.

Re: 4 - It is likely plugins trying to compute SPP from tempo that causes the issue right now. They really cannot do this accurately because they do not know the ramp function. However, the error could be fairly small, particularly if the host sends tempo every 1/32nd note as @Bremmers has described.

[baconpaul]

right so yes when you get the tempo that is the tempo. That makes sense

so if i get a tempo event every 64 samples, like i do a poly mod, i just update as i go. no problem. and then i get some drift in my internal ppqpos calculation.

so what do you think is a way to improve the situation? It seems like at least clarifying the spec that 'plugins shouldn't smooth tempo and should just accept it' is a good start. And maybe encourage hosts to send tempo changes with a resolution such that piecewise continuous interpretation of tempo is a good approximation to the actual curve?

I actually haven't checked if BWS sends me mid-block tempo changes, but i know surge would not listen to the tempo events properly yet. Although no reason it couldn't. It does everything on a 32 block basis.

[Bremmers]

I'm not sure why a plugin developer would think smoothing the tempo is a good idea. I'd think the problems come from incorrectly predicting the future. Getting arpeggiator note-offs right may be rather difficult if there are 40 tempo changes spread out over 28 process blocks.

The comments could say this:

• Tempo is a staircase function, ie. the tempo is constant until the next tempo event arrives.
• Hosts should send tempo/timesig events whenever something changes.
• Hosts that have an unusually high tempo resolution should limit the tempo event rate to something reasonable like 1/64 notes.

[Bremmers]

On a related note:

I can imagine an extension that lets plugins see the hosts tempo/timesig map. This could also include functions for beat <--> frame conversion, so the host can do the math.

There are two ways such an extension would be useful:

1. Plugins can use this to easily calculate things in the future, like a note-off position.
2. Plugins like a drummer that makes up the drum part all by itself can get an overview of time sig changes and the tempo beforehand. So they can show the song structure in their UI without the user having to play back the whole song first. And the poor drummer can play fills without suddenly being told the time sig will change 1/44100 seconds from now ;-)

[pauldavisthefirst]

I can imagine an extension that lets plugins see the hosts tempo/timesig map.

Please don't try for this. Coming up with a tempo map structure that everyone can agree on is hard. We wanted this in JACK since about 2005, but even in that "limited context" (FOSS applications), it was too difficult to come up with a structure that works for everyone.

It's fine if you think of a tempo map as just as list of static changes to tempo, with musical time monotonically increase through time. That's a common structure, but it doesn't deal with accelerando/decelarando, or rubato. It also doesn't deal very well with tempo maps generated from audio analysis. Finally, in the specific case of Ardour/Mixbus, we allow users to define the location of specific BBT times in a completely arbitrary way (you can have multiple instances of 1|1|0 for example).

I think it is much better for the information flow between the host and the plugin to be based around an API that is as "model free" as possible (for tempo, anyway).

[urshimself]

Let me try to add some perspective from the plug-in side.

The timing of built-in sequencers, arpeggiators etc. is usually not such a big deal with tempo changes, as these can typically "snap in" to the PPQ position: "Oh, we're past due time? Fire that NoteOn!"

Likewise, host synced delays and such are also no biggie as they need to be smoothed and they're fine as long as the input material is on time. Delays that are adjusted in length will have pitch and comb filters artefacts anyway. Delays that adjust in sample rate are seldom and might drift for a short time when the tempo changes. Depending on feedback, they catch up.

The main problem in my experience are tempo synced modulations which can be modulated in rate. For example, take an LFO which is synced to 1/4 but which can be modulated to double speed via aftertouch. It can't be "snapped in" with PPQ position, because that's relative to song start. The solution is to "take it off strict sync" and have it free running. What then happens easily with tempo changes, is a drift. This is the main issue with tempo changes.

Furthermore, LFOs and such are typically calculated at control rate, which may be something like 1/16th of the sample rate. So even if tempo ramps were perfectly known to plug-ins, it would be difficult to calculate the correct progression of such a modulator unless control rate and tempo map coincide.

I think a really, really good solution would be something as trivial as a very good example implementation of an LFO-style modulator which runs at a common control rate, deals with sudden or smooth tempo changes and finds a way to "snap back in" after a short period of rate modulation (think square wave 4x rate/quarter rate). This example implementation could then serve as test case for hosts as well.

[signaldust]

I feel like from a plugin point of view there's three sensible options for dealing with continuous modulators:

1. snap at event (will potentially cause a glitch)
2. smooth over the snap by "chasing" the host time (causes some lag in reaction to tempo changes, but workable)
3. interpolate time from one event to the next

Option 3 is ideal, but can only be done when the next event is known.. so I'd support a proposal that encourages (if not mandates) that hosts send a time event at the very end of the block, at least when there's some tempo-ramp going on (ie. when it's likely you'd otherwise have to snap/chase more than the typical floating point rounding drift).

[pauldavisthefirst]

I'd support a proposal that encourages (if not mandates) that hosts send a time event at the very end of the block

Yes, I think this would be good as a mandate, though stated slightly differently: the most recently received tempo event always defines the tempo for all samples timestamped after the event, BUT if the tempo is changing in a non-instantaneous fashion, then the host should always deliver a tempo event timestamped for the end of the process() cycle. Plugins may look ahead in the event queue to determine if there are (a) more than 1 tempo events (b) an "end" tempo events. If neither condition is satisfied, any tempo changes should be considered instantaneous (non-ramped). If these
conditions are satisifed, the plugin should not interpolate between the values provided by the host.

This works even for extremely small sample sizes (say, 8 samples); the host simply sets the tempo for the start of teh process block, and if ramping, includes an "end" tempo value to indicate to the plugin that it should not interpolate. This leaves the host free to use continuously valued tempo functions (like Ardour) or discrete ones (like Live).

I don't like (2) because it implies that following tempo also means following host time, and requires plugins to implement a DLL or equivalent merely to handle tempo changes correctly.

[signaldust]

I feel like it'd be better to signify instantaneous tempo-changes with one event at the last sample of the old tempo period and another event on the next sample to signify a new tempo. Otherwise you can't represent a situation where you have (part of) a ramp AND an instantaneous change within a single processing block.

[pauldavisthefirst]

That's an excellent point, and I fully concur.

[baconpaul]

One thought on this, just reading the spec

the transport event has a tempo and a tempo inc field.

clap/include/clap/events.h

Line 223 in e60671b

double tempo_inc; // tempo increment for each samples and until the next

In theory this obviates the need to look ahead in the queue and the job of the host is to send events such that a plugin using piecewise linear interpolation between events of tempo + s * d tempo where s is samples since last event woild sufficiently approximate the underlying tempo curve

I hadn’t seen that mentioned in this thread before, and I certainly know surge doesn’t currently use these events accurately (but we can fix that). But I do wonder if from an api design point this solves the problems stated?

I’m also going to tag @robbert-vdh since these events are things we should add to the emerging clap validator he is working on

[baconpaul]

Oh and of course a host which preferred stepwise tempo could just set that field to zero and send events as it sees fit.

[robbert-vdh]

Yes I thought about tempo_inc a couple days ago, but I have no idea how you'd test whether a plugin handles transport changes and tempo ramps (at all and/or correctly) without actually listening to the plugin's output.

[abique]

With clap you know exactly the value of the tempo for every samples of the block.
If there is a small drift in the beat time calculation, it probably won't be noticeable by a user, and at the start of the next block the plugin can do some drift correction.

I believe that clap provides enough info to have "reproduce-able" output. At least it is for sure good enough compared to today standards.

[signaldust]

Yeah, tempo_inc is fine, must have missed it above. It is somewhat unfortunate that no similar _inc field exists for modulation (or automation) events, but I guess it might be too late for those?

[robbert-vdh]

@signaldust The problems here are mostly that you may not know in advance what the future automation/modulation events look like (for instance, when them modulation is an envelope follower attached to some other plugin's output), and that plugins will be low-passing these changes internally anyways. So having ramps to linearly interpolate between automation/modulation events (which Bitwig for instance sends every 64 samples) tends to not add much here when account for that smoothing, but it does add a lot of complexity. And since the availability of this information will differ between hosts and likely even between modulation sources within the same host you also wouldn't really be able to rely on them.

[signaldust]

That said... I guess the original question about exponential (or more generally non-linear) vs. linear ramps is that if you sample the smooth tempo curve at some reasonable modulation rate and send it to the plugin as linear segments (using tempo_inc), then you have to make the choice between continuous tempo vs. continuous PPQ time. Continuous tempo is a matter of setting tempo_inc to the difference of the tempo values at the end-points. Continuous PPQ time is a matter of computing the average tempo over the interval (which one can do with or without tempo_inc).

From the point of view of synchronizing something like LFOs having continuous PPQ time is preferable. From the point of view of something like tempo-synced delays, one could make the case that continuous tempo might be preferable. Personally I feel like if I had to choose one or the other, then continuous PPQ time would probably be preferable most of the time.

[baconpaul]

I think I agree yeah. We snap our lfo phase to ppq in freerun mode when transport is active

my point was more: I think the api as written let’s us address the original points, albeit with a trade off where over some n samples (n being small like 64 or so) you need to approximate your desired tempo curve as piecewise linear. That’s a pretty good approx thoigh for continuous tempo changes I think?

[signaldust]

Yeah... also if you split each interval into two linear segments (to get another degree of freedom to play with), then I think it'd typically be possible to solve for both continuous tempo and continuous PPQ time at the end-points.