Adaptive RTP Jitter buffer or aubuf

[cspiel1]

@sreimers You told us that you plan to make aubuf something more intelligent. The audio frames already have timestamps and the aubuf may do the jitter and reorder handling by using the timestamp of the auframe struct.

[ RTP socket] --> [decoder] --> [adaptive aubuf] --> [speaker]

Question: Doesn't the decoder need the RTP packets in order?

Ref: baresip/baresip#1659

[sreimers]

From my observations and tests reordered packets are very rare. I think only for PLC/Opus FEC, pakets have to be in order but also Opus can only handle one lost paket at a time. So we can keep an optional small RTP reorder buffer with 2-3 pakets for PLC/FEC and handle jitter (network spikes etc.) within adaptive aubuf.

A more complicated approach could be this:

[cspiel1]

• The PLC could be added in a second step.
• The jitter computation and adaptive algorithm of jbuf could be a reference for the adaptive aubuf.

[sreimers]

Another problem in my opinion is "software" jitter. The RTP socket and decoder are running in re_main thread. So any timer or FD (network etc.) callback introduces jitter. Maybe we can run the RTP fd_listen in a separate worker re thread (re_thread_init()). For sending pakets we already have a separate thread (audio_txmode thread).

[cspiel1]

The receiving of RTP, decoding and pushing to the sound card has real time constraints. The fd_listen()/re main loop does not support higher ranking of individual file descriptors. Should we add a second epoll loop, a "real-time" loop?

[cspiel1]

Then we could give the RX audio processing thread a higher priority.

I have some commits for RX audio and will select those that make sense independent of your aubuf plans or those that could be helpful to be discussed/merged before.

[sreimers]

I think i will start by implementing @alfredh `s suggested audio pipeline baresip/baresip#1443 (comment)

This way we can cleanup a bit first (move resampling to module) and have a more flexible pipeline. Maybe PLC/FEC can integrated as aufilt module. And we can keep the core code nice and clean.

[cspiel1]

Another kind of jitter is coming from audio buffer output. How steady are audio frames read. E.g. by the pulse player, alsa player.

I am currently investigating the "output jitter" for different audio players. Here is a branch for ajb.c:
https://github.com/cspiel1/rem/commit/8e3be023098d8d2fc8465d27f0975117df74c144

• pulse.so: about 2ms jitter
• alsa.so: about 5ms jitter

Currently we are evaluating a new audio module for pulseaudio that uses the native (async) API. The jitter is better because we set the minimum request size of pulseaudio to 1/4 of ptime.

• pulse_async.so: about 500us jitter

These jitter values are relatively low. So I am not sure if it is worth to add these extra code for the output jitter. TODO: More tests with full CPU load, e.g. full HD video + Opus audio SIP calls on imx6.

[sreimers]

If the re_printf is also involved, this alone can lead to ~5ms jitter itself. I have added the TRACE API for debug time critical stuff.