Video delay, and visualisations (and playback speed) #750
Comments
RJVB
changed the title
|
Bluetooth speakers has variable audio latency - depends on codec and on signal strength - low signal might increase latency because of larger buffer to prevent audio stuttering (and also can lower the bitrate). Use cable whenever possible (many BT speakers has JACK input). With cable you're always full quality, lowest and fixed latencies and you can see your "signal" range. Visualizations doesn't respect video latency - I don't want to buffer it there and complicate things. It's not video anyway 😄 About playback speed - I don't know, but I think when speed is too low, the latency doesn't matter that much. When speed is too high, you should still hear the latency. Also rubberband filter has latency, but it should be implicitly added to video delay. |
Well, DOH, but a cable isn't always feasible, is it. Can't just span one the middle of a (kitchen) table across the space to where my speaker sits...
For visualisations it's one thing, for the FFT spectrum another, IMHO. I don't know how you implemented the former, but I suspect you have to buffer for the latter anyway.
I don't see how that could make sense, unless the slowing down introduces its own delay w.r.t. the audio. Which might be the case since there is indeed a noticeable sync problem when I play the audio of a slowed-down video over the builtin speakers.
Is the rubberband filter always in the loop, with just an on/off switch to determine whether it does something to pitch? Because that noticeable sync problem mentioned above is there whether I keep audio pitch constant or not. I think I had actually noticed it before, as the slowed-down playback isn't as helpful in figuring out what musicians are doing with their hands (which is what I hoped to use the constant-pitch feature for). I'm attaching the audio/video sync test I use - and just FWIW: visualisations and FFT appear to be completely out of sync with the audio for me. Audio.Video.Sync.Test.60.FPS.mp4 |
|
For me this video plays perfect in QMPlay2 regardless of keep pitch and playback speed (using PC's sound card and pipewire).
Rubberband filter is running only when your playback speed != 1.0 and you have keep pich enabled. Otherwise it's unused, resources are freed.
How about video? As I told before - video delay parameter doesn't work in visualizations .
What do you mean? The only buffer there is to grab enough samples to display and/or calculate FFT. They should display at same time (I mean some milliseconds doesn't matter like 1-2x screen refresh rate or so). |
|
For me this video plays perfect in QMPlay2 regardless of keep pitch and playback speed (using PC's sound card and pipewire).
I suppose your PC is a lot faster than mine, but is there a reason to expect that slowing down the video increases computational load such that video playback becomes delayed w.r.t. audio playback? On the Linux system where I observed the sync issues I actually have to dial in a negative video delay when I slow down playback to 0.25x!
I'll see what happens on my Mac.
> For visualisations it's one thing, for the FFT spectrum another, IMHO.
What do you mean?
I consider visualisations mostly eye candy, but the FFT spectrum is supposed to correspond to what you're hearing. Showing it for something else doesn't really make sense (not much more than showing a random spectrum).
The only buffer there is to grab enough samples to display and/or calculate FFT.
Let's just stick to the situation where the video delay setting is used to dial in a transport delay in the audio playback (so it's a positive delay). In that case, can't you simply fill the buffer for calculating the FFT with audio samples from X seconds ago? Don't you keep any decoded audio data after it has been handed off to the audio device?
They should display at same time (I mean some milliseconds doesn't matter like 1-2x screen refresh rate or so).
Who are "they" here?
( 1 screen refresh is already almost 17 ms for a 60Hz display ;) )
|
|
On Friday July 12 2024 13:12:52 Błażej Szczygieł wrote:
For me this video plays perfect in QMPlay2 regardless of keep pitch and playback speed (using PC's sound card and pipewire).
Basically idem on my Mac. At some point I thought I noticed a slight offset at 0.25x speed but couldn't reproduce that.
A few important differences of course:
- VTB acceleration rather than VAAPI
- OpenGL rendering instead of Vulkan
- "direct" audio rendering rather than via Pulseaudio.
I did manage to build Pipewire on my Linux rig but haven't yet figured out how to start it instead of Pulseaudio. (I tried once to launch it in addition to assuming the daemons shouldn't bite but it messed up the sound volume levels until I restarted the PA daemon.)
Did you notice that QMPlay2 identifies the Twitch video as 120Hz in the Info panel instead of the actual 60Hz? `ffprobe` shows it's also a "120 tbr" (??) video track; maybe that's where the confusion comes from?
|
|
Notes about Bluetooth audio: yes, this method has variable audio latency (depends on devices, codec used, signal strength). You can compensate it manually for video playback in QMPlay2, but not for visualizations. Visualizations in QMPlay2 doesn't have additional buffering, so it's impossible. Adding this is possible, but too much work and too much code for this use-case. Sorry, I can't complete every feature request 😞 Also bluetooth audio latency can change at "random" in some cases, so you'll have to compensate it again manually. Once I had two Bluetooth mono speakers (no JACK cable input...) and I wanted to use them as stereo. After manual latency synchronization between them (in PipeWire), they played quite well. After some time they desynced and echo appeared again. |
|
If memory serves me right the main thing here is the fact that I often lose some sound at the beginning of a video started with latency compensation, esp. noticeable in music videos where the music starts at the 1st frame. It's as if the audio is being delayed, which is the opposite of what should happen.
Or is this related to bluetooth too? IOW, would it help to "prime" the audio channel maybe 1s before playback starts - which could be hidden easily in the network delays when starting a YT video?
|
|
Hmm. I experience it on HDMI when HDMI is in... power-saving mode? One it wakes-up, it plays normally, but usually I lost begin of audio. Maybe same with bluetooth? Is your issue related to QMPlay2 or it's a general issue? |
|
I have the impression it's more noticeable when playing YT videos using QMPlay2's latency correction. In this video e.g. (https://www.youtube.com/watch?v=soDuDgnqjE4) the 1st note swells into existence after a short initial silence. When I just start the playback with +0.3s latency it starts abruptly, also when I stop and restart the video so nothing is supposedly gone into standby. But when I just drag the cursor back to the beginning I do hear that initial silence and the swelling of the note.
I'm still using Pulseaudio and don't even know if the latency you can configure there (via e.g. pavucontrol) actually does something - it certainly doesn't interfere with QMplay2's correction (QP2 playback uses an "ALSA Plugin"). I'm also using a USB BT DAC (Sennheiser BTD600) which may add its own side-effects?)
|
I think for PulseAudio you should use PulseAudio plugin. ALSA is for
I forgot about it long time ago and my sound issues are gone with PipeWire.
Exactly same here on HDMI regardless of player used. I noticed that the latency when audio starts to be hear depends on connected hardware (sometimes is 1 second, sometimes more). I rarely use HDMI, so workaround is unknown so far. If you add additional latency via pavucontrol, you simply has more time to "wait" for the audio device to be ready to output sound. I still don't know if it's hardware issue or driver issue. |
QMPlay2 corrects A/V sync only. You need to add about 0.5 seconds silence on playback start (when your audio device is open or resumed) to workaround hardware/driver issue. |
|
On Saturday December 14 2024 15:47:05 Błażej Szczygieł wrote:
> it certainly doesn't interfere with QMplay2's correction
QMPlay2 corrects A/V sync only.
Hmmm, of course, I was thinking of additional compensation added by downstream layers. But a transport delay in the audio channel can only be compensated by delaying the video and I don't see how PA could do that in an application that handles video playback itself.
I keep the latency correction configured via pavucontrol because it seems this improves sync in in-browser content playback - possibly because the browser queries the value from PA and adds the corresponding delay to the appropriate channel?
I have a vague memory that I kept QMPlay2 set to ALSA/PA Sound Server because the PA ALSA plugin ignores the latency setting, but the above explanation seems more plausible.
Still, after some more testing right now it does seem like the PA latency offset is to blame?!
You need to add about 0.5 seconds silence on playback start (when your audio device is open or resumed) to workaround hardware/driver issue.
Play 0.5s of silence (more or less what I called priming) before actually starting the content playback? That's currently not possible to achieve, is it?
And that still doesn't explain why the phenomenon is more noticeable when correcting for an audio transport delay! It seems to me that you shouldn't even notice the correction when the latency is set correctly (and when you switch to the internal audio device with 0 latency you should see a short delay in the video). Again, right now it's more like the video starts playing without sound, and then all of a sudden sound starts.
Same with a negative or a positive PA latency offset, and that just doesn't seem to make sense.
I do have pipewire installed nowadays, including its pulseaudio drop-in replacement. I haven't yet figured out how to start it correctly; the one time I did try it messed up the max. volume I could get through PA when I went back to that.
|
Latency in pavucontrol shouldn't be used for your issue. It may generate other issues. Found solution, it's here for PipeWire with WirePlumber >= 0.5.0. You shouldn't set it for all devices, just set for your device (you can use My config for HDMI ( Your first audio playback in system will be cut, but every other will not cut the sound. |
|
On Sunday December 15 2024 00:57:36 Błażej Szczygieł wrote:
> Still, after some more testing right now it does seem like the PA latency offset is to blame?!
Latency in pavucontrol shouldn't be used for your issue. It may generate other issues.
I agree, at least not for compensating an audio transport delay by adding yet more delay ;) But it clearly does do something. Guess I'll have to RTFM ;)
Found solution, it's [here](https://wiki.archlinux.org/title/PipeWire#Noticeable_audio_delay_or_audible_pop/crack_when_starting_playback) for PipeWire with WirePlumber >= 0.5.0.
You shouldn't set it for all devices, just set for your device (you can use `pw-top` to obtain the name) or maybe set the config for bluetooth devices only. `pw-top` will show you is it suspends or not.
Worth a try if I can figure out how to switch to pipewire without breaking anything, including my PulseAudio config.
Meanwhile, my BT device is seen as a USB DAC by the system, and the LEDs on it go a bit duller after about 5 seconds of not receiving any signal.
Also, I can reproduce the issue with the internal sound device too. Not for small latency offsets nor for long ones >= 900ms or so, so it's not something like a glitch in a different code path that's taken when lat.offset!=0 .
Your first audio playback in system will be cut, but every other will not cut the sound.
Sorry, I can't make sense of that. You mean the 1st time I play audio, or the 1st device in the list?
|
Yes, this is probably the reason why you can't hear begin of the audio - the device is waking-up from sleep. If it's a USB DAC, just set
🤔 Don't understand.
I mean - when you start any playback on your device when suspend is disabled in PipeWire - the device will not go sleep anymore, so it'll work. However the device has to be accessed at least once to wake-up, so first audio you play after device connection or PC boot - begin of the audio will not be emited (only once). Also you can try (not working for HDMI to me): echo 0 | sudo tee /sys/module/snd_hda_intel/parameters/power_save
echo 0 | sudo tee /sys/module/snd_hda_intel/parameters/power_save_controller |
|
On Sunday December 15 2024 07:13:32 Błażej Szczygieł wrote:
Yes, this is probably the reason why you can't hear begin of the audio - the device is waking-up from sleep. If it's a USB DAC, just set `node.name` according to what `pw-top` sees on your device.
I'm not convinced, because I can see the device wake up while QMPlay2 still seems to be starting the playback. There may be the tiniest bit of an artefact when I start playback from the standby state without latency correction but it's definitely more pronounced when there is a latency correction.
Would it be an invasive change to open the configured audio channel (and send "something" into it?) as soon as the user hits play, if that's not already being done? That might fix or alleviate the issue in >75% of the cases. And I'm guessing the same issue will exist on platforms where you can't configure suspension like you showed for pipewire
Except on Mac with bitperfect playback ... that clearly wakes the audio device up :)
> Also, I can reproduce the issue with the internal sound device too. Not for small latency offsets nor for long ones >= 900ms or so, so it's not something like a glitch in a different code path that's taken when lat.offset!=0 .
:thinking: Don't understand.
You don't understand my words or what I describe?
However the device has to be accessed at least once to wake-up, so first audio you play after device connection or PC boot - begin of the audio will not be emited (only once).
Yes, that makes sense.
Also you can try (not working for HDMI to me):
```sh
echo 0 | sudo tee /sys/module/snd_hda_intel/parameters/power_save
echo 0 | sudo tee /sys/module/snd_hda_intel/parameters/power_save_controller
```
I must already have set those ("N" for the controller but that's surely the same thing), if it's not the default. Possibly when trying to minimise the delay in notification sounds.
|
On your internal sound - are you adding latency offset in pavucontrol? Do you have any issues when you don't add any latency in pavucontrol?
I don't know, I can't see/touch this 😄 Maybe it's waiting and after this still not playing from the begin. I have only HDMI, same behavior (all applications, not only QMPlay2). Sometimes it "cuts" 1 sec of audio, sometimes 4 sec of audio...
I don't know how long the silence must be sent - it depends on hardware/driver. Also it must be send on every resume from pause, because I don't know when device really suspends (maybe can ask PipeWire for suspend time, but it's not universal way and much more complicated). And finally it'll be done for QMPlay2 only, other software will still have this problem. If you just push 1 sec silence before first audio write, QMPlay2 will discard begin on audio due to video sync. |
|
On Sunday December 15 2024 10:32:55 Błażej Szczygieł wrote:
> You don't understand my words or what I describe?
On your internal sound - are you adding latency offset in pavucontrol?
I did only for testing if I could reproduce the issue that way.
Do you have any issues when you don't add any latency in pavucontrol?
Nope. And based on what you've been suggesting that's probably because the internal DAC is configured not to resume.
Also it must be send on every resume from pause, because I don't know when device really suspends
Ah, yes! I guess I must be expecting glitches when resuming from pause because the effect doesn't bother me there. There's of course also the effect that it's not trivial to reproduce the exact glitch by pausing at the exact same time.
But isn't there a way to keep the device from suspending when playback is paused?
And finally it'll be done for QMPlay2 only, other software will still have this problem.
Well, that's fine with me - I'm using QMPlay2 for low-resource, high-audio-quality playback (except on MSWin when I use Foobar2000 for audio).
If you just push 1 sec silence before first audio write, QMPlay2 will discard begin on audio due to video sync.
Even if you pretend the video also starts with the same duration of something else - say a dissolve effect of the rest screen?
|
Video needs to be informed about it somehow. Need to look at code, don't remember (also it's very old code).
😄
PipeWire metod as I gave you before (also PulseAudio can do this). QMPlay2 - need to play silence all the time when paused. Alternative - play silence somewhere in background process all the time - it'll never suspend. |
|
PipeWire metod as I gave you before (also PulseAudio can do this).
Nothing to do with delays or latencies, but I just read that PW resamples everything to a "central frequency" and from there to whatever the output device is set to.
Unless PA does the same thing that doesn't sound like a recipe for getting better output quality nor shorter latencies...?!
In short, neither system will do anything to get even close to bit-perfect transfer? I thought PA would switch to its alternate frequency (usually to 48k from 44.1k) if that allowed an integer factor resampling but it looks like it isn't doing that on my system.
|
You can bypass PipeWire and use ALSA output in QMPlay2 set to Another solution is to create config in Restart services and observe |
|
On Thursday December 19 2024 12:44:56 Błażej Szczygieł wrote:
You can bypass PipeWire and use ALSA output in QMPlay2 set to `hw:n` to use the device directly
Does that work with PW? With PA you can't unless you exclude the device you want. I think it has to do with the compulsory "monitor of XXX" devices.
Another solution is to create config in `/etc/pipewire/pipewire.conf.d/` and set array of allowed frequencies (set freq you want, you can set more than one):
```
context.properties = {
default.clock.allowed-rates = [ 48000 ]
}
I read about that, but 1) there's apparently a kernel bug that hasn't been fixed in the kernel I'm using. IIUC it has to do with recording at a different rate than playing back, which I'd never do so it may not be relevant. And 2) I haven't yet seen any indication that this would bypass to initial resampling to some internal rate.
|
|
Try this command Also open
If PW closes audio device (suspend must be enabled), you can open it using ALSA plugin using |
|
On Friday December 20 2024 09:38:47 Błażej Szczygieł wrote:
If all are the same - I guess there's no resampling from the player to the audio device.
That's what I'd hope too, but I suppose it depends what that internal frequency is - probably not 2116mHz (44.1k * 48k) ;)
If PW closes audio device (suspend must be enabled), you can open it using ALSA plugin using `hw:n,m` device in settings. It's direct access to the device, it skips PipeWire/JACK/Pulseaudio/whatever completely.
I saw that, yes. Should work like that in PA too, but I guess it's those pesky monitors that wreak havoc.
|
What driver or hardware is doing is - let's say - out of scope. I'm talking about not using PipeWire/PulseAudio for this. Or configure PipeWire/PulseAudio to use different sample rate to match your audio file (so it'll depend on driver/hardware, PipeWire will not resample). This ALSA settings works for my integrated audio device:
In this case, PipeWire is not used:
192kHz stereo samplerate in QMPlay2, the driver
I can measure high frequencies (generated by Tone generator in QMPlay2) e.g. 80kHz on JACK output, so it's really 192kHz (or around 192kHz - idk what driver/hardware is doing). |
I just noticed that visualisations do not take the configured delay into account, causing a noticeable lag when listening over my BT speaker. It's a nice speaker for the place it's in (Klipsch The One Mk2) but it uses the basic audio codec, so I have to set a "video delay" (it's actually a transport delay of course) of +0.3 seconds.
More importantly, I wonder if there's something that can be done to improve the experience at the beginning.
That latter symptom is something I cannot really get my head around when the correct transport has been set, and if I were to guess I'd say that this initial bit of "silent movie" is actually longer than those 0.3 seconds.
EDIT: something strange goes on when reducing playback speed, regardless of whether this is done keeping the audio pitch constant or not. At 0.75x speed I need to configure a transport delay of about 0.1 seconds only, and at 0.5x it has to be set to 0.
It looks like speeding up the playback doesn't require changing the transport delay though of course it becomes increasingly difficult to detect sync issues.
The text was updated successfully, but these errors were encountered: