Feedback on audio capture stats

[padenot]

Being able to know the input latency is a welcome change, but there are a number of problems with the prose:

totalSamplesCaptured: this should be named totalFramesCaptured, to handle multichannel and to be consistent with the rest of the Web Platform. A frame can contain multiple samples (one per audio channel).

droppedSamplesDuration: it's unclear what this means. When does a capture device drop audio? It is because of a catastrophic real-time load problem? A bug in the implementation?

The frequency of the media source is necessarily the same as the frequency of encoders later in pipeline.

This is using incorrect terminology, I think this is referring to the sampling-rate or sampling-frequency, not the frequency.

[henbos]

Samples rather than frames were chosen for consistency with other metrics (we consistently use samples rather than frames when it comes to audio, e.g. totalSamplesReceived). This might not be ideal in hindsight, but it is addressed in the Terminology section:

An audio sample refers to having a sample in any channel of an audio track - if multiple audio channels are used, metrics based on samples do not increment at a higher rate, simultaneously having samples in multiple channels counts as a single sample.

When does a capture device drop audio?

If the browser has a polling based model it may not be able to keep up with all the samples in all circumstances, though I am not the audio expert here. I can try to figure out more information about this if you like

[henbos]

Asked the audio experts, they said "The capture device is a realtime device and if the browser does not process them in a timely manner it has to drop them"

[padenot]

Right, so this is when a "catastrophic real-time load problem" occurs, which makes sense.

We need to revert this and have something on the MediaStream that represents an audio input device, or something like this.

It's not desirable to have coupling between the audio input system and WebRTC, and it's best to use the terms used in the industry and the rest of the Web Platform. gUM is frequently used without anything related to WebRTC, and knowing the input latency is capital in those scenarios (e.g., anything related to synchronizing recordings).

[henbos]

Having media source metrics is not new and having measurements of glitches seem to fit well together with existing sample counters and audio energy metrics that we already get from the track and expose via getStats.

That said, general solutions are nice to have. Historically it has been difficult to add any metrics to the MediaStreamTrack interface, but given the recent extension API MediaStreamTrack.getFrameStats() (which expose similar counters to media-source.frames, but also counters for frame drops and don't require attachment to WebRTC) I could certainly imagine something like track.getAudioStats() providing information about sample drops.

[vr000m]

Linking to semi related for multi-channel, stereo metrics #686

[dontcallmedom-bot]

This issue was mentioned in WebRTC WG 2023-04-18 – (Issue 93: MediaStreamTrack audio delay/glitch capture stats (Henrik))

[henbos]

This work was moved to the track stats (webrtc-extension) spec, if that PR can merge then we should delete the stats from this document