Toggle Spectrogram Preview for Audio

[Path-A]

Is your feature request related to a problem? Please describe.
Classifying or segmenting audio with only a waveform preview can be time-consuming or difficult, especially with noisy audio data. Some data is more easily segmented by looking at frequency content over time.

Describe the solution you'd like
Include a toggle to preview a spectrogram representation of an audio clip. Some common python libraries to generate these are Librosa or Scipy.signal.

Describe alternatives you've considered
I've manually generated the spectrograms and saved them as images to be used within the image classification labeling tool. The downsides of this are threefold.

1. Labeling audio this way does not allow for temporal segmentation. The user must classify the entire spectrogram, not simply a vertical fraction of it. A user could, in theory, use the image annotation tool, but it would be tedious and the user would need to convert bounding boxes to its corresponding time in the audio clip.
2. The user can no longer listen to the audio clip while viewing the spectrogram image.
3. The user generated spectograms require temporary additional storage requirements.

Additional context
Each user's spectrogram needs may differ, such as their sound of interest being within the low or high frequency areas of the spectrogram. To keep implementation simple, use default spectrogram parameters that generalize well and potentially allow users to zoom in on this general spectrogram. A more robust solution would allow the user to specify a few parameters to generate the spectrogram that they would want. Lastly, I include an example of a log-scaled spectrogram with its accompanying waveform.

[makseq]

@Path-A Thank you for your issue. Are you familiar with React?

[Path-A]

@makseq Unfortunately not, although I've been wanting to learn.

[feddybear]

For future reference, I think the challenge here is more of understanding the multicanvas code of wavesurfer.
I was able to successfully use the spectrogram functions but could only implement it on the older single canvas implementation. Unfortunately, for long audio files, this is impractical because of the need to recalculate and redraw spectra when zooming, etc. If pre-segmentation is done, it becomes more practical. But segmentation of audio (especially for speech technology applications) isn't perfect.
Here's a sample demonstration as reference: LINK
It takes about 12-14 seconds per zoom value on a 3-minute file (using N=512 fft samples).

[makseq]

@feddybear Wow! It's very impressive! Do you have an account on our slack? https://label-studio.slack.com/

[feddybear]

@feddybear Wow! It's very impressive! Do you have an account on our slack? https://label-studio.slack.com/

Hi @makseq yeah I also mentioned this on one of the spectrogram inquiries there. But I'm leaving it to someone more capable, especially in reading the wavesurfer multicanvas codes. Hopefully it's also someone who knows signal processing, as the older implementation of drawing spectrogram on wavesurfer had some really weird canvas settings that didn't make sense (e.g. height of the spectrogram).

[makseq]

@feddybear Let's move to slack. I know DSP, also we'll include our frontend team there.

[makseq]

@feddybear please, tag me again (@makseq). I can't find my mention there.

[Tom-Lu]

I'm also seeking for similar feature, any progress so far?

[makseq]

@Tom-Lu
We have some news from our contributor:
https://github.com/feddybear/label-studio-frontend

I hope we will make this work to the end.

[tpeet]

Would be also very interested in this feature, as it is currently hard to select regions for audio with low SNR

[Path-A]

Has there been any progress regarding development of this feature?

[makseq]

Only if @feddybear has any news.
We are currently focusing on the image / html tagging. Audio updates are planned for next year.

[feddybear]

Sorry, I have yet to integrate the spectrogram-related edits from the previous version to the latest one. Also, kinda occupied with other stuff outside of annotation.

[Selimonder]

Hello,

Has there been any progress regarding the development of this feature?

[mikolajpabiszczak]

I know it's irritating with people asking again & again, but it would be really useful to have this, any news about this?

[makseq]

Thank you for asking. By your activity we prioritize features, so it isn't irritating :-)
@nicholasrq had some progress in Audio Plus Engine, but I heard that we haven't still implemented spectrograms :-( I will draw the attention of our team to this feature request.

[faroit]

@makseq @feddybear also 👍 for spectrograms!

[mcgee0916]

Hello,
Has there been any progress about this feature?

[cspindler]

👍 Yes, spectrogram annotation would be fantastic. But it would be just the start - with the spectrogram view available, the following features would be super useful:

• RectangleLabels in the spectrogram (time and frequency range)
• Audio playback speed control (with and without time-stretching), especially slowing down to 0.1 of original speed.
• Choices of frequency scaling (lin, log, mel, bark)
• Zoom (time and frequency)

I'm not familiar with React at all, but that could change.

We're building a pipeline to annotate bats in high-frequency audio recordings, based on batdetect2. They also have a labelling UI that checks many of the boxes UI-wise, except the handling of large amount of tasks, users, storage backends etc. - all the golden labelstud.io features.

[samvelkoch]

Up for specs in audio labeling

[sajarin]

/jira create

Workflow run
Jira issue TRIAG-527 is created