Skip to content

Commit

Permalink
Update benchmarks.html
Browse files Browse the repository at this point in the history
  • Loading branch information
@mravanelli
mravanelli authored Dec 9, 2024
1 parent 628437d commit f316ffb
Showing 1 changed file with 79 additions and 24 deletions.
103 changes: 79 additions & 24 deletions benchmarks.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,30 +74,85 @@ <h2> 📊 Available Benchmarks</h2>
The following benchmarks are currently available:
</p>
<br>
<h3> 🌟 Zen of SpeechBrain </h3>
<p class="justified large"> SpeechBrain could be used for research, academic, commercial, non-commercial purposes.If you want to contribute, keep in mind the following features:
<br/><br/>
<b>Simplicity:</b> the code must be easy to understand even by students or users that are not professional programmers or speech researchers.
Design your code such that it can be easily read. Given alternatives with the same level of performance, code the simplest one. <br/><br/>
<b>Modularity:</b> Write your code to be modular and well-fitting with the other functionalities of the toolkit. The idea is to develop a bunch of models that can be naturally interconnected with each other. <br/><br/>
<b>Efficiency:</b> The code should be as efficient as possible. Contributors should maximize the use of pytorch native operations
<br/><br/>
<b>Documentation:</b> Given the goals of SpeechBrain, writing rich and good documentation is a crucial step. Write docstrings with runnable examples (as done in PyTorch code).
<br/><br/>
<h3> 🔧 How to get my code into SpeechBrain? </h3>
<p class="justified large"> SpeechBrain is hosted via <a href="https://github.com/speechbrain/speechbrain"> GitHub </a>. Contributing requires three steps: <br/><br/>
<b>1.</b> Fork, clone the <a href="https://github.com/speechbrain/speechbrain">repository</a> and install our test suite as detailed in <a href="https://speechbrain.readthedocs.io/en/latest/contributing.html"> the documentation </a>. <br/>
<b>2.</b> Write your code and test it properly. Commit your changes to your fork with our pre-commit tests to ensure tests are passing.
Then open a pull request on the official repository.<br/>
<b>3.</b> Participate in the review process. Each pull request is reviewed by one or two reviewers.
Please integrate their feedback into your code. Once reviewers are happy with your pull request, they will merge it into the official code. <br/><br/>
<b>Details about this process (i.e including steps for installing the tests) are given in <a href="https://speechbrain.readthedocs.io/en/latest/contributing.html"> the documentation </a>.</b>
</p>
<br/><br/>
<h3> 🙌 How can I help? </h3>
<p class="justified large"> Examples of contributions include new recipes, new models, new external functionalities, solving issues/bugs.
</p>
</div>
<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MOABB" target="_blank">
<img src="logo_image_url_here" alt="SpeechBrain-MOABB Logo" style="width: 50px; vertical-align: middle; margin-right: 10px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MOABB" target="_blank">SpeechBrain-MOABB</a> is an open-source Python library for benchmarking deep neural networks applied to EEG signals. <br/><br/> This repository provides a set of recipes for processing electroencephalographic (EEG) signals based on the popular <a href="https://github.com/NeuroTechX/moabb" target="_blank">Mother of all BCI Benchmarks (MOABB)</a> , seamlessly integrated with SpeechBrain.
<br/><br/>
This package facilitates the integration and evaluation of new algorithms (e.g., a novel deep learning architecture or a novel data augmentation strategy) in standardized EEG decoding pipelines based on MOABB-supported tasks, i.e., motor imagery (MI), P300, and steady-state visual evoked potential (SSVEP).
<br/><br/>
It not only offers an interface for easy model integration and testing but also proposes a fair and robust protocol for comparing different decoding pipelines, fully described in our paper:
<ul>
<li>
Davide Borra, Francesco Paissan, and Mirco Ravanelli. <i>SpeechBrain-MOABB: An open-source Python library for benchmarking deep neural networks applied to EEG signals.</i> Computers in Biology and Medicine, Volume 182, 2024. <a href="https://www.sciencedirect.com/science/article/pii/S001048252401182X" target="_blank">[Paper]</a>
</li> <br/><br/>
<li>
Davide Borra, Elisa Magosso, and Mirco Ravanelli. <i>https://www.sciencedirect.com/science/article/pii/S0893608024007718</i> Neural Networks, Page 106847, 2024. <a href="https://www.sciencedirect.com/science/article/pii/S001048252401182X" target="_blank">[Paper]</a>
</li>
</ul> <br><br>


<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/DASB" target="_blank">
<img src="logo_image_url_here" alt="DASB Logo" style="width: 50px; vertical-align: middle; margin-right: 10px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/DASB" target="_blank">DASB - Discrete Audio and Speech Benchmark</a> is a benchmark for evaluating discrete audio representations using popular audio tokenizers like EnCodec, DAC, and many more, integrated with SpeechBrain.
<br><br>
The package helps integrate and evaluate new audio tokenizers in speech tasks of great interest such as <i>speech recognition</i>,  <i>speaker identification</i>, <i>emotion recognition</i>, <i>keyword spotting</i>, <i>intent classification</i>, <i>speech enhancement</i>, <i>separation</i>, <i>text-to-speech</i>, and many more.
<br><br>
It offers an interface for easy model integration and testing and a protocol for comparing different audio tokenizers.

<ul>
<li>
Pooneh Mousavi, Luca Della Libera, Jarod Duret, Arten Ploujnikov, Cem Subakan, Mirco Ravanelli,
<em>DASB - Discrete Audio and Speech Benchmark</em>, 2024
arXiv preprint arXiv:2406.14294.
</li> <a href="https://arxiv.org/abs/2406.14294" target="_blank">[Paper]
</ul> <br><br>


<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/CL_MASR" target="_blank">
<img src="logo_image_url_here" alt="CL-MASR Logo" style="width: 50px; vertical-align: middle; margin-right: 10px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/CL_MASR" target="_blank">CL-MASR</a>
is a Continual Learning Benchmark for Multilingual ASR.
<br><br>
It includes scripts to train Whisper and WavLM-based ASR systems on a subset of 20 languages selected from Common Voice 13 in a continual learning fashion using a handful of methods including rehearsal-based, architecture-based, and regularization-based approaches.
<br><br>
The goal is to continually learn new languages while limiting forgetting the previously learned ones.

<br><br>
An ideal method should achieve both positive forward transfer (i.e. improve performance on new tasks leveraging shared knowledge from previous tasks) and positive backward transfer (i.e. improve performance on previous tasks leveraging shared knowledge from new tasks).

<ul>
<li>
Luca Della Libera, Pooneh Mousavi, Salah Zaiem, Cem Subakan, Mirco Ravanelli, (2024). CL-MASR: A continual learning benchmark for multilingual ASR. <i>IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32</i>, 4931–4944.

</li> <a href="https://arxiv.org/abs/2310.16931" target="_blank">[Paper]
</ul> <br><br>


<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MP3S" target="_blank">
<img src="logo_image_url_here" alt="MP3S Logo" style="width: 50px; vertical-align: middle; margin-right: 10px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MP3S" target="_blank">MP23 - Multi-probe Speech Self Supervision Benchmark</a> aims to evaluate self-supervised representations on various downstream tasks, including <i>ASR</i>, <i>speaker verification</i>, <i>emotion recognition</i>, and <i>intent classification</i>.
<br><br>
The key feature of this benchmark is that it allows users to choose their desired probing head for downstream training.
<br><br>
This is why we called it the Multi-probe Speech Self Supervision Benchmark (MP3S). It has been demonstrated that the performance of the model is greatly influenced by this selection
<br><br>


<ul>
<li>
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, Mirco Ravanelli, (2023). Speech Self-Supervised Representation Benchmarking: Are We Doing it Right? <i>Proceedings of Interspeech 2023</i>

</li> <a href="https://arxiv.org/abs/2306.00452" target="_blank">[Paper]</a>
<br><br>
<li>
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, Mirco Ravanelli, (2023). Speech self-supervised representations benchmarking: a case for larger probing heads. <i>Computer Speech & Language, 89</i>, 101695.</i>

</li> <a href="https://www.sciencedirect.com/science/article/pii/S0885230824000780" target="_blank">[Paper]
</ul> <br><br>


</section>
Expand Down

0 comments on commit f316ffb

Please sign in to comment.