-
Notifications
You must be signed in to change notification settings - Fork 29
/
Copy pathbenchmarks.html
175 lines (161 loc) · 11.5 KB
/
benchmarks.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
<!doctype html>
<html lang="en">
<head>
<!-- Required meta tags -->
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="icon" href="img/favicon.ico">
<title>How to Contribute</title>
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="css/bootstrap.css">
<link rel="stylesheet" href="vendors/linericon/style.css">
<link rel="stylesheet" href="css/font-awesome.min.css">
<link rel="stylesheet" href="vendors/owl-carousel/owl.carousel.min.css">
<link rel="stylesheet" href="vendors/lightbox/simpleLightbox.css">
<link rel="stylesheet" href="vendors/nice-select/css/nice-select.css">
<link rel="stylesheet" href="vendors/animate-css/animate.css">
<!-- main css -->
<link rel="stylesheet" href="css/style.css">
<link rel="stylesheet" href="css/responsive.css">
</head>
<body>
<!--================Header Menu Area =================-->
<header class="header_area">
<div class="main_menu">
<nav class="navbar navbar-expand-lg navbar-light">
<div class="container box_1620">
<!-- Brand and toggle get grouped for better mobile display -->
<a class="navbar-brand logo_h" href="index.html"><img src="img/speechbrain-horiz-logo.svg" width="175px" alt=""></a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse offset" id="navbarSupportedContent">
<ul class="nav navbar-nav menu_nav justify-content-center">
<li class="nav-item active"><a class="nav-link" href="index.html">Home</a></li>
<li class="nav-item"><a class="nav-link" href="about.html">About SpeechBrain</a></li>
<li class="nav-item"><a class="nav-link" href="contributing.html">Contributing</a></li>
<li class="nav-item"><a class="nav-link" href="https://speechbrain.readthedocs.io/">Documentation</a></li>
<li class="nav-item"><a class="nav-link" href="https://speechbrain.readthedocs.io/">Tutorials</a></li>
<li class="nav-item"><a class="nav-link" href="benchmarks.html">Benchmarks</a></li>
</ul>
</div>
</div>
</nav>
</div>
</header>
<!--================Header Menu Area =================-->
<!--================Home Banner Area =================-->
<section class="home_banner_area blog_banner">
<div class="banner_inner d-flex align-items-center">
<div class="overlay bg-parallax" data-stellar-ratio="0.9" data-stellar-vertical-offset="0" data-background=""></div>
<div class="container">
<div class="blog_b_text text-center">
<h2>Benchmarks</h2>
<h3>Advancing Reproducibility and Transparency in Research</h3>
</div>
</div>
</div>
</section>
<!--================End Home Banner Area =================-->
<div class="alert alert-primary" role="alert" style="text-align: center; font-size: 24px;">
🎉 SpeechBrain 1.0 is out. <a href="https://colab.research.google.com/drive/1IEPfKRuvJRSjoxu22GZhb3czfVHsAy0s?usp=sharing"><strong> Check out what's new!</strong></a>
</div>
<!--================Work Area =================-->
<section class="work_area p_120">
<div class="main_title">
<h2> 📊 Available Benchmarks</h2>
<p class="justified large"> SpeechBrain aims to promote transparent and reproducible research.<br><br>
To advance this mission, we develop benchmarks that help researchers conduct fair, robust, and standardized performance comparisons. We have created a dedicated <a href="https://github.com/speechbrain/benchmarks"> repository </a> for this purpose. <br><br>
The following benchmarks are currently available:
</p>
<br>
<hr class="separation-line">
<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MOABB" target="_blank">
<img src="img/benchmarks/sb-moabb-logo.svg" alt="SpeechBrain-MOABB Logo" style="width: 500px; vertical-align: middle; margin-right: 100px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MOABB" target="_blank">SpeechBrain-MOABB</a> is an open-source Python library for benchmarking deep neural networks applied to EEG signals. <br/><br/> This repository provides a set of recipes for processing electroencephalographic (EEG) signals based on the popular <a href="https://github.com/NeuroTechX/moabb" target="_blank">Mother of all BCI Benchmarks (MOABB)</a> , seamlessly integrated with SpeechBrain.
<br/><br/>
This package facilitates the integration and evaluation of new algorithms (e.g., a novel deep learning architecture or a novel data augmentation strategy) in standardized EEG decoding pipelines based on MOABB-supported tasks, i.e., motor imagery (MI), P300, and steady-state visual evoked potential (SSVEP).
<br/><br/>
<b>Reference Papers:</b>
<br/><br/>
Davide Borra, Francesco Paissan, and Mirco Ravanelli. <i>SpeechBrain-MOABB: An open-source Python library for benchmarking deep neural networks applied to EEG signals.</i> Computers in Biology and Medicine, Volume 182, 2024. <a href="https://www.sciencedirect.com/science/article/pii/S001048252401182X" target="_blank">[Paper]</a>
<br/><br/>
Davide Borra, Elisa Magosso, and Mirco Ravanelli. <i>Neural Networks, Page 106847, 2024. <a href="https://www.sciencedirect.com/science/article/pii/S001048252401182X" target="_blank">[Paper]</a>
<br><br>
<hr class="separation-line">
<a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/DASB" target="_blank">
<img src="img/benchmarks/DASB_logo.png" alt="DASB Logo" style="width: 250px; vertical-align: middle; margin-right: 250px;">
</a>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/DASB" target="_blank">DASB - Discrete Audio and Speech Benchmark</a> is a benchmark for evaluating discrete audio representations using popular audio tokenizers like EnCodec, DAC, and many more, integrated with SpeechBrain.
<br><br>
The package helps integrate and evaluate new audio tokenizers in speech tasks of great interest such as <i>speech recognition</i>, <i>speaker identification</i>, <i>emotion recognition</i>, <i>keyword spotting</i>, <i>intent classification</i>, <i>speech enhancement</i>, <i>separation</i>, <i>text-to-speech</i>, and many more.
<br><br>
It offers an interface for easy model integration and testing and a protocol for comparing different audio tokenizers.
<br><br>
<b>Reference Paper:</b>
<br/><br/>
Pooneh Mousavi, Luca Della Libera, Jarod Duret, Arten Ploujnikov, Cem Subakan, Mirco Ravanelli,
<em>DASB - Discrete Audio and Speech Benchmark</em>, 2024
arXiv preprint arXiv:2406.14294.
<a href="https://arxiv.org/abs/2406.14294" target="_blank">[Paper]
<br><br>
<hr class="separation-line">
<h3>CL-MASR: Continual Learning for Multilingual ASR</h3>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/CL_MASR" target="_blank">CL-MASR</a>
is a Continual Learning Benchmark for Multilingual ASR.
<br><br>
It includes scripts to train Whisper and WavLM-based ASR systems on a subset of 20 languages selected from Common Voice 13 in a continual learning fashion using a handful of methods including rehearsal-based, architecture-based, and regularization-based approaches.
<br><br>
The goal is to continually learn new languages while limiting forgetting the previously learned ones.
<br><br>
An ideal method should achieve both positive forward transfer (i.e. improve performance on new tasks leveraging shared knowledge from previous tasks) and positive backward transfer (i.e. improve performance on previous tasks leveraging shared knowledge from new tasks).
<br><br>
<b>Reference Paper:</b>
<br/><br/>
Luca Della Libera, Pooneh Mousavi, Salah Zaiem, Cem Subakan, Mirco Ravanelli, (2024). CL-MASR: A continual learning benchmark for multilingual ASR. <i>IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32</i>, 4931–4944.
<a href="https://arxiv.org/abs/2310.16931" target="_blank">[Paper]
<br><br>
<hr class="separation-line">
<h3>MP23: Multi-probe Speech Self Supervision Benchmark</h3>
<p class="justified large"> <a href="https://github.com/speechbrain/benchmarks/tree/main/benchmarks/MP3S" target="_blank">MP23 - Multi-probe Speech Self Supervision Benchmark</a> aims to evaluate self-supervised representations on various downstream tasks, including <i>ASR</i>, <i>speaker verification</i>, <i>emotion recognition</i>, and <i>intent classification</i>.
<br><br>
The key feature of this benchmark is that it allows users to choose their desired probing head for downstream training.
<br><br>
This is why we called it the Multi-probe Speech Self Supervision Benchmark (MP3S). It has been demonstrated that the performance of the model is greatly influenced by this selection
<br><br>
<b>Reference Papers:</b>
<br/><br/>
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, Mirco Ravanelli, (2023). Speech Self-Supervised Representation Benchmarking: Are We Doing it Right? <i>Proceedings of Interspeech 2023</i>
<a href="https://arxiv.org/abs/2306.00452" target="_blank">[Paper]</a>
<br><br>
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, Mirco Ravanelli, (2023). Speech self-supervised representations benchmarking: a case for larger probing heads. <i>Computer Speech & Language, 89</i>, 101695.</i>
<a href="https://www.sciencedirect.com/science/article/pii/S0885230824000780" target="_blank">[Paper]
<br><br>
<hr class="separation-line">
</section>
<!--================End Work Area =================-->
<!-- Optional JavaScript -->
<!-- jQuery first, then Popper.js, then Bootstrap JS -->
<script src="js/jquery-3.2.1.min.js"></script>
<script src="js/popper.js"></script>
<script src="js/bootstrap.min.js"></script>
<script src="js/stellar.js"></script>
<script src="vendors/lightbox/simpleLightbox.min.js"></script>
<script src="vendors/nice-select/js/jquery.nice-select.min.js"></script>
<script src="vendors/isotope/imagesloaded.pkgd.min.js"></script>
<script src="vendors/isotope/isotope-min.js"></script>
<script src="vendors/owl-carousel/owl.carousel.min.js"></script>
<script src="js/jquery.ajaxchimp.min.js"></script>
<script src="js/mail-script.js"></script>
<script src="vendors/counter-up/jquery.waypoints.min.js"></script>
<script src="vendors/counter-up/jquery.counterup.min.js"></script>
<script src="js/theme.js"></script>
<link rel="stylesheet" href="js/styles/monokai-sublime.css">
<script src="js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
</body>
</html>