-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
443 lines (397 loc) · 21.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
<!DOCTYPE html>
<head>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-178132094-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag() {
dataLayer.push(arguments);
}
gtag("js", new Date());
gtag("config", "UA-178132094-1");
</script>
<meta charset="UTF-8" />
<!-- <meta name="viewport" content="width=device-width, initial-scale=1" />-->
<meta name="viewport" content="width=1024" />
<title>SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/MathJax.js?config=TeX-MML-AM_CHTML">
</script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/foundation/6.4.3/css/foundation.min.css" />
<link rel="stylesheet" href="https://cdn.rawgit.com/jpswalsh/academicons/master/css/academicons.min.css" />
<script src="https://kit.fontawesome.com/b939870cfb.js" crossorigin="anonymous"></script>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/foundation/6.4.3/css/foundation.min.css">
<link rel="stylesheet" href="https://cdn.datatables.net/1.10.24/css/dataTables.foundation.min.css">
<script type="text/javascript" src="https://code.jquery.com/jquery-3.5.1.js"></script>
<script type="text/javascript" src="https://cdn.datatables.net/1.10.24/js/jquery.dataTables.min.js"></script>
<script type="text/javascript" src="https://cdn.datatables.net/1.10.24/js/dataTables.foundation.min.js"></script>
<link rel="stylesheet" href="./css/main.css" />
</head>
<body>
<nav class="navbar navbar-expand-md">
<div class="container">
<a class="navbar-brand" href="./index.html"
>SysNoise</a>
<button
class="navbar-toggler navbar-light"
type="button"
data-toggle="collapse"
data-target="#main-navigation"
>
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="main-navigation">
<ul class="navbar-nav">
<li class="nav-item">
<a class="nav-link" href="#leaderboard">Leaderboards</a>
</li>
<li>
<a class="nav-link" href="https://modeltc.github.io/systemnoise_web/MLSys2023.pdf">Paper</a>
</li>
<li>
<a class="nav-link" href="#dataset"
>Dataset</a
>
</li>
<!-- <li>-->
<!-- <a class="nav-link" href="#contribute">Contribute</a>-->
<!-- </li>-->
<li>
<a class="nav-link text-nowrap" href="https://github.com/double-fire-0/SystemNoise"
>Code Tools</a
>
</li>
</ul>
</div>
</div>
</nav>
<!-- <hr class="toprule" /> -->
<header>
<div class="header-block container">
<div class="title">SysNoise</div>
<div class="description">
Exploring and Benchmarking Training-Deployment System Inconsistency
</div>
</div>
</header>
<!-- <hr class="toprule" /> -->
<div class="container">
<section id="introduction">
<div class="overview">
<p class="doublealign">
Extensive studies have shown that deep learning models are vulnerable to adversarial and natural noises,
yet little is known about model robustness on noises caused by different system implementations. In this
paper, we for the first time introduce SysNoise, a frequently occurred but often overlooked noise in the
deep learning training-deployment cycle. In particular, SysNoise happens when the source training system
switches to a disparate target system in deployments, where various tiny system mismatch adds up to a
non-negligible difference. We first identify and classify SysNoise into three categories based on the
inference stage; we then build a holistic benchmark to quantitatively measure the impact of SysNoise on
20+ models, comprehending image classification, object detection, and instance segmentation tasks. Our
extensive experiments revealed that SysNoise could bring certain impacts on model robustness across different
tasks and common mitigations like data augmentation and adversarial training show limited effects on it.
Together, our findings open a new research topic and we hope this work will raise research attention to deep
learning deployment systems accounting for model performance.
<br><br>
<!-- <strong> News:</strong>-->
<!-- <ul>-->
<!-- <li> <strong>May 2022:</strong> -->
<!-- We have extended the common corruptions leaderboard on ImageNet with <a href="https://3dcommoncorruptions.epfl.ch">3D Common Corruptions</a> (ImageNet-3DCC). ImageNet-3DCC evaluation is interesting since (1) it includes more realistic corruptions and (2) it can be used to assess generalization of the existing models which may have overfitted to ImageNet-C. For a quickstart, click <a href="https://github.com/RobustBench/robustbench#new-evaluating-robustness-of-imagenet-models-against-3d-common-corruptions-imagenet-3dcc">here</a>. See the new leaderboard with ImageNet-C and ImageNet-3DCC <a href="https://robustbench.github.io/#div_imagenet_corruptions_heading">here</a> (also mCE metrics can be found <a href="https://github.com/RobustBench/robustbench#corruptions-imagenet-c--imagenet-3dcc">here</a>).-->
<!-- </li>-->
<!-- <li> <strong>May 2022:</strong> -->
<!-- We fixed the preprocessing issue for ImageNet corruption evaluations: previously we used resize to 256x256 and central crop to 224x224 which wasn't necessary since the ImageNet-C images are already 224x224. Note that this changed the ranking between the top-1 and top-2 entries.-->
<!-- </li>-->
<!-- </ul>-->
</p>
<!-- <div class="flexbox-container features">-->
<!-- <div class="element">-->
<!-- <div class="icon">-->
<!-- <img src="https://img.icons8.com/wired/100/000000/leaderboard.png" />-->
<!-- </div>-->
<!-- <p>-->
<!-- Up-to-date leaderboard based <br />-->
<!-- on 120+ models-->
<!-- </p>-->
<!-- </div>-->
<!-- <div class="element">-->
<!-- <div class="icon">-->
<!-- <img src="https://img.icons8.com/ios-glyphs/80/000000/user-credentials.png" />-->
<!-- </div>-->
<!-- <p>-->
<!-- Unified access to 80+ state-of-the-art <br />robust models via-->
<!-- Model Zoo-->
<!-- </p>-->
<!-- </div>-->
<!-- </div>-->
<!-- </div>-->
<!-- <div class="details">-->
<!-- <div class="box usage">-->
<!-- <p>Model Zoo</p>-->
<!-- <div class="divider">-->
<!-- <hr />-->
<!-- </div>-->
<!-- Check out the-->
<!-- <a href="https://github.com/RobustBench/robustbench#model-zoo">available models</a>-->
<!-- and our-->
<!-- <a href="https://github.com/RobustBench/robustbench#notebooks">Colab tutorials</a>.-->
<!-- <div class="codeblock">-->
<!-- <div class="vspace10"></div>-->
<!--
# !pip install git+https://github.com/RobustBench/[email protected]
from robustbench.utils import load_model
# Load a model from the model zoo
model = load_model(model_name='Rebuffi2021Fixing_70_16_cutmix_extra',
dataset='cifar10',
threat_model='Linf')
# Evaluate the Linf robustness of the model using AutoAttack
from robustbench.eval import benchmark
clean_acc, robust_acc = benchmark(model,
dataset='cifar10',
threat_model='Linf')
-->
<!-- HTML generated using hilite.me -->
<!-- <div style="background: #ffffff; overflow:auto;width:auto;padding:.2em .6em;">-->
<!-- <pre style="margin: 0; line-height: 125%"><span style="color: #888888"># !pip install git+https://github.com/RobustBench/[email protected]</span>-->
<!--<span style="color: #008800; font-weight: bold">from</span> <span style="color: #0e84b5; font-weight: bold">robustbench.utils</span> <span style="color: #008800; font-weight: bold">import</span> load_model-->
<!--<span style="color: #888888"># Load a model from the model zoo</span>-->
<!--model <span style="color: #333333">=</span> load_model(model_name<span style="color: #333333">=</span><span style="background-color: #fff0f0">'Rebuffi2021Fixing_70_16_cutmix_extra'</span>,-->
<!-- dataset<span style="color: #333333">=</span><span style="background-color: #fff0f0">'cifar10'</span>,-->
<!-- threat_model<span style="color: #333333">=</span><span style="background-color: #fff0f0">'Linf'</span>)-->
<!--<span style="color: #888888"># Evaluate the Linf robustness of the model using AutoAttack</span>-->
<!--<span style="color: #008800; font-weight: bold">from</span> <span style="color: #0e84b5; font-weight: bold">robustbench.eval</span> <span style="color: #008800; font-weight: bold">import</span> benchmark-->
<!--clean_acc, robust_acc <span style="color: #333333">=</span> benchmark(model,-->
<!-- dataset<span style="color: #333333">=</span><span style="background-color: #fff0f0">'cifar10'</span>,-->
<!-- threat_model<span style="color: #333333">=</span><span style="background-color: #fff0f0">'Linf'</span>)-->
<!--</pre>-->
<!-- </div>-->
<!-- <!– HTML generated using hilite.me –>-->
<!-- <div style="-->
<!-- background: #ffffff;-->
<!-- overflow: auto;-->
<!-- width: auto;-->
<!-- border: solid gray;-->
<!-- border-width: 0em 0em 0em 0em;-->
<!-- padding: 0.2em 0.6em;-->
<!-- ">-->
<!-- </pre>-->
<!-- </div>-->
<!-- </div>-->
<!-- </div>-->
<!-- <div class="box images">-->
<!-- <p>Analysis</p>-->
<!-- <div class="divider">-->
<!-- <hr />-->
<!-- </div>-->
<!-- Check out <a href="https://arxiv.org/abs/2010.09670">our paper</a> with a detailed analysis.-->
<!-- <div>-->
<!-- <!– <div class="scroller analysis-images">–>-->
<!-- <img class="analysis" src="./images/aa_robustness_vs_venues_Linf.png" alt="robustness_vs_venues" />-->
<!-- <!– <img–>-->
<!-- <!– src="./images/aa_robustness_vs_standard_Linf.png"–>-->
<!-- <!– alt="robustness_vs_clean"–>-->
<!-- <!– />–>-->
<!-- </div>-->
<!-- </div>-->
<!-- </div>-->
<!-- <div class="vspace10"></div>-->
<!-- </section>-->
<div id="leaderboard" class="container button-list">
<div class="heading">
<u>
Available Leaderboards
</u>
</div>
<a class="btn btn-secondary" href="#div_cifar10_Linf_heading">Classification Task</a>
<a class="btn btn-secondary" href="#div_cifar10_L2_heading">Object Detection Task</a>
<a class="btn btn-secondary" href="#div_cifar10_corruptions_heading">Instance Segmentation Task</a>
</div>
<section class="container" id="div_cifar10_Linf_heading">
<div class="heading">
<p>
<!-- <div style="opacity:0;">1<br>1<br>1<br>1</div> <!– Needed to introduce a margin for the navigation bar between leaderboards –>-->
Leaderboard:
<span class="heading-math"> Classification Task </span>
</p>
</div>
<div id="cls"></div>
</section>
<section>
<div class="heading" id="div_cifar10_L2_heading">
<p>
<div style="opacity:0;">1<br>1<br>1<br>1</div> <!-- Needed to introduce a margin for the navigation bar between leaderboards -->
Leaderboard:
<span class="heading-math"> Object Detection Task</span>
</p>
</div>
<div id="det"></div>
</section>
<section>
<div class="heading" id="div_cifar10_corruptions_heading">
<p>
<div style="opacity:0;">1<br>1<br>1<br>1</div> <!-- Needed to introduce a margin for the navigation bar between leaderboards -->
Leaderboard:
<span class="heading-math"> Instance Segmentation Task </span>
</p>
</div>
<div id="seg"></div>
</section>
<div class="vspace50"></div>
<section id="dataset">
<div class="heading">
<p>Dataset</p>
</div>
<div class="vspace30"></div>
<div id="leaderboard" class="container button-list">
<div class="heading">
<u>
SysNoise on ImageNet
</u>
</div>
<a class="btn btn-secondary" href="https://drive.google.com/file/d/1j7Ox169iTetr0fNkifzzK14WlqxRwmNH/view?usp=sharing"> ImageNet: Color_Mode Noise</a>
<a class="btn btn-secondary" href="https://drive.google.com/file/d/1j7Ox169iTetr0fNkifzzK14WlqxRwmNH/view?usp=sharing"> ImageNet: Decoder+Resize+Color_Mode Noise</a>
<div class="heading">
<u>
SysNoise on MSCOCO
</u>
</div>
<a class="btn btn-secondary" href="https://drive.google.com/file/d/1HEZSVas_SERh8tudRb_i9jOkzzweJUwz/view?usp=sharing"> MSCOCO: Color_Mode Noise </a>
<a class="btn btn-secondary" href="https://drive.google.com/file/d/1_3w7a8fW44PpIqOVliDhPyqV9JSNnpvR/view?usp=sharing"> MSCOCO: Decoder+Resize+Color_Mode Noise </a>
</div>
</section>
<div class="vspace50"></div>
<section id="future_work">
<div class="heading">
<p>Future Work</p>
</div>
<div class="vspace30"></div>
Based on the research conducted in this paper, our future work will focus on extending the SysNoise to other fields such as speech and audio.
We will explore how SysNoise occurs in the different steps of the ML pipeline and benchmark it. We will keep updating our website and the final
results will release on it.
So far, we have found that model quantization will make an influence on the text to speech task. A preliminary speech example can be found at
<a href="https://modeltc.github.io/systemnoise_web/wav_quant.tar">here</a>, which contains some text-to-speech results with and
without quantization noise. An example of how to visually see audio differences is provided <a href="https://github.com/ModelTC/systemnoise_web/blob/master/diff_wav.ipynb">here</a>
</section>
<div class="vspace50"></div>
<section id="citation">
<div class="heading">
<p>Citation</p>
</div>
Consider citing our work by following bibtex:
<!-- Model Zoo:-->
<!-- @article{croce2020robustbench,-->
<!-- title={RobustBench: a standardized adversarial robustness benchmark},-->
<!-- author={Croce, Francesco and Andriushchenko, Maksym and Sehwag, Vikash and Flammarion, Nicolas and Chiang, Mung and Mittal, Prateek and Matthias Hein},-->
<!-- journal={arXiv preprint arXiv:2010.09670},-->
<!-- year={2020}-->
<!-- }-->
<!-- HTML generated using hilite.me -->
<!-- @article{2022sysnoise,-->
<!-- title={SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency},-->
<!-- author={Yan Wang and Yuhang Li and Ruihao Gong and Aishan Liu and Yanfei Wang-->
<!-- and Jian Hu and Yongqiang Yao and Tianzi Xiao and Fengwei Yu and Xianglong Liu},-->
<!-- year={2022}-->
}
<div
style="background: #ffffff; overflow:auto;width:auto;border:solid gray;border-width:.0em .0em .0em .0em;padding:.2em .6em;">
<pre style="margin: 0; line-height: 125%"><span style="color: #555555; font-weight: bold">@article</span>{yan2022sysnoise,
title<span style="color: #333333">=</span>{SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency},
author<span style="color: #333333">=</span>{Yan Wang <span style="color: #000000; font-weight: bold">and</span> Yuhang Li <span style="color: #000000; font-weight: bold">and</span> Ruihao Gong <span style="color: #000000; font-weight: bold">and</span> Aishan Liu <span style="color: #000000; font-weight: bold">and</span> Yanfei Wang
<span style="color: #000000; font-weight: bold">and</span> Jian Hu <span style="color: #000000; font-weight: bold">and</span> Yongqiang Yao <span style="color: #000000; font-weight: bold">and</span> Tianzi Xiao
<span style="color: #000000; font-weight: bold">and</span> Fengwei Yu <span style="color: #000000; font-weight: bold">and</span> Xianglong Liu},
year<span style="color: #333333">=</span>{2022}
}</pre>
</div>
</section>
<div class="vspace50"></div>
<section id="citation">
<div class="heading">
<p>Contribute to Us</p>
</div>
Welcome to provide us with test results for new models, as well as models that are robust to SysNoise.
Contact me to have your model and results updated in the leaderboard.
If you have other ideas about this noise, you can contact me at
<a href="mailto:[email protected]">[email protected]</a>
</section>
<!-- <section id="contribute">-->
<!-- <div class="details">-->
<!-- <div class="box2">-->
<!-- <p>Contribute to RobustBench!</p>-->
<!-- <div class="divider">-->
<!-- <hr />-->
<!-- </div>-->
<!-- We welcome any contribution in terms of both new robust models and-->
<!-- evaluations. Please check-->
<!-- <a href="https://github.com/RobustBench/robustbench#how-to-contribute">here</a>-->
<!-- for more details.-->
<!-- <br />-->
<!-- <br />-->
<!-- Feel free to contact us at-->
<!-- <a href="mailto:[email protected]">[email protected]</a>-->
<!-- </div>-->
<!-- <div class="box2">-->
<!-- <p>Maintainers</p>-->
<!-- <div class="divider">-->
<!-- <hr />-->
<!-- </div>-->
<!-- <ul>-->
<!-- <li>-->
<!-- <a href="https://twitter.com/fra__31" target="_blank">Francesco Croce-->
<!-- </a>-->
<!-- <a href="https://twitter.com/fra__31"><i class="fas fa-globe"></i></a>-->
<!-- <a href="https://github.com/fra31"><i class="fab fa-github"></i></a>-->
<!-- <a href="https://scholar.google.com/citations?user=laq9cq0AAAAJ"><i class="ai ai-google-scholar"></i></a>-->
<!-- </li>-->
<!-- <li>-->
<!-- <a href="https://people.epfl.ch/maksym.andriushchenko" target="_blank">Maksym Andriushchenko</a>-->
<!-- <a href="https://people.epfl.ch/maksym.andriushchenko"><i class="fas fa-globe"></i></a>-->
<!-- <a href="https://github.com/max-andr"><i class="fab fa-github"></i></a>-->
<!-- <a href="https://scholar.google.com/citations?user=ZNtuJYoAAAAJ"><i class="ai ai-google-scholar"></i></a>-->
<!-- </li>-->
<!-- <li>-->
<!-- <a href="https://vsehwag.github.io/" target="_blank">Vikash Sehwag</a>-->
<!-- <a href="https://vsehwag.github.io/"><i class="fas fa-globe"></i></a>-->
<!-- <a href="https://github.com/VSehwag"><i class="fab fa-github"></i></a>-->
<!-- <a href="https://scholar.google.com/citations?user=JAkeEG8AAAAJ"><i class="ai ai-google-scholar"></i></a>-->
<!-- </li>-->
<!-- <li>-->
<!-- <a href="https://edoardo.science" target="_blank">Edoardo Debenedetti</a>-->
<!-- <a href="https://edoardo.science"><i class="fas fa-globe"></i></a>-->
<!-- <a href="https://github.com/dedeswim"><i class="fab fa-github"></i></a>-->
<!-- <a href="https://twitter.com/edoardo_debe"><i class="fab fa-twitter"></i></a>-->
<!-- </li>-->
<!-- </ul>-->
<!-- </div>-->
<!-- </div>-->
<!-- </section>-->
<!-- </div>-->
<hr class="bottomrule" />
<footer>
<small>© 2022, SysNoise, SenseTime;
</footer>
<script>
// When the user scrolls the page, execute myFunction
window.onscroll = function () {
myFunction();
};
// Get the navbar
var navbar = document.getElementById("navbar");
// Get the offset position of the navbar
var sticky = navbar.offsetTop;
// Add the sticky class to the navbar when you reach its scroll position. Remove "sticky" when you leave the scroll position
function myFunction() {
if (window.pageYOffset >= sticky) {
navbar.classList.add("sticky");
} else {
navbar.classList.remove("sticky");
}
}
</script>
<script>
$("#cls").load("./sys/classification.html");
$("#det").load("./sys/det.html");
$("#seg").load("./sys/seg.html");
</script>
</body>