| layout |
|---|
default |
<div id="intro-description">
<p>The ACM SIGSOFT Empirical Standards are the official evidence standards (models of a community's expectations for conducting and reporting studies) for software engineering research.</p>
</div>
<a class="page-link" href="https://www2.sigsoft.org/EmpiricalStandards/tools/">Review a paper</a>
</div>
<div id="image-container">
<img src="assets/img/landing-image.jpg">
</div>
Different kinds of research have different norms, so each common research method has a unique standard
<div class="feature-box">
<img src="assets/img/list.png">
<h3>Checklists for authors and reviewers</h3>
<p>Use standards-based checklists to improve your methods, papers, and peer reviews</p>
</div>
<div class="feature-box">
<img src="assets/img/medal.png">
<h3>Desirable and extraordinary attributes</h3>
<p>Standards and checklists clearly differentiate must-haves from exceptional research</p>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Specific attributes</h3>
<p>A list of properties the paper should possess, grouped into essential, desirable, and extraordinary</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=EngineeringResearch#">Conceptually evaluates the proposed artifact; discusses its strengths, weaknesses and limitations.</q><br><br><span>Engineering Research</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>General quality criteria</h3>
<p>Qualitative and quantitative quality criteria the paper should meet</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=Experiments#">Conclusion validity, construct validity, internal validity, reliability, objectivity, reproducibility.</q><br><br><span>Experiment</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Acceptable deviations</h3>
<p>Circumstances where the paper is permitted to deviate from a standard</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=DataScience#">Data not shared because it is impractical (e.g. too large) or unethical (e.g. too sensitive)</q><br><br><span>Data Science</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Antipatterns</h3>
<p>Common problems with this methodology that papers should avoid</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=GroundedTheory#">Data analysis focusing on counting words, codes, concepts, or categories instead of interpreting.</q><br><br><span>Grounded Theory</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Invalid criticisms</h3>
<p>Unreasonable arguments against a paper that reviewers should not make</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=Replication#">The replication merely confirms the findings of the original study; no inconsistencies are reported.</q><br><br><span>Replication</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Suggested readings</h3>
<p>Additional scholarship on the method upon which the standard is based</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=SystematicReviews#">Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering.</q><br><br><span>Systematic Review</span></p>
</div>
</div>
<div class="feature-list-item">
<div class="feature-list-text">
<h3>Exemplars</h3>
<p>Good examples of the method that authors should emulate</p>
</div>
<div class="feature-list-example">
<p><q cite="https://www2.sigsoft.org/EmpiricalStandards/docs/standards?standard=CaseStudy">Diomidis Spinellis and Paris C. Avgeriou. Evolution of the Unix System Architecture: An Exploratory Case Study. IEEE Transactions on Software Engineering. (2019).</q><br><br><span>Case Study</span></p>
</div>
</div>
<div class="instruction-box">
<div class="instruction-text">
<p>Interactive checklists</p>
<p>Interactive checklists based on the standards make peer review more specific, technical, and reliable.
Customized diagnostics help reviewers make more reasonable and actionable suggestions.</p>
<ul class="benefit-list">
<li>More effective, transparent peer reviews</li>
<li>Faster publication times</li>
<li>Reduced reviewer workload</li>
<li>Higher-quality papers</li>
</ul>
</div>
<img src="assets/img/filling_checklist_crop.PNG">
</div>
<div class="timeline">
<div class="timeline-box left">
<p class="timeline-date">MAY 2019</p>
<p class="timeline-content">At ICSE town hall, SIGSOFT launches the "Improving Paper and Peer Review Quality Initiative"</p>
</div>
<div class="timeline-box right">
<p class="timeline-date">MAY 2020</p>
<p class="timeline-content">First 8 empirical standards drafted</p>
</div>
<div class="timeline-box left">
<p class="timeline-date">OCT 2020</p>
<p class="timeline-content">
Empirical Standards Report made accessible on arXiv <!-- Link to report -->
<a href="https://arxiv.org/abs/2010.03525">Read the Empirical Standards Report →</a>
</p>
</div>
<div class="timeline-box right">
<p class="timeline-date">JAN 2021</p>
<p class="timeline-content">
First 8 standards made available on GitHub for public comment <!-- Link to git repo -->
<a href="https://github.com/acmsigsoft/EmpiricalStandards">Visit GitHub repository →</a>
</p>
<!-- https://github.com/acmsigsoft/EmpiricalStandards/blob/bf2bfaecc5a9e7659e66237994da7f93433c1e45/empiricalStandards.md#grounded-theory -->
</div>
<div class="timeline-box left">
<p class="timeline-date">MAY 2021</p>
<p class="timeline-content">First review checklists available on the web</p>
<!-- https://github.com/acmsigsoft/EmpiricalStandards/commit/3462acf033baa670f4dd869be1d85d642688da51 -->
</div>
<div class="timeline-box right">
<p class="timeline-date">JUNE 2021</p>
<p class="timeline-content">First recommendation of the standards by a conference (EASE 2021)</p>
</div>
<div class="timeline-box left">
<p class="timeline-date">JUNE 2023</p>
<p class="timeline-content">Field experiment at EASE showing standards improve reliability</p>
</div>
<div id="timeline-roadmap">
<h3>Roadmap</h3>
</div>
<div class="timeline-box right">
<p class="timeline-content">EASE experiment and standards published in journal</p>
</div>
<div class="timeline-box left">
<p class="timeline-content">Empirical standards listed on EQUATOR Network</p>
</div>
</div>
Ready to try?