Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarks #29

Open
wants to merge 1 commit into
base: old-discuss
Choose a base branch
from
Open

Update benchmarks #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,11 @@ As the scheme removes the overhead of including root and intermediate certificat

## Preliminary Evaluation

This draft is a work in progress, however a preliminary evaluation is available:

| Scheme | p5 | p50 | p95 |
|------------------------------------------------------|------|-------|-------|
| Original | 2308 | 4032 | 5609 |
| TLS Cert Compression | 1619 | 3243 | 3821 |
| Intermediate Suppression and TLS Cert Compression | 1020 | 1445 | 3303 |
| **This Draft** | 661 | 1060 | 1437 |
| Original / Uncompressed | 2308 | 4032 | 5609 |
| Existing TLS Certificate Compression | 1619 | 3243 | 3821 |
| **This Draft** | 881 | 1256 | 1716 |
| Hypothetical Optimal Compression | 377 | 742 | 1075 |

A complete table of results and benchmarking scripts can be found in [benchmarks](benchmarks/).
Expand Down
44 changes: 20 additions & 24 deletions benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,27 @@

This folder contains scripts for benchmarking the various compression schemes.

| Scheme | Storage Footprint | p5 | p50 | p95 |
|------------------------------------------------------|---------------------|------|-------|-------|
| Original | 0 | 2308 | 4032 | 5609 |
| TLS Cert Compression | 0 | 1619 | 3243 | 3821 |
| Intermediate Suppression | 0 | 1315 | 1688 | 4227 |
| Intermediate Suppression and TLS Cert Compression | 0 | 1020 | 1445 | 3303 |
| Pass 1 only (intermediate and root compression) | 0 | 1001 | 1429 | 2456 |
| Dictionary composed all intermediate and root certs | 3455467 | 721 | 1094 | 1631 |
| Pass 1 plus popular strings | 1848 | 718 | 1128 | 1627 |
| This Draft | 65336 | 661 | 1060 | 1437 |
| Pass 1 plus trained end-entity zstd dict | 3000 | 562 | 931 | 1454 |
| Pass 1 plus trained end-entity zstd dict | 100000 | 520 | 894 | 1291 |
| Hypothetical Optimal Compression | 0 | 377 | 742 | 1075 |
| Scheme | Storage Footprint | p5 | p50 | p95 |
|-------------------------------------------------------------------------------------------------|---------------------|------|-------|-------|
| Original | 0 | 2308 | 4031 | 5636 |
| TLS Cert Compression | 0 | 1673 | 3319 | 3963 |
| Intermediate Suppression | 0 | 1316 | 1689 | 4220 |
| Intermediate Suppression and TLS Cert Compression | 0 | 1035 | 1467 | 3370 |
| Hypothetical Optimal Compression | 0 | 380 | 746 | 1078 |
| Leaf Certificate Metadata Estimate | 0 | 568 | 721 | 1072 |
| Leaf Certificate Compressed Domains Estimate | 0 | 25 | 39 | 273 |
| CA Prefix Only | 0 | 1005 | 1440 | 2498 |
| Base: Dictionary Compressor Base: Zstandard + Offline Compression:False | 0 | 877 | 1293 | 1797 |
| Base: Dictionary Compressor Base: Zstandard + Offline Compression:True | 0 | 868 | 1286 | 1757 |
| **This Draft | 0 | 881 | 1256 | 1716 |
| Method 1: Baseline Base: Zstandard + Offline Compression:True | 3455467 | 721 | 1095 | 1633 |
| Method 1: Baseline Base: Zstandard + Offline Compression:False | 3455467 | 1179 | 2874 | 3344 |
| Method 2: CA Prefix with Training redacted=True, offlineComp=True | 3000 | 582 | 959 | 1538 |
| Method 2: CA Prefix with Training redacted=True, offlineComp=True | 100000 | 548 | 931 | 1393 |
| Method 2: CA Prefix and CommonStrings threshold=2000 Base: Zstandard + Offline Compression:True | 1848 | 724 | 1131 | 1641 |
| Method 2: CA Prefix and SystematicStrings Base: Zstandard + Offline Compression:True | 65336 | 661 | 1061 | 1447 |
| Method 2: CA Prefix and SystematicStrings Base: Zstandard + Offline Compression:False | 65336 | 690 | 1087 | 1515 |


## Evaluated Schemes

* **TLS Certificate Compression** - Using zstandard tuned for maximum compression
* **Intermediate Suppression** - Removes the intermediate and root certificates from the chain. Has no effect if the chain contains a certificate not in Mozilla's list of intermediates and roots.
* **Pass 1** - Just the compression scheme defined in the draft for intermediate and root certificates.
* **Dictionary of all intermediate and root certs** - Pure Zstandard with a very large dictionary composed of the concatination of all intermediate and root certificates in the CCADB.
* **Pass 1 plus popular strings** - Pass 1 followed by compression of the end-entity certificate using a dictionary of common strings extracted from witnessed certificate chains.
* **This Draft** - As described in this document.
* **Pass 1 plus trained zstd dict** - The end entity certiifcates from a number of certificate chains have their subject-specific (name, domains) removed and then are passed to the Zstandard dictionary training function with a size of 3 KB or 100 KB.
* **Hypothetical Optimal Compression** - This assumes an end-entity certificate can be reduced to purely the compressed domain names and public key, CA signature and SCTs.
## Methodology

These compression schemes are defined in the associated scripts in the schemes folder. Each scheme is evaluated over a sample of certificate chains fetched from the Tranco top 100k. The confidence interval for each percentile is calculated and the upper bound is taken.
22 changes: 9 additions & 13 deletions draft-ietf-tls-cert-abridge.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,27 +222,23 @@ The second pass uses Brotli {{BROTLI}} to compress any redundant data in the end
* `quality=5`
* `lgwindow=17`

Benchmarks on real world certificate chains suggest that in this context higher values require greater CPU usage but do not result in better compression,
Benchmarks on real world certificate chains suggest that higher values require greater CPU usage but do not result in better compression.

# Preliminary Evaluation {#eval}
# Evaluation {#eval}

[[**NOTE:** This section to be removed prior to publication.]]

The storage footprint refers to the on-disk size required for the end-entity dictionary. The other columns report the 5th, 50th and 95th percentile of the resulting certificate chains. The evaluation set was a ~75,000 certificate chains from the Tranco list using the python scripts in the draft's Github repository.
The columns report the 5th, 50th and 95th percentile of the resulting certificate chains wire sizes in bytes. The evaluation set was ~75000 certificate chains from the Tranco list using the python scripts in the draft's Github repository.

| Scheme | Storage Footprint | p5 | p50 | p95 |
|------------------------------------------------------|---------------------|------|-------|-------|
| Original | 0 | 2308 | 4032 | 5609 |
| TLS Cert Compression | 0 | 1619 | 3243 | 3821 |
| Intermediate Suppression and TLS Cert Compression | 0 | 1020 | 1445 | 3303 |
| **This Draft** | 65336 | 661 | 1060 | 1437 |
| **This Draft with opaque trained dictionary** | 3000 | 562 | 931 | 1454 |
| Hypothetical Optimal Compression | 0 | 377 | 742 | 1075 |
| Scheme | p5 | p50 | p95 |
|------------------------------------------------------|------|-------|-------|
| Original / Uncompressed | 2308 | 4032 | 5609 |
| Existing TLS Certificate Compression | 1619 | 3243 | 3821 |
| **This Draft** | 881 | 1256 | 1716 |
| Hypothetical Optimal Compression | 377 | 742 | 1075 |

* 'Original' refers to the sampled certificate chains without any compression.
* 'TLS Cert Compression' used ZStandard with the parameters configured for maximum compression as defined in {{TLSCertCompress}}.
* 'Intermediate Suppression and TLS Cert Compression' was modelled as the elimination of all certificates in the intermediate and root certificates with the Basic Constraints CA value set to true. If a cert chain included an unrecognized certificate with CA status, then no CA certificates were removed from that chain. The cert chain was then passed to 'TLS Cert Compression' as a second pass.
* 'This Draft with opaque trained dictionary' refers to pass 1 and pass 2 as defined by this draft, but instead using a 3000 byte dictionary for pass 2 which was produced by the Zstandard dictionary training algorithm. This illustrates a ceiling on what ought to be possible by improving the construction of the pass 2 dictionary in this document. However, using this trained dictionary directly will not treat all CA's equitably, as the dictionary will be biased towards compressing the most popular CAs more effectively.
* 'Hypothetical Optimal Compression' is the resulting size of the cert chain after reducing it to only the public key in the end-entity certificate, the CA signature over the EE cert, the embedded SCT signatures and a compressed list of domains in the SAN extension. This represents the best possible compression as it entirely removes any CA certs, identifiers, field tags and lengths and non-critical extensions such as OCSP, CRL and policy extensions.

# Deployment Considerations {#deployment}
Expand Down
Loading