From a76f72751692a991f36efff3253f29cd58c09185 Mon Sep 17 00:00:00 2001 From: Janosh Riebesell Date: Mon, 13 Nov 2023 12:01:47 -0800 Subject: [PATCH 1/4] docs_src/index.md remove outdated matbench-discovery metrics table --- docs_src/index.md | 412 +--------------------------------------------- 1 file changed, 1 insertion(+), 411 deletions(-) diff --git a/docs_src/index.md b/docs_src/index.md index 174de541..bb1d3865 100644 --- a/docs_src/index.md +++ b/docs_src/index.md @@ -32,11 +32,8 @@ Find more information about this benchmark on [the benchmark info page](Benchmar | [matbench_mp_is_metal](Leaderboards%20Per-Task/matbench_v0.1_matbench_mp_is_metal.md) | 106,113 | [CGCNN v2019](Full%20Benchmark%20Data/matbench_v0.1_cgcnnv2019.md) | **0.9520** | structure required | | [matbench_mp_e_form](Leaderboards%20Per-Task/matbench_v0.1_matbench_mp_e_form.md) | 132,752 | [coGN](Full%20Benchmark%20Data/matbench_v0.1_coGN.md) | **0.0170 (eV/atom)** | structure required | - - - Scaled errors for regressions on this leaderboard plot are assessed as the ratio of mean absolute error to mean absolute deviation: @@ -47,414 +44,7 @@ $$ ## Leaderboard-Discovery: General Purpose Algorithms on `matbench_discovery 0.1.0` -[Matbench Discovery](https://matbench-discovery.materialsproject.org/) is an interactive leaderboard and associated PyPI package which together make it easy to benchmark ML energy models on a task designed to closely simulate a high-throughput discovery campaign for new stable inorganic crystals. Matbench-discovery compares ML structure-relaxation methods on the [WBM dataset](https://www.nature.com/articles/s41524-020-00481-6) for ranking ~250k generated structures according to predicted hull stability (42k stable). [Matbench Discovery](https://matbench-discovery.materialsproject.org/) is developed by Janosh Riebesell. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
modelF1DAFPrecisionTPRTNRAccuracyMAERMSER²
CHGNet0.593.060.520.670.870.840.070.110.61
M3GNet0.582.660.450.790.800.800.070.120.59
MEGNet0.522.700.460.590.860.810.130.20-0.27
CGCNN0.522.620.450.600.850.810.140.23-0.61
CGCNN+P0.512.380.410.690.790.780.110.180.02
Wrenformer0.482.130.360.710.740.740.100.18-0.04
BOWSR + MEGNet0.441.900.320.740.670.680.110.160.15
Voronoi RF0.341.510.260.520.690.660.140.21-0.32
dummy0.191.010.170.230.770.680.120.180.00
- - +[Matbench Discovery](https://matbench-discovery.materialsproject.org) is an interactive leaderboard and associated PyPI package which together make it easy to benchmark ML energy models on a task designed to closely simulate a high-throughput discovery campaign for new stable inorganic crystals. Matbench-discovery compares ML structure-relaxation methods on the [WBM dataset](https://www.nature.com/articles/s41524-020-00481-6) for ranking ~250k generated structures according to predicted hull stability (42k stable).
## Overview From e086986b9afd17935145cfa8fec47d6e69cce705 Mon Sep 17 00:00:00 2001 From: Janosh Riebesell Date: Mon, 13 Nov 2023 12:14:31 -0800 Subject: [PATCH 2/4] dynamically load matbench discovery metric table in iframe to keep leaderboard up-to-date automatically --- docs_src/index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs_src/index.md b/docs_src/index.md index bb1d3865..67427954 100644 --- a/docs_src/index.md +++ b/docs_src/index.md @@ -45,6 +45,17 @@ $$ ## Leaderboard-Discovery: General Purpose Algorithms on `matbench_discovery 0.1.0` [Matbench Discovery](https://matbench-discovery.materialsproject.org) is an interactive leaderboard and associated PyPI package which together make it easy to benchmark ML energy models on a task designed to closely simulate a high-throughput discovery campaign for new stable inorganic crystals. Matbench-discovery compares ML structure-relaxation methods on the [WBM dataset](https://www.nature.com/articles/s41524-020-00481-6) for ranking ~250k generated structures according to predicted hull stability (42k stable). + + + +
## Overview From c54db71101762ac6fa1ccb28a53ab8835a52f012 Mon Sep 17 00:00:00 2001 From: Janosh Riebesell Date: Mon, 13 Nov 2023 12:16:19 -0800 Subject: [PATCH 3/4] remove outdated cumulative precision/recall curves --- docs_src/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs_src/index.md b/docs_src/index.md index 67427954..e2d4ed9c 100644 --- a/docs_src/index.md +++ b/docs_src/index.md @@ -56,7 +56,6 @@ $$ }); -
## Overview [Matbench](https://doi.org/10.1038/s41524-020-00406-3) is an [ImageNet](http://www.image-net.org) for **materials science**; a From 53c2d5c8f026af753a4f76d776b33bfe3f574dfb Mon Sep 17 00:00:00 2001 From: Janosh Riebesell Date: Mon, 13 Nov 2023 12:35:03 -0800 Subject: [PATCH 4/4] add transparent white background to improve text legibility --- docs_src/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs_src/index.md b/docs_src/index.md index e2d4ed9c..97853716 100644 --- a/docs_src/index.md +++ b/docs_src/index.md @@ -46,14 +46,14 @@ $$ [Matbench Discovery](https://matbench-discovery.materialsproject.org) is an interactive leaderboard and associated PyPI package which together make it easy to benchmark ML energy models on a task designed to closely simulate a high-throughput discovery campaign for new stable inorganic crystals. Matbench-discovery compares ML structure-relaxation methods on the [WBM dataset](https://www.nature.com/articles/s41524-020-00481-6) for ranking ~250k generated structures according to predicted hull stability (42k stable). - + ## Overview