chore: Update benchmarks results based on 0.3.0-rc1 (#969)

* add TPC-H benchmark results * updat TPC-H results * update charts * improve charts * improve charts * add old results back
apache · Sep 25, 2024 · b4de8e0 · b4de8e0
1 parent 94093f3
commit b4de8e0
Show file tree

Hide file tree

Showing 27 changed files with 1,484 additions and 30 deletions.
diff --git a/README.md b/README.md
@@ -44,25 +44,30 @@ The following chart shows the time it takes to run the 22 TPC-H queries against
 using a single executor with 8 cores. See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html)
 for details of the environment used for these benchmarks.
 
-When using Comet, the overall run time is reduced from 616 seconds to 379 seconds, a 1.62x speedup, with query 1
-running more than 7x faster than Spark.
+When using Comet, the overall run time is reduced from 616 seconds to 374 seconds, a 1.6x speedup, with query 1
+running 9x faster than Spark.
 
 Running the same queries with DataFusion standalone (without Spark) using the same number of cores results in a 3.6x 
 speedup compared to Spark.
 
 Comet is not yet achieving full DataFusion speeds in all cases, but with future work we aim to provide a 2x-4x speedup 
 for a broader set of queries.
 
-![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_allqueries.png)
+![](docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png)
 
 Here is a breakdown showing relative performance of Spark, Comet, and DataFusion for each TPC-H query.
 
-![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_compare.png)
+![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png)
 
-The following chart shows how much Comet currently accelerates each query from the benchmark. Performance optimization
-is an ongoing task, and we welcome contributions from the community to help achieve even greater speedups in the future.
+The following charts shows how much Comet currently accelerates each query from the benchmark.
 
-![](docs/source/_static/images/benchmark-results/2024-08-23/tpch_queries_speedup_rel.png)
+### Relative speedup
+
+![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png)
+
+### Absolute speedup
+
+![](docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png)
 
 These benchmarks can be reproduced in any environment using the documentation in the 
 [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html). We encourage 

diff --git a/...k-results/2024-08-23/tpcds_allqueries.png → ...chmark-results/0.2.0/tpcds_allqueries.png b/...k-results/2024-08-23/tpcds_allqueries.png → ...chmark-results/0.2.0/tpcds_allqueries.png
diff --git a/...ults/2024-08-23/tpcds_queries_compare.png → ...k-results/0.2.0/tpcds_queries_compare.png b/...ults/2024-08-23/tpcds_queries_compare.png → ...k-results/0.2.0/tpcds_queries_compare.png
diff --git a/.../2024-08-23/tpcds_queries_speedup_abs.png → ...sults/0.2.0/tpcds_queries_speedup_abs.png b/.../2024-08-23/tpcds_queries_speedup_abs.png → ...sults/0.2.0/tpcds_queries_speedup_abs.png
diff --git a/...rk-results/2024-08-23/tpch_allqueries.png → ...nchmark-results/0.2.0/tpch_allqueries.png b/...rk-results/2024-08-23/tpch_allqueries.png → ...nchmark-results/0.2.0/tpch_allqueries.png
diff --git a/...sults/2024-08-23/tpch_queries_compare.png → ...rk-results/0.2.0/tpch_queries_compare.png b/...sults/2024-08-23/tpch_queries_compare.png → ...rk-results/0.2.0/tpch_queries_compare.png
diff --git a/...s/2024-08-23/tpch_queries_speedup_rel.png → ...esults/0.2.0/tpch_queries_speedup_rel.png b/...s/2024-08-23/tpch_queries_speedup_rel.png → ...esults/0.2.0/tpch_queries_speedup_rel.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_allqueries.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_allqueries.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_compare.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_abs.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png b/docs/source/_static/images/benchmark-results/0.3.0/tpcds_queries_speedup_rel.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_allqueries.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_compare.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_abs.png
diff --git a/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png b/docs/source/_static/images/benchmark-results/0.3.0/tpch_queries_speedup_rel.png
diff --git a/...hmark-results/2024-08-23/comet-tpcds.json → .../benchmark-results/0.2.0/comet-tpcds.json b/...hmark-results/2024-08-23/comet-tpcds.json → .../benchmark-results/0.2.0/comet-tpcds.json
diff --git a/...chmark-results/2024-08-23/comet-tpch.json → ...e/benchmark-results/0.2.0/comet-tpch.json b/...chmark-results/2024-08-23/comet-tpch.json → ...e/benchmark-results/0.2.0/comet-tpch.json
diff --git a/...ts/2024-08-23/datafusion-python-tpch.json → ...results/0.2.0/datafusion-python-tpch.json b/...ts/2024-08-23/datafusion-python-tpch.json → ...results/0.2.0/datafusion-python-tpch.json
diff --git a/...hmark-results/2024-08-23/spark-tpcds.json → .../benchmark-results/0.2.0/spark-tpcds.json b/...hmark-results/2024-08-23/spark-tpcds.json → .../benchmark-results/0.2.0/spark-tpcds.json
diff --git a/...chmark-results/2024-08-23/spark-tpch.json → ...e/benchmark-results/0.2.0/spark-tpch.json b/...chmark-results/2024-08-23/spark-tpch.json → ...e/benchmark-results/0.2.0/spark-tpch.json