Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bench: change the benchmark tool from vitest to tinybench #710

Open
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

dayongkr
Copy link
Contributor

@dayongkr dayongkr commented Oct 14, 2024

Description

Vitest introduces overhead from vite, which significantly impacts functions in utility tools with low computation, making it unsuitable for es-toolkit.

To address this, I implemented benchmarking using only tinybench. The process works as follows:

  1. When a user executes a command, the bench-runner.mjs script is triggered.
  2. This script compares the input file with files located in benchmarks/performance to determine which files to benchmark.
  3. The selected benchmark files are bundled with rollup and stored in the .bench-bundle folder.
  4. The bundled files are then executed using node <built-file>.
  5. This is repeated for all relevant files.

Additionally, to maintain the usability of vitest, I implemented the same describe and bench interface in the benchmark/bench.ts file.

The main differences are that, instead of importing fromvitest, the functions are now imported from ../bench, and the output is somewhat simpler. Other than that, commands are executed the same way, such as yarn bench chunk.

I’m sorry for opening such a large PR.🙏 If there are any questions or concerns, ask for a review at any time.

Screenshot 2024-10-14 at 4 06 23 PM Screenshot 2024-10-14 at 4 08 29 PM

I will gradually improve the output.

  • ✅ I ran through all the files.
  • ⚠️ But I didn’t compare them all against vitest again.

TODO

  • Implement bundling scripts for es-toolkit and a benchmark file

Utilize the existing bundling setup to closely resemble the production environment.

  • Abstract tinybench to have the same interface as vitest for convenient writing of benchmark code
  • Implement the ability to run benchmarks via a command, specifying the file name
  • Modify existing benchmark files:
    • Remove reassigned parts
    • Remove Vitest dependency

- Update .gitignore to include .bench-bundle file
- Add new script .scripts/bench-runner.mjs for running benchmarks
- Update benchmarks/package.json to add devDependencies
- Add new benchmark file benchmarks/performance/bench.ts
- Update package.json to add new devDependencies
- Update rollup.config.mjs to export entrypoints
- Update yarn.lock to add new devDependencies
Copy link

vercel bot commented Oct 14, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
es-toolkit ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 19, 2024 2:22pm

@dayongkr dayongkr changed the title bench: Change the benchmark tool from vitest to tinybench bench: change the benchmark tool from vitest to tinybench Oct 14, 2024
@codecov-commenter
Copy link

codecov-commenter commented Oct 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.03%. Comparing base (4b2de1f) to head (6c985f9).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #710   +/-   ##
=======================================
  Coverage   99.03%   99.03%           
=======================================
  Files         285      285           
  Lines        2483     2483           
  Branches      706      706           
=======================================
  Hits         2459     2459           
  Misses         23       23           
  Partials        1        1           

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants