Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(schema-compiler): Boost models transpilation 10-13x times (using SWC instead of Babel) #9225

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Feb 14, 2025

This PR brings further improvements into data model compilation.
Cube transpilers are rewritten in Rust and are called asynchronously, moving transpilation jobs to the internal native threads under the hood. More improvements are achieved by caching the transpilation metadata between files transpile calls within one compilation phase.

This implementation supersedes other approaches taken in #9217 as it boosts performance more.

There were a bunch of iterations, research, and improvements. But long story short: now js files transpilation is 11−14 times faster than it was before!! 🚀

Here are a few numbers and charts for comparison:

Input: 300 js files with ~400 members each

We have 3 phases of compilation. That's why there are 3 messages with times for each phase.

Original master: synchronous sequential traversing in main node.js thread.

Execution Time for transpiling all 301 files: 11657.56 ms
Execution Time for transpiling all 301 files: 11871.40 ms
Execution Time for transpiling all 301 files: 12725.75 ms

In total, it's ~36 seconds.
What is more important — we observe event loop delays up to 50,000 ms!

comp1 — Original master

And here is what we have now: asynchronous parallel processing in native worker threads. (defaults to 4 libuv threads)

Execution Time for transpiling all 301 files: 222.43 ms
Execution Time for transpiling all 301 files: 190.51 ms
Execution Time for transpiling all 301 files: 2357.43 ms

In total, it's ~2,8 seconds.

Comparing to master: 36 / 2.8 = 12.8x!!!

What's also important is that I can't get rid of event loop delays totally, but now there are smaller portions of delays, and they are way shorter (5,000 VS 50,000).

comp7 — SWC async backend-native threads-2

Check List

  • Tests have been run in packages where changes made if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet
  • Docs have been added / updated if required

Copy link

codecov bot commented Feb 14, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.57%. Comparing base (671bcdb) to head (bcce65d).
Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #9225   +/-   ##
=======================================
  Coverage   83.57%   83.57%           
=======================================
  Files         227      227           
  Lines       81627    81627           
=======================================
  Hits        68216    68216           
  Misses      13411    13411           
Flag Coverage Δ
cubesql 83.57% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KSDaemon KSDaemon marked this pull request as ready for review February 15, 2025 10:41
@KSDaemon KSDaemon requested review from a team as code owners February 15, 2025 10:41
@KSDaemon KSDaemon changed the title feat(schema-compiler): Use SWC compiler instead of Babel for js models transpilation feat(schema-compiler): Boost models transpilation 12x times (using SWC instead of Babel) Feb 15, 2025
@KSDaemon KSDaemon changed the title feat(schema-compiler): Boost models transpilation 12x times (using SWC instead of Babel) feat(schema-compiler): Boost models transpilation 10-13x times (using SWC instead of Babel) Feb 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant