Investigate Whether QueryCompiler
Can Reuse JavaFileManager
for Server Duration
#4814
Labels
Milestone
QueryCompiler
Can Reuse JavaFileManager
for Server Duration
#4814
While working on #4808, I discovered that there is potential to cut formula compile time by ~50% if the
JavaFileManager
is reused across all calls toQueryCompiler#maybeCreateClassHelper
. See the test, the test results, and accompanying image below.The concerns / things that need follow up validation:
JavaFileManager
is certainly not thread safe. Is it sufficient to implement a delegating implementation that synchronizes all public methods?The Test:
-- the first batch is run sharing a single, initial,
JavaFileManager
(JFM
) that was also used in the first query.-- the second batch disables sharing the
JFM
and instead creates a new one that it closes at the end of the compilation (engine behavior as ofTableServiceAsyncTest
: Eliminate Compiler Overhead Variance; CloseQueryCompiler
'sJavaFileManager
#4808).-- the third batch goes back to sharing a single
JFM
but creates a new instance first.The first batch of group 1 has an average operation time of 48ms. (shared JFM)
The first batch of group 2 has an average operation time of 113ms. (new JFM per compilation)
The first batch of group 3 has an average operation time of 58ms. (shared JFM, but validating that the JFM is not responsible for the time creep)
Note that over time the simple formula compilation time creeps from 48ms to 69ms over ~3k formulas. This is possibly due to the temp dir that we write formula results into - I couldn't put my finger on the time creep except to eliminate blame on the
JFM
.The text was updated successfully, but these errors were encountered: