Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
WIP: Set empirically determined Adaptive values
Based on some experiments on the cluster, it appears that it takes anywhere from 9s-15s to startup a single worker. So set the startup cost to the average value of 12s. Given it takes a bit to startup workers, bump the interval for rechecking whether to adjust workers to 6s (leaving the number of checks the same as the default of 3). Thus the time for a job to startup and get processing some data should occur a little after if not at least by the 2nd check, which avoid rapid adjustment of workers, while still allowing it to be frequent enough to notice if things need a slight tweak. This should slow down rapid downscaling and upscaling a bit hopefully avoiding fluctuations.
- Loading branch information