Skip to content

Commit

Permalink
WIP: Set empirically determined Adaptive values
Browse files Browse the repository at this point in the history
Based on some experiments on the cluster, it appears that it takes
anywhere from 9s-15s to startup a single worker. So set the startup cost
to the average value of 12s. Given it takes a bit to startup workers,
bump the interval for rechecking whether to adjust workers to 6s
(leaving the number of checks the same as the default of 3). Thus the
time for a job to startup and get processing some data should occur a
little after if not at least by the 2nd check, which avoid rapid
adjustment of workers, while still allowing it to be frequent enough to
notice if things need a slight tweak. This should slow down rapid
downscaling and upscaling a bit hopefully avoiding fluctuations.
  • Loading branch information
jakirkham committed Apr 5, 2018
1 parent 271a997 commit c47df22
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions nanshe_ipython.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@
"}\n",
"client_kwargs = {}\n",
"adaptive_kwargs = {\n",
" \"startup_cost\": \"12s\",\n",
" \"interval\": \"6s\",\n",
" \"minimum\": 0,\n",
" \"maximum\": int(os.environ.get(\"CORES\", cpu_count())) - 1\n",
"}"
Expand Down

0 comments on commit c47df22

Please sign in to comment.