Skip to content

Commit

Permalink
WIP: Set empirically determined Adaptive values
Browse files Browse the repository at this point in the history
Based on some experiments on the cluster, it appears that it takes
anywhere from 9s-15s to startup a single worker. So set the startup cost
to the average value of 12s. Given it takes a bit to startup workers,
bump the interval for rechecking whether to adjust workers to 6s
(leaving the number of checks the same as the default of 3). Thus the
time for a job to startup and get processing some data should occur a
little after if not at least by the 2nd check, which avoid rapid
adjustment of workers, while still allowing it to be frequent enough to
notice if things need a slight tweak. This should slow down rapid
downscaling and upscaling a bit hopefully avoiding fluctuations.
  • Loading branch information
jakirkham committed Apr 5, 2018
1 parent 14ee9e0 commit 8d0c6cf
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions nanshe_ipython.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@
"}\n",
"client_kwargs = {}\n",
"adaptive_kwargs = {\n",
" \"startup_cost\": \"12s\",\n",
" \"interval\": \"6s\",\n",
" \"minimum\": 0,\n",
" \"maximum\": int(os.environ.get(\"CORES\", cpu_count())) - 1\n",
"}"
Expand Down

0 comments on commit 8d0c6cf

Please sign in to comment.