Skip to content
This repository has been archived by the owner on Apr 24, 2020. It is now read-only.

Commit

Permalink
Don't shrink the executor count to non-zero number
Browse files Browse the repository at this point in the history
Fixes #14.

When using dynamic allocation, a smaller executor count than
what is currently running might be requested. Since we don't
want to kill existing allocations (they have shuffle services,
as well as executors that are potentially still running tasks),
we won't reduce the number of executor task groups unless we
are asked to reduce it to 0.
  • Loading branch information
barnardb committed Jul 22, 2018
1 parent 73fe132 commit 4e27e7f
Showing 1 changed file with 4 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,10 @@ private[spark] class SparkNomadJobController(jobManipulator: NomadJobManipulator

def setExecutorCount(count: Int): Unit = {
jobManipulator.updateJob(startIfNotYetRunning = count > 0) { job =>
SparkNomadJob.find(job, ExecutorTaskGroup).get
.setCount(count)
val executorGroup = SparkNomadJob.find(job, ExecutorTaskGroup).get
if (count > 0 && executorGroup.getCount < count) {
executorGroup.setCount(count)
}
}
}

Expand Down

0 comments on commit 4e27e7f

Please sign in to comment.