Skip to content

Commit

Permalink
[GLUTEN-6864] Error during spilling: The current running driver and t…
Browse files Browse the repository at this point in the history
…he request driver must be from the same task
  • Loading branch information
zhztheplayer committed Aug 15, 2024
1 parent 4ae3cb1 commit 774e48e
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions cpp/velox/compute/WholeStageResultIterator.cc
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,15 @@ int64_t WholeStageResultIterator::spillFixedSize(int64_t size) {
int64_t shrunken = memoryManager_->shrink(size);
// todo return the actual spilled size?
if (spillStrategy_ == "auto") {
exec::DriverThreadContext* driverThreadCtx = exec::driverThreadContext();
if (driverThreadCtx != nullptr) {
if (driverThreadCtx->driverCtx.task->taskId() != task_->taskId()) {
// Disallow one task from spilling another. This is by Velox's restriction. See
// https://github.com/facebookincubator/velox/blob/9523840ac329be4f7318338fb92cb92bcb0a8c07/velox/exec/Operator.cpp#L566-L575.
VLOG(2) << logPrefix << "One Velox task is trying to spill another, ignoring.";
return shrunken;
}
}
int64_t remaining = size - shrunken;
LOG(INFO) << logPrefix << "Trying to request spilling for " << remaining << " bytes...";
// suspend the driver when we are on it
Expand Down

0 comments on commit 774e48e

Please sign in to comment.