Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Following #6988, move a warning from core to Velox backend #7010

Merged
merged 1 commit into from
Aug 26, 2024
Merged

[VL] Following #6988, move a warning from core to Velox backend #7010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxListenerApi.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ class VeloxListenerApi extends ListenerApi with Logging {
override def onDriverStart(sc: SparkContext, pc: PluginContext): Unit = {
val conf = pc.conf()

// FIXME: The following is a workaround. Remove once the causes are fixed.
conf.set(GlutenConfig.GLUTEN_OVERHEAD_SIZE_IN_BYTES_KEY, Long.MaxValue.toString)
logWarning(
"Setting overhead memory that Gluten can use to UNLIMITED. This is currently a" +
" temporary solution to avoid OOM by Velox's global memory pools." +
" See GLUTEN-6960 for more information.")

// Sql table cache serializer.
if (conf.getBoolean(GlutenConfig.COLUMNAR_TABLE_CACHE_ENABLED.key, defaultValue = false)) {
conf.set(
Expand Down
7 changes: 0 additions & 7 deletions gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,13 +184,6 @@ private[gluten] class GlutenDriverPlugin extends DriverPlugin with Logging {
val overheadSize: Long = SparkResourceUtil.getMemoryOverheadSize(conf)
conf.set(GlutenConfig.GLUTEN_OVERHEAD_SIZE_IN_BYTES_KEY, overheadSize.toString)

// FIXME: The following is a workaround. Remove once the causes are fixed.
conf.set(GlutenConfig.GLUTEN_OVERHEAD_SIZE_IN_BYTES_KEY, Long.MaxValue.toString)
logWarning(
"Setting overhead memory that Gluten can use to UNLIMITED. This is currently a" +
" temporary solution to avoid OOM by Velox's global memory pools." +
" See GLUTEN-6960 for more information.")

// If dynamic off-heap sizing is enabled, the off-heap size is calculated based on the on-heap
// size. Otherwise, the off-heap size is set to the value specified by the user (if any).
// Note that this means that we will IGNORE the off-heap size specified by the user if the
Expand Down
Loading