Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-6600]Fix NPE issue when running window sql #6803

Merged
merged 1 commit into from
Aug 14, 2024
Merged

[GLUTEN-6600]Fix NPE issue when running window sql #6803

merged 1 commit into from
Aug 14, 2024

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Aug 13, 2024 •
edited
Loading

What changes were proposed in this pull request?

When running the following sql in yarn client mode, we got the following exception:

spark.sql("create temporary view d233 using parquet options ( path 'file:///home/jk/d233.parquet')")
spark.sql("select sum(impression_count_last7days ) from (     SELECT sum(impression_count) OVER (         PARTITION BY user_id,         surface         ORDER BY date RANGE BETWEEN 6 preceding AND CURRENT ROW       ) AS impression_count_last7days     from d233 )").show()

24/08/12 16:13:26 WARN TaskSetManager: Lost task 86.1 in stage 3.0 (TID 263) (sr246 executor 8): TaskKilled (Stage cancelled: Job aborted due to stage failure: Task 90 in stage 3.0 failed 4 times, most recent failure: Lost task 90.3 in stage 3.0 (TID 254) (sr246 executor 6): java.lang.NullPointerException
        at org.apache.gluten.expression.ExpressionMappings$.expressionsMap(ExpressionMappings.scala:341)
        at org.apache.gluten.expression.ExpressionConverter$.replaceWithExpressionTransformer(ExpressionConverter.scala:54)
        at org.apache.gluten.expression.ExpressionConverter.replaceWithExpressionTransformer(ExpressionConverter.scala)
        at org.apache.gluten.substrait.expression.WindowFunctionNode.setBound(WindowFunctionNode.java:102)
        at org.apache.gluten.substrait.expression.WindowFunctionNode.toProtobuf(WindowFunctionNode.java:180)
        at org.apache.gluten.substrait.rel.WindowRelNode.toProtobuf(WindowRelNode.java:77)
        at org.apache.gluten.substrait.rel.ProjectRelNode.toProtobuf(ProjectRelNode.java:69)
        at org.apache.gluten.substrait.rel.AggregateRelNode.toProtobuf(AggregateRelNode.java:89)
        at org.apache.gluten.substrait.plan.PlanNode.toProtobuf(PlanNode.java:74)
        at org.apache.gluten.backendsapi.velox.VeloxIteratorApi.genFinalStageIterator(VeloxIteratorApi.scala:238)
        at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:59)
        at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
        at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
        at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
        at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
        at org.apache.spark.scheduler.Task.run(Task.scala:141)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
        at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

How was this patch tested?

local verified.

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions github-actions bot added the CORE works for Gluten Core label Aug 13, 2024
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf changed the title fix NPE issue [GLUTEN-6600]Fix NPE issue when running window sql Aug 13, 2024
@github-actions GitHub Actions
Copy link

#6600

@JkSelf JkSelf requested review from zzcclp and ulysses-you August 13, 2024 08:14
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

ulysses-you
ulysses-you reviewed Aug 13, 2024
View reviewed changes
gluten-core/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala Outdated
expressionExtensionTransformer.extensionExpressionsMapping

var supportedExprs = defaultExpressionsMap
if (expressionExtensionTransformer != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about giving expressionExtensionTransformer a initial value DefaultExpressionExtensionTransformer ? it seems quite risk to check if not null at everywhere..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ulysses-you It's strange that the ExpressionUtil.extendedExpressionTransformer method is returning null. I'm not sure why this is happening. It should be return the DefaultExpressionExtensionTransformer. Do you have any insights?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you mean ExpressionUtil.extendedExpressionTransformer returns null but not DefaultExpressionExtensionTransformer() when extendedExpressionTransformer is empty ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zzcclp Yes.

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

ulysses-you
ulysses-you approved these changes Aug 14, 2024
View reviewed changes
Copy link
Contributor

@ulysses-you ulysses-you left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @JkSelf

@FelixYBW FelixYBW mentioned this pull request Aug 14, 2024
Closed
@JkSelf JkSelf merged commit bf9ebd5 into apache:main Aug 14, 2024
43 checks passed
@JkSelf JkSelf mentioned this pull request Aug 14, 2024
Closed
sharkdtu pushed a commit to sharkdtu/gluten that referenced this pull request Nov 11, 2024
@JkSelf
[GLUTEN-6600]Fix NPE issue when running window query(apache#6803)
45991bc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ulysses-you ulysses-you ulysses-you approved these changes

@zzcclp zzcclp Awaiting requested review from zzcclp

Assignees
No one assigned
Labels
CORE works for Gluten Core
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JkSelf @zzcclp @ulysses-you