Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use makeCopy to change relation in FileSourceScanExec #207

Merged
merged 1 commit into from
Mar 14, 2024

Conversation

viirya
Copy link
Member

@viirya viirya commented Mar 13, 2024

Which issue does this PR close?

Relates #190.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

@viirya viirya force-pushed the use_make_copy branch 2 times, most recently from bc12dc3 to 8f992c0 Compare March 13, 2024 20:42
@viirya
Copy link
Member Author

viirya commented Mar 13, 2024

mapProductIterator in class TreeNode cannot be accessed in org.apache.spark.sql.execution.FileSourceScanExec

Ah, mapProductIterator is protected method. Let me update this later.

@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 84.61538% with 2 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (main@9aa42cd). Click here to learn what that means.

Files Patch % Lines
...ala/org/apache/spark/sql/comet/CometScanExec.scala 84.61% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #207   +/-   ##
=======================================
  Coverage        ?   33.33%           
  Complexity      ?      769           
=======================================
  Files           ?      107           
  Lines           ?    35406           
  Branches        ?     7673           
=======================================
  Hits            ?    11804           
  Misses          ?    21147           
  Partials        ?     2455           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@@ -439,8 +440,29 @@ case class CometScanExec(

object CometScanExec {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps we can also consider CometScanExec extends TreeNode to remove mapProductIterator method copy?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to run mapProductIterator on FileSourceScanExec. I'm not sure how object CometScanExec extends TreeNode can help us?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, its needed to run mapProductIterator on FileSourceScanExec, I wrongly assumed this method needed for CometScanExec .

@viirya
Copy link
Member Author

viirya commented Mar 14, 2024

This patch cannot completely resolve the ticket #190 but only fixes the first error. After the first error is fixed, there comes out another API incompatibility issue around DataSourceScanExec.

I think it is still good to have this patch to fix the first error.

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @viirya

@viirya viirya merged commit 4b66a78 into apache:main Mar 14, 2024
27 checks passed
@viirya
Copy link
Member Author

viirya commented Mar 14, 2024

Merged. Thanks.

@viirya viirya deleted the use_make_copy branch March 14, 2024 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants