Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove sink desc #3170

Merged
merged 20 commits into from
Dec 20, 2024
Merged

Remove sink desc #3170

merged 20 commits into from
Dec 20, 2024

Conversation

Yicong-Huang
Copy link
Collaborator

@Yicong-Huang Yicong-Huang commented Dec 19, 2024

This PR refactors the handling of sink operators in Texera by removing the sink descriptor and introducing a streamlined approach to creating physical sink operators during compilation and scheduling. Additionally, it shifts the storage assignment logic from the logical layer to the physical layer.

  1. Sink Descriptor Removal: Removed the sink descriptor, physical sink operators are no longer created through descriptors. In the future, we will remove physical sink operators.
  2. Sink Operator Creation:
  • Introduced a temporary factory for creating physical sink operators without relying on a descriptor.
  • Physical sink operators are now considered part of the sub-plan of their upstream logical operator. For example: If the HashJoin logical operator requires a sink, its physical sub-plan includes the building physicalOp, probing physicalOp, and the sink physicalOp.
  1. Storage Assignment Refactor:
  • Merged the storage assignment logic into the physical layer, removing it from the logical layer.
  • When a physical sink operator is created (either during compilation or scheduling), its associated storage is also created at the same moment.

@Yicong-Huang Yicong-Huang self-assigned this Dec 19, 2024
Copy link
Collaborator

@shengquan-ni shengquan-ni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with minor comments.

@Yicong-Huang Yicong-Huang added the refactoring Refactor the code label Dec 20, 2024
@Yicong-Huang Yicong-Huang merged commit f90362e into master Dec 20, 2024
8 checks passed
@Yicong-Huang Yicong-Huang deleted the yicong-remove-sink-desc branch December 20, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactoring Refactor the code
Projects
Development

Successfully merging this pull request may close these issues.

2 participants