feat: support RightAnti for SortMergeJoin
#13680
Conversation
|
@comphead - are you able to review this? |
| select 41 a, 42 b union all | ||
| select 41 a, 45 b | ||
| ) | ||
| select t2.* from t1 right anti join t2 on t1.a = t2.a and t1.b = t2.b |
There was a problem hiding this comment.
interesting thing: right ANTI/SEMI naturally not supported by SQL text, however DF supports it, DuckDB/PG will fail on query like that because of parse error.
DF runs it and picks the join type properly
| plan_type | plan |
+---------------+---------------------------------------------------------------------------------------------+
| logical_plan | Sort: t2.a ASC NULLS LAST, t2.b ASC NULLS LAST |
| | RightAnti Join: t1.a = t2.a, t1.b = t2.b |
From one side it is really convenient to test out queries with RIGHT ANTI/SEMI using SQL, from another it is not very expected behavior comparing to other engines.
WDYT should we do something with that? @alamb @Dandandan @ozankabak @jayzhan-synnada @findepi
There was a problem hiding this comment.
DuckDB/PG failed because of not supported, in this case, I think it is fine that we have different result because we support the query.
There was a problem hiding this comment.
I think it's fine to support the syntax, although it's not common for other engines to support it.
| @@ -2910,6 +2984,310 @@ mod tests { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn join_right_anti_one() -> Result<()> { | |||
There was a problem hiding this comment.
The meaning is "a column." This naming has already been updated.
| @@ -1920,6 +1939,10 @@ impl SortMergeJoinStream { | |||
| let output_column_indices = (0..left_columns_length).collect::<Vec<_>>(); | |||
| filtered_record_batch = | |||
| filtered_record_batch.project(&output_column_indices)?; | |||
| } else if matches!(self.join_type, JoinType::RightAnti) { | |||
| let output_column_indices = (0..right_columns_length).collect::<Vec<_>>(); | |||
There was a problem hiding this comment.
I think we can factor out this part into separate method
|
Thanks @comphead . I've made the changes based on your suggestions. Could you please take another look? |
| @@ -2910,6 +2992,310 @@ mod tests { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn join_right_anti_one_one() -> Result<()> { | |||
There was a problem hiding this comment.
Thanks @irenjj sorry I still not very sure what is one_one?
is it join by one column?
Which issue does this PR close?
Closes #13472
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?