feat: support asof join

[zenus]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

1. thanks to @xudong963 for the nice work of range join
2. thanks to duckdb for the nice idea of refactor window functions to implement asof join

Currently, due to different ways of implementing range join, the order of results obtained by asof join is random. I may be able to get help from @xudong963。that's why i do not add any test case.

Benchmark

build side : 5w probe side : 5w

• Fixes #[https://github.com/feat: support asof join  #15410]

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[xudong963]

Thanks @zenus

I think we don't need to ensure the order of results. It makes sense for different databases to have different results order.

I've skimmed through the code and there are a few points to note:

1. todo!() needs to be replaced with code that makes sense or provides a specific error message.
2. build_asof_join method can be further split into smaller functions.
3. Regarding tests, if you are going to borrow the test set from duckdb, you can put the tests in the directory https://github.com/datafuselabs/databend/tree/main/tests/sqllogictests/suites/duckdb/asof_ join
4. AsOf join can be used as an alternative to window to semantically express temporal relationships, if so, can you provide a performance comparison of asof join vs window?

In addition, I've opened a tracking issue on asof join, if you have time you can continue to finish some sub-issues in it.

[zenus]

Thanks @zenus

I think we don't need to ensure the order of results. It makes sense for different databases to have different results order.

I've skimmed through the code and there are a few points to note:

1. todo!() needs to be replaced with code that makes sense or provides a specific error message.
2. build_asof_join method can be further split into smaller functions.
3. Regarding tests, if you are going to borrow the test set from duckdb, you can put the tests in the directory https://github.com/datafuselabs/databend/tree/main/tests/sqllogictests/suites/duckdb/asof_ join
4. AsOf join can be used as an alternative to window to semantically express temporal relationships, if so, can you provide a performance comparison of asof join vs window?

In addition, I've opened a tracking issue on asof join, if you have time you can continue to finish some sub-issues in it.

nice advice , my pleasure.

[zenus]

@Dousir9 could you help me , i can not pass all the case, in my laptop, i passed all new test case.

[Dousir9]

@Dousir9 could you help me , i can not pass all the case, in my laptop, i passed all new test case.

@zenus Yeah, I will review this PR today.

[Dousir9]

@zenus Sorry for the wait, I was really busy, let's continue this excellent work.

1. For test_asof_join_ints.test:65, we can fix it by adding an order by column to this sqllogictest.
2. For test_asof_join_inequal.test:23, it looks like we need to support distributed execution for asof join, cloud you show the query plan for this sqllogictest under distributed mode ? you can use scripts/ci/deploy/databend-query-cluster-3-nodes.sh to start a cluster, and then execute explain ....
3. Resolving conflicting files with the main branch.

[zenus]

@zenus Sorry for the wait, I was really busy, let's continue this excellent work.

1. For test_asof_join_ints.test:65, we can fix it by adding an order by column to this sqllogictest.
2. For test_asof_join_inequal.test:23, it looks like we need to support distributed execution for asof join, cloud you show the query plan for this sqllogictest under distributed mode ? you can use scripts/ci/deploy/databend-query-cluster-3-nodes.sh to start a cluster, and then execute explain ....
3. Resolving conflicting files with the main branch.

@Dousir9 i have fixed the first and third , and for the second one i have no idea after i took a week to read the code.

[zenus]

@Dousir9 i have fixed the first and third , and for the second one i have no idea after i took a week to read the code.

@xudong963 could you help me out ?

[xudong963]

Hi @zenus , sorry for the late reply, we've solved some issues in iejoin by now, could you please update your PR?

[zenus]

ocessors/transforms/hash_join/hash_join_probe_state.rs
src/query/service/src/pipelines/processors/transforms/hash_join/mod.rs
src/query/service/src/pipelines/processors/transforms/range_join/ie_join_state.rs
src/query/sql/src/planner/binder/window.rs
src/query/sql/src/planner/plans/join.rs

/

Hi @zenus , sorry for the late reply, we've solved some issues in iejoin by now, could you please update your PR?

ok

[xudong963]

Invalid Asof join, missing inequality condition