feat(query): partition sort spill #16987

forsaken628 · 2024-12-03T06:25:23Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Implemente TransformStreamSortSpill.

In the previous implementation, the spill phase would immediately spill to storage when it received a data block, but in this PR, it is changed to buffer a batch, sort it first, and then spill to storage.
The spill phase samples the data blocks proportionally and determines the partition boundaries from them.
The restore phase restores and sorts according to the partition order, avoiding unnecessary IO as much as possible.
~~Unimplemented~~ optimization: lazy spill, i.e., delaying the timing of the spill to fully utilize all available memory and avoid unnecessary IO.
Unimplemented optimization: sort row rebuild, i.e., discard sort row columns during spill and rebuild them during restore to reduce IO.
Unimplemented optimization: memory centralization, centralized memory is better for reducing IO than scattered memory.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

Signed-off-by: coldWater <[email protected]>

github-actions · 2024-12-05T08:56:57Z

Docker Image for PR

tag: pr-16987-e723c92-1733388924

note: this image tag is only available for internal use,
please check the internal doc for more details.

forsaken628 · 2024-12-05T10:47:32Z

benchmark:

data set
tpch_sf_100

sql

EXPLAIN ANALYZE SELECT * FROM customer ORDER BY c_custkey,c_nationkey ignore_result;

settings

set max_threads = 3;
set sort_spilling_bytes_threshold_per_proc = 1024*1024*500;

this pr
version: pr-16987-e723c92-1733388924
size: XSmall
total execution time: 75.3s
estimated rows: 15000000.00
cpu time: 26.781757575s
wait time: 94.645316229s
output rows: 15 million
output bytes: 3.05 GiB
numbers remote spilled by write: 443
bytes remote spilled by write: 1.32 GiB
remote spilled time by write: 74.059s
numbers remote spilled by read: 443
bytes remote spilled by read: 1.32 GiB
remote spilled time by read: 37.55s

main
version: Databend Query v1.2.668-nightly-35c1c24742(rust-1.81.0-nightly-2024-12-02T06:44:39.368245594Z)
size: XSmall
total execution time: 124.6s
estimated rows: 15000000.00
cpu time: 51.686441524s
wait time: 201.063679221s
output rows: 15 million
output bytes: 3.04 GiB
numbers remote spilled by write: 965
bytes remote spilled by write: 2.90 GiB
remote spilled time by write: 154.229s
numbers remote spilled by read: 965
bytes remote spilled by read: 2.90 GiB
remote spilled time by read: 84.61s

lazy spill
version: pr-16987-aa68ff9-1733728820
size: XSmall
total execution time: 60.2s
estimated rows: 15000000.00
cpu time: 25.603120611s
wait time: 67.799707406s
output rows: 15 million
output bytes: 3.09 GiB
numbers remote spilled by write: 411
bytes remote spilled by write: 1.21 GiB
remote spilled time by write: 53.808s
numbers remote spilled by read: 411
bytes remote spilled by read: 1.21 GiB
remote spilled time by read: 29.813s

remove sort_desc parameter from various sorting functions and implementations. Adjusted implementations of `Rows` and `SortAlgorithm` traits to eliminate unnecessary `Send` bounds. sort_row_offset.

github-actions · 2024-12-07T13:45:43Z

Docker Image for PR

tag: pr-16987-c00c0cd-1733579081

note: this image tag is only available for internal use,
please check the internal doc for more details.

src/query/expression/src/lib.rs

github-actions · 2024-12-09T07:21:47Z

Docker Image for PR

tag: pr-16987-aa68ff9-1733728820

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2024-12-11T04:11:36Z

Docker Image for PR

tag: pr-16987-e55e2c0-1733890235

note: this image tag is only available for internal use,
please check the internal doc for more details.

github-actions · 2024-12-11T04:54:40Z

ClickBench Report

forsaken628 added 10 commits November 27, 2024 15:52

init

628febe

Signed-off-by: coldWater <[email protected]>

update

899b55d

Signed-off-by: coldWater <[email protected]>

x

c5807d7

Signed-off-by: coldWater <[email protected]>

test

41252f6

Signed-off-by: coldWater <[email protected]>

update

0a9182b

Signed-off-by: coldWater <[email protected]>

fixed_size_sampler

df595e1

Signed-off-by: coldWater <[email protected]>

fixed_rate_sampler

a2d1499

Signed-off-by: coldWater <[email protected]>

x

4805fe4

Signed-off-by: coldWater <[email protected]>

compact_blocks

b90461d

Signed-off-by: coldWater <[email protected]>

update

5358470

Signed-off-by: coldWater <[email protected]>

github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Dec 3, 2024

forsaken628 added 8 commits December 3, 2024 14:48

fix

f9422af

Signed-off-by: coldWater <[email protected]>

fix

bc210e2

Signed-off-by: coldWater <[email protected]>

fix

f5578a0

Signed-off-by: coldWater <[email protected]>

fix

f343945

Signed-off-by: coldWater <[email protected]>

fix

fc1221a

Signed-off-by: coldWater <[email protected]>

fix

7b616e1

Signed-off-by: coldWater <[email protected]>

fix

dd26305

Signed-off-by: coldWater <[email protected]>

fix

7455048

forsaken628 force-pushed the stream-sort-spill branch from 8faef81 to 7455048 Compare December 5, 2024 08:19

forsaken628 added the ci-cloud Build docker image for cloud test label Dec 5, 2024

wubx mentioned this pull request Dec 5, 2024

Refactor: improve query spill #16605

Open

3 tasks

forsaken628 requested review from sundy-li and Dousir9 and removed request for sundy-li December 6, 2024 02:22

forsaken628 marked this pull request as ready for review December 6, 2024 02:22

Merge branch 'main' into stream-sort-spill

8962ff9

refactor:

952bf44

remove sort_desc parameter from various sorting functions and implementations. Adjusted implementations of `Rows` and `SortAlgorithm` traits to eliminate unnecessary `Send` bounds. sort_row_offset.

forsaken628 force-pushed the stream-sort-spill branch from 570d2b2 to 985397d Compare December 7, 2024 08:17

fix

a74522c

forsaken628 force-pushed the stream-sort-spill branch from 985397d to a74522c Compare December 7, 2024 11:39

forsaken628 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Dec 7, 2024

sundy-li reviewed Dec 9, 2024

View reviewed changes

src/query/expression/src/lib.rs Outdated Show resolved Hide resolved

forsaken628 added 5 commits December 9, 2024 10:22

typo

f612c69

lazy spill

3681d90

take_next_bounded_block

b17939b

Merge remote-tracking branch 'up/main' into stream-sort-spill

81b924a

sort entire block

a9ba2dd

forsaken628 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Dec 9, 2024

forsaken628 added 2 commits December 10, 2024 11:01

Merge remote-tracking branch 'up/main' into stream-sort-spill

93713a0

fix

7fab593

sundy-li self-requested a review December 11, 2024 01:46

forsaken628 added the ci-benchmark Benchmark: run all test label Dec 11, 2024

sundy-li approved these changes Dec 11, 2024

View reviewed changes

forsaken628 removed the ci-benchmark Benchmark: run all test label Dec 11, 2024

forsaken628 added 2 commits December 11, 2024 15:18

format_memory_usage

c1a178a

Merge remote-tracking branch 'up/main' into stream-sort-spill

0a81447

sundy-li added this pull request to the merge queue Dec 12, 2024

BohuTANG removed this pull request from the merge queue due to a manual request Dec 12, 2024

BohuTANG merged commit 236fcc4 into databendlabs:main Dec 12, 2024
72 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(query): partition sort spill #16987

feat(query): partition sort spill #16987

forsaken628 commented Dec 3, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024

forsaken628 commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 7, 2024

github-actions bot commented Dec 9, 2024

github-actions bot commented Dec 11, 2024

github-actions bot commented Dec 11, 2024

feat(query): partition sort spill #16987

feat(query): partition sort spill #16987

Conversation

forsaken628 commented Dec 3, 2024 • edited Loading

Summary

Tests

Type of change

github-actions bot commented Dec 5, 2024

Docker Image for PR

forsaken628 commented Dec 5, 2024 • edited Loading

benchmark:

github-actions bot commented Dec 7, 2024

Docker Image for PR

github-actions bot commented Dec 9, 2024

Docker Image for PR

github-actions bot commented Dec 11, 2024

Docker Image for PR

github-actions bot commented Dec 11, 2024

ClickBench Report

forsaken628 commented Dec 3, 2024 •

edited

Loading

forsaken628 commented Dec 5, 2024 •

edited

Loading