The destruction of GroupState in high cardinality aggregation takes a lot of time

[ic4y]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The test is as follows（4core 16G MacOS）
select count(1) from (select user_id from event group by user_id)a

The total data is 350 million, and the user_id deduplication number is 5 million. The entire query takes 15s. Viewing through pprf, it is found that about 60% of the time is destructing the GroupState.

Describe the solution you'd like

Using bumpalo to allocate GroupState to a chunk of memory as much as possible, and then release it wholely. Will the destruction time be much better in this way?

[ic4y]

By using --features "mimalloc", it was found that the test results did not differ much.

[rdettai]

Thanks for the analysis @ic4y ! I am quite surprised we pay the fragmentation that comes from the row oriented structure of Accumulators that much more at de-allocation time than when computing the actual aggregates.

I guess the arena strategy that you are suggesting should work, though I don't know bumpalo specifically.

It is worth referencing your discussion about making accumulators column based in #956, which I believe should also solve this issue.

[ic4y]

By using --features "mimalloc", it was found that the test results did not differ much.

Using --features "mimalloc" did not take effect.

Add the following code in the main.rs (this is mentioned in the user guide) can solve this problem

#[global_allocator]
static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;