Description

This PR implements streaming DeviceSelect and DevicePartition that, for very large inputs exceeding INT_MAX number of items, splits up the input into partitions of at most INT_MAX number of items and processes one partition at a time.

Closes #2238
Closes #1422
Closes #1437
Closes #1614

TODOs

• Adapt DeviceSelect interface to take num_items as int64_t.
• Adapt DevicePartition interface to take templatized num_items, use the kernel with minimal changes for int32_t and smaller NumItemsT and use full-fledged streaming kernel for the remaining types.
• Add tests for large number of items for DeviceSelect::Flagged.
• Add tests for large number of items for DevicePartition::If.
• Add tests for large number of items for DevicePartition::Flagged.
• Add tests for large number of items for DeviceSelect::Unique.
• Add benchmarks for DevicePartition with two distinct iterators (one for selected and one for rejected items).
• Mitigate performance degradation for DeviceSelect::If.
• Mitigate performance degradation for DeviceSelect::Flagged.
• Mitigate performance degradation for DeviceSelect::Unique.
• Mitigate performance degradation for DevicePartition::If.
• Mitigate performance degradation for DevicePartition::Flagged.
• Mitigate performance degradation for DevicePartition::If with distinct partitions.
• Update tests for large number of items for DeviceSelect::If.
• Open issue for adding support for large number of items for DevicePartition::If using ThreeWayPartition and tests.
• Account in thrust dispatch for copy_if et al. for the optimal choice of offset types.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

Latest benchmark results on 95de26f

Approach

• The streaming DeviceSelect and DevicePartition splits up inputs larger than INT_MAX into partitions of up to INT_MAX items each, repeatedly invoking the respective algorithm
• We provide a streaming_context object to the algorithm that provides all information about the current partition (i.e., for the current kernel invocation), like offsets into the input and output iterators.
• For DevicePartition::{If,Flagged}
  • we use the streaming_context, iff the user-provided offset type is uint_32 or wider than 4 bytes
  • otherwise, we use a dummy streaming_context that basically returns a 0 immediate value for the offsets et al. to keep performance impact minimal
• For DeviceSelect::{If,Flagged,Unique} we always use the streaming_context as there's negligible performance downside. Always using the streaming_context provides the benefit that we compile just one single kernel no matter the user-provided offset type here. Another benefit is that, in future, we only have to tune one kernel template specialization.

Summaries

How to interpret:

• Diff i32 vs i32.main: We use only a dummy streaming_context. These columns compare the algorithm with a dummy streaming_context to the performance numbers we got in main (using i32 offsets) today.
• Diff i64 vs i32.main: We use a streaming_context that provides offsets using i64. I.e., whenever we need to index into the overall inputs / outputs across partitions, we use i64 offsets, other indexing happens using i32. These columns compare the algorithm with a proper streaming_context to what we have in main today using i32(!) offsets.

What to focus on:

• DeviceSelect is, from now on, always using i64, because there is only very limited performance downside from using i64 instead of i32 with the streaming approach, while we have the benefit of just having to maintain and tune a single kernel template instantiating going forward. Given that, we want to focus on the rightmost two columns for DeviceSelect in the following summary.
• DevicePartition is using a static dispatch, i.e., i32 (dummy streaming context) and i64 (full-fledged streaming), depending on the user-provided OffsetT. A use of i32 retains sass compatibility to what we have in main today. So, basically unchanged for i32 user-provided offset type.

Summary on benefits of streaming i64 versus main i64 offset type:
In the following we compare the worst-case slowdown of 2^28 number of items of the two mentioned approaches:

• DeviceSelect::If: 4.64% versus 169%
• DeviceSelect::Flagged: 1.5% versus 91%
• DevicePartition::If: 18.9% versus 30.95%
• DevicePartition::Flagged: 11.8% versus 36.9%

Select.If

Select.Flagged

Select.Unique

Partition.If

Partition.Flagged

Detailed benchmark results