GH-44629: [C++][Acero] Use implicit_ordering for asof_join rather than require_sequenced_output

[EnricoMi]

Rationale for this change

Changes in #44083 (GH-41706) unnecessarily sequences batches retrieved from scanner where it only requires the batches to provide index according to implicit input order.

What changes are included in this PR?

Setting implicit_ordering causes existing code to set batch index, which is then available to the asof_join node to sequence the batches int input order. This replaces some of #44083 changes.

Some code introduced by #44083 turns out to not be required and has therefore been reverted.

Are these changes tested?

Existing unit tests still pass.

Are there any user-facing changes?

Reverts some breaking user-facing changes done by #44083.

• GitHub Issue: [Python] join_asof out-of-order error for big sorted tables #41706
• GitHub Issue: [C++][Acero] Use implicit_ordering for asof_join rather than require_sequenced_output #44629

[gitmodimo]

All looks good. I think all source nodes need an option to assert Implicit ordering but I am not sure if it is in the scope of this PR.
Is there currently any use for require_sequenced_output?

[EnricoMi]

@westonpace are you happy with this?

[kou]

Could you open a new issue instead of reusing closed issue?

[github-actions]

⚠️ GitHub issue #44629 has been automatically assigned in GitHub to PR creator.

[EnricoMi]

@kou done

[EnricoMi]

@lidavidm what do you think?

[gitmodimo]

I don't know if it in the scope of this PR, but Implicit Ordering in other source nodes also need some consideration. My thoughts:
SourceNode - ordering depends on generator and can be asserted by option - seems reasonable
TableSourceNode - currently has implicit ordering which seems reasonable since tables do have an order. Probably little to none benefit in dropping the it.
SchemaSourceNode,RecordBatchSourceNode - currently has implicit ordering, but ordering depends on generator/iterator and should not default to either. I think it should be on option like in SourceNode
RecordBatchReaderSourceNode - currently has no implicit ordering - Same case as SchemaSourceNode

I think either all source nodes should have ordering option or the ordering should originate from arrow::AsyncGenerator ?

Other than that it looks good. Thanks for fixing my mistakes.

[lidavidm]

Sorry @EnricoMi I wasn't involved in most of the Acero development so I can't really speak to the workings here. @bkietz would probably know better