[lapack][blas][cuda] Update host task impl to use enqueue_native_command

[JackAKirk]

Description

Update host task impl to use enqueue_native_command for blas/lapack using the cuda backend (cublas/cusolver). I did both backends in a single PR because the cusolver backend uses the cublas backend of oneMKL.

The sycl_ext_codeplay_enqueue_native_command extension reduces latency wrt the host_task for native library submissions, and allows integration with sycl task_graph / events. See https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/experimental/sycl_ext_codeplay_enqueue_native_command.asciidoc
for details.

This extension has already been shown to lead to considerable performance improvements for applications that call oneMKL, such as Gromacs for the oneMKL fft backend. We expect similar improvements for the lapack and blas backends implemented here.

I had to update the lapack tests because they previously relied on the synchronous behaviour of the native calls due to the fact we had to sync the native streams, since previously with standard host_task we are not able to integrate the native event into the sycl task_graph/ sycl::event.
I did not need to update the blas tests since they already take into account asynchronous behaviour.

Checklist

#216 is for the most part fixed, but technically this PR maximally enables ooo queue interoperability so we can say that this
fixes #216

All Submissions

• Do all unit tests pass locally? Attach a log.

I've added a test for each backend for each of the possible codepaths:

• SYCL_EXT_ONEAPI_ENQUEUE_NATIVE_COMMAND is defined: "..native_command"
• SYCL_EXT_ONEAPI_ENQUEUE_NATIVE_COMMAND is not defined so use previous code path with standard host_task

test_main_blas_ct_host_task.txt
test_main_blas_rt_host_task.txt
test_main_lapack_rt_native_command.txt
test_main_lapack_ct_native_command.txt
test_main_lapack_ct_host_task.txt
test_main_lapack_rt_host_task.txt
test_main_blas_ct_res_native_command.txt
test_main_blas_rt_res_native_command.txt

[Rbiessy]

LGTM, thanks for the PR

[JackAKirk]

@hdelan please review this when you are back.

[JackAKirk]

I have a small patch ready to update cublas backend just a little to implement missing GEMV_BATCH.
This I think puts cublas backend to a status where everything that maps directly between oneMKL and cublas APIs is supported to some degree (some types remain unimplemented, such as bfloat16/some mixed precisions already identified in the issues board etc).
Is it OK for me to add it here, to save the PR review overhead?
@Rbiessy what do you think?

[Rbiessy]

I would prefer to have a separate PR to make it clearer which commit implements what.

[JackAKirk]

I would prefer to have a separate PR to make it clearer which commit implements what.

Yeah OK, I'll be patient, thanks.

[JackAKirk]

Hi @oneapi-src/onemkl-blas-write @oneapi-src/onemkl-lapack-write
would it be possible for you to review this?

Thanks

[hdelan]

LGTM

[andrewtbarker]

The BLAS part looks good, thank you. I'm less familiar with the LAPACK part.

[sknepper]

Could you please provide more details about why the LAPACK tests required updating? I see your comment in the description of the PR, but I still have a few questions. Thanks!
Also, if the LAPACK tests do need updating, how will that impact testing the other (non-cuSOLVER) backends?

[sknepper]

Thanks for all your work here to improve performance, @JackAKirk !