Support UR program creation from multiple device binaries

[againull]

To support multi-device AOT scenario in SYCL we need an ability to create UR program from multiple device binaries.

Changes in this PR:

• Modify the core function urProgramCreateWithBinary to support program creation from multiple device binaries.
• Add methods to ur_program to get/set per-device data like L0 module handle, build log etc. Otherwise any change in the structure of the class requires multiple changes in all UR functions which work with ur_program, in addition to this it allows to handle interop case (which implementation is an exception right now) in a single place. Also make State and some other info per-device because it is possible that UR program is associated with multiple devices and urProgramBuildExp is called multiple times for subsets, and we have to know the state per-device.

Corresponding intel/llvm PR with E2E tests: intel/llvm#15546
Also outlined minimal changes just to uplift UR tag: intel/llvm#15710

[aarongreig]

There's an ongoing effort to remove this extension by making it the default behaviour for the affected entry points #1195 as it was a workaround for an issue discovered late in the last release cycle. Since it seems like there's a compelling use case I think this should be a change to urProgramCreateWithBinary, maybe with a device info flag for adapters report whether they support creating binaries for more than one device at a time

[againull]

There's an ongoing effort to remove this extension by making it the default behaviour for the affected entry points #1195 as it was a workaround for an issue discovered late in the last release cycle. Since it seems like there's a compelling use case I think this should be a change to urProgramCreateWithBinary, maybe with a device info flag for adapters report whether they support creating binaries for more than one device at a time

Thank you, I've changed the PR to modify the core urProgramCreateWithBinary instead.

[aarongreig]

we should add this to the yml so it gets generated and used by all adapters, I think it should be an INVALID_SIZE return:

diff --git a/scripts/core/program.yml b/scripts/core/program.yml
index 3fe2632c..2a0bc14a 100644
--- a/scripts/core/program.yml
+++ b/scripts/core/program.yml
@@ -158,6 +158,7 @@ returns:
         - "`NULL != pProperties && pProperties->count > 0 && NULL == pProperties->pMetadatas`"
     - $X_RESULT_ERROR_INVALID_SIZE:
         - "`NULL != pProperties && NULL != pProperties->pMetadatas && pProperties->count == 0`"
+        - "`numDevices == 0`"
     - $X_RESULT_ERROR_INVALID_NATIVE_BINARY:
         - "If any binary in `ppBinaries` isn't a valid binary for the corresponding device in `phDevices.`"
 --- #--------------------------------------------------------------------------

[againull]

Thanks, fixed.

[kbenzie]

@againull looks like there are conformance tests which still need updated. Some conformance tests are only enabled when the UR_DPCXX CMake variable points at clang++ from an intel/llvm build as they depend on generating program binaries.

[againull]

@againull looks like there are conformance tests which still need updated. Some conformance tests are only enabled when the UR_DPCXX CMake variable points at clang++ from an intel/llvm build as they depend on generating program binaries.

Oh, I see, thanks, I will check and fix that.

[againull]

@againull looks like there are conformance tests which still need updated. Some conformance tests are only enabled when the UR_DPCXX CMake variable points at clang++ from an intel/llvm build as they depend on generating program binaries.

Fixed remaining failures.

[againull]

By default I don't think the pre-commit runs on PVC or DG2 or anything other than GEN12 for the most part.

just fyi - we have a multi-device PVC system in UR CI. You just need to add tests, which this doesn't.

@nrspruit @pbalcer I have enabled existing "program" conformance test (f3fa464), as well as added new multi-device test (88fb72a) to verify urProgramCreateWithBinary.

[againull]

@oneapi-src/unified-runtime-level-zero-write @oneapi-src/unified-runtime-native-cpu-write @oneapi-src/unified-runtime-command-buffer-write Could you please take a look.

[nrspruit]

LGTM, thank you for adding the UR tests!

[EwanC]

@oneapi-src/unified-runtime-level-zero-write @oneapi-src/unified-runtime-native-cpu-write @oneapi-src/unified-runtime-command-buffer-write Could you please take a look.

I can't see any files owned by @oneapi-src/unified-runtime-command-buffer-write in this PR diff, I assume the change has been refactored in some way to avoid touching these files since this reviewer group was requested.

[PietroGhg]

Native CPU lgtm, thank you

[againull]

@oneapi-src/unified-runtime-maintain Could you please merge this PR.