GH-39301: [Archery][CI][Integration] Add nanoarrow to archery + integration setup

[paleolimbot]

Rationale for this change

The ability to add integration testing was added in nanoarrow however, the infrastructure for running these tests currently lives in the arrow monorepo.

What changes are included in this PR?

• Added the relevant code to Archery such that these tests can be run
• Added the relevant scripts/environment variables to CI such that these tests run in the integration CI job

Are these changes tested?

Yes, via the "Integration" CI job.

Are there any user-facing changes?

No.

This PR still needs #41264 for the integration tests to pass.

• Closes: [Archery][CI][Integration] Add nanoarrow integration tests to archery/CI #39301
• GitHub Issue: [Archery][CI][Integration] Add nanoarrow integration tests to archery/CI #39301

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #39301 has been automatically assigned in GitHub to PR creator.

[paleolimbot]

A few outstanding issues:

I'm not sure how to skip based on type in Archery. nanoarrow doesn't support the new types yet, so it doesn't add them in integration testing. There must be an example of this but I can't seem to find it!

FAILED TEST: run_end_encoded C++ producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Unsupported Type name: 'runendencoded'

FAILED TEST: binary_view C++ producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Unsupported Type name: 'binaryview'

FAILED TEST: list_view C++ producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Unsupported Type name: 'listview'

FAILED TEST: run_end_encoded C++ producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Unsupported Type name: 'runendencoded'

C# appears to export record batches as nullable structs, whereas nanoarrow expects them to be non-nullable. In nanoarrow I should probably just ignore the difference for a top-level "batch".

FAILED TEST: recursive_nested C# producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Found 1 differences:
Path: 
- .flags: 2
+ .flags: 0

Java's metadata probably goes through a hash map or something because it looks like the order is not always maintained. We can relax the comparison to consider all keys/values of the metadata as a whole:

 FAILED TEST: custom_metadata Java producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Found 2 differences:
Path: .children[1]
- {"name": "lots_of_meta", "nullable": true, "type": {"name": "int", "bitWidth": 8, "isSigned": true}, "children": [], "metadata": [{"key": "..", "value": "{}"}, {"key": "a", "value": "{}"}, {"key": "b", "value": "{}"}, {"key": "c", "value": "{}"}, {"key": "d", "value": "{}"}, {"key": "w", "value": "{}"}, {"key": "x", "value": "{}"}, {"key": "y", "value": "{}"}, {"key": "z", "value": "{}"}]}
+ {"name": "lots_of_meta", "nullable": true, "type": {"name": "int", "bitWidth": 8, "isSigned": true}, "children": [], "metadata": [{"key": "a", "value": "{}"}, {"key": "b", "value": "{}"}, {"key": "c", "value": "{}"}, {"key": "d", "value": "{}"}, {"key": "..", "value": "{}"}, {"key": "w", "value": "{}"}, {"key": "x", "value": "{}"}, {"key": "y", "value": "{}"}, {"key": "z", "value": "{}"}]}

It looks like Go exports zero-length metadata (i.e., b"\x00\x00\x00\x00") instead of NULL metadata. We can relax that check in the comparison.

 FAILED TEST: primitive_no_batches Go producing,  nanoarrow consuming
<class 'RuntimeError'>: nanoarrow C Data Integration call failed: Found 31 differences:
Path: .children[0]
- {"name": "bool_nullable", "nullable": true, "type": {"name": "bool"}, "children": [], "metadata": []}
+ {"name": "bool_nullable", "nullable": true, "type": {"name": "bool"}, "children": []}

It looks like C# has some issues with the arrays produced by nanoarrow (or at least by the JSON reader):

 FAILED TEST: nested_dictionary nanoarrow producing,  C# consuming
<class 'Xunit.Sdk.TrueException'>: Validity buffers do not match.
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 146
   at Apache.Arrow.Tests.ArrowReaderVerifier.ArrayComparer.CompareValidityBuffer(Int32 nullCount, Int32 arrayLength, ArrowBuffer expectedValidityBuffer, ArrowBuffer actualValidityBuffer) in /arrow/csharp/test/Apache.Arrow.Tests/ArrowReaderVerifier.cs:line 435

...and C# reports a memory leak. I would have assumed that a memory leak would have been consistent between languages so I'm puzzled by this one.

 FAILED TEST: extension nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 694, after = 734 (should have been equal)

[paleolimbot]

Ok, we're now down to:

################# FAILURES #################
FAILED TEST: primitive_zerolength nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 0, after = 16 (should have been equal)

FAILED TEST: recursive_nested nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 16, after = 668 (should have been equal)

FAILED TEST: union nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 668, after = 676 (should have been equal)

FAILED TEST: custom_metadata nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 676, after = 691 (should have been equal)

FAILED TEST: extension nanoarrow producing,  C# consuming
<class 'RuntimeError'>: Memory was not released correctly after roundtrip: before = 691, after = 731 (should have been equal)

(i.e., only very specific files, but the same files each time, are reporting leaks when C# is responsible for releasing)

[felipecrv]

LGTM, but I'm not to be trusted reviewing shell script and Python.

[raulcd]

Unless we clone nanoarrow inside the arrow repo ARROW_ROOT_DEFAULT would never be the correct path here as it will include the arrow folder.

[paleolimbot]

If you wanted to do this interactively that is basically what you would have to do (I've tried this and it does work)...this is the approach used by the Rust tester that I copied here. In docker-compose.yml this is overridden such that the build happens elsewhere.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 899422e.

None of the specified runs were found on the Conbench server.

The full Conbench report has more details.

[pitrou]

Thanks for this @paleolimbot !