destination-s3: Use airbyte/java-connector-base:2.0.0

[alafanechere]

What

Make the connector use our official non-root base image for java connectors

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 17, 2025 8:46pm

[alafanechere]

• destination-s3: Use airbyte/java-connector-base:2.0.0 #51536 👈 (View in Graphite)
• airbyte-ci: correct mounted volumes ownership  #51582 : 2 other dependent PRs (#43869 , #48656 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[edgao]

resolve: process "sh /tmp/finalize_build.sh" did not complete successfully: exit code: 1
Stdout:
Running destination-s3 docker custom steps...
x86_64
Stderr:
Error: This command has to be run with superuser privileges (under the root user on most systems).

... the dest-s3 build is running yum install, which I guess needs sudo :/ is there some way to run the build as root, but run the actual container as non-root?

[edgao]

oh right, this is in the finalize script, not even the docker build itself. @alafanechere do we have a way to run that script as root?

[alafanechere]

oh right, this is in the finalize script, not even the docker build itself. @alafanechere do we have a way to run that script as root?

You're right. It should be run as root at build time then. It'll require an airbyte-ci change. I'll do that.

[alafanechere]

@edgao airbyte-ci now runs the finalize script as root. The build is passing 👍
Unfortunately the integration tests are not...
I'm seeing errors like:

Caused by: software.amazon.awssdk.crt.CrtRuntimeException: software.amazon.awssdk.crt.CrtRuntimeException: Invalid directory: /tmp UNKNOWN(-1) UNKNOWN(-1)

I'm surprised because it should be allowed to write to /tmp according to our base image definition. . But when I shell in the built image I can't indeed write to this path. I'm investigating now.

[alafanechere]

I think I understand what's happenning:

• When we run finalize as route some files get written to /tmp, and the owner is root. Which supersedes the original ownership of this dir to the airbyte user.
  I'm going to change airbyte-ci to give /tmp ownership back to airbyte

[alafanechere]

@edgao so to summarize I made two changes to airbyte-ci:

• Run finalize as root - which could change the ownership of /tmp
• When finalize is done give back ownership of /tmp to airbyte/

I confirmed that build is passing and when shelling into the built container I'm able to read and write to /tmp.
But we still have this error:

Caused by: software.amazon.awssdk.crt.CrtRuntimeException: software.amazon.awssdk.crt.CrtRuntimeException: Invalid directory: /tmp UNKNOWN(-1) UNKNOWN(-1)

I think that it's the same problem we have with other python connectors we're trying to make non root.
I've probably identified the root cause: we mount a volume at test time to /tmp and the owner of this volume is root. I'm on it.