Refactor compute dockerfile #10371
Open
Refactor compute dockerfile #10371
+591
−295
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6740 tests run: 6407 passed, 0 failed, 333 skipped (full report)Flaky tests (2)Postgres 17
Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
321ba26 at 2025-01-31T00:40:52.274Z :recycle: |
hlinnaka
force-pushed
the
refactor-compute-dockerfile
branch
8 times, most recently
from
fbe2a18 to
ef9d7d3
Compare
hlinnaka
requested review from
knizhnik,
lubennikovaav and
tristan957
and removed request for
knizhnik
tristan957
approved these changes
Jan 21, 2025
hlinnaka
force-pushed
the
refactor-compute-dockerfile
branch
7 times, most recently
from
1e837ce to
98b15a6
Compare
bayandin
reviewed
Jan 29, 2025
There was a problem hiding this comment.
It seems tests for extension tried to get regression.out even from all layers/directories (for example from onnxruntime-src)
+ docker cp docker-compose-neon-test-extensions-1:/ext-src/onnxruntime-src/regression.out onnxruntime-src
Error response from daemon: Could not find the file /ext-src/onnxruntime-src/regression.out in container docker-compose-neon-test-extensions-1
+ true
https://github.com/neondatabase/neon/actions/runs/12997051648/job/36281529046?pr=10371#step:7:1897
myrrc
reviewed
Jan 29, 2025
hlinnaka
force-pushed
the
refactor-compute-dockerfile
branch
from
98b15a6 to
8552cbb
Compare
1. Rename some of the extension layers, so that names correspond more
precisely to the upstream repository name and the source directory
name. For example, instead of "pg-jsonschema-pg-build", spell it
"pg_jsonschema-build". Some of the layer names had the extra "pg-"
part, and some didn't; harmonize on not having it. And use an
underscore if the upstream project name uses an underscore.
2. Each extension now consists of two dockerfile targets:
[extension]-src and [extension]-build. By convention, the -src
target downloads the sources and applies any neon-specific patches
if necessary. The source tarball is downloaded and extracted under
/ext-src. For example, the 'pgvector' extension creates the
following files and directory:
/ext-src/pgvector.tar.gz # original tarball
/ext-src/pgvector.patch # neon-specific patch, copied from patches/ dir
/ext-src/pgvector-src/ # extracted tarball, with patch applied
This separation avoids re-downloading the sources every time the
extension is recompiled. The 'extension-tests' target also uses the
[extension]-src layers, by copying the /ext-src/ dirs from all
the extensions together into one image.
This refactoring came about when I was experimenting with different
ways of splitting up the Dockerfile so that each extension would be in
a separate file. That's not part of this PR yet, but this is a good
step in modularizing the extensions.
That was removed in 'main' already, but got reintroduced when I resolve merge conflicts.
The run_tests.sh script tries to run "neon-test.sh" or "make installcheck" for every directory in /ext-src. Hide the directories for extensions that we don't want to test. Or in case of onnxbuild-src, have a no-op "neon-test.sh" file to silence the error that you get on "make installcheck" in that directory.
hlinnaka
force-pushed
the
refactor-compute-dockerfile
branch
from
8552cbb to
321ba26
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor how extensions are built in compute Dockerfile
Rename some of the extension layers, so that names correspond more
precisely to the upstream repository name and the source directory
name. For example, instead of "pg-jsonschema-pg-build", spell it
"pg_jsonschema-build". Some of the layer names had the extra "pg-"
part, and some didn't; harmonize on not having it. And use an
underscore if the upstream project name uses an underscore.
Each extension now consists of two dockerfile targets:
[extension]-src and [extension]-build. By convention, the -src
target downloads the sources and applies any neon-specific patches
if necessary. The source tarball is downloaded and extracted under
/ext-src. For example, the 'pgvector' extension creates the
following files and directory:
This separation avoids re-downloading the sources every time the
extension is recompiled. The 'extension-tests' target also uses the
[extension]-src layers, by copying the /ext-src/ dirs from all
the extensions together into one image.
This refactoring came about when I was experimenting with different
ways of splitting up the Dockerfile so that each extension would be in
a separate file. That's not part of this PR yet, but this is a good
step in modularizing the extensions.