Add support for docker image with presto java #7920
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for meta-velox canceled.
|
facebook-github-bot
added
the
CLA Signed
kgpai
force-pushed
the
add_prestojava_image
branch
4 times, most recently
from
44b450c to
a27a51c
Compare
kgpai
force-pushed
the
add_prestojava_image
branch
from
a27a51c to
f956dab
Compare
assignUser
requested changes
Dec 8, 2023
docker-compose.yml
Outdated
| # or | ||
| # docker-compose run -e NUM_THREADS=<NUMBER_OF_THREADS_TO_USE> --rm presto-java | ||
| # to set the number of threads used during compilation | ||
| image: ghcr.io/facebookincubator/velox-dev:amd64-centos-8-avx |
There was a problem hiding this comment.
Suggested change
| image: ghcr.io/facebookincubator/velox-dev:amd64-centos-8-avx | |
| image: ghcr.io/facebookincubator/velox-dev:presto-java |
This will be the name of the build image :)
kgpai
commented
Dec 8, 2023
| @@ -38,7 +38,33 @@ permissions: | |||
|
|
|||
| jobs: | |||
| linux: | |||
| name: "Build and Push ${{ matrix.name }}" | |||
There was a problem hiding this comment.
Thanks to @assignUser .
|
@kgpai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
assignUser
approved these changes
Dec 8, 2023
gggrace14
approved these changes
Dec 8, 2023
facebook-github-bot
pushed a commit
that referenced
this pull request
Jan 16, 2024
…tal jobs. (#8314) Summary: **What ?** Recently support for [PrestoQueryRunner ](#7628 added to AggregationFuzzer, so that it can use Presto as a source of truth. This PR sets up an experimental job that uses the Presto Java container to run Aggregation Fuzzer with Presto as source of truth. **Why ?** Running the aggregation fuzzer against Presto gives us greater confidence on the correctness of our aggregate functions. **How ?** 1. [Previously we created a docker image with support for java and Presto](#7920) 2. This still requires some changes to get Presto to work with AggregationFuzzer. These changes are listed below. 3. Adds support for Hive metastore - we do this via a file based metastore by adding the hive.properties file to $PRESTO_HOME/etc/catalog (see #8111) 4. We also set the pid file on the launcher to ensure there is only one run of the Presto server 5. Another thing to note is that we set the timezone to be ['Americas/Los_Angeles' due to details described here ](#8127). **Testing:** There is no automated way to test this. Here are manual steps I follow: 1. Create presto-java-container by running `docker-compose build presto-java` 6. Ssh into the presto-java-container 7. Copy Velox sources into the presto-java-container and build Velox 8. Launch Presto service: /opt/start-prestojava.sh 9. Run Aggregation fuzzer with --presto_url=http://127.0.0.1:8080 flag Example Run: Here is an example run https://github.com/facebookincubator/velox/actions/runs/7508139395/job/20443035814 Pull Request resolved: #8314 Reviewed By: kagamiori Differential Revision: D52701237 Pulled By: kgpai fbshipit-source-id: e4ba284aed469bc69cc95c90f708a0c92a040205
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This image will allow us to run Velox + Presto java and allow Velox to use Presto java as a source of truth for its fuzzer runs.
Once the image is built , you can run presto java on it by calling /opt/start-prestojava.sh