Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup AggregationFuzzer to use Presto as source of truth in experimental jobs. #8314

Closed
wants to merge 3 commits into from
Closed

Setup AggregationFuzzer to use Presto as source of truth in experimental jobs. #8314

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db5aa54
Setup AggregationFuzzer against Presto in experimental jobs.
kgpai Jan 9, 2024
57d8781
Add support for another repo to take reference from.
kgpai Jan 10, 2024
93eec92
Ensure that server logs are archived.
kgpai Jan 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/experimental.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ jobs:
uses: actions/upload-artifact@v3
with:
name: aggregation
path: velox/_build/debug/velox/exec/tests/velox_aggregation_fuzzer_test
path: velox/_build/debug/velox/functions/prestosql/fuzzer/velox_aggregation_fuzzer_test

- name: Upload spark fuzzer
uses: actions/upload-artifact@v3
Expand Down Expand Up @@ -117,6 +117,7 @@ jobs:

- name: "Run Aggregate Fuzzer"
run: |
/opt/start-prestojava.sh 2>&1 >> /tmp/server.log &
mkdir -p /tmp/aggregate_fuzzer_repro/
rm -rfv /tmp/aggregate_fuzzer_repro/*
chmod -R 777 /tmp/aggregate_fuzzer_repro
Expand All @@ -128,7 +129,9 @@ jobs:
--minloglevel=0 \
--repro_persist_path=/tmp/aggregate_fuzzer_repro \
--enable_sorted_aggregations=true \
--presto_url=http://127.0.0.1:8080 \
&& echo -e "\n\nAggregation fuzzer run finished successfully."

- name: Archive aggregate production artifacts
if: always()
uses: actions/upload-artifact@v3
Expand Down
4 changes: 4 additions & 0 deletions scripts/etc/hive.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
connector.name=hive-hadoop2
hive.metastore=file
hive.metastore.catalog.dir=file:/opt/presto-server/etc/data
hive.allow-drop-table=true
5 changes: 3 additions & 2 deletions scripts/etc/node.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
node.environment=test
node.data-dir=/var/lib/presto/data
node.environment=production
node.id=ffffffff-ffff-ffff-ffff-ffffffffffff
node.data-dir=/tmp/presto/data
9 changes: 7 additions & 2 deletions scripts/prestojava-container.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,20 +28,25 @@ ARG PRESTO_CLI_JAR=presto-cli-$PRESTO_VERSION-executable.jar
ENV PRESTO_HOME="/opt/presto-server"
RUN cp $PRESTO_CLI_JAR /opt/presto-cli

RUN dnf install -y java-11-openjdk less procps python3 \
RUN dnf install -y java-11-openjdk less procps python3 tzdata \
&& ln -s $(which python3) /usr/bin/python \
&& tar -zxf $PRESTO_PKG \
&& mv ./presto-server-$PRESTO_VERSION $PRESTO_HOME \
&& chmod +x /opt/presto-cli \
&& ln -s /opt/presto-cli /usr/local/bin/ \
&& mkdir -p $PRESTO_HOME/etc \
&& mkdir -p $PRESTO_HOME/etc/catalog \
&& mkdir -p /var/lib/presto/data \
&& mkdir -p $PRESTO_HOME/etc/data \
&& mkdir -p /usr/lib/presto/utils


# Setting timezone to deal with Velox timezone conversion problem
ENV TZ=America/Bahia_Banderas

COPY scripts/etc/config.properties.example $PRESTO_HOME/etc/config.properties
COPY scripts/etc/jvm.config.example $PRESTO_HOME/etc/jvm.config
COPY scripts/etc/node.properties $PRESTO_HOME/etc/node.properties
COPY scripts/etc/hive.properties $PRESTO_HOME/etc/catalog
COPY scripts/start-prestojava.sh /opt

WORKDIR /velox
7 changes: 5 additions & 2 deletions scripts/start-prestojava.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@
# limitations under the License.

set -e
echo "node.id=$HOSTNAME" >> $PRESTO_HOME/etc/node.properties

$PRESTO_HOME/bin/launcher run
nohup $PRESTO_HOME/bin/launcher --pid-file=/tmp/pidfile run > /tmp/server.log
#wait a few seconds for presto to start
sleep 60
echo 'CREATE SCHEMA hive.tpch;' > /tmp/hive_create.sql
/opt/presto-cli --server 127.0.0.1:8080 --file /tmp/hive_create.sql
Loading