Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add join fuzzer to experimental jobs #8319

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 59 additions & 8 deletions .github/workflows/experimental.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ permissions:

jobs:
compile:
runs-on: 16-core
runs-on: 8-core
timeout-minutes: 120
env:
CCACHE_DIR: "${{ github.workspace }}/.ccache/"
Expand Down Expand Up @@ -86,13 +86,23 @@ jobs:
name: aggregation
path: velox/_build/debug/velox/exec/tests/velox_aggregation_fuzzer_test

- name: Upload spark fuzzer
- name: Upload spark aggregation fuzzer
uses: actions/upload-artifact@v3
with:
name: spark
path: |
velox/_build/debug/velox/expression/tests/spark_expression_fuzzer_test
velox/_build/debug/velox/expression/tests/spark_aggregation_fuzzer_test
name: spark_aggregation_fuzzer
path: velox/_build/debug/velox/functions/sparksql/fuzzer/spark_aggregation_fuzzer_test

- name: Upload aggregation fuzzer
uses: actions/upload-artifact@v3
with:
name: aggregation
path: velox/_build/debug/velox/functions/prestosql/fuzzer/velox_aggregation_fuzzer_test

- name: Upload join fuzzer
uses: actions/upload-artifact@v3
with:
name: join
path: velox/_build/debug/velox/exec/tests/velox_join_fuzzer_test


presto-java-aggregation-fuzzer-run:
Expand Down Expand Up @@ -151,10 +161,10 @@ jobs:
- name: "Install dependencies"
run: source ./scripts/setup-ubuntu.sh

- name: Download spark fuzzer
- name: Download spark aggregation fuzzer
uses: actions/download-artifact@v3
with:
name: spark
name: spark_aggregation_fuzzer

- name: "Run Spark Aggregate Fuzzer"
run: |
Expand All @@ -177,3 +187,44 @@ jobs:
name: spark-agg-fuzzer-failure-artifacts
path: |
/tmp/spark_aggregate_fuzzer_repro

linux-join-fuzzer-run:
runs-on: ubuntu-latest
needs: compile
timeout-minutes: 120
steps:

- name: "Checkout Repo"
uses: actions/checkout@v3
with:
ref: "${{ inputs.ref || 'main' }}"

- name: "Install dependencies"
run: source ./scripts/setup-ubuntu.sh

- name: Download join fuzzer
uses: actions/download-artifact@v3
with:
name: join

- name: "Run Join Fuzzer"
run: |
ls /lib64
mkdir -p /tmp/join_fuzzer_repro/
rm -rfv /tmp/join_fuzzer_repro/*
chmod -R 777 /tmp/join_fuzzer_repro
chmod +x velox_join_fuzzer_test
./velox_join_fuzzer_test \
--seed ${RANDOM} \
--duration_sec 1800 \
--logtostderr=1 \
--minloglevel=0 \
&& echo -e "\n\nAggregation fuzzer run finished successfully."

- name: Archive aggregate production artifacts
if: always()
uses: actions/upload-artifact@v3
with:
name: join-fuzzer-failure-artifacts
path: |
/tmp/join_fuzzer_repro
1 change: 1 addition & 0 deletions scripts/setup-ubuntu.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export CMAKE_BUILD_TYPE=Release
# Install all velox and folly dependencies.
# The is an issue on 22.04 where a version conflict prevents glog install,
# installing libunwind first fixes this.
apt update && apt install sudo
sudo --preserve-env apt update && sudo --preserve-env apt install -y libunwind-dev && \
sudo --preserve-env apt install -y \
g++ \
Expand Down
Loading