Skip to content

Commit

Permalink
Add join fuzzer to experimental jobs (facebookincubator#8319)
Browse files Browse the repository at this point in the history
Summary:
This will allow us to run experimental fuzzer on an ad-hoc basis against any branch.
Also fixes the path of the spark fuzzer artifacts.

Pull Request resolved: facebookincubator#8319

Reviewed By: mbasmanova

Differential Revision: D52646532

Pulled By: kgpai

fbshipit-source-id: 4d341c142a1aadcdfce6d9a145f1d208c0a3941d
  • Loading branch information
kgpai authored and liujiayi771 committed Jan 16, 2024
1 parent f0adb66 commit a310728
Show file tree
Hide file tree
Showing 2 changed files with 60 additions and 8 deletions.
67 changes: 59 additions & 8 deletions .github/workflows/experimental.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ permissions:

jobs:
compile:
runs-on: 16-core
runs-on: 8-core
timeout-minutes: 120
env:
CCACHE_DIR: "${{ github.workspace }}/.ccache/"
Expand Down Expand Up @@ -86,13 +86,23 @@ jobs:
name: aggregation
path: velox/_build/debug/velox/exec/tests/velox_aggregation_fuzzer_test

- name: Upload spark fuzzer
- name: Upload spark aggregation fuzzer
uses: actions/upload-artifact@v3
with:
name: spark
path: |
velox/_build/debug/velox/expression/tests/spark_expression_fuzzer_test
velox/_build/debug/velox/expression/tests/spark_aggregation_fuzzer_test
name: spark_aggregation_fuzzer
path: velox/_build/debug/velox/functions/sparksql/fuzzer/spark_aggregation_fuzzer_test

- name: Upload aggregation fuzzer
uses: actions/upload-artifact@v3
with:
name: aggregation
path: velox/_build/debug/velox/functions/prestosql/fuzzer/velox_aggregation_fuzzer_test

- name: Upload join fuzzer
uses: actions/upload-artifact@v3
with:
name: join
path: velox/_build/debug/velox/exec/tests/velox_join_fuzzer_test


presto-java-aggregation-fuzzer-run:
Expand Down Expand Up @@ -151,10 +161,10 @@ jobs:
- name: "Install dependencies"
run: source ./scripts/setup-ubuntu.sh

- name: Download spark fuzzer
- name: Download spark aggregation fuzzer
uses: actions/download-artifact@v3
with:
name: spark
name: spark_aggregation_fuzzer

- name: "Run Spark Aggregate Fuzzer"
run: |
Expand All @@ -177,3 +187,44 @@ jobs:
name: spark-agg-fuzzer-failure-artifacts
path: |
/tmp/spark_aggregate_fuzzer_repro
linux-join-fuzzer-run:
runs-on: ubuntu-latest
needs: compile
timeout-minutes: 120
steps:

- name: "Checkout Repo"
uses: actions/checkout@v3
with:
ref: "${{ inputs.ref || 'main' }}"

- name: "Install dependencies"
run: source ./scripts/setup-ubuntu.sh

- name: Download join fuzzer
uses: actions/download-artifact@v3
with:
name: join

- name: "Run Join Fuzzer"
run: |
ls /lib64
mkdir -p /tmp/join_fuzzer_repro/
rm -rfv /tmp/join_fuzzer_repro/*
chmod -R 777 /tmp/join_fuzzer_repro
chmod +x velox_join_fuzzer_test
./velox_join_fuzzer_test \
--seed ${RANDOM} \
--duration_sec 1800 \
--logtostderr=1 \
--minloglevel=0 \
&& echo -e "\n\nAggregation fuzzer run finished successfully."
- name: Archive aggregate production artifacts
if: always()
uses: actions/upload-artifact@v3
with:
name: join-fuzzer-failure-artifacts
path: |
/tmp/join_fuzzer_repro
1 change: 1 addition & 0 deletions scripts/setup-ubuntu.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ export CMAKE_BUILD_TYPE=Release
# Install all velox and folly dependencies.
# The is an issue on 22.04 where a version conflict prevents glog install,
# installing libunwind first fixes this.
apt update && apt install sudo
sudo --preserve-env apt update && sudo --preserve-env apt install -y libunwind-dev && \
sudo --preserve-env apt install -y \
g++ \
Expand Down

0 comments on commit a310728

Please sign in to comment.