Add basic WindowFuzzerTest #7917

kagamiori · 2023-12-07T18:51:27Z

This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions.

At every iteration, this fuzzer chooses a random signature among all registered aggregation and window function signatures (excluding functions in the skipFunctions list), and execute a plan of window operation with randomly generated set of partition keys, an optional set of sorting keys (with 50% chance), and the default frame (RANGE between UNBOUNDED PRECEDING and CURRENT ROW). For functions in the orderDependentFunctions list, the set of sorting keys include the row_number column to ensure the input is fully sorted. The input columns are either generated randomly, or generated via the custom input generator if provided. A GFLAG enable_window_reference_verification (with the default value false) controls whether the result is verified against DuckDB. For functions in the customVerificationFunctions list, results are not verified even if enable_window_reference_verification is set to true.

The following GFLAGs are supported in the basic version:
enable_window_reference_verification
seed
only
steps
duration_sec
batch_size
num_batches
max_num_varargs
null_ratio
repro_persist_path
persist_and_run_once
log_signature_stats

Function stats are printed before iteration starts, e.g.,

31219 20:14:25.472924 2372852 AggregationFuzzerBase.cpp:541] Total functions: 87 (509 signatures)
I20231219 20:14:25.472947 2372852 AggregationFuzzerBase.cpp:545] Functions with at least one supported signature: 87 (100.00%)
I20231219 20:14:25.472965 2372852 AggregationFuzzerBase.cpp:551] Functions with no supported signature: 0 (0.00%)
I20231219 20:14:25.472975 2372852 AggregationFuzzerBase.cpp:554] Supported function signatures: 497 (97.64%)
I20231219 20:14:25.472982 2372852 AggregationFuzzerBase.cpp:560] Unsupported function signatures: 12 (2.36%)
I202

Test stats are printed after all iterations complete, e.g.,

I20231219 20:14:54.414773 2372852 WindowFuzzer.cpp:224] Total functions tested: 3
I20231219 20:14:54.414824 2372852 WindowFuzzer.cpp:225] Total functions requiring sorted inputs: 0 (0.00%)
I20231219 20:14:54.414860 2372852 WindowFuzzer.cpp:227] Total functions verified against reference DB: 1 (25.00%)
I20231219 20:14:54.414870 2372852 WindowFuzzer.cpp:229] Total functions not verified (verification skipped / not supported by reference DB / reference DB failed): 2 (50.00%) / 0 (0.00%) / 0 (0.00%)
I20231219 20:14:54.414880 2372852 WindowFuzzer.cpp:234] Total failed functions: 1 (25.00%)
I20231219 20:15:59.899459 2372852 AggregationFuzzerBase.cpp:344] Signature #49 failed 1 out of 1 times: approx_most_frequent( BIGINT SMALLINT BIGINT ) -> MAP<SMALLINT,BIGINT>

The window fuzzer test currently have the following limitations:

Not supporting reproduction of test failures from saved files.
Not using randomly-generated frames.
Not allowing custom result verifiers.
Verify results against DuckDB instead of Presto.

The limitations will be addressed next in the subsequent PRs.

This is the second piece of #7754.

Differential Revision: D51880255

netlify · 2023-12-07T18:51:33Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`7875449`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/65dcd8c126b265000959e3a2

facebook-github-bot · 2023-12-07T18:51:39Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2023-12-09T00:33:27Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2023-12-12T01:58:33Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2023-12-12T02:59:48Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2023-12-12T03:02:44Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2023-12-14T02:30:08Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: Pull Request resolved: facebookincubator#7917 This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255 fbshipit-source-id: 0c9f88d9877b5d24051f66073b9996de30149ff8

Summary: Pull Request resolved: facebookincubator#7917 This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: https://internalfb.com/D51880255 fbshipit-source-id: 4e03cf0020151f9bfe96d1d27182ad3b27f70b24

facebook-github-bot · 2023-12-14T21:27:08Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: Pull Request resolved: facebookincubator#7917 This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. * Verify results against DuckDB instead of Presto. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255 fbshipit-source-id: 0f44d3cfd973c68730b8dbcef231e395b6124d3c

facebook-github-bot · 2023-12-14T21:32:50Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2024-02-15T21:55:46Z

This pull request was exported from Phabricator. Differential Revision: D51880255

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2024-02-15T21:56:22Z

This pull request was exported from Phabricator. Differential Revision: D51880255

kagamiori · 2024-02-15T23:26:12Z

Hi @aditi-pandit and @kgpai, thank you for your review! I've addressed your comments. Please take a look again. Thanks!

kgpai

Looks good - can you add documentation in fuzzer.rst

kgpai · 2024-02-16T23:38:17Z

velox/exec/fuzzer/AggregationFuzzerBase.cpp

+    ++numReferenceQueryFailed;
+  } else if (errorCode == ReferenceQueryErrorCode::kReferenceQueryUnsupported) {
+    ++numReferenceQueryNotSupported;
+  }


nit : in case we ever change it to add some other kind of failure we can do ++numReferenceQueryFailedOther.

kgpai · 2024-02-17T00:12:53Z

velox/exec/fuzzer/WindowFuzzerRunner.h

+
+namespace facebook::velox::exec::test {
+
+class WindowFuzzerRunner {


nit: Can you add documentation for WindowFuzzer's in fuzzer.rst

Updated fuzzer.rst. Please take a look and let me know any further comments. Thanks!

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Differential Revision: D51880255

facebook-github-bot · 2024-02-21T01:33:14Z

This pull request was exported from Phabricator. Differential Revision: D51880255

kgpai

Thanks, LGTM.

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Reviewed By: kgpai Differential Revision: D51880255

Summary: Pull Request resolved: facebookincubator#8681 Reviewed By: kgpai Differential Revision: D53459530

Summary: This diff adds a basic version of window fuzzer test that tests the Window operator with window functions and aggregation functions. The window fuzzer test currently have the following limitations: * Not supporting reproduction of test failures from saved files. * Not using randomly-generated frames. * Not allowing custom result verifiers. The limitations will be addressed next in the subsequent PRs. This is the second piece of facebookincubator#7754. Reviewed By: kgpai Differential Revision: D51880255

facebook-github-bot · 2024-02-26T22:07:10Z

This pull request has been merged in 4b439db.

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 7, 2023

facebook-github-bot added the fb-exported label Dec 7, 2023

kagamiori force-pushed the export-D51880255 branch from f8c2a63 to 7276d4b Compare December 9, 2023 00:33

kagamiori changed the title ~~Add WindowFuzzerTest~~ Add Basic WindowFuzzerTest Dec 9, 2023

kagamiori changed the title ~~Add Basic WindowFuzzerTest~~ Add basic WindowFuzzerTest Dec 9, 2023

kagamiori force-pushed the export-D51880255 branch from 7276d4b to 054c44f Compare December 12, 2023 01:58

kagamiori force-pushed the export-D51880255 branch from 054c44f to 962fc55 Compare December 12, 2023 02:59

kagamiori force-pushed the export-D51880255 branch from 962fc55 to cb4f4c2 Compare December 12, 2023 03:02

kagamiori requested review from mbasmanova and kgpai December 12, 2023 19:07

kagamiori force-pushed the export-D51880255 branch from cb4f4c2 to 3c89a2d Compare December 14, 2023 02:30

kagamiori force-pushed the export-D51880255 branch from 3c89a2d to 0cfd7c7 Compare December 14, 2023 21:27

kagamiori force-pushed the export-D51880255 branch from 0cfd7c7 to df68513 Compare December 14, 2023 21:32

kagamiori force-pushed the export-D51880255 branch from 9f4a0ef to 8e88a9d Compare February 15, 2024 21:55

kagamiori requested review from aditi-pandit and kgpai February 15, 2024 23:25

kgpai reviewed Feb 17, 2024

View reviewed changes

kagamiori force-pushed the export-D51880255 branch from 8e88a9d to 3f45a1e Compare February 21, 2024 01:32

kgpai approved these changes Feb 21, 2024

View reviewed changes

kagamiori force-pushed the export-D51880255 branch from 3f45a1e to 7e31b5b Compare February 23, 2024 22:03

kagamiori force-pushed the export-D51880255 branch 2 times, most recently from a557d74 to eb65427 Compare February 26, 2024 17:58

kagamiori force-pushed the export-D51880255 branch from eb65427 to 7084f67 Compare February 26, 2024 18:28

kagamiori force-pushed the export-D51880255 branch from 7084f67 to 5ef9e5c Compare February 26, 2024 18:28

kagamiori added 2 commits February 26, 2024 10:30

Add unit test for PrestoQueryRunner::toSql() (facebookincubator#8681)

3d5cd4f

Summary: Pull Request resolved: facebookincubator#8681 Reviewed By: kgpai Differential Revision: D53459530

kagamiori force-pushed the export-D51880255 branch from 5ef9e5c to 7875449 Compare February 26, 2024 18:30

facebook-github-bot closed this in 4b439db Feb 26, 2024

facebook-github-bot added the Merged label Feb 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic WindowFuzzerTest #7917

Add basic WindowFuzzerTest #7917

kagamiori commented Dec 7, 2023 •

edited

Loading

netlify bot commented Dec 7, 2023 •

edited

Loading

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 9, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Feb 15, 2024

facebook-github-bot commented Feb 15, 2024

kagamiori commented Feb 15, 2024

kgpai left a comment

kgpai Feb 16, 2024

kgpai Feb 17, 2024

kagamiori Feb 21, 2024

facebook-github-bot commented Feb 21, 2024

kgpai left a comment

facebook-github-bot commented Feb 26, 2024


		namespace facebook::velox::exec::test {

		class WindowFuzzerRunner {

Add basic WindowFuzzerTest #7917

Add basic WindowFuzzerTest #7917

Conversation

kagamiori commented Dec 7, 2023 • edited Loading

netlify bot commented Dec 7, 2023 • edited Loading

✅ Deploy Preview for meta-velox canceled.

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 9, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 12, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Dec 14, 2023

facebook-github-bot commented Feb 15, 2024

facebook-github-bot commented Feb 15, 2024

kagamiori commented Feb 15, 2024

kgpai left a comment

Choose a reason for hiding this comment

kgpai Feb 16, 2024

Choose a reason for hiding this comment

kgpai Feb 17, 2024

Choose a reason for hiding this comment

kagamiori Feb 21, 2024

Choose a reason for hiding this comment

facebook-github-bot commented Feb 21, 2024

kgpai left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 26, 2024

kagamiori commented Dec 7, 2023 •

edited

Loading

netlify bot commented Dec 7, 2023 •

edited

Loading