Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: improve test case inputs in selector table fuzz #3625

Conversation

charles-cooper
Copy link
Member

@charles-cooper charles-cooper commented Sep 26, 2023

What I did

How I did it

How to verify it

Commit message

this commit improves the fuzz examples for the selector table. the
nested `@given` tests too many "dumb" examples (ex. 0, 1, max_value)
when `max_examples` is not large enough. the nested `@given` strategy
can find falsifying inputs, but it requires the inner `max_examples` to
be set much higher, and the shrinking takes much longer. this setting
of `max_examples=125` with a single `@given` using the `@composite`
strategy in this commit finds the selector table bug (that was fixed in
823675a8dc) after an average of 3 runs.

Description for the changelog

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

this commit improves the entropy provided to the selector table fuzzer,
by varying `seed` in the inner hypothesis function as well as the outer
one.
@codecov-commenter
Copy link

codecov-commenter commented Sep 26, 2023

Codecov Report

Merging #3625 (f8e1cff) into master (1e9922d) will increase coverage by 0.03%.
Report is 5 commits behind head on master.
The diff coverage is n/a.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##           master    #3625      +/-   ##
==========================================
+ Coverage   89.01%   89.04%   +0.03%     
==========================================
  Files          86       86              
  Lines       11455    11460       +5     
  Branches     2607     2606       -1     
==========================================
+ Hits        10197    10205       +8     
+ Misses        836      832       -4     
- Partials      422      423       +1     

see 6 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@charles-cooper charles-cooper marked this pull request as ready for review September 28, 2023 18:20
the nested given has too many "dumb" examples. nested given can find
falsifying inputs, but it requires the inner `max_examples` to be set
much higher. this setting with a single `@given` using `@composite`
finds the selector table bug after an average of 3 runs.
@charles-cooper charles-cooper force-pushed the fix/improve-selector-table-fuzz-entropy branch from 8b226dc to f8e1cff Compare September 28, 2023 18:21
@charles-cooper charles-cooper changed the title fix: improve entropy in selector table fuzz fix: improve test case inputs in selector table fuzz Sep 28, 2023
@charles-cooper charles-cooper merged commit 4281780 into vyperlang:master Sep 28, 2023
82 checks passed
@charles-cooper charles-cooper deleted the fix/improve-selector-table-fuzz-entropy branch September 28, 2023 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants