Refactor pattern-matching usefulness algorithm #65160

Nadrieril · 2019-10-06T11:29:30Z

While working on #53820, I felt uneasy because the usefulness algorithm was quite a complex piece of code and it wasn't clear to me how the different parts interacted. The basic algorithm is clear enough and well explained at the top of the file, but additions were made to support integer ranges, slice patterns and various edge cases like private fields. Those additions made some sense individually but didn't seem to fit in a larger coherent whole.
In this PR, I propose a rework of the algorithm, that integrates advanced features like slice patterns natively. The algorithm describes different steps from the one in the paper, but it actually does the exact same work when restricted to normal constructors + wildcard patterns. This rewrite in particular removes the special-casing of the wildcard pattern, which I found added a lot of cognitive complexity to the code. This PR also largely increases separation of concerns between specialization, witness reconstruction, and handling of special features like integer ranges. I personally feel that the code is now much easier to follow.

This is a fairly large PR; I felt it only made sense as a whole, but if needed I could try splitting it up.
This is for the most part pure refactoring: a lot of the commits simply move code around. I made them small so it should be easy to see that each step preserves behaviour. The only real change in behaviour is regarding exhaustiveness diagnostics for array/slice patterns: they have been slightly altered and are no longer cut off at some arbitrary-seeming length. They have also improved when slice_patterns is enabled.
I recommend the following reading order: first read the description of the new algorithm at the top of the lastest version of _match.rs, then commit-by-commit. I believe tests pass after every commit.

This PR is mostly me being unable to resist refactoring everything I see, so the changes may be a bit all over the place. I hope it all makes sense in the end.
Thanks for taking some time to review !

rust-highfive · 2019-10-06T11:29:34Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @eddyb (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

eddyb · 2019-10-06T13:43:44Z

r? @pnkfelix or @nikomatsakis cc @varkor @Centril

bors · 2019-10-06T16:35:20Z

☔ The latest upstream changes (presumably #65089) made this pull request unmergeable. Please resolve the merge conflicts.

src/librustc_mir/hair/pattern/_match.rs

Centril · 2019-10-06T21:31:14Z

src/librustc_mir/hair/pattern/_match.rs

-                }).collect()
-            }
+        PatKind::Or { ref pats } => {
+            pats.iter().flat_map(|pat| pat_constructors(tcx, param_env, pat, ty)).collect()


Since you've familiarized yourself in-depth with this code, would you like to tackle exhaustiveness for or-patterns as well? (Hope you don't mind @dlrobertson -- then you can focus more on MIR)

Yeah, don't mind at all. Having more eyes on the code is always better 😄

I would like to, yes !

varkor · 2019-10-07T19:08:05Z

I've started reviewing this. Let's check to see whether this affects perf at all.

@bors try

bors · 2019-10-07T19:08:17Z

⌛ Trying commit 92bf5a2d22ffd7cdefb0ed22ceaf18e291a1cfa5 with merge 3c8eb891c46a383549ee6d6d3936bec0dee6f238...

varkor · 2019-10-07T19:11:57Z

cc @arielb1

bors · 2019-10-07T22:08:44Z

☀️ Try build successful - checks-azure
Build commit: 3c8eb891c46a383549ee6d6d3936bec0dee6f238 (3c8eb891c46a383549ee6d6d3936bec0dee6f238)

varkor

I've left some comments. Some of them may be out of date, because I was commenting as I was looking through each commit, which often got refactored later, so ignore any that don't seem relevant any more. Overall, these changes seem really good, and make things a lot clearer ❤️ The diagnostics improvements as a result are a nice bonus, too!

Let's just check that nothing regresses, but other than that, I'm pretty happy with how things are here. You've put a lot of effort into improving this code; thank you!

src/librustc_mir/hair/pattern/_match.rs

src/test/ui/pattern/usefulness/match-slice-patterns.rs

src/librustc_mir/hair/pattern/_match.rs

varkor · 2019-10-07T22:24:21Z

@rust-timer build 3c8eb891c46a383549ee6d6d3936bec0dee6f238

Mark-Simulacrum · 2019-10-07T23:16:33Z

@rust-timer build 3c8eb891c46a383549ee6d6d3936bec0dee6f238

rust-timer · 2019-10-07T23:16:35Z

Queued 3c8eb891c46a383549ee6d6d3936bec0dee6f238 with parent 59a31c8, future comparison URL.

Centril · 2019-10-08T02:59:01Z

r? @varkor

As suggested by arielb1 rust-lang#65160 (review)

All the constructed PatStacks end up having `head_ctors` called on them, so this does not unnecessarily precompute anything.

Stop checking that cached constructors are valid. Avoid cloning constructors when not necessary.

This reverts some of the or-patterns future-proofing, for the sake of performance.

pietroalbini · 2019-10-18T13:06:09Z

/AzurePipelines run

pietroalbini · 2019-10-18T13:07:11Z

Sorry for the noise, trying to figure out why PR builders don't work.

/AzurePipelines help

azure-pipelines · 2019-10-18T16:27:13Z

Azure Pipelines successfully started running 1 pipeline(s).

bors · 2019-10-19T17:48:51Z

☔ The latest upstream changes (presumably #65598) made this pull request unmergeable. Please resolve the merge conflicts.

Nadrieril · 2019-10-20T13:38:26Z

So the lastest merge conflict is taking a big toll on me to rebase. I also have the feeling that this PR has become so unwieldy that maintainers will struggle to find time to review it. If it is going to wait around a while (and need several more complex rebases), maybe splitting this PR up into smaller meaningful chunks could make everyone's job easier; but that would not be worth it if the PR is almost mergeable.
Does this sound like it would be worth the additional effort ?

Centril · 2019-10-20T13:41:36Z

@Nadrieril I would personally try to split it up; what should make it both easier to merge and review, and especially the latter is important as match checking is important for soundness.

varkor · 2019-10-20T13:50:15Z

I think that as long as the rebases aren't requiring significant changes in how things work, and the performance issue can be resolved, it could be easier to just go for this one PR, as it's already been looked at by two of us.

That said, it's up to you: if splitting up would be easier, then go for that.

JohnCSimon · 2019-10-26T02:13:48Z

Ping from triage.
@Nadrieril This PR has sat idle for the last few days. Can you please address the merge conflicts?
CC @varkor
Thank you!

Nadrieril · 2019-10-27T21:03:08Z

I have decided to split this into smaller bits

Clarify pattern-matching usefulness algorithm This PR clarifies a bit the usefulness algorithm by emphasizing that each row of the matrix can be seen as a sort of stack from which we pop constructors. It also moves code around to increase separation of concerns. This is part of my splitting of #65160 into smaller PRs.

rust-highfive assigned eddyb Oct 6, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 6, 2019

Nadrieril mentioned this pull request Oct 6, 2019

Stack overflow compiling slice pattern for big array #53820

Closed

rust-highfive assigned pnkfelix and unassigned eddyb Oct 6, 2019

Nadrieril force-pushed the refactor-usefulness branch from 5c849c5 to bae5adb Compare October 6, 2019 14:32

Nadrieril force-pushed the refactor-usefulness branch from 4a1e85b to b1c0113 Compare October 6, 2019 17:31

Centril reviewed Oct 6, 2019

View reviewed changes

Centril added F-or_patterns `#![feature(or_patterns)]` F-slice_patterns `#![feature(slice_patterns)]` labels Oct 6, 2019

This comment has been minimized.

Sign in to view

varkor reviewed Oct 7, 2019

View reviewed changes

rust-highfive assigned varkor and unassigned pnkfelix Oct 8, 2019

Nadrieril added 4 commits October 18, 2019 14:01

Clarify why the naive algorithm may not terminate

1f6a5a5

As suggested by arielb1 rust-lang#65160 (review)

Remove unneeded lifetime extensions

455095a

Use special Captures trait to solve issues with impl Iterator

d379a43

pat_constructors doesn't need a ty parameter

1de1683

pietroalbini closed this Oct 18, 2019

pietroalbini reopened this Oct 18, 2019

Nadrieril added 8 commits October 18, 2019 14:02

Extract head constructors construction

3aa5fb3

Name the field of PatStack

c956c65

Cache pat_constructors invocations

368c71b

Compute constructors on PatStack construction

6df1815

All the constructed PatStacks end up having `head_ctors` called on them, so this does not unnecessarily precompute anything.

Restore performance

ed3ea67

Stop checking that cached constructors are valid. Avoid cloning constructors when not necessary.

Don't rebuild the same matrix twice

9c3682f

Don't use SmallVec when we could use Option

b052036

This reverts some of the or-patterns future-proofing, for the sake of performance.

Integrate comment fixes suggested by arielb1

fd3ec66

Nadrieril force-pushed the refactor-usefulness branch from ad6ad6b to fd3ec66 Compare October 18, 2019 13:02

Nadrieril mentioned this pull request Oct 27, 2019

Clarify pattern-matching usefulness algorithm #65874

Merged

Nadrieril closed this Oct 27, 2019

varkor mentioned this pull request Nov 5, 2019

Refactor slice pattern usefulness checking #66129

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor pattern-matching usefulness algorithm #65160

Refactor pattern-matching usefulness algorithm #65160

Nadrieril commented Oct 6, 2019

rust-highfive commented Oct 6, 2019

eddyb commented Oct 6, 2019

bors commented Oct 6, 2019

Centril Oct 6, 2019

dlrobertson Oct 7, 2019

Nadrieril Oct 7, 2019 •

edited

Loading

varkor commented Oct 7, 2019

bors commented Oct 7, 2019

varkor commented Oct 7, 2019

bors commented Oct 7, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

varkor left a comment

varkor commented Oct 7, 2019

Mark-Simulacrum commented Oct 7, 2019

rust-timer commented Oct 7, 2019

Centril commented Oct 8, 2019

pietroalbini commented Oct 18, 2019

pietroalbini commented Oct 18, 2019

azure-pipelines bot commented Oct 18, 2019

bors commented Oct 19, 2019

Nadrieril commented Oct 20, 2019

Centril commented Oct 20, 2019

varkor commented Oct 20, 2019 •

edited

Loading

JohnCSimon commented Oct 26, 2019

Nadrieril commented Oct 27, 2019

Refactor pattern-matching usefulness algorithm #65160

Refactor pattern-matching usefulness algorithm #65160

Conversation

Nadrieril commented Oct 6, 2019

rust-highfive commented Oct 6, 2019

eddyb commented Oct 6, 2019

bors commented Oct 6, 2019

Centril Oct 6, 2019

Choose a reason for hiding this comment

dlrobertson Oct 7, 2019

Choose a reason for hiding this comment

Nadrieril Oct 7, 2019 • edited Loading

Choose a reason for hiding this comment

varkor commented Oct 7, 2019

bors commented Oct 7, 2019

varkor commented Oct 7, 2019

bors commented Oct 7, 2019

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

varkor left a comment

Choose a reason for hiding this comment

varkor commented Oct 7, 2019

Mark-Simulacrum commented Oct 7, 2019

rust-timer commented Oct 7, 2019

Centril commented Oct 8, 2019

pietroalbini commented Oct 18, 2019

pietroalbini commented Oct 18, 2019

azure-pipelines bot commented Oct 18, 2019

bors commented Oct 19, 2019

Nadrieril commented Oct 20, 2019

Centril commented Oct 20, 2019

varkor commented Oct 20, 2019 • edited Loading

JohnCSimon commented Oct 26, 2019

Nadrieril commented Oct 27, 2019

Nadrieril Oct 7, 2019 •

edited

Loading

varkor commented Oct 20, 2019 •

edited

Loading