Exception magic makes debugging difficult

[aengelberg]

Currently if an operation's apply-op throws an exception, it will get automatically caught and surfaced as the return value, without halting the test. This means in the check method, the result input may be either a successful value or an exception object.

This makes it convenient to validate certain expected failure modes of my system ((is (instance? Exception result))), but in truly "exceptional" cases it's tough to investigate what went wrong.

• If my check assertion happens to look like (is (= ... result)), the resulting failure message will show me the exception's class and message, but not a stack trace.
• Otherwise, depending on the structure of my assertion, like (= 5 (:total result)), (every? integer? result), or (nil? (:cache result)), I'll get either mysterious failures, a ClassCastException, or worst of all, a false positive.

Based on my experience working with both vanilla clojure.test and clojure.test.check, I consider uncaught exceptions to be an instant failure of a test run. i.e. my expectation would be for the test suite to stop the world, then maybe try to shrink the inputs a bit before blowing up with a full stack trace. I expect other developers would fall into this trap as well.

While this alternative behavior would force the user to explicitly catch any of their known exception types, I believe the friendlier surfacing of unexpected exceptions (which are far more common in my experience) would be worth it.