-
-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add meta checks #59
base: master
Are you sure you want to change the base?
Add meta checks #59
Conversation
Why is check-error called |
Symmetry with the output of checks, since check failures print |
How does I tried a quick example and
Reminds me though, it would be good to have a |
That example works because (check-exn (lambda (v) (equal? v 'oops))
(lambda () (check-pred (lambda (v) (raise 'oops)) 'foo))) |
(with-expected pred-or-msg | ||
(assert-failure failure) | ||
(assert-check-failure failure) | ||
(if (procedure? pred-or-msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think (predicate/c pred-or-msg)
will do the right thing (and also check (procedure-arity-includes? pred-or-msg 1)
)
(or use contract-out
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add a procedure-arity-includes?
check here?
(unless (pred-or-msg failure) | ||
(fail-check "Check failure didn't pass predicate")) | ||
(unless (regexp-match? pred-or-msg (exn-message failure)) | ||
(fail-check "Check failure message didn't match regexp"))))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These fail-check
should print failure
somehow. (Do they?)
I'd prefer they mirror check-exn
, and also say "Wrong failure raised"
in both cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. They now print the message and the failure value itself, just like check-exn
.
(unless (equal? info expected-info) | ||
(with-actual info | ||
(fail-check | ||
"Check failure contained an unexpected info value"))))))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This stops at the first unexpected info, right?
I think it'd be better to print all the unexpecteds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hadn't thought about if the info stack contains multiple infos with the expected name but different values. This needs to be reworked to deal with that.
(assert-failure failure) | ||
(assert-not-check-failure failure) | ||
(cond | ||
[(procedure? pred-or-msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(predicate/c pred-or-msg)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a higher order contract, so I don't think it would work here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here, add procedure-arity-includes?
(define-simple-macro (with-expected exp:expr body:expr ...) | ||
(with-check-info* (list (make-check-expected exp)) (λ () body ...))) | ||
|
||
;; Pseudo-contract helpers, to be replaced with real check contracts eventually |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
eventually = this pull request?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By "real check contracts", I mean #55. I don't plan on implementing those in this PR.
;; Pseudo-contract helpers, to be replaced with real check contracts eventually | ||
|
||
(define (contract-pred-or-msg! name pred-or-msg) | ||
(unless (or (procedure? pred-or-msg) (regexp? pred-or-msg)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
predicate/c
(next line too)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather not add the complexity of real higher order contracts here. Additionally, the function used as a predicate can return non-boolean values which predicate/c
doesn't allow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok. then (and (procedure? pred-or-msg) (procedure-arity-includes? pred-or-msg 1))
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
(parameterize ([current-check-handler raise] | ||
[test-log-enabled? #f] | ||
[current-check-info (list)]) | ||
(with-handlers ([(λ (e) (not (exn:break? e))) values]) (chk-thnk) #f))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use negate
?
I don't really care but racket/function
's already imported
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@bennn Added docs and addressed all your comments (I think), this PR is ready for a final review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good timing :)
[message string? ""]) | ||
void?]{ | ||
Checks that @racket[thunk] evaluates a failing check and that | ||
@racket[fail-exn-predicate], it it's a function, returns a true value when |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it it's
=> if it's
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@racket[fail-check], not the optional @racket[message] argument that all checks | ||
accept. See also @racket[check-exn] and @racket[check-error]. | ||
|
||
@(interaction |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should get the same output with @examples[#:label #f ....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Didn't add #:label #f
because using the label seems appropriate. They are examples, after all.
[message string? ""]) | ||
void?]{ | ||
Checks that @racket[thunk] evaluates a check that raises an error value instead | ||
of passing or failing, and checks that the raised value, if it it's a function, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if it
=> if
Also "checks that the raised value" should refer to fail-exn-predicate
.
(The sentence is a little long & awkward ... maybe start a new sentence after "passing or failing"? or end it with "checks that the raised value satisfies the predicate. Satisfies means ....")
(with-expected pred-or-msg | ||
(assert-failure failure) | ||
(assert-check-failure failure) | ||
(if (procedure? pred-or-msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add a procedure-arity-includes?
check here?
(assert-failure failure) | ||
(assert-not-check-failure failure) | ||
(cond | ||
[(procedure? pred-or-msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here, add procedure-arity-includes?
Would it be better if |
@AlexKnauth How about |
@bennn For some reason I can't respond directly to your comments about |
(Re: |
I think this should be a new module, maybe I'd prefer if
More descriptive, less ambiguous names would be |
@rmculpepper Thanks for taking a look at this! Here are my thoughts: A To me, the (test-begin
(check-fail (list #rx"foo" pred) thnk))
(test-begin
(check-fail #rx"foo" thnk)
(check-fail pred thnk))
(test-begin
(check-fail (list (list #rx"foo") (list (list pred))) thnk)) One of these could be preferred, but will they have subtle differences that a user has to remember? What about the extra location info provided by separate checks? And when using only a single predicate / regex / info, does it needs to be wrapped in a list to be considered a tree? Can you mix predicates / regexes / infos in the same tree? Is the order important? Is the tree traversed depth-first or breadth-first? Why is it a tree and not a list? If multiple parts of the tree fail, which message is shown? These are all questions I'd have to ask myself as a user of this API, and I'd likely forget the answers to some of them frequently and have to consult the docs. Not to mention there's the increased implementation and contract complexity. I don't see the benefits of this approach outweighing the costs. |
I like @rmculpepper 's suggestions.
|
Re As for the tree API: A tree API wouldn't quite be a single form, because it would still need separate forms for normal failures and errors. A third form that that supplied an empty tree by default would also be needed if we wanted to match the simplicity of A tree of renderers in A tree API also makes it harder to distinguish between different kinds of failures. Each leaf of the tree would have the same check location information. DrRacket's integration with the location information makes it easy to instantly see exactly which cases failed; I consider it bad practice for a rackunit API to encourage users to subvert that feature. Maybe if we had some way to point to different leaves of the tree in the location message? |
I still like the tree API
|
I'd like to get this feature in and it seems I'm the only one doubting a tree API, so I shall defer to your judgment. Tree API it is! With multiple errors, how should the checks report failures? Multiple failures for one check is something I've been working on in racket-expect and it can be complex enough that I'd rather avoid implementing it entirely here. |
I think just normal messages separated by whitespace would be OK. Example:
But since name & location are always the same, maybe each failed check in the tree could register a unique check-info key & all these keys could be printed on failure. |
Just printing like this would be fine with me:
Anything more complicated I'd defer to a separate package designed with multiple failures in mind. |
Can simplify 1 more step and only print |
4f365d9
to
8efcbe5
Compare
This is ready for another review. Now, the only export of
The See the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About the error message; I like how the expected looks, but it's hard for me to line it up with the "actual".
Can you change the error message to something like this:
--------------------
FAILURE
name: check-fail
location: unsaved-editor:13:0
actual:
exn: (exn:test:check "some message from foo" #<continuation-mark-set> ...)
message: "some message from foo"
info: ((check-info 'foo 1))
expected:
exn-predicate: #<procedure:has-bar-info?>
message: #rx"message from bar"
info: (check-info 'bar 1)
--------------------
And also sort the expected data, so all predicates, messages, and infos are grouped together?
amount of logic. Consequently, custom checks can be buggy and should be tested. | ||
RackUnit provides a few checks explicitly designed for testing the behavior of | ||
other checks; they allow verifying checks pass and fail when expected or that | ||
checks add certain information to the check information stack. These bindings |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
check information stack
=> @tech{check-info stack}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@@ -1,10 +1,11 @@ | |||
#lang scribble/doc | |||
@(require "base.rkt") | |||
@(require (except-in "base.rkt" examples) | |||
scribble/example) | |||
|
|||
@(require (for-label racket/match)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
treeof
needs a for-label
require ... not sure which one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's in plot
somewhere, but I'm wary of adding a dependency on the plot package's docs to rackunit-doc
. Maybe treeof
should be moved out of the plot package?
[message string? ""]) | ||
void?]{ | ||
Checks that @racket[thunk] raises a check failure and that the failure | ||
satisfies @racket[assertion-tree]. The tree is checked in the following manner: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the wording here is too low-level (sounds too much like an algorithm to me).
suggestion: change this line & the list to say
satisfies every assertion in
assertion-tree
. A failure can satisfy an assertion in one of three ways, depending on the type of the assertion:
- a failure
f
satisfies a predicatep
if(p f)
is a true value- a failure satisfies a regular expression
r
ifr
matches the failure's message,- a failure satisfies a
check-info
structure if the failure's check stack contains the expected value
... okay I don't like what I wrote very much either. Maybe it'd be better to have a deftech
for the (treeof ....)
contract and define "satisfies" in that?
Anyway, what's important to me is that the 3 cases for an "assertion" have 3 clear meanings. (I don't think the 4-point list & focus on "satisfying a tree" does that.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like what you wrote and definitely agree about having three list cases instead of four. I used slightly different wording though.
A deftech
seems out of place here, especially since it would only be used in this one check.
Checks that @racket[thunk] raises a check failure and that the failure | ||
satisfies @racket[assertion-tree]. The tree is checked in the following manner: | ||
|
||
@(itemlist |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@itemlist[ .... ]
(same for the examples
below)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
rackunit-lib/rackunit/meta.rkt
Outdated
(define failure (check-raise-value chk-thnk)) | ||
(unless (exn:test:check? failure) | ||
(with-actual failure | ||
(fail-check "Check raised error instead of failing"))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
replace "failing" with "calling fail-check
"?
Because I think:
- "failing" is a technical term, it really means to throw an
exn:test:check
- "failing" isn't described in the docs (anyway, someone writing their first
define-check
will have trouble with the current error message)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, very good idea.
rackunit-lib/rackunit/meta.rkt
Outdated
(define-simple-macro (with-actual act:expr body:expr ...) | ||
(with-check-info* (error-info act) (λ () body ...))) | ||
|
||
(define (list/if . vs) (filter values vs)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this go in a common place?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've written this in half a dozen different codebases so really I'd prefer to see it added to racket/list
or some list-more
package. But I'll put it in rackunit/private/util-list
for now (not util
, because rackunit/private/check
duplicates this and it can't depend on util
).
are provided by @racketmodname[rackunit/meta], not @racketmodname[rackunit]. | ||
|
||
@defproc[(check-fail [assertion-tree | ||
(treeof (or/c (-> exn:test:check? any/c) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess any/c
should be any
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The predicates have to return one value so I think any/c
is appropriate here.
rackunit-lib/rackunit/meta.rkt
Outdated
(procedure-arity-includes? v 1)) | ||
(regexp? v) | ||
(check-info? v)) | ||
(define ctrct "(or/c (-> any/c boolean?) regexp? check-info?)") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
boolean?
=> any
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed to any/c
(see earlier comment about any
vs any/c
)
(λ () (check-fail (list #rx"foo" 'partial-nonsense) | ||
fail-check))) | ||
(check-exn exn:fail:contract? | ||
(λ () (check-fail accepts-no-args fail-check))))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add some tests with silly lists?
like '(((#rx"hello") ()) #rx"world")
... just non-flat variations on the tests you already have
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. I think one test that uses all three kinds of assertions in a single silly list ought to be enough; more tests than that seem redundant.
Also add exception and exception message to check info in regex failures instead of only the message.
e5e7e84
to
fabfad0
Compare
1.8 was the dynamic-info feature, which was merged before this.
Sorting and grouping infos in the failure message is now implemented. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, I like the new docs
(define (exn-info) (make-check-info 'exn (pretty-info raised))) | ||
(define (msg-info) (make-check-info 'message (exn-message raised))) | ||
(define (info-info) | ||
(make-check-info 'info (nested-info (exn:test:check-stack raised)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wish the actual
and expected
info fields looked more similar:
actual:
exn: (exn:test:check "some message from foo" #<continuation-mark-set> ...)
message: "some message from foo"
info:
foo: 1
expected:
predicate: #<procedure:has-bar-info?>
message: #rx"message from bar"
info: (check-info 'bar 1)
Because, what if foo
isn't a check-info
struct, but something else that prints the same way?
I'd like to do (nested-info (map pretty-info (exn:test:check-stack raised)))
, but that's a contract error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean what if foo
is a check-info
struct who's value is not 1
, but something that prints the same such as (string-info "1")
? Calling exn:test:check-stack
will always return a list of check-info
values modulo unreasonable shenanigans.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't know that about exn:test:check-stack
.
Okay though, my real complaint is that foo: 1
and (check-info 'bar 1)
look too different. I'd rather the first looked like (check-info 'foo 1)
.
(λ () (check-equal? 'foo 'foo)))] | ||
|
||
Note that a single predicate, regular expression, or @tech{check-info} value is | ||
considered a legal tree, as well as the empty list. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think the docs really need this note.
But maybe, add a reminder note not to abuse to tree API? (A check should test "1 thing".)
(with-check-info* (error-info act) (λ () body ...))) | ||
|
||
(define (error-info raised) | ||
(define (exn-info) (make-check-info 'exn (pretty-info raised))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these thunks are more time/space expensive than just doing (define exn-info (make....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The optimizer should be able to inline them, so I don't think that's worth worrying about.
Closes #47
This PR adds the following checks:
(check-fail <pred-or-exn> <thunk>)
- asserts that<thunk>
evaluates a failing check and that the check failure exception satisfies<pred-or-exn>
in the same way ascheck-exn
(check-fail* <thunk>)
- likecheck-fail
, but without caring about the specifics of the thrown failure(check-fail/info <check-info> <thunk>)
- asserts that<thunk>
evaluates a failing check and that the check failure exception contains an info that isequal?
to<check-info>
(check-error <pred-or-exn> <thunk>)
- asserts that<thunk>
evaluates a check that raises an error instead of failing and that the raised value satisfies<pred-or-exn>
I went with
check-fail/info
instead of adding an info case tocheck-fail
to avoid confusion over whethercheck-error
has an info-accepting variant. The existence ofcheck-fail/info
and nonexistence ofcheck-error/info
is a clear signal in documentation search results and playing at the REPL that wouldn't be present ifcheck-fail/info
was merged withcheck-fail
.The only addition from the proposed API is
check-error
, which I realized I'd need to test to the contract failure cases of these checks.