From b566d4ceadab2bff0ae77765be63aadb4d3660d2 Mon Sep 17 00:00:00 2001 From: Greg Hale Date: Tue, 26 Nov 2024 15:04:43 -0800 Subject: [PATCH] Add test-block constraints docs (#1198) > [!IMPORTANT] > Adds documentation for BAML test-block constraints, renames a file for clarity, and updates navigation paths. > > - **Documentation**: > - Adds documentation for test-block constraints in `testing-functions.mdx`, detailing the use of `@@check` and `@@assert`. > - Explains variables `_` and `this` in the context of test assertions. > - Provides examples of using `@@check` and `@@assert` in BAML tests. > - **File Renaming**: > - Renames `validations.mdx` to `checks-and-asserts.mdx` to better reflect content. > - **Navigation Update**: > - Updates navigation path in `docs.yml` from `validations.mdx` to `checks-and-asserts.mdx`. > > This description was created by [Ellipsis](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral) for 4e13977411433ea2f270252ace9b019f053e2459. It will automatically update as commits are pushed. --- .../04-baml-basics/testing-functions.mdx | 41 ++++++++++++++++++- ...validations.mdx => checks-and-asserts.mdx} | 8 ++-- fern/docs.yml | 4 +- 3 files changed, 46 insertions(+), 7 deletions(-) rename fern/01-guide/05-baml-advanced/{validations.mdx => checks-and-asserts.mdx} (97%) diff --git a/fern/01-guide/04-baml-basics/testing-functions.mdx b/fern/01-guide/04-baml-basics/testing-functions.mdx index 292edda43..a1a1763ab 100644 --- a/fern/01-guide/04-baml-basics/testing-functions.mdx +++ b/fern/01-guide/04-baml-basics/testing-functions.mdx @@ -27,6 +27,7 @@ test Test1 { // input is the first argument of ClassifyMessage input "Can't access my account using my usual login credentials, and each attempt results in an error message stating 'Invalid username or password.' I have tried resetting my password using the 'Forgot Password' link, but I haven't received the promised password reset email." } + @@assert( {{ this == AccountIssue }}) } ``` See the [interactive examples](https://promptfiddle.com) @@ -274,4 +275,42 @@ test Test1 { ## Assertions -This is coming soon! We'll be supporting assertions in test cases. For now -- when you run a test you'll only see errors parsing the output into the right schema, or LLM-provider errors. \ No newline at end of file +Test blocks in BAML code may contain checks and asserts. These attributes +behave similarly to value-level [Checks and Asserts](/guide/baml-advanced/checks-and-asserts), +with several additional variables available in the context of the jinja +expressions you can write in a test: + + - The `_` variable contains fields `result`, `checks` and `latency_ms`. + - The `this` variable refers to the value computed by the test, and is + shorthand for `_.result`. + - In a given check or assert, `_.checks.$NAME` can refer to the NAME of any earlier + check that was run in the same test block. By referring to prior checks, + you can build compound checks and asserts, for example asserting that all + checks of a certain type passed. + +The following example illustrates how each of these features can be used to +validate a test result. + +```rust +test MyTest { + functions [EchoString] + args { + input "example input" + } + @@check( nonempty, {{ this|length > 0 }} ) + @@check( small_enough, {{ _.result|length < 1000 }} ) + @@assert( {{ _.checks.nonempty and _.checks.small_enough }}) + @@assert( {{ _.latency_ms < 1000 }}) +} +``` + +`@@check` and `@@assert` behave differently: + - A `@@check` represents a property + of the test result that should either be manually checked or checked by a + subsequent stage in the test. Multiple `@@check` predicates can fail + without causing a hard failure of the test. + - An `@@assert` represents a hard guarantee. The first failing assert will halt + the remainder of the checks and asserts in this particular test. + +For more information about the syntax used inside `@@check` and `@@assert` +attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts) \ No newline at end of file diff --git a/fern/01-guide/05-baml-advanced/validations.mdx b/fern/01-guide/05-baml-advanced/checks-and-asserts.mdx similarity index 97% rename from fern/01-guide/05-baml-advanced/validations.mdx rename to fern/01-guide/05-baml-advanced/checks-and-asserts.mdx index 6fe3d7d03..b93a4b07d 100644 --- a/fern/01-guide/05-baml-advanced/validations.mdx +++ b/fern/01-guide/05-baml-advanced/checks-and-asserts.mdx @@ -1,5 +1,5 @@ -With custom type validations, you can set specific rules to ensure your data's +With checks and asserts, you can set specific rules to ensure your data's value falls within an acceptable range. BAML provides two types of validations: @@ -117,7 +117,7 @@ class Person { ## Assertion Errors -When validations fail, your BAML function will raise a `BamlValidationError` +When asserts fail, your BAML function will raise a `BamlValidationError` exception, same as when parsing fails. You can catch this exception and handle it as you see fit. @@ -199,8 +199,8 @@ const main = () => { ## Checks -`@check` attributes add validations without raising exceptions if they fail. -Types with `@check` attributes allow the validations to be inspected at +`@check` attributes add validation without raising exceptions if they fail. +Types with `@check` attributes allow the checks to be inspected at runtime. diff --git a/fern/docs.yml b/fern/docs.yml index c3091d67d..28ebd1fc0 100644 --- a/fern/docs.yml +++ b/fern/docs.yml @@ -151,9 +151,9 @@ navigation: - page: Prompt Caching / Message Role Metadata icon: fa-regular fa-database path: 01-guide/05-baml-advanced/prompt-caching.mdx - - page: Validations + - page: Checks and Asserts icon: fa-regular fa-check-circle - path: 01-guide/05-baml-advanced/validations.mdx + path: 01-guide/05-baml-advanced/checks-and-asserts.mdx - section: Boundary Cloud contents: - section: Functions