diff --git a/docs/book/src/README.md b/docs/book/src/README.md index 4624294..e142a6a 100644 --- a/docs/book/src/README.md +++ b/docs/book/src/README.md @@ -16,12 +16,11 @@ Out of the box `typst-test` supports the following features: This book contains a few sections aimed at answering the most common questions right out the gate. - [Installation](./quickstart/install.md) outlines various ways to install `typst-test`. - [Usage](./quickstart/usage.md) goes over some basic commands to get started with `typst-test`. -- [Features](./quickstart/features.md) introduces various concepts to help you use `typst-test` effectively. -- [Configuration](.quickstart/config.md) explains commonly used configuration keys. After the quick start, a few guides delve deeper into some advanced topics. -- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters. +- [Writing Tests](./guides/tests.md) inspects adding, removing, updating and editing tests more closely. - [Using Test Sets](./guides/test-sets.md) delves into the test set language and how it can be used to isolate tests and speed up your TDD workflow. +- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters. - [Setting Up CI](./guides/ci.md) shows how to set up `typst-test` to continuously test all changes to your package. The later sections of the book are a technical reference to `typst-test` and its various features or concepts. diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index 18e3014..f49d3d8 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -4,19 +4,17 @@ # Quickstart - [Installation](./quickstart/install.md) - [Usage](./quickstart/usage.md) -- [Concepts]() -- [Configuration]() # Guides -- [Automation]() +- [Writing Tests]() - [Using Test Sets](./guides/test-sets.md) +- [Automation]() - [Setting Up CI](./guides/ci.md) # Reference -- [Tests]() - - [Reference Kinds]() - - [Annotations]() - - [Directory Structure]() +- [Tests](./reference/tests/README.md) + - [Annotations](./reference/tests/annotations.md) + - [Test Library](./reference/tests/lib.md) - [Test Set Language](./reference/test-sets/README.md) - [Grammar](./reference/test-sets/grammar.md) - [Built-in Test Sets](./reference/test-sets/built-in.md) diff --git a/docs/book/src/reference/tests/README.md b/docs/book/src/reference/tests/README.md new file mode 100644 index 0000000..6d335ec --- /dev/null +++ b/docs/book/src/reference/tests/README.md @@ -0,0 +1,86 @@ +# Tests + +There are currently three types of tests: +- Unit tests, tests which are run to test regressions on code changes mostly through comparison to reference documents. +- Template tests, special tests for template packages which take a scaffold document and attempt to compile it and optionally compare it. +- Doc tests, example code in documentation comments which are compiled but not compared. + +
+ +`typst-test` can currently only operate on unit tests found as individual files in the test root. + +In the future, template tests and doc tests will be added, see [#34] and [#49]. + +
+ +Tests get access to a special [test library](./lib.md) and can use [annotations](./annotations.md) configuration. + +## Unit Tests +Unit tests are found in the test root as individual scripts and are the most versatile type of test. +There are three kinds of unit tests: +- compile only, tests which are compiled, but not compared +- compared + - persistent, tests which are compared to reference persistent documents + - ephemeral, tests which are compared to the output of another script which is compiled on the fly + +> Each of those can be selected using one of the [built-in test sets](../test-sets/built-in.md#constants). + +Unit tests are the only tests which have access to an extended Typst standard library. +This extended standard library provides curently provides panic-helpers for catching and comparing panics. + +A test is a directory somehwere within the test root (commonly `/tests`), which contains the following entries: +- `test.typ`: as the entry point +- `ref.typ` (optional): for ephemeral tests as the reference entry point +- `ref/` (optional, temporary): for persistent or ephemeral tests for the reference documents +- `out/` (temporary) for the test documents +- `diff/` (temporary) for the diff documents + +The path from the test root to the test script marks the test's identifier. Its test kind is determined by the existence of the ref script and ref directory: +- If it contains a `ref` directory but no `ref.typ` script, it is considered a persistent test. +- If it a `ref.typ` script, it is considered an ephermeral test. +- If it contains neither, it is considered compile only. + +Tests may contain other tests at the moment, e.g the following is valid +```txt +tests/ + foo + foo/test.typ + foo/bar + foo/bar/test.typ +``` + +and contains the tests `foo` and `foo/bar`. + +Unit tests are compiled with the project root as typst root, such that they can easily access package internals. +They can also access test library items such as `catch` for catching and binding panics for testing error reporting: + +```typst +/// [annotation] +/// +/// Description + +// access to internals +#import "/src/internal.typ": foo + +#let panics = () => { + foo("bar") +} + +// ensures there's a panic +#assert-panic(panics) + +// unwraps the panic if there is one +#assert.eq( + catch(panics).first(), + "panicked with: Invalid arg, expected `int`, got `str`", +) +``` + +## Documentation Tests +TODO: See [#34]. + +## Template Tests +TODO: See [#49]. + +[#34]: https://github.com/tingerrr/typst-test/issues/34 +[#49]: https://github.com/tingerrr/typst-test/issues/49 diff --git a/docs/book/src/reference/tests/annotations.md b/docs/book/src/reference/tests/annotations.md new file mode 100644 index 0000000..0b7393e --- /dev/null +++ b/docs/book/src/reference/tests/annotations.md @@ -0,0 +1,25 @@ +# Annotations +Tests may contain annotations at the start of the file. +These annotations are placed on the leading doc comment of the file itself. + +```typst +/// [ignore] +/// [custom: foo] +/// +/// Synopsis: +/// ... + +#import "/src/internal.typ": foo +... +``` + +Annotations may only be placed at the start of the doc comment on individual lines without anything between them (no empty lines or other content). + +The following annotations exist: + +|Annotation|Description| +|---|---| +|`ignore`|Takes not arguments, marks the test as part of the `ignored` test set, can only be used once.| +|`custom`|Takes a single identifier as argument, marks the test as part of a custom test set of the given identifier, can be used multiple times.| + +A test with an annotation like `[custom: foo]` can be selected with a test set like `custom(foo)`. diff --git a/docs/book/src/reference/tests/lib.md b/docs/book/src/reference/tests/lib.md new file mode 100644 index 0000000..48ac0eb --- /dev/null +++ b/docs/book/src/reference/tests/lib.md @@ -0,0 +1,68 @@ +# Test Library +The test library is an augmented standard library, it contains all definitions in the standard library plus some additional modules and functions which help testing packages and debug regressions. + +It defines the following modules: +- `test`: a module with various testing helpers such as `catch` and additonal asserts. + +The following items are re-exorted in the global scope as well: +- `assert-panic`: originally `test.assert-panic` +- `catch`: originally `test.catch` + +## `test` +Contains the main testing utilities. + +### `assert-panic` +Ensures that a function panics. + +Fails with an error if the function does not panic. Does not produce any output in the document. + +#### Example +```typst +#assert-panic(() => {}, message: "I panic!") +#assert-panic(() => panic(), message: "I don't!") +``` + +#### Parameters +```txt +assert-panic( + function, + message: str | auto, +) +``` + +> ##### `function: function` +> - `required` +> - `positional` +> +> The function to test. + +> ##### `message: str | auto` +> +> The error message when the assertion fails. + +### `catch` +Unwraps and returns the panics generated by a function, if there were any. + +Does not produce any output in the document. + +#### Example +```typst +#assert.eq(catch(() => {}), none) +#assert.eq( + catch(panics).first(), + "panicked with: Invalid arg, expected `int`, got `str`", +) +``` + +#### Parameters +```txt +catch( + function, +) +``` + +> ##### `function: function` +> - `required` +> - `positional` +> +> The function to test.