Book: Add test reference

docs/book/src/README.md

@@ -16,17 +16,16 @@ Out of the box `typst-test` supports the following features:
 This book contains a few sections aimed at answering the most common questions right out the gate.
 - [Installation](./quickstart/install.md) outlines various ways to install `typst-test`.
 - [Usage](./quickstart/usage.md) goes over some basic commands to get started with `typst-test`.
-- [Features](./quickstart/features.md) introduces various concepts to help you use `typst-test` effectively.
-- [Configuration](.quickstart/config.md) explains commonly used configuration keys.
 
 After the quick start, a few guides delve deeper into some advanced topics.
-- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters.
+- [Writing Tests](./guides/tests.md) inspects adding, removing, updating and editing tests more closely.
 - [Using Test Sets](./guides/test-sets.md) delves into the test set language and how it can be used to isolate tests and speed up your TDD workflow.
+- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters.
 - [Setting Up CI](./guides/ci.md) shows how to set up `typst-test` to continuously test all changes to your package.
 
 The later sections of the book are a technical reference to `typst-test` and its various features or concepts.
-- [Tests](./reference/tests.md) outlines which types of tests `typst-test` supports, how they can be customized and which features are offered within the test scripts.
-- [Test Set Language](./reference/test-set-dsl.md) defines the test set language and its built in test sets.
+- [Tests](./reference/tests/index.md) outlines which types of tests `typst-test` supports, how they can be customized and which features are offered within the test scripts.
+- [Test Set Language](./reference/test-sets/index.md) defines the test set language and its built in test sets.
 - [Configuration Schema](./reference/config.md) lists all existing config options, their expected types and default values.
-- [Command Line Tool](./reference/cli.md) goes over `typst-test`s various sub commands, arguments and options.
+- [Command Line Tool](./reference/cli/index.md) goes over `typst-test`s various sub commands, arguments and options.
 

docs/book/src/SUMMARY.md

@@ -4,36 +4,23 @@
 # Quickstart
 - [Installation](./quickstart/install.md)
 - [Usage](./quickstart/usage.md)
-- [Concepts]()
-- [Configuration]()
 
 # Guides
-- [Automation]()
+- [Writing Tests]()
 - [Using Test Sets](./guides/test-sets.md)
+- [Automation]()
 - [Setting Up CI](./guides/ci.md)
 
 # Reference
-- [Tests]()
-  - [Reference Kinds]()
-  - [Annotations]()
-  - [Directory Structure]()
-- [Test Set Language](./test-sets/README.md)
-  - [Grammar](./test-sets/grammar.md)
-  - [Built-in Test Sets](./test-sets/built-in.md)
+- [Tests](./reference/tests/README.md)
+  - [Annotations](./reference/tests/annotations.md)
+  - [Test Library](./reference/tests/lib.md)
+- [Test Set Language](./reference/test-sets/README.md)
+  - [Grammar](./reference/test-sets/grammar.md)
+  - [Built-in Test Sets](./reference/test-sets/built-in.md)
 - [Configuration Schema]()
   - [Template]()
   - [Hooks]()
 - [Command Line Tool]()
   - [Global Options]()
   - [Command Reference]()
-    - [init]()
-    - [uninit]()
-    - [status]()
-    - [list]()
-    - [compile]()
-    - [run]()
-    - [update]()
-    - [add]()
-    - [edit]()
-    - [remove]()
-    - [util]()

docs/book/src/guides/test-sets.md

@@ -38,7 +38,7 @@ Tying it all together, we can describe what this expression matches in a sentenc
 
 > Select all tests which are not marked ignore and are inside a module starting with `foo-`, include also the test `bar/baz`.
 
-Trying to describe this relationship using options on the command line would be cumbersome, error prone and, depending on the options present, impossible.
+Trying to describe this relationship using options on the command line would be cumbersome, error prone and, depending on the options present, impossible. [^ref]
 
 ## Default Test Sets
 Many operations take either a set of tests as positional arguments, which are matched exactly, or a test set expression.
@@ -52,13 +52,14 @@ See [#40](https://github.com/tingerrr/typst-test/issues/40).
 </div>
 
 More concretely given the invocation
+
 ```bash
 tt list test1 test2 ...
 ```
 
 is equivalent to the following invocation
 
-```txt
+```bash
 tt list --expression 'default & (id(=test1) | id(=test2) | ...)'
 ```
 
@@ -98,3 +99,5 @@ Specifically the `all` and `none` test sets can be used as identity sets for cer
 
 Some of the syntax used in test sets may interfere with your shell, especially the use of whitespace.
 Use non-interpreting quotes around the test set expression (commonly single quotes `'...'`) to avoid interpreting them as shell specific sequences.
+
+[^ref]: To get a more complete look at test sets, take a look at the [reference](../reference/test-sets.md).

docs/book/src/reference/tests/README.md

@@ -0,0 +1,86 @@
+# Tests
+
+There are currently three types of tests:
+- Unit tests, tests which are run to test regressions on code changes mostly through comparison to reference documents.
+- Template tests, special tests for template packages which take a scaffold document and attempt to compile it and optionally compare it.
+- Doc tests, example code in documentation comments which are compiled but not compared.
+
+<div class="warning">
+
+`typst-test` can currently only operate on unit tests found as individual files in the test root.
+
+In the future, template tests and doc tests will be added, see [#34] and [#49].
+
+</div>
+
+Tests get access to a special [test library](./lib.md) and can use [annotations](./annotations.md) configuration.
+
+## Unit Tests
+Unit tests are found in the test root as individual scripts and are the most versatile type of test.
+There are three kinds of unit tests:
+- compile only, tests which are compiled, but not compared
+- compared
+  - persistent, tests which are compared to reference persistent documents
+  - ephemeral, tests which are compared to the output of another script which is compiled on the fly
+
+> Each of those can be selected using one of the [built-in test sets](../test-sets/built-in.md#constants).
+
+Unit tests are the only tests which have access to an extended Typst standard library.
+This extended standard library provides curently provides panic-helpers for catching and comparing panics.
+
+A test is a directory somehwere within the test root (commonly `<project>/tests`), which contains the following entries:
+- `test.typ`: as the entry point
+- `ref.typ` (optional): for ephemeral tests as the reference entry point
+- `ref/` (optional, temporary): for persistent or ephemeral tests for the reference documents
+- `out/` (temporary) for the test documents
+- `diff/` (temporary) for the diff documents
+
+The path from the test root to the test script marks the test's identifier. Its test kind is determined by the existence of the ref script and ref directory:
+- If it contains a `ref` directory but no `ref.typ` script, it is considered a persistent test.
+- If it a `ref.typ` script, it is considered an ephermeral test.
+- If it contains neither, it is considered compile only.
+
+Tests may contain other tests at the moment, e.g the following is valid
+```txt
+tests/
+  foo
+  foo/test.typ
+  foo/bar
+  foo/bar/test.typ
+```
+
+and contains the tests `foo` and `foo/bar`.
+
+Unit tests are compiled with the project root as typst root, such that they can easily access package internals.
+They can also access test library items such as `catch` for catching and binding panics for testing error reporting:
+
+```typst
+/// [annotation]
+///
+/// Description
+
+// access to internals
+#import "/src/internal.typ": foo
+
+#let panics = () => {
+  foo("bar")
+}
+
+// ensures there's a panic
+#assert-panic(panics)
+
+// unwraps the panic if there is one
+#assert.eq(
+  catch(panics).first(),
+  "panicked with: Invalid arg, expected `int`, got `str`",
+)
+```
+
+## Documentation Tests
+TODO: See [#34].
+
+## Template Tests
+TODO: See [#49].
+
+[#34]: https://github.com/tingerrr/typst-test/issues/34
+[#49]: https://github.com/tingerrr/typst-test/issues/49

docs/book/src/reference/tests/annotations.md

@@ -0,0 +1,25 @@
+# Annotations
+Tests may contain annotations at the start of the file.
+These annotations are placed on the leading doc comment of the file itself.
+
+```typst
+/// [ignore]
+/// [custom: foo]
+///
+/// Synopsis:
+/// ...
+
+#import "/src/internal.typ": foo
+...
+```
+
+Annotations may only be placed at the start of the doc comment on individual lines without anything between them (no empty lines or other content).
+
+The following annotations exist:
+
+|Annotation|Description|
+|---|---|
+|`ignore`|Takes not arguments, marks the test as part of the `ignored` test set, can only be used once.|
+|`custom`|Takes a single identifier as argument, marks the test as part of a custom test set of the given identifier, can be used multiple times.|
+
+A test with an annotation like `[custom: foo]` can be selected with a test set like `custom(foo)`.

docs/book/src/reference/tests/lib.md

@@ -0,0 +1,68 @@
+# Test Library
+The test library is an augmented standard library, it contains all definitions in the standard library plus some additional modules and functions which help testing packages and debug regressions.
+
+It defines the following modules:
+- `test`: a module with various testing helpers such as `catch` and additonal asserts.
+
+The following items are re-exorted in the global scope as well:
+- `assert-panic`: originally `test.assert-panic`
+- `catch`: originally `test.catch`
+
+## `test`
+Contains the main testing utilities.
+
+### `assert-panic`
+Ensures that a function panics.
+
+Fails with an error if the function does not panic. Does not produce any output in the document.
+
+#### Example
+```typst
+#assert-panic(() => {},      message: "I panic!")
+#assert-panic(() => panic(), message: "I don't!")
+```
+
+#### Parameters
+```txt
+assert-panic(
+  function,
+  message: str | auto,
+)
+```
+
+> ##### `function: function`
+> - `required`
+> - `positional`
+>
+> The function to test.
+
+> ##### `message: str | auto`
+>
+> The error message when the assertion fails.
+
+### `catch`
+Unwraps and returns the panics generated by a function, if there were any.
+
+Does not produce any output in the document.
+
+#### Example
+```typst
+#assert.eq(catch(() => {}), none)
+#assert.eq(
+  catch(panics).first(),
+  "panicked with: Invalid arg, expected `int`, got `str`",
+)
+```
+
+#### Parameters
+```txt
+catch(
+  function,
+)
+```
+
+> ##### `function: function`
+> - `required`
+> - `positional`
+>
+> The function to test.