From da5b8aae4c0c818fb7b5a67652142dbbc9a83418 Mon Sep 17 00:00:00 2001 From: tinger Date: Thu, 25 Jul 2024 19:43:05 +0200 Subject: [PATCH 1/3] docs(book): Use correct language in test set guide example --- docs/book/src/guides/test-sets.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/book/src/guides/test-sets.md b/docs/book/src/guides/test-sets.md index 428f649..d2e43fd 100644 --- a/docs/book/src/guides/test-sets.md +++ b/docs/book/src/guides/test-sets.md @@ -52,13 +52,14 @@ See [#40](https://github.com/tingerrr/typst-test/issues/40). More concretely given the invocation + ```bash tt list test1 test2 ... ``` is equivalent to the following invocation -```txt +```bash tt list --expression 'default & (id(=test1) | id(=test2) | ...)' ``` From 013cb7d550be5f836ef7030f6f3d35ca74124101 Mon Sep 17 00:00:00 2001 From: tinger Date: Thu, 25 Jul 2024 19:56:11 +0200 Subject: [PATCH 2/3] docs(book): Fix broken internal links When refering to `README.md` from anywhere outside the SUMMARY.md, it will not correctly translate them into links to `index.html`, but refering to `index.md` will. This also moves `./test-sets` directly into `./reference` and removes some overly specific cli reference sections. --- docs/book/src/README.md | 6 +++--- docs/book/src/SUMMARY.md | 17 +++-------------- docs/book/src/guides/test-sets.md | 4 +++- .../src/{ => reference}/test-sets/README.md | 0 .../src/{ => reference}/test-sets/built-in.md | 0 .../src/{ => reference}/test-sets/grammar.md | 0 6 files changed, 9 insertions(+), 18 deletions(-) rename docs/book/src/{ => reference}/test-sets/README.md (100%) rename docs/book/src/{ => reference}/test-sets/built-in.md (100%) rename docs/book/src/{ => reference}/test-sets/grammar.md (100%) diff --git a/docs/book/src/README.md b/docs/book/src/README.md index c97da5f..4624294 100644 --- a/docs/book/src/README.md +++ b/docs/book/src/README.md @@ -25,8 +25,8 @@ After the quick start, a few guides delve deeper into some advanced topics. - [Setting Up CI](./guides/ci.md) shows how to set up `typst-test` to continuously test all changes to your package. The later sections of the book are a technical reference to `typst-test` and its various features or concepts. -- [Tests](./reference/tests.md) outlines which types of tests `typst-test` supports, how they can be customized and which features are offered within the test scripts. -- [Test Set Language](./reference/test-set-dsl.md) defines the test set language and its built in test sets. +- [Tests](./reference/tests/index.md) outlines which types of tests `typst-test` supports, how they can be customized and which features are offered within the test scripts. +- [Test Set Language](./reference/test-sets/index.md) defines the test set language and its built in test sets. - [Configuration Schema](./reference/config.md) lists all existing config options, their expected types and default values. -- [Command Line Tool](./reference/cli.md) goes over `typst-test`s various sub commands, arguments and options. +- [Command Line Tool](./reference/cli/index.md) goes over `typst-test`s various sub commands, arguments and options. diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index 3a4d0e9..18e3014 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -17,23 +17,12 @@ - [Reference Kinds]() - [Annotations]() - [Directory Structure]() -- [Test Set Language](./test-sets/README.md) - - [Grammar](./test-sets/grammar.md) - - [Built-in Test Sets](./test-sets/built-in.md) +- [Test Set Language](./reference/test-sets/README.md) + - [Grammar](./reference/test-sets/grammar.md) + - [Built-in Test Sets](./reference/test-sets/built-in.md) - [Configuration Schema]() - [Template]() - [Hooks]() - [Command Line Tool]() - [Global Options]() - [Command Reference]() - - [init]() - - [uninit]() - - [status]() - - [list]() - - [compile]() - - [run]() - - [update]() - - [add]() - - [edit]() - - [remove]() - - [util]() diff --git a/docs/book/src/guides/test-sets.md b/docs/book/src/guides/test-sets.md index d2e43fd..444e548 100644 --- a/docs/book/src/guides/test-sets.md +++ b/docs/book/src/guides/test-sets.md @@ -38,7 +38,7 @@ Tying it all together, we can describe what this expression matches in a sentenc > Select all tests which are not marked ignore and are inside a module starting with `foo-`, include also the test `bar/baz`. -Trying to describe this relationship using options on the command line would be cumbersome, error prone and, depending on the options present, impossible. +Trying to describe this relationship using options on the command line would be cumbersome, error prone and, depending on the options present, impossible. [^ref] ## Default Test Sets Many operations take either a set of tests as positional arguments, which are matched exactly, or a test set expression. @@ -99,3 +99,5 @@ Specifically the `all` and `none` test sets can be used as identity sets for cer Some of the syntax used in test sets may interfere with your shell, especially the use of whitespace. Use non-interpreting quotes around the test set expression (commonly single quotes `'...'`) to avoid interpreting them as shell specific sequences. + +[^ref]: To get a more complete look at test sets, take a look at the [reference](../reference/test-sets.md). diff --git a/docs/book/src/test-sets/README.md b/docs/book/src/reference/test-sets/README.md similarity index 100% rename from docs/book/src/test-sets/README.md rename to docs/book/src/reference/test-sets/README.md diff --git a/docs/book/src/test-sets/built-in.md b/docs/book/src/reference/test-sets/built-in.md similarity index 100% rename from docs/book/src/test-sets/built-in.md rename to docs/book/src/reference/test-sets/built-in.md diff --git a/docs/book/src/test-sets/grammar.md b/docs/book/src/reference/test-sets/grammar.md similarity index 100% rename from docs/book/src/test-sets/grammar.md rename to docs/book/src/reference/test-sets/grammar.md From 4dd5665eea1a93d429b9f6cb73d747f4c8254740 Mon Sep 17 00:00:00 2001 From: tinger Date: Thu, 25 Jul 2024 20:27:10 +0200 Subject: [PATCH 3/3] docs(book): Add tests reference chapter Documents and outlines the types and kinds of tests, their expected structure, as well as the test library available to them. --- docs/book/src/README.md | 5 +- docs/book/src/SUMMARY.md | 12 ++- docs/book/src/reference/tests/README.md | 86 ++++++++++++++++++++ docs/book/src/reference/tests/annotations.md | 25 ++++++ docs/book/src/reference/tests/lib.md | 68 ++++++++++++++++ 5 files changed, 186 insertions(+), 10 deletions(-) create mode 100644 docs/book/src/reference/tests/README.md create mode 100644 docs/book/src/reference/tests/annotations.md create mode 100644 docs/book/src/reference/tests/lib.md diff --git a/docs/book/src/README.md b/docs/book/src/README.md index 4624294..e142a6a 100644 --- a/docs/book/src/README.md +++ b/docs/book/src/README.md @@ -16,12 +16,11 @@ Out of the box `typst-test` supports the following features: This book contains a few sections aimed at answering the most common questions right out the gate. - [Installation](./quickstart/install.md) outlines various ways to install `typst-test`. - [Usage](./quickstart/usage.md) goes over some basic commands to get started with `typst-test`. -- [Features](./quickstart/features.md) introduces various concepts to help you use `typst-test` effectively. -- [Configuration](.quickstart/config.md) explains commonly used configuration keys. After the quick start, a few guides delve deeper into some advanced topics. -- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters. +- [Writing Tests](./guides/tests.md) inspects adding, removing, updating and editing tests more closely. - [Using Test Sets](./guides/test-sets.md) delves into the test set language and how it can be used to isolate tests and speed up your TDD workflow. +- [Automation](./guides/automation.md) explains the ins and outs of hooks and how they can be used for testing typst preprocessors or formatters. - [Setting Up CI](./guides/ci.md) shows how to set up `typst-test` to continuously test all changes to your package. The later sections of the book are a technical reference to `typst-test` and its various features or concepts. diff --git a/docs/book/src/SUMMARY.md b/docs/book/src/SUMMARY.md index 18e3014..f49d3d8 100644 --- a/docs/book/src/SUMMARY.md +++ b/docs/book/src/SUMMARY.md @@ -4,19 +4,17 @@ # Quickstart - [Installation](./quickstart/install.md) - [Usage](./quickstart/usage.md) -- [Concepts]() -- [Configuration]() # Guides -- [Automation]() +- [Writing Tests]() - [Using Test Sets](./guides/test-sets.md) +- [Automation]() - [Setting Up CI](./guides/ci.md) # Reference -- [Tests]() - - [Reference Kinds]() - - [Annotations]() - - [Directory Structure]() +- [Tests](./reference/tests/README.md) + - [Annotations](./reference/tests/annotations.md) + - [Test Library](./reference/tests/lib.md) - [Test Set Language](./reference/test-sets/README.md) - [Grammar](./reference/test-sets/grammar.md) - [Built-in Test Sets](./reference/test-sets/built-in.md) diff --git a/docs/book/src/reference/tests/README.md b/docs/book/src/reference/tests/README.md new file mode 100644 index 0000000..6d335ec --- /dev/null +++ b/docs/book/src/reference/tests/README.md @@ -0,0 +1,86 @@ +# Tests + +There are currently three types of tests: +- Unit tests, tests which are run to test regressions on code changes mostly through comparison to reference documents. +- Template tests, special tests for template packages which take a scaffold document and attempt to compile it and optionally compare it. +- Doc tests, example code in documentation comments which are compiled but not compared. + +
+ +`typst-test` can currently only operate on unit tests found as individual files in the test root. + +In the future, template tests and doc tests will be added, see [#34] and [#49]. + +
+ +Tests get access to a special [test library](./lib.md) and can use [annotations](./annotations.md) configuration. + +## Unit Tests +Unit tests are found in the test root as individual scripts and are the most versatile type of test. +There are three kinds of unit tests: +- compile only, tests which are compiled, but not compared +- compared + - persistent, tests which are compared to reference persistent documents + - ephemeral, tests which are compared to the output of another script which is compiled on the fly + +> Each of those can be selected using one of the [built-in test sets](../test-sets/built-in.md#constants). + +Unit tests are the only tests which have access to an extended Typst standard library. +This extended standard library provides curently provides panic-helpers for catching and comparing panics. + +A test is a directory somehwere within the test root (commonly `/tests`), which contains the following entries: +- `test.typ`: as the entry point +- `ref.typ` (optional): for ephemeral tests as the reference entry point +- `ref/` (optional, temporary): for persistent or ephemeral tests for the reference documents +- `out/` (temporary) for the test documents +- `diff/` (temporary) for the diff documents + +The path from the test root to the test script marks the test's identifier. Its test kind is determined by the existence of the ref script and ref directory: +- If it contains a `ref` directory but no `ref.typ` script, it is considered a persistent test. +- If it a `ref.typ` script, it is considered an ephermeral test. +- If it contains neither, it is considered compile only. + +Tests may contain other tests at the moment, e.g the following is valid +```txt +tests/ + foo + foo/test.typ + foo/bar + foo/bar/test.typ +``` + +and contains the tests `foo` and `foo/bar`. + +Unit tests are compiled with the project root as typst root, such that they can easily access package internals. +They can also access test library items such as `catch` for catching and binding panics for testing error reporting: + +```typst +/// [annotation] +/// +/// Description + +// access to internals +#import "/src/internal.typ": foo + +#let panics = () => { + foo("bar") +} + +// ensures there's a panic +#assert-panic(panics) + +// unwraps the panic if there is one +#assert.eq( + catch(panics).first(), + "panicked with: Invalid arg, expected `int`, got `str`", +) +``` + +## Documentation Tests +TODO: See [#34]. + +## Template Tests +TODO: See [#49]. + +[#34]: https://github.com/tingerrr/typst-test/issues/34 +[#49]: https://github.com/tingerrr/typst-test/issues/49 diff --git a/docs/book/src/reference/tests/annotations.md b/docs/book/src/reference/tests/annotations.md new file mode 100644 index 0000000..0b7393e --- /dev/null +++ b/docs/book/src/reference/tests/annotations.md @@ -0,0 +1,25 @@ +# Annotations +Tests may contain annotations at the start of the file. +These annotations are placed on the leading doc comment of the file itself. + +```typst +/// [ignore] +/// [custom: foo] +/// +/// Synopsis: +/// ... + +#import "/src/internal.typ": foo +... +``` + +Annotations may only be placed at the start of the doc comment on individual lines without anything between them (no empty lines or other content). + +The following annotations exist: + +|Annotation|Description| +|---|---| +|`ignore`|Takes not arguments, marks the test as part of the `ignored` test set, can only be used once.| +|`custom`|Takes a single identifier as argument, marks the test as part of a custom test set of the given identifier, can be used multiple times.| + +A test with an annotation like `[custom: foo]` can be selected with a test set like `custom(foo)`. diff --git a/docs/book/src/reference/tests/lib.md b/docs/book/src/reference/tests/lib.md new file mode 100644 index 0000000..48ac0eb --- /dev/null +++ b/docs/book/src/reference/tests/lib.md @@ -0,0 +1,68 @@ +# Test Library +The test library is an augmented standard library, it contains all definitions in the standard library plus some additional modules and functions which help testing packages and debug regressions. + +It defines the following modules: +- `test`: a module with various testing helpers such as `catch` and additonal asserts. + +The following items are re-exorted in the global scope as well: +- `assert-panic`: originally `test.assert-panic` +- `catch`: originally `test.catch` + +## `test` +Contains the main testing utilities. + +### `assert-panic` +Ensures that a function panics. + +Fails with an error if the function does not panic. Does not produce any output in the document. + +#### Example +```typst +#assert-panic(() => {}, message: "I panic!") +#assert-panic(() => panic(), message: "I don't!") +``` + +#### Parameters +```txt +assert-panic( + function, + message: str | auto, +) +``` + +> ##### `function: function` +> - `required` +> - `positional` +> +> The function to test. + +> ##### `message: str | auto` +> +> The error message when the assertion fails. + +### `catch` +Unwraps and returns the panics generated by a function, if there were any. + +Does not produce any output in the document. + +#### Example +```typst +#assert.eq(catch(() => {}), none) +#assert.eq( + catch(panics).first(), + "panicked with: Invalid arg, expected `int`, got `str`", +) +``` + +#### Parameters +```txt +catch( + function, +) +``` + +> ##### `function: function` +> - `required` +> - `positional` +> +> The function to test.