diff --git a/README.md b/README.md index 984efea9..5570b850 100644 --- a/README.md +++ b/README.md @@ -2,51 +2,117 @@ [![Coverage Status](https://coveralls.io/repos/github/tarantool/test-run/badge.svg)](https://coveralls.io/github/tarantool/test-run) -### Test Suite - -Bunch of tests, that lay down in the subfolder (recursively) with `suite.ini` -file. `suite.ini` is basic ini-file, that consists of one section `default`, -and a number of fields: - -* `core` -* `description` - Test Suite description -* `script` - shebang file to start tarantool with -* disables: - * `disabled` - tests that must be skipped - * `release_disabled` - tests that must be skipped when Tarantool has been - builded with `Release` - * `valgrind_disabled` - tests that must be skipped when Valgrind is enabled -* `lua_libs` - paths for lua files, that should be copied into the folder, - where server is started (delimited with the space, e.g. `lua_libs=lua/1.lua - lua/2.lua`) -* `long_run` - mark tests as long, enabled only with `--long` option (delimited - with the space, e.g. `long_run=t1.test.lua t2.test.lua`) -* `config` - test configuration file name - -Field `core` must be one of: - -* `tarantool` - Test-Suite for Functional Testing -* `app` - Another functional Test-Suite -* `unittest` - Unit-Testing Test Suite - -### Test - -Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, and may have -skip condition file `*.skipcond`. On first run (without `.result`) `.result` -is generated from output. Each run, in the beggining, `.skipcond` file is -executed. In the local env there's object `self`, that's `Test` object. If test -must be skipped - you must put `self.skip = 1` in this file. Next, -`.test(.lua|.py)?` is executed and file `.reject` is created, then `.reject` is -compared with `.result`. If something differs, then 15 last string of this diff -file are printed and `.reject` file is saving in the `/rejects/` -subfolder given in options or set localy as `var/rejects/` by default. -If not, then `.reject` file is deleted. +## Test suite configuration + +Test suite is a bunch of tests located in a subdirectory (recursively) with a `suite.ini` +file. The `suite.ini` is a basic ini-file with one section `[default]`. + +```ini +[default] +core = luatest +description = Example test suite using luatest +; comment line +... +``` + +All list values in `suite.ini` are delimited by spaces. +Lines starting with `;` are comment lines. + +Below is a list of configuration values (fields) in the `suite.ini` file. + +### General configuration + +* `core` — major testing dependency or method + Should have one of the following values: + + * `tarantool` — test suite used for functional testing + * `app` — TAP-tests, another functional test suite + * `unittest` — unit testing test suite + * `luatest` — test suite using luatest library + +* `description` — test suite description +* `script` (optional) — shebang file to start tarantool with +* `config` — name of a test configuration file + TODO: what's a test configuration file? + +### Skipping tests + +A number of fields are used to disable (skip) certain tests: + +* `disabled` — list of tests that should be skipped +* `release_disabled` — list of tests that should be skipped when Tarantool is + built with `Release` +* `valgrind_disabled` — list of tests that should be skipped when Valgrind is enabled +* `long_run` — mark tests as long, such tests will run only with `--long` option. + + ```ini + long_run = t1.test.lua t2.test.lua + ``` + +### Other parameters + +* `lua_libs` — paths for lua files, that should be copied into the folder, + where server is started. For example: + + ```ini + lua_libs = lua/require_mod.lua lua/serializer_test.lua lua/process_timeout.lua + ``` + +* `show_reproduce_content` (optional, `True` by default) — TODO: what's this? +* `is_parallel` (optional, `False` by default) — whether the tests in the suite can run in parallel + +* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets +* `use_unix_sockets_iproto` (optional, `False` by default) — use hard-coded unix sockets for iproto. + This parameter is specific to `core = app`. + +### Fragile tests + +Tests which fail sometimes due to external reasons can be marked as fragile (flaky). +Test-run will retry each test if it fails. +It's recommended to provide a list of related +[tarantool/tarantool](https://github.com/tarantool/tarantool) issues +next to each fragile test. + +```ini +fragile = { + "retries": 10, + "tests": { + "tarantoolctl.test.lua": { + "issues": [ "gh-5059", "gh-5346" ], + }, + {...}, + {...}, + } +} +``` + +## Steps of a test run + +Each test consists of files `*.test(.lua|.sql|.py)?`, `*.result`, +and an optional skip condition file `*.skipcond`. + +In the beginning of each test run, the `.skipcond` file is executed as a Python script. +In the local Python environment there's a `self` object, which is an instance of the [`Test` class](./lib/test.py). +To skip the test under some condition, set `self.skip = 1`. + +Next, the test in `.test(.lua|.py)?` is executed. +If this is a first test run and there's no `.result` file yet, it is generated from the output. +If there is a `.result` file, output is written to a `.reject` file, +which is then compared to `.result`. + +If there's a difference between `.reject` and `.result`, the following happens: + +1. The last 15 lines of diff are printed to output. +2. The `.reject` file is saved in the `/rejects/` + subfolder given in options or set locally as `var/rejects/` by default. + +If there is no difference between `.reject` and `.result`, the `.reject` file is deleted. ### Test configuration -Test configuration file contains config for multiple run. For each test section -system runs separated test and compares result with common `.result` file. For -example we need to run one test for different db engines("*" means default +Test configuration file contains configuration for multiple runs. +For each test section, system runs a separate test and compares the result to the common `.result` file. +For example, we need to run a test with different DB engines (`"*"` means the default configuration): ```json @@ -62,7 +128,7 @@ configuration): } ``` -In test case we can get configuration from inspector: +In the test case we can get configuration from the inspector: ```lua engine = test_run:get_cfg('engine') @@ -70,7 +136,7 @@ engine = test_run:get_cfg('engine') -- second run engine is 'sophia' ``` -"engine" value has a special meaning for *.test.sql files: if it is "memtx" or +"engine" value has a special meaning for `*.test.sql` files: if it is "memtx" or "vinyl", then the corresponding default engine will be set before executing commands from a test file. An engine is set with the following commands: @@ -79,11 +145,11 @@ UPDATE "_session_settings" SET "value" = 'memtx|vinyl' WHERE "name" = 'sql_defau pragma sql_default_engine='memtx|vinyl' ``` -If the first fails, then the second will be executed. When both fails, fail the test. +If the first fails, then the second will be executed. When both fail, the test fails. -#### Python +#### Python tests -Files: `.test.py`, `.result` and `.skipcond`(optionaly). +Files: `.test.py`, `.result` and `.skipcond` (optionally). Environment: @@ -173,7 +239,7 @@ Tests interact only with `AdminConnection`. Supports some preprocessor functions **Delimiter example:** -``` +```lua env = require('test_run') test_run = env.new() box.schema.space.create('temp') @@ -194,7 +260,7 @@ test **Delimiter result:** -``` +```yaml env = require('test_run') test_run = env.new() box.schema.space.create('temp')