From a755d4c843d02f074fa8006d5dade945649b4c1c Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Fri, 20 Dec 2024 10:26:13 -0800 Subject: [PATCH 1/2] Add some basic docs about writing ktest tests --- docs/testing/automated-testing.md | 56 ----------- docs/testing/release-testing.md | 2 +- docs/testing/running-tests.md | 15 +-- docs/testing/writing-tests.md | 150 ++++++++++++++++++++++++++++++ 4 files changed, 155 insertions(+), 68 deletions(-) delete mode 100644 docs/testing/automated-testing.md create mode 100644 docs/testing/writing-tests.md diff --git a/docs/testing/automated-testing.md b/docs/testing/automated-testing.md deleted file mode 100644 index 8bbe409f18..0000000000 --- a/docs/testing/automated-testing.md +++ /dev/null @@ -1,56 +0,0 @@ -# Automated Testing - -## Testing with gtest/gmock - -Before feature requests or bug fixes can be merged into master, the folowing -steps should be taken: - -- Create a new test suite named after the issue, e.g. `Issue840`. -- Add a test case named `Reproduces` that reproduces the bug. It should fail if - the bug is present and pass if the bug is fixed. -- Merge the proposed fix into a temporary testing branch. -- The reproduction test should fail. -- Add a test called "HasNotRegresed" that detects a potential regression. - It should pass with the fix and fail before the fix. -- Comment out and keep the `Reproduces` test case as documentation. - -For an example, see keyboardio:Kaleidoscope/tests/issue_840. - -### Adding a New Test Case - -For general information on writing test with gtest/gmock see [gtest -docs](https://github.com/google/googletest/tree/master/googletest/docs) and -[gmock docs](https://github.com/google/googletest/tree/master/googlemock/docs). - -1. Create a new test file, if appropriate. -1. Choose a new test suite name and create a new test fixture, if appropriate. -1. Write your test case. - -The final include in any test file should be `#include -"testing/setup-googletest.h"` which should be followed by the macro -invocation `SETUP_GOOGLETEST()`. This will take care of including headers -commonly used in tests in addtion to gtest and gmock headers. - -Any test fixtures should inherit from `VirtualDeviceTest` which wraps the test -sim APIs and provides common setup and teardown functionality. The appropriate -header is already imported by `setup-googletest.h` - -### Test Infrastructure - -If you need to modify or extend test infrastructure to support your use case, -it can currently be found under `keyboardio:Kaleidoscope/testing`. - -### Style - -TODO(obra): Fill out this section to your liking. - -You can see samples of the desired test style in the [example tests](#examples). - -### Examples - -All existing tests are examples and may be found under -`keyboardio:Kaleidoscope/tests`. - -## Testing with Aglais/Papilio - -TODO(obra): Write (or delegate the writing of) this section. diff --git a/docs/testing/release-testing.md b/docs/testing/release-testing.md index 3def39d132..80fd81c189 100644 --- a/docs/testing/release-testing.md +++ b/docs/testing/release-testing.md @@ -2,7 +2,7 @@ Before a new release of Kaleidoscope, the following test process should be run through on all supported operating systems. -Always run all of the [automated tests](automated-testing.md) to verify there are no regressions. +Always run all of the [running tests](automated-testing.md) to verify there are no regressions. (As of August 2017, this whole thing really applies to Model01-Firmware, but we hope to generalize it to Kaleidoscope) diff --git a/docs/testing/running-tests.md b/docs/testing/running-tests.md index 946baa2ed6..9f39bafa3d 100644 --- a/docs/testing/running-tests.md +++ b/docs/testing/running-tests.md @@ -1,7 +1,5 @@ # Testing Kaleidoscope -This is not yet proper documentation about running or writing tests, just some rough notes. - Kaleidoscope includes a simulator that can pretend (to a certain extent) to be a keyboard for the purpose of testing. On most UNIX-like systems, you can run Kaleidoscope's simulator tests by running @@ -10,12 +8,6 @@ On most UNIX-like systems, you can run Kaleidoscope's simulator tests by running make simulator-tests ``` -Our simulator currently has some weird linking issues on macOS, so the easiest way to run tests on macOS is using Docker. - - -``` -make docker-simulator-tests -``` During development, when you may be running your tests very frequently, it's sometimes useful to run a subset of tests. @@ -24,11 +16,12 @@ You can control the directory that Kaleidoscope searches for test suites with th To only run tests in subdirectories of the 'tests/hid' directory, you'd write: ``` -make simulator-tests TEST_PATH=tests/hid +make simulator-tests TEST_PATH=hid ``` -or + +If you'd rather run the simulator in a Docker environment, you can do that with the following command, although it will be a good deal slower than running them directly on your system: ``` -make docker-simulator-tests TEST_PATH=tests/hid +make docker-simulator-tests ``` diff --git a/docs/testing/writing-tests.md b/docs/testing/writing-tests.md new file mode 100644 index 0000000000..9919f07ccb --- /dev/null +++ b/docs/testing/writing-tests.md @@ -0,0 +1,150 @@ +# Writing tests + +## Simulator tests + +### Adding a New Test Case + +If you're just getting started with testing in Kaleidoscope, you almost certainly want to add a simulator test. + +Simulator tests are composed of a `.ino` sketch file which sets up a keyboard layout and any plugins you need for your test and a `test.ktest` file which simulates a typist's behavior and checks the keyboard's response. The `.ino` file must be named the same as the directory it's in. So if you're creating a new test in `tests/features/ai-keypresses`, your `.ino` file would be named `ai-keypresses.ino` + +Each simulator test lives in its own directory underneath `tests/` + +The simplest test is `tests/examples/basic-keypress`: + +``` +basic-keypress.ino +sketch.yaml +test.ktest +``` + +`basic-keypress.ino` looks like this: + +``` +#include + +KEYMAPS( + [0] = KEYMAP_STACKED + ( + ___, ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, ___, ___, ___, + ___, Key_A, Key_S, ___, ___, ___, + ___, ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, + ___, + + ___, ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, ___, ___, ___, + ___, ___, ___, ___, + ___ + ), +) + +void setup() { + Kaleidoscope.setup(); +} + +void loop() { + Kaleidoscope.loop(); +} +``` + +All it does is set up the keys we need to test. + +The `test.ktest` file looks like this: + +``` +VERSION 1 + +KEYSWITCH A 2 1 +KEYSWITCH S 2 2 + +# ============================================================================== +NAME Keys active when pressed + +RUN 4 ms +PRESS A +RUN 1 cycle +EXPECT keyboard-report Key_A # Key A should be pressed + +RUN 4 ms +RELEASE A +RUN 1 cycle +EXPECT keyboard-report empty # No keys should be pressed + +RUN 4 ms +PRESS A +PRESS S +RUN 1 ms +EXPECT keyboard-report Key_A # Key A should be pressed +EXPECT keyboard-report Key_A, Key_S # A and S should be pressed + +RUN 4 ms +RELEASE A +RELEASE S +RUN 1 cycle +# A is released first because of scan order +EXPECT keyboard-report Key_S # Key S should be pressed +EXPECT keyboard-report empty # No keys should be pressed + +RUN 5 ms +``` + +The most complicated part of it is the `KEYSWITCH` mapping at the top, which tells the test infrastructure what you mean when you say things like `PRESS ` or `RELEASE `. The `A` and `B` in the example above are just human readable names for which simulated physical keys are getting `PRESS`ed, `RELEASE`ed or `TAP`ped. + +The `EXPECT` lines for the `keyboard-report` are looking for the same `Key_` identifiers you'd put in an arduino sketch. + +The third file you need in your example is the `sketch.yaml` file. This tell the simulator what kind of virtual device to test on. As of right now, the only devices that we know work in the simulator are the `atreus` and the `model01`. + +Unless you have reason to do something else, your `sketch.yaml` should look like this: + +``` +default_fqbn: keyboardio:virtual:model01 +``` + +To learn how to run the simulator tests, head on over to [Running Tests](running-tests.md) + +## Testing with gtest/gmock + +Before feature requests or bug fixes can be merged into master, the folowing +steps should be taken: + +- Create a new test suite named after the issue, e.g. `Issue840`. +- Add a test case named `Reproduces` that reproduces the bug. It should fail if + the bug is present and pass if the bug is fixed. +- Merge the proposed fix into a temporary testing branch. +- The reproduction test should fail. +- Add a test called "HasNotRegresed" that detects a potential regression. + It should pass with the fix and fail before the fix. +- Comment out and keep the `Reproduces` test case as documentation. + +For an example, see keyboardio:Kaleidoscope/tests/issue_840. + + + + +### Adding a low-level test + +For general information on writing test with gtest/gmock see [gtest +docs](https://github.com/google/googletest/tree/master/googletest/docs) and +[gmock docs](https://github.com/google/googletest/tree/master/googlemock/docs). + +1. Create a new test file, if appropriate. +1. Choose a new test suite name and create a new test fixture, if appropriate. +1. Write your test case. + +The final include in any test file should be `#include +"testing/setup-googletest.h"` which should be followed by the macro +invocation `SETUP_GOOGLETEST()`. This will take care of including headers +commonly used in tests in addtion to gtest and gmock headers. + +Any test fixtures should inherit from `VirtualDeviceTest` which wraps the test +sim APIs and provides common setup and teardown functionality. The appropriate +header is already imported by `setup-googletest.h` + +### Test Infrastructure + +If you need to modify or extend test infrastructure to support your use case, +it can currently be found under `keyboardio:Kaleidoscope/testing`. From df643ea3aa53910918709625dbcc3a2b2273607b Mon Sep 17 00:00:00 2001 From: Jesse Vincent Date: Fri, 20 Dec 2024 10:28:01 -0800 Subject: [PATCH 2/2] remove the old sketch.json file from the basic-keypress test as a first test of being able to remove them all --- tests/examples/basic-keypress/sketch.json | 6 ------ 1 file changed, 6 deletions(-) delete mode 100644 tests/examples/basic-keypress/sketch.json diff --git a/tests/examples/basic-keypress/sketch.json b/tests/examples/basic-keypress/sketch.json deleted file mode 100644 index 43dc4c7e2d..0000000000 --- a/tests/examples/basic-keypress/sketch.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "cpu": { - "fqbn": "keyboardio:virtual:model01", - "port": "" - } -} \ No newline at end of file