forked from autotest/autotest
-
Notifications
You must be signed in to change notification settings - Fork 0
Requirements
Lucas Meneghel Rodrigues edited this page Sep 22, 2011
·
3 revisions
Make it simple to use
- Make the system as user-friendly as possible, whilst still allowing power users (defaults with overrides!)
- Provide web front-ends where possible.
- Capture the "magic" knowledge of how to complex or fiddly operations within the harness, not in a person.
- Low barrier to entry for use and development
Gather as much information as possible
- Collect stdout and stderr. Break them out per test.
- Collect dmesg, and serial console where available. Fall back to netconsole where not.
- Gather profiling data from oprofile, vmstat etc.
- On a hang, gather alt+sysrq+t, etc.
- Monitor the machine via ssh and ICMP ping for it going down
Allow the developers to DEBUG the test failures
- Allow them to rerun the exact same test by hand easily.
- Keep the tests as simple as possible.
- Provide tracebacks on a failure
- Provide a flexible control file format that allows developers to do custom modifications easily.
Support all types of testing
- Allow tests to run in parallel
- Provide reproducible performance benchmarks
- Allow multiple iterations to be done cleanly for performance testing.
- Support filesystem tests (mkfs, mount, umount, fsck, etc)
- Provide test grouping into single units (build, filesystem, etc).
- Support multi-machine testing and provide syncronization barriers
- Support virtualized machines (Containers, KVM, Xen)
An OPEN harness
- Allow us to interact with vendors by sharing tests and problem scenarios easily
- Allow us to interact with the open source community by sharing tests and problem scenarios easily
- Encourage others to contribute to development.
- Also cleanly support proprietary tests where necessary, and code extensions.
Robust operation
- Allow reinstall of machines from scratch
- Support power cycle on failure
Scheduling and automation
- Provide one job queue per machine, or machine group
- Collect results to a central repository
- Automatically watch for new software releases, and kick off any job based on that.
Provide back-end analysis
- Suck all the results into a simple, well formatted database.
- Give a clear PASS/FAIL indication from the client test
- Allow arbitrary key-value pairs per test iteration
- Provide clear display of which tests passed on which machines.
- Graph performance results over time, indicating errors, etc.
- Compare two releases for statistically significant performance differences.
Download in other formats: