Make test generation pluggable

[cdent]

It ought to be possible to generate tests from something other than yaml files.

From IRC converations with @elmiko one idea was:

[10:24pm] cdent: what about passing a generator class to build_tests that defaults to the yaml reader?
[10:24pm] elmiko: yea, exactly
[10:24pm] elmiko: that would make nice room for a possible plugin at some point

As discussed there this would likely require some adjustments to the flow of info through driver.py but the result would allow a lot of flexibility.

I reckon this should be post 1.0 because we don't want to be adding large features now.

Basically there would be a TestSuiteGenerator class with a sub class that is generate-from-yaml-files-dir that behaves as the current code. While it would be nice to preserve the build_tests signature, we don't have to make it that if the new signature was sufficiently powerful.

Should a class be passed or an instantiated object or even a list of well-formed objects? As things stand now there is tight coupling between the gathering of the test data and the assembly of that data into tests cases. This could easily be two separate steps: Make the data, traverse it to make it into cases. The way it is now is simply the result of making the initial use case work.

(readers feel free to add comments with additional ideas)

[elmiko]

i think the TestSuiteGenerator sounds like a fine idea.

as for how it should be passed into the test gathering code, i'm not sure. initially i was thinking that it would be nice to have some sort of generator/iterator class that could be walked at the time of testing. my reasoning for this is that if the generator code is large we might not want to pre-generate the entire test. this way if an early test fails it wouldn't be wasted effort generating the entire suite.

[cdent]

Unfortunately one of the early design constraints in gabbi was that it work properly with the kinds of test discovery and reporting that's used in the belly of openstack ci. The main implication of that is that before any tests actually run they are all created. You can see this in the way that the load_tests method creates a suite of TestSuites. When being used with testr the list of tests is discovered, split for concurrency, and then fed back as patterns into the test discovery process. It gets a bit confused and confusing.

Does that mean an iterator that can be interrupted is not an option? No. Just that we need to be aware of the constraints that come about by being somewhat wed to testr+subunit and that failure handling is different depending on whether it is failure in creating a test or in running the test.

Note: When I discovered all this (what felt like) chaos it was after coming from what felt like the much purer world of py.test and it made me shake my head. Alot.

[elmiko]

thanks for the clarification, there may not be a better way to do this than just create all the tests ahead of time.

[cdent]

Now that we've made some refactorings in gabbi.driver the interface for this has become a bit simpler. What we need, essentially, is a callable that returns a list of dicts of the right structure:

[{
   fixtures: [list of fixture class names],
   defaults: dict of test defaults,
   tests: [list of individual tests]
}]

argument handling would need to be resolved but the simplest thing is presumably some kind of iterator. This would replace the cuts of build_tests which currently loops over a list of yaml file names and load_yaml on each one and creates a dict.

[cdent]

Commit 1b31ef0 extracted test making to the suitemaker module, so something could programmatically call test_suite_from_dict with inputs that are generated by...whatever.

[cdent]

While this is still entirely possible to do, at least theoretically, years of use have made me think that the key special value in gabbi is the syntax of the yaml files. It somehow manages to just work and feel good.

[elmiko]

total necro bump XD