[Feature Request] It might be useful to have a utility function for integration testing pipelines built with Hydra that expose command line interfaces through config files.

[mmcdermott]

🚀 Feature Request

I have written a few pipelines that use Hydra for configuration and workflow management. I have integration tests in these codebases to verify that the CLI tools exposed run, end to end, using realistic configurations. To aid this, I have a custom built function, here which takes a set of command line arguments within a script and a dictionary of configuration keywords and constructs a hydra config file in a temporary directory on disk then runs the target command using the built-in hydra args to route to that config file dynamically. This functionality may be useful to folks in general, so that others can more easily run tests like test_pipeline_runs(pipeline_command, {dict of config kwargs}) without needing to duplicate the logic of building and storing the config on disk and routing to it each time. The code I link to above could serve as starter code for integrating this if such functionality would be helpful to others. Obviously this is low priority as one can also just duplicate this code manually or otherwise construct these files in each test, but I figured I post something here in case others would value it.

[mmcdermott]

Note that this part of the functionality which converts a dictionary of kwargs to a hydra/omegaconf command line string is a more annoying block of code to repeat, though there may be better ways of doing this or this maybe should be referenced on omegaconf directly rather than here.