Sweeper PR

[SabhyaC26]

Fixes #143 #181

Changes proposed in this pull request:

• new sweeper class for performing hyperparameter sweeps in tango

Before submitting

• I've read and followed all steps in the Making a pull request
  section of the CONTRIBUTING docs.
• I've updated or added any relevant docstrings following the syntax described in the
  Writing docstrings section of the CONTRIBUTING docs.
• If this PR fixes a bug, I've added a test that will fail without my fix.
• If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

• All GitHub Actions jobs for my pull request have passed.

[epwalsh]

run_experiment is just for testing. We probably want to run our sweep experiments in subprocesses so that we can execute them in parallel. You could do something like this:

Suggested change

	main_config = self.override_hyperparameters(combination)
	# TODO: need to figure where & how to store results / way to track runs
	with run_experiment(main_config, include_package=[self.components]) as run_dir:
	# TODO: fill in something here?
	pass
	subprocess.call(["tango", "run", self.main_config_path, "--overrides", json.dumps(overrides)])

Of course this doesn't make anything run in parallel, but would be a good first step.

[epwalsh]

The Sweeper will also need to be aware of the target_step that spits out metric we're trying to optimize for, and the workspace.

Suggested change

	def __init__(self, main_config_path: str, sweeps_config_path: str, components: str):
	def __init__(self, main_config_path: str, sweeps_config_path: str, components: str, target_step: str, workspace: Workspace):

[epwalsh]

You may want to start by implementing #142 in a separate PR

[epwalsh]

Suggested change

	class SweepConfig(Params):
	class SweepConfig(FromParams):

[epwalsh]

See

tango/tango/settings.py

Line 13 in 54c4a8d

class TangoGlobalSettings(FromParams):

for example

[epwalsh]

Can you remove this from the PR? You can tell git to ignore this file by adding it to your global git ignore

[epwalsh]

Suggested change

	main_config_path = (
	"/Users/sabhyac/Desktop/sabhya/tango/test_fixtures/sweeps/basic_test/config.jsonnet"
	)
	sweeps_config_path = (
	"/Users/sabhyac/Desktop/sabhya/tango/test_fixtures/sweeps/basic_test/sweeps-config.jsonnet"
	)
	components = (
	"/Users/sabhyac/Desktop/sabhya/tango/test_fixtures/sweeps/basic_test/basic_arithmetic.py"
	)

[epwalsh]

The Sweeper class needs to be aware of the Workspace being used in order to get the results of each target step, and to pass the workspace URL when it runs each experiment (with the -w option).

[epwalsh]

For compat with older Python versions:

Suggested change

	def get_combinations(self) -> list:
	def get_combinations(self) -> List:

[epwalsh]

It would probably be better if this method was a generator function so that implementations could dynamically pick the next combination based on the history so far.

[epwalsh]

Docstrings should be written like this:

Suggested change

	# function to override all the hyperparameters in the current experiment_config
	def override_hyperparameters(self, experiment_tuple: tuple) -> dict:
	def override_hyperparameters(self, experiment_tuple: tuple) -> dict:
	"""
	Generates a overrides dictionary for the current experiment config.
	"""

[epwalsh]

In recent versions of Python you can just do this:

Suggested change

	super(Registrable, self).__init__()
	super().__init__()