[Feature request]: consolidate "jobs" and "slots"/"chains" for simulation / projection

[pearsonca]

Label

meta/workflow

Priority Label

low priority

Is your feature request related to a problem? Please describe.

User interface item. Currently, simulation / projection takes both a jobs (parallelization) and slots/chains argument (which appears to be used in a ... samples? parallelization? other?) sense.

Is your feature request related to a new application, scenario round, pathogen? Please describe.

No response

Describe the solution you'd like

Eliminate (or otherwise clarify) the point of this argument.

[TimothyWillard]

GH-394 is still a work in progress, but I believe the gempyor.batch.JobSize class in that PR addresses, or at least provides a start for, this issue. See:

flepiMoP/flepimop/gempyor_pkg/src/gempyor/batch.py

Lines 102 to 153 in 1316ebe

	@dataclass(frozen=True, slots=True)
	class JobSize:
	"""
	A batch submission job size.
	Attributes:
	jobs: The number of jobs to use.
	simulations: The number of simulations to run per a block.
	blocks: The number of sequential blocks to run per a job.
	Raises:
	ValueError: If any of the attributes are less than 1.
	"""
	jobs: int
	simulations: int
	blocks: int
	def __post_init__(self) -> None:
	for p in self.__slots__:
	if (val := getattr(self, p)) < 1:
	raise ValueError(
	(
	f"The '{p}' attribute must be greater than 0, "
	f"but instead was given '{val}'."
	)
	)
	@classmethod
	def size_from_jobs_sims_blocks(
	cls,
	jobs: int | None,
	simulations: int | None,
	blocks: int | None,
	inference_method: Literal["emcee"] | None,
	) -> "JobSize":
	"""
	Infer a job size from several explicit and implicit parameters.
	Args:
	jobs: An explicit number of jobs.
	simulations: An explicit number of simulations per a block.
	blocks: An explicit number of blocks per a job.
	inference_method: The inference method being used as different methods have
	different restrictions.
	Returns:
	A job size instance with either the explicit or inferred job sizing.
	"""
	if inference_method == "emcee":
	return cls(jobs=jobs, simulations=blocks * simulations, blocks=1)
	return cls(jobs=jobs, simulations=simulations, blocks=blocks)

[pearsonca]

GH-394 is still a work in progress, but I believe the gempyor.batch.JobSize class in that PR addresses, or at least provides a start for, this issue. See:

flepiMoP/flepimop/gempyor_pkg/src/gempyor/batch.py

Lines 102 to 153 in 1316ebe

	@dataclass(frozen=True, slots=True)
	class JobSize:
	"""
	A batch submission job size.
	Attributes:
	jobs: The number of jobs to use.
	simulations: The number of simulations to run per a block.
	blocks: The number of sequential blocks to run per a job.
	Raises:
	ValueError: If any of the attributes are less than 1.
	"""
	jobs: int
	simulations: int
	blocks: int
	def __post_init__(self) -> None:
	for p in self.__slots__:
	if (val := getattr(self, p)) < 1:
	raise ValueError(
	(
	f"The '{p}' attribute must be greater than 0, "
	f"but instead was given '{val}'."
	)
	)
	@classmethod
	def size_from_jobs_sims_blocks(
	cls,
	jobs: int | None,
	simulations: int | None,
	blocks: int | None,
	inference_method: Literal["emcee"] | None,
	) -> "JobSize":
	"""
	Infer a job size from several explicit and implicit parameters.
	Args:
	jobs: An explicit number of jobs.
	simulations: An explicit number of simulations per a block.
	blocks: An explicit number of blocks per a job.
	inference_method: The inference method being used as different methods have
	different restrictions.
	Returns:
	A job size instance with either the explicit or inferred job sizing.
	"""
	if inference_method == "emcee":
	return cls(jobs=jobs, simulations=blocks * simulations, blocks=1)
	return cls(jobs=jobs, simulations=simulations, blocks=blocks)

Seems likely - there will also need to be some adaptation at the dispatch stage, as these arguments get passed several layers deep from the simulate method.

Some of the problem here seems to be muddying together two considerations which are theoretically orthogonal (how much parallelization, how many independent inference processes) but are practically pretty much always the same, and then propagating that confusion to places where it doesn't actually apply (projection shouldn't know about inference chains; probably needs to know about parallelization and maybe samples?).

[jcblemai]

which are theoretically orthogonal (how much parallelization, how many independent inference processes) but are practically pretty much always the same, a

Just wanted to add to tgat that with some methods such as emcee, there is often need for more chains that parallel jobs depending on the number of parameters and the size of compute boxes.