Code

EMCEE Batch Submit CLI Tool#394

Code

Draft

TimothyWillard wants to merge 78 commits intodevHopkinsIDD/flepiMoP:devfrom GH-365/emcee-submitHopkinsIDD/flepiMoP:GH-365/emcee-submitCopy head branch name to clipboard

+3,889-8

Commits

Commits on Nov 8, 2024

• Jinja2 templating infrastructure

  Show description for ce065d4
  TimothyWillard

  committed
  ce065d4

  Copy full SHA for ce065d4

• Logging infrastructure for gempyor

  Show description for 8ecc3e0
  TimothyWillard

  committed
  8ecc3e0

  Copy full SHA for 8ecc3e0

• Add private shutil.which wrapper

  Show description for 1afe3c8
  TimothyWillard

  committed
  1afe3c8

  Copy full SHA for 1afe3c8

• Only propagate logger in pytest

  Show description for cd3ca2d
  TimothyWillard

  committed
  cd3ca2d

  Copy full SHA for cd3ca2d

• Add _sbatch to _slurm

  Show description for e6c86ef
  TimothyWillard

  committed
  e6c86ef

  Copy full SHA for e6c86ef

• Add gempyor.utils._git_head

  Show description for 51ae598
  TimothyWillard

  committed
  51ae598

  Copy full SHA for 51ae598

• Add write_manifest for batch manifest files

  Show description for d642739
  TimothyWillard

  committed
  d642739

  Copy full SHA for d642739

• Move manifest module into batch

  Show description for 75304cd
  TimothyWillard

  committed
  75304cd

  Copy full SHA for 75304cd

• Moved _slurm module into batch

  Show description for 3eb1cb2
  TimothyWillard

  committed
  3eb1cb2

  Copy full SHA for 3eb1cb2

• Initial JobSize class

  Show description for 06dbf0d
  TimothyWillard

  committed
  06dbf0d

  Copy full SHA for 06dbf0d

• Add JobSize.size_from_jobs_sims_blocks

  Show description for 10faf77
  TimothyWillard

  committed
  10faf77

  Copy full SHA for 10faf77

• Reformat new files with line length 92

  Show description for 498197c
  TimothyWillard

  committed
  498197c

  Copy full SHA for 498197c

• Add _format_cli_options helper

  Show description for d9b03a7
  TimothyWillard

  committed
  d9b03a7

  Copy full SHA for d9b03a7

• Draft _sbatch_template function

  Show description for 23b315e
  TimothyWillard

  committed
  23b315e

  Copy full SHA for 23b315e

• Initial pass at generic cluster config

  Show description for ff2d1f5
  TimothyWillard

  committed
  ff2d1f5

  Copy full SHA for ff2d1f5

• Initial EMCEE inference sbatch script

  Show description for 05c119b
  TimothyWillard

  committed
  05c119b

  Copy full SHA for 05c119b

• Add path modifications to cluster info

  Show description for d98eb80
  TimothyWillard

  committed
  d98eb80

  Copy full SHA for d98eb80

• Skeleton of sbatch script

  Show description for abd92e6
  TimothyWillard

  committed
  abd92e6

  Copy full SHA for abd92e6

• Apply black formatter

  Show description for e3e373d
  TimothyWillard

  committed
  e3e373d

  Copy full SHA for e3e373d

• Add log_cli_inputs utility

  Show description for dc67254
  TimothyWillard

  committed
  dc67254

  Copy full SHA for dc67254

• Add gempyor.batch._job_name

  Show description for 3089358
  TimothyWillard

  committed
  3089358

  Copy full SHA for 3089358

• Add _resolve_batch_system utility

  Show description for d6773ff
  TimothyWillard

  committed
  d6773ff

  Copy full SHA for d6773ff

• Initial work on flepimop batch CLI

  Show description for 42ceda3
  TimothyWillard

  committed
  42ceda3

  Copy full SHA for 42ceda3

• Minor additions to flepimop batch

  Show description for ed4ddc7
  TimothyWillard

  committed
  ed4ddc7

  Copy full SHA for ed4ddc7

• Add JobTimeLimit class

  Show description for 382a616
  TimothyWillard

  committed
  382a616

  Copy full SHA for 382a616

• Add from_per_simulation_time method

  Show description for 28965ff
  TimothyWillard

  committed
  28965ff

  Copy full SHA for 28965ff

• Begin constructing sbatch call

  Show description for 772e872
  TimothyWillard

  committed
  772e872

  Copy full SHA for 772e872

• Add custom duration click param type

  Show description for a8ab8db
  TimothyWillard

  committed
  a8ab8db

  Copy full SHA for a8ab8db

• Fix bug in time defaults, add in cluster info

  Show description for 8db1e4a
  TimothyWillard

  committed
  8db1e4a

  Copy full SHA for 8db1e4a

• Write config for batch, work on batch script

  Show description for 5a95683
  TimothyWillard

  committed
  5a95683

  Copy full SHA for 5a95683

• Implement options for flepimop-calibrate

  Show description for 3fc27b9
  TimothyWillard

  committed
  3fc27b9

  Copy full SHA for 3fc27b9

• Add --email to flepimop batch

  Show description for c40aed9
  TimothyWillard

  committed
  c40aed9

  Copy full SHA for c40aed9

• Correct param_decls arg to click.Option

  Show description for 458ac00
  TimothyWillard

  committed
  458ac00

  Copy full SHA for 458ac00

• New lines in emcee_inference template

  Show description for b4c028d
  TimothyWillard

  committed
  b4c028d

  Copy full SHA for b4c028d

• Add slurm+cluster option value error unit test

  Show description for 1e7e1e4
  TimothyWillard

  committed
  1e7e1e4

  Copy full SHA for 1e7e1e4

• Add BatchSystem enum

  Show description for a18af5f
  TimothyWillard

  committed
  a18af5f

  Copy full SHA for a18af5f

• Greatly simplify JobSize for first draft

  Show description for b0d1852
  TimothyWillard

  committed
  b0d1852

  Copy full SHA for b0d1852

• Add JobResources class

  Show description for 770d6f6
  TimothyWillard

  committed
  770d6f6

  Copy full SHA for 770d6f6

• Further unit tests of JobResources

  Show description for e2f3e13
  TimothyWillard

  committed
  e2f3e13

  Copy full SHA for e2f3e13

• Remove unneeded warnings import

  TimothyWillard

  committed
  c710522

  Copy full SHA for c710522

• Rename flepimop batch to submit

  Show description for fcc1dbc
  TimothyWillard

  committed
  fcc1dbc

  Copy full SHA for fcc1dbc

• Dump config to actual rather than temp file

  Show description for a893b00
  TimothyWillard

  committed
  a893b00

  Copy full SHA for a893b00

• Remove --nthin from flepimop submit

  TimothyWillard

  committed
  d5eab61

  Copy full SHA for d5eab61

• Optional overrides to job resource presets

  TimothyWillard

  committed
  1316ebe

  Copy full SHA for 1316ebe

Commits on Nov 11, 2024

• Initial pass at incorporating R inference

  Show description for d4a86a9
  TimothyWillard

  committed
  d4a86a9

  Copy full SHA for d4a86a9

• Clean whitespace in _sbatch logging

  TimothyWillard

  committed
  82dfe8f

  Copy full SHA for 82dfe8f

• Add _local internal utility

  Show description for 3d5484b
  TimothyWillard

  committed
  3d5484b

  Copy full SHA for 3d5484b

• Splitting out templates

  Show description for a2fe5df
  TimothyWillard

  committed
  a2fe5df

  Copy full SHA for a2fe5df

• Initial template testing

  Show description for 641bc70
  TimothyWillard

  committed
  641bc70

  Copy full SHA for 641bc70

Commits on Nov 12, 2024

• Add exact results for conda activate template

  Show description for 4eecc0b
  TimothyWillard

  committed
  4eecc0b

  Copy full SHA for 4eecc0b

• Add _local_template utility

  Show description for a842342
  TimothyWillard

  committed
  a842342

  Copy full SHA for a842342

• Incorporate batch system into JobSize

  Show description for 398c352
  TimothyWillard

  committed
  398c352

  Copy full SHA for 398c352

• Add conda init shim

  Show description for 75448f7
  TimothyWillard

  committed
  75448f7

  Copy full SHA for 75448f7

• Split env vars out into its own template

  TimothyWillard

  committed
  e3e4adc

  Copy full SHA for e3e4adc

• Now supports --local and non-EMCEE inference

  Show description for e9bcb23
  TimothyWillard

  committed
  e9bcb23

  Copy full SHA for e9bcb23

• Fix bug in --skip-manifest flag

  Show description for 08ec61e
  TimothyWillard

  committed
  08ec61e

  Copy full SHA for 08ec61e

• Add git branch checkout to preserve run.

  Show description for b6c95ca
  TimothyWillard

  committed
  b6c95ca

  Copy full SHA for b6c95ca

• Remove custom interpreter for sbatch

  Show description for 3fb4dbe
  TimothyWillard

  committed
  3fb4dbe

  Copy full SHA for 3fb4dbe

Commits on Nov 14, 2024

• Legacy inference use slurm array jobs

  TimothyWillard

  committed
  93ab660

  Copy full SHA for 93ab660

• Override nodes for legacy inference on slurm

  TimothyWillard

  committed
  0490895

  Copy full SHA for 0490895

• Infer cluster from FQDN

  Show description for 7e7efd5
  TimothyWillard

  committed
  7e7efd5

  Copy full SHA for 7e7efd5

• Add --samples option to flepimop submit

  TimothyWillard

  committed
  821210d

  Copy full SHA for 821210d

Commits on Nov 25, 2024

• Add detailed documentation for flepimop submit

  TimothyWillard

  committed
  9353b66

  Copy full SHA for 9353b66

• Add custom click param type for memory

  Show description for a56a4a8
  TimothyWillard

  committed
  a56a4a8

  Copy full SHA for a56a4a8

• Replace memory option IntRange with MEMORY_MB

  TimothyWillard

  committed
  b3961f6

  Copy full SHA for b3961f6

• Document info/shared_cli, MemoryParamType int

  Show description for 0fa1f9b
  TimothyWillard

  committed
  0fa1f9b

  Copy full SHA for 0fa1f9b

• Assume unitless numbers are in default unit

  Show description for 6e38dd2
  TimothyWillard

  committed
  6e38dd2

  Copy full SHA for 6e38dd2

• Move get_cluster_info call into _click_submit

  Show description for 4f936db
  TimothyWillard

  committed
  4f936db

  Copy full SHA for 4f936db

• Helpful logging if using --local on a cluster

  Show description for 8d7b3d8
  TimothyWillard

  committed
  8d7b3d8

  Copy full SHA for 8d7b3d8

Commits on Nov 26, 2024

• Add args documentation for _submit_scenario_job

  TimothyWillard

  committed
  5027499

  Copy full SHA for 5027499

• Remove temporary comments

  TimothyWillard

  committed
  d549728

  Copy full SHA for d549728

• Add FLEPI_PATH to GH action, fix Jinja2 loader

  Show description for b76bb3b
  TimothyWillard

  committed
  b76bb3b

  Copy full SHA for b76bb3b

• Add Self shim for python 3.10

  TimothyWillard

  committed
  2f46391

  Copy full SHA for 2f46391

• Escape \ in flepimop submit --help man page

  Show description for c14552e
  TimothyWillard

  committed
  c14552e

  Copy full SHA for c14552e

• Provide only pkg name to Jinja2 PackageLoader

  TimothyWillard

  committed
  a64ca7e

  Copy full SHA for a64ca7e

• Prefer FileSystemLoader over PackageLoader

  TimothyWillard

  committed
  7c03736

  Copy full SHA for 7c03736

• Set include-package-data to true

  TimothyWillard

  committed
  0149bc7

  Copy full SHA for 0149bc7

Commits on Dec 2, 2024

• Include templates with include-data directive

  TimothyWillard

  committed
  ad11fae

  Copy full SHA for ad11fae