make the batch submisson cluster agnostic

[jcblemai]

There are hardcoded paths and slurm loaded modules, these need to be provided as separate scripts -- that the user can override from its home directory, such that we can run on any slurm cluster

[TimothyWillard]

@shauntruelove I saw you assigned this issue to me and it's labeled as high priority, do we have a desired outcome and deadline for this since this is now high priority? I'm not super familiar with the details of this issue, @jcblemai or @shauntruelove would you be able to fill those in?

[shauntruelove]

Notes:

• need agnostic handling of secrets
• remove hard-coded specifications with Rockfish
• make work for Longleaf and others
• add workflow section of the config to tell where everything goes

[TimothyWillard]

@pearsonca @jcblemai I've attached the branch I'm doing the documentation updates to to this issue.

[jcblemai]

[twillard@longleaf-login6 ~]$ sg rc_jlessler_pi -c "touch foobar.txt"
[twillard@longleaf-login6 ~]$ ls -lah foobar.txt
-rw-r--r-- 1 twillard rc_jlessler_pi 0 Sep 13 11:25 foobar.txt

We need to ensure that everyone can access the run, so commands should be optionally run for a configurable certain group of user.
su requires sudo, sg does not (edited)

[TimothyWillard]

Update as of EOD 2024-09-16:

• It is now possible to cleanly and consistently install flepiMoP on Longleaf using the build/hpc_install.sh script on the GH-191/longleaf-batch-submission branch. Yet to add rockfish, pending access to be able to work through adding that as a supported cluster.
• Corresponding documentation changes have been started on the GH-191/longleaf-batch-submission-documentation branch
• Added support for providing a slurm --partition to the batch/inference_job_launcher.py script, see bba583a.
• Still yet to successfully submit a batch run on Longleaf. Managed to successfully submit a batch run on Longleaf, yet to verify it worked correctly.