Feature/slurm no shared filesystem

[PerilousApricot]

These are the changes needed to support configurations where the CEs and worker nodes don't share a filesystem. There's two parts:

1. Changes to blah to move the files back and forth via curl, instead of assuming there are shared filesystems
2. A small python REST server to send/receive the files

Unfortunately, SLURM doesn't have a native functionality to move files from the submission node to/from the execution node (even though it's a spiritual successor to PBS, which does), so there's two choices, each with their own downsides:

1. Provide a new service
2. Use rsh/ssh

I went with 1 because it seemed less difficult than trying to sort out the authentication hassle for 2.

There's still work that needs to be done to generalize it (see the commit messages), but it's what we've been using at Vanderbilt for a few months now.

[PerilousApricot]

This is an output submission script generated by this code:

[root@ce4 ~]# cat /tmp/bl_0c860312570c.debug/submit.script 
#!/bin/bash
# SLURM job wrapper generated by slurm2_submit.sh
# on Tue Jul 25 17:25:30 CDT 2017
#
#
# SLURM directives:
#   #  SBATCH -o /home/meloam/debug_slurm_log/slurm-ce4-%j.out
#SBATCH -o /dev/null
# Begin  /etc/blahp/slurm_local_submit_attributes.sh
#SBATCH --priority 900
#SBATCH --mem=5000
#SBATCH --time=48:00:00
export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# End    /etc/blahp/slurm_local_submit_attributes.sh
# Begin file staging
env
set -x
cd $HOME
curl --retry 5 -s -o glidein_startup.sh.29607.643511.1501021530 http://ce4:8080/var/lib/condor-ce/spool/7813/0/cluster427813.proc0.subproc0/glidein_startup.sh
curl --retry 5 -s -o bl_0c860312570c.proxy http://ce4:8080/var/lib/condor-ce/spool/7813/0/cluster427813.proc0.subproc0/credential_CMSG-v1_0.main_411868
chmod go-rwx glidein_startup.sh.29607.643511.1501021530
chmod go-rwx bl_0c860312570c.proxy
function blah_stageout_trap() {
set -x
curl --retry 5 -s -F 'data=@out_bl_0c860312570c__condor_stdout' http://ce4:8080/var/lib/condor-ce/spool/7813/0/cluster427813.proc0.subproc0/_condor_stdout
curl --retry 5 -s -F 'data=@err_bl_0c860312570c__condor_stderr' http://ce4:8080/var/lib/condor-ce/spool/7813/0/cluster427813.proc0.subproc0/_condor_stderr
rm -f out_bl_0c860312570c__condor_stdout
rm -f err_bl_0c860312570c__condor_stderr
    sleep 5
}
# End file staging

# Setting the environment:
export "HOME=/home/cmsglidein"
export "CONDORCE_COLLECTOR_HOST=ce4.accre.vanderbilt.edu:9619"
export "OSG_GRID=/etc/osg/wn-client/"
export "OSG_APP=/cvmfs/cms.cern.ch"
export "OSG_GLEXEC_LOCATION=/usr/sbin/glexec"
export "OSG_DATA=UNAVAILABLE"
export "OSG_HOSTNAME=ce4.accre.vanderbilt.edu"
export "OSG_STORAGE_ELEMENT=False"
export "OSG_SITE_NAME=Vanderbilt_CE4"
export "GLOBUS_LOCATION=/usr"
export "OSG_WN_TMP=/tmp"
export ""
GLOBUS_LOCATION="/usr"
OSG_APP="/cvmfs/cms.cern.ch"
OSG_DATA="UNAVAILABLE"
OSG_GLEXEC_LOCATION="/usr/sbin/glexec"
OSG_GRID="/etc/osg/wn-client/"
OSG_HOSTNAME="ce4.accre.vanderbilt.edu"
OSG_SITE_NAME="Vanderbilt_CE4"
OSG_STORAGE_ELEMENT="False"
OSG_WN_TMP="/tmp"
export GLOBUS_LOCATION
export OSG_APP
export OSG_DATA
export OSG_GLEXEC_LOCATION
export OSG_GRID
export OSG_HOSTNAME
export OSG_SITE_NAME
export OSG_STORAGE_ELEMENT
export OSG_WN_TMP

old_home=`pwd`
new_home=/tmp//home_bl_ce4.accre.vanderbilt.edu_9619_ce4.accre.vanderbilt.edu#427814.0#1501021520
mkdir $new_home
trap 'wait $job_pid; cd $old_home; rm -rf $new_home; exit 255' 1 2 3 15 24
trap 'wait $job_pid; cd $old_home; rm -rf $new_home' 0
# Copy into new home any shared input sandbox file
# Move into new home any relative input sandbox file
mv "glidein_startup.sh.29607.643511.1501021530" "$new_home/glidein_startup.sh" &> /dev/null
mv "bl_0c860312570c.proxy" "$new_home/bl_0c860312570c.proxy" &> /dev/null
export HOME=$new_home
cd $new_home
# Resetting proxy to local position
export X509_USER_PROXY=$new_home/bl_0c860312570c.proxy

# Command to execute:
if [ ! -x ./glidein_startup.sh ]; then chmod u+x ./glidein_startup.sh; fi
if [ -x ${GLITE_LOCATION:-/opt/glite}/libexec/jobwrapper ]
then
${GLITE_LOCATION:-/opt/glite}/libexec/jobwrapper ./glidein_startup.sh ##SNIP ARGS## < "/dev/null" > "out_bl_0c860312570c__condor_stdout" 2> "err_bl_0c860312570c__condor_stderr" &
elif [ -x /opt/lcg/libexec/jobwrapper ]
then
/opt/lcg/libexec/jobwrapper ./glidein_startup.sh ##SNIP ARGS## < "/dev/null" > "out_bl_0c860312570c__condor_stdout" 2> "err_bl_0c860312570c__condor_stderr" &
elif [ -x $BLAH_AUX_JOBWRAPPER ]
then
$BLAH_AUX_JOBWRAPPER ./glidein_startup.sh ##SNIP ARGS## < "/dev/null" > "out_bl_0c860312570c__condor_stdout" 2> "err_bl_0c860312570c__condor_stderr" &
else
$new_home/glidein_startup.sh ##SNIP ARGS## < "/dev/null" > "out_bl_0c860312570c__condor_stdout" 2> "err_bl_0c860312570c__condor_stderr" &
fi
job_pid=$!

# Wait for the user job to finish
wait $job_pid
user_retcode=$?

# Move all relative outputsand paths out of temp home
cd $new_home
mv "out_bl_0c860312570c__condor_stdout" "$old_home/out_bl_0c860312570c__condor_stdout" 2> /dev/null
mv "err_bl_0c860312570c__condor_stderr" "$old_home/err_bl_0c860312570c__condor_stderr" 2> /dev/null
if declare -f blah_stageout_trap &>/dev/null; then
  pushd $old_home
  blah_stageout_trap
  popd
fi
# Move any remapped outputsand file to shared directories
# Remove the staged files, if any
rm "glidein_startup.sh" 2> /dev/null
rm "bl_0c860312570c.proxy" 2> /dev/null
cd $old_home
rm -f /var/lib/condor-ce/spool/7813/0/cluster427813.proc0.subproc0/credential_CMSG-v1_0.main_411868

exit $user_retcode

[brianhlin]

Ah, I thought this was somehow going to be using built-in slurm file transfer. This is a little scarier so I'll review when I have more time.

[PerilousApricot]

Yeah, unfortunately SLURM doesn't have its own inbuilt file transfer mechanism :/