| title | description | category | subcategory | tags | ||
|---|---|---|---|---|---|---|
Transfer files inside cluster |
This code helps with transfer files inside cluster |
computing |
tips_tricks |
|
Useful for file transfers on O2's new transfer cluster (transfer.rc.hms.harvard.edu).
The nohup command can be prepended to the bash command and the command will keep running after you logout (or have your connection interrupted).
From HMS RC:
From one of the file transfer systems under transfer.rc.hms.harvard.edu , you can prefix your command with "nohup" to put it in the background and be able to log out without interrupting the process.
For example, after logging in to e.g. the transfer01 host, run your command:
nohup rsync -av /dir1 /dir2
and then log out. rsync will keep running.
To check in on the process later, just remember which machine you ran rsync and you can directly re-login to that system if you like.
For example:
1. ssh transfer.rc.hms.harvard.edu (let's say you land on transfer03), and then:
2. ssh transfer01
-- from there you can run the "ps" command or however you like to monitor the process.
If you run tmux from the login node before you ssh to the transfer node to xfer files, you can drop your connection and then re-attach to your tmux session later. It should still be running your transfer.
General steps
- Login to O2
- write down what login node your are on (usually something like login0#)
at login node - Start a new tmux session
tmux new -s myname - SSH to the transfer node
ssh [email protected]
on transfer node - start transfer with rsync, scp etc.
- close terminal window without logging out
time passes - Login to O2 again
- ssh to the login node you wrote down above
ssh user@login0# - Reattach to your tmux session
tmux a -t myname - Profit
You can get around having to remember which node you logged into by alwasys logging into the same node. For example you can add this to your .bash_profile on OSX:
alias ssho2='ssh -XY -l user login05.o2.rc.hms.harvard.edu'