Data_Transfer_Utah_CHPC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Tools for Data Transfer**\n",
    "In case of large data transfer, using the ***Data Transfer Nodes*** is more properly, because it has higher bandwith and was designed for that sort of transfer. So it won't bottleneck anyone else's traffic on the Utah University network.\n",
    "\n",
    "It is important to note, as well, that you can use ***mapping for a file system*** to access your files directly instead of copying them to and from your computer. Map the file system if you want to work with one copy of your file instead of having different versions.\n",
    "\n",
    "In this case, we are going to treat the casse where you want to have two separate files: \n",
    "\n",
    "* One in your local machine\n",
    "* One on CHPC resources.\n",
    "\n",
    "## **scp**\n",
    "Relatively simple command-line utility, perfect for small transfers and things you don't really need to rename any files or preserve any symbolic links or anymore complicated task that you would normally do with a ***shell script***, for instance. It works with `ssh`.\n",
    "\n",
    " **From your local computer to Utah CHPC File System**\n",
    "```bash\n",
    "scp [file_name].[extension] [uNID]@[cluster_name].chpc.utah.edu:~/[specified_path]/[file_(re)name].[extension]\n",
    "```\n",
    "Once you have run this command, you will be required enter your password. Could appear some errors related to the sell itself, but you can see that the command have actually run succesfully.\n",
    "\n",
    "The time it takes to copy it will depend entirely on the file size and your internet speed, but it is pretty fast because scp compresses the files when it sends them.\n",
    "\n",
    "**From CHPC File System to my local computer:**\n",
    "```bash\n",
    "scp [uNID]@[cluster_name].chpc.utah.edu:~/[specified_path]/[file_name].[extension] [file_(re)name].[extension]\n",
    "```\n",
    "Then you have to enter your campus password.\n",
    "\n",
    "### **If you want to copy multiple files**\n",
    "Utah CHPC Staff suggest to use just a directory with the files we want to copy and do it recursively. Consider we have, at somewhere in our local machine, a directory that contains multiple files. If we want to copy all of these files, we can use `scp -r`.\n",
    "\n",
    "**From local machine to CHPC system:**\n",
    "\n",
    "```bash\n",
    "scp -r [directory_name] [uNID]@[cluster_name].chpc.utah.edu:~/[specified_path]/[new_directory_name]\n",
    "```\n",
    "\n",
    "**From CHPC system to local machine:**\n",
    "```bash\n",
    "scp -r [uNID]@[cluster_name].chpc.utah.edu:~/[specified_path]/[directory_name_inside_cluster] [new_directory_name_local_machine]\n",
    "```\n",
    "\n",
    "## **rsync**\n",
    "It is another command-line utility and is a bit more robust. It has a few more options like tools to allow the interruption of transfers, so if something happens with your internet connection or the remote, that you're trying to push files to, goes down; you can actually just pick up where you left off and not have to transfer all your files. \n",
    "\n",
    "`rsync` is better to larger transfers. And also has tools to let you rename multiple files and preserves your directory structures and simbolic links. `rsync` is suggested if you have a more complicated setup that you need to copy over.\n",
    "\n",
    "You can check `rsync --help` for more information about. In particular, `rsync -P` is interesting when there is internet connections issues, it will allow your transaction to be interrupted and picked up a later time.\n",
    "\n",
    "**From local machine to CHPC system:**\n",
    "```bash\n",
    "rsync -hacvzP [directory_name_local_machine] [uNID]@[cluster_name].chpc.utah.edu:~/[specified_path]/[new_directory_name_cluster]\n",
    "```\n",
    "\n",
    "## **Cyberduck**\n",
    "Graphical utility for both macOS and Windows and has drag-and-drop interface. It has multiple protocols available for transfers, including some that are based on `ssh`, so you don't really have to find your location on the filesystems.\n",
    "\n",
    "There are a number of protocols we can use, but the easiest probably going be **STFP** or any other SSH-based file transfer protocol. Set the **port at 22**.\n",
    "\n",
    "At the **server** input, you need to put (e.g, if you want to use `kingspeak`):\n",
    "\n",
    "```bash\n",
    "kingspeak.chpc.utah.edu\n",
    "```\n",
    "\n",
    "Furthermore, you must to put your `[uNID]` as your **username** and your campus password as your **password**.\n",
    "\n",
    "Then, in **Cyberduck**, you can directly upload a local machine file to your HPC account through the **Upload** option. In **FileZilla** you can only drag-and-drop for do that.\n",
    "\n",
    "# **WinSCP**\n",
    "\n",
    "Supports a number of file protocols, but the easies are probably SFTP or SCP because you can connect to a computer like `kingspeak.chpc.utah.edu` instead of actually connecting to the file. Only log in a new session in WinSCP. In WinSCP you can see both your local computer and the remote computer. WinSCP has also a **Upload** botton at the local side of the interface, and a **Download** option at the remote side of the interface."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}