Skip to content

Latest commit

 

History

History
138 lines (131 loc) · 6.46 KB

using-for-uploads.md

File metadata and controls

138 lines (131 loc) · 6.46 KB
Raw
layout
default

Options for uploading large files

If a user needs to get data to VisIt developers for any reason, the options available depend on the size of the files. In all of the scenarios below, compression can help mitgate size limits. best to use an archiver tool such as tar, zip, etc.

  • Email to an individual developer
    • Size Limit: 5-10 MB
    • The actual size limit depends on several factors which are likely outside the knowledge of sender and recipeint.
    • Our cyber-filtering will remove file attachments with certain extensions. If this is an issue, try adding a funky file extension like .gorfo (e.g. name your attachment foo.tar.gz.gorfo instead of foo.tar.gz).
  • Attachments to Discussions
    • Only certain file extensions are supported.
    • Size Limits:
      • Image files (.gif., .png, .jpeg and .jpg) < 10MB.
      • Raw text files (.txt, and .log) < 25MB.
      • Document files (.pdf, .docx, .pptx, and .xlsx) < 25MB.
      • Compressed files (.gz, .zip) < 25MB.
    • Because GitHub currently keys off only the file's extension, you can attach any format by simply spoofing the extension. For example, to attach foo.obj just rename it to foo.obj.not.gz. Or, better yet, just compress foo.obj with gzip to produce foo.obj.gz.
  • LLNL Anonymous FTP server, ftp://ftp.llnl.gov
    • Size limit: probably < 1GB
    • Works only to get data to VisIt developers with LLNL accounts.
    • It is not possible to use a web browser to upload data to ftp://ftp.llnl.gov. Only command-line FTP tools that do not attempt to show directory listings will work.
    • Uploaders need to upload their files to the incoming directory on the server.
      • Note: VisIt developers may also download data to users via the outgoing directory.
    • Whether uploading or downloading data via ftp.llnl.gov, everyone needs to know the full URLs of the associated files in order to access them.
    • Using the standard ftp command-line tool... 
      % ftp ftp.llnl.gov
% Connected to ftp.llnl.gov.
220-        **WARNING**WARNING**WARNING**WARNING**WARNING**
220-        
220- This is a Department of Energy (DOE) computer system. DOE
.
.
.
Name (ftp.llnl.gov:<someuser>): anonymous
331 Please specify email address as password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
Unknown command.
ftp> bin # Switch data mode to binary when sending binary (e.g. not ascii) files
200 Switching to Binary mode.
ftp> cd incoming
ftp> put foo.tar.gz someuser-upload-1.tar.gz # be sure to create a likely unique file name on the server
local: foo.tar.gz remote: someuser-upload-1.tar.gz
500 Unknown command.
ALLO64 83 - command not accepted - Reverting to ALLO command
200 The filesize has been allocated.
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.
443 bytes sent in 0.0075 seconds (0.011 MBytes/sec)
ftp> ctrl-D
ftp> 221 Goodbye.
    • Using curl 
      curl -T foo.tar.gz --ftp-create-dirs ftp://ftp.llnl.gov/incoming/someuser/upload-1.tar.gz
      Note: the command above also demonstrates the creation of a new directory in the ftp server's incoming directory using curl. We currently know of no way to use wget to in the same way.
    • Email VisIt developers with the full URL name(s) of the files you uploaded there so they know to go get them. For example... 
      ftp://ftp.llnl.gov/incoming/someuser/upload-1.tar.gz
    • Files uploaded there are readable only from LLNL firewalled networks and are purged regularly.
  • Submit a PR from a fork of this repository
    • Size limit: < 2GB
    • Works for any VisIt developer
    • See below for detailed instructions.

Sending data via a Pull Request

This involves using a combination of a web browser on GitHub and a shell command-line.

  1. First, fork the repository on GitHub

    • Open a browser to https://github.com/visit-dav/largedata
    • Find and press the Fork button in the upper right
    • GitHub will ask you where you want to create the fork.
    • Create the fork in your personal space on GitHub.
    • Lets assume your GitHub name is fizbo.
    • Once you create the fork, clone it

  2. Create a local clone

    • Its best to use ssh to create the clone.
    git clone [email protected]:fizbo/largedata.git

    git clone https://github.com/fizbo/largedata.git

  3. Add, commit and push file(s) to the bindata directory

    cd largedata/bindata
git branch -b add-data-from-fizbo
git add foo_data.tar.gz
git commit -a -m 'adding my data file for upload'
git push
    • Note: depending on the size of your files, it may take a long time for the git push operation to complete.
    • Even after completing the steps above, the files are on GitHub but not accessible by anyone other than you, fizbo. To make them accessible to the VisIt team, you need to create a pull request.

  4. Create a pull request

  5. After whatever issues associated the data are resolved, VisIt team members will close the pull request.