Data transfer

Transfer protocols

A number of methods allow transferring data in/out of Panthera. For most cases, we recommend using SSH-based file transfer commands, such as scp, sftp, or rsync. They will provide the best performance for data transfers from and to campus.

SCP (Secure Copy)

The easiest command to use to transfer files to/from Panthera is scp. It works like the cp command, except it can work over the network to copy files from one computer to another, using the secure SSH protocol.

The general syntax to copy a file to a remote server is:

$ scp <source_file_path> <username>@<remote_host>:<destination_path>

For instance, the following command will copy the file named foo from your local machine to your home directory on Panthera:

$ scp foo u111111@login.hpc.iut.ac.ir:

Note the : character, that separates the hostname from the destination path. Here, the destination path is empty, which will instruct scp to copy the file in your home directory.

You can copy foo under a different name, or to another directory, with the following commands:

$ scp foo u111111@login.hpc.iut.ac.ir:bar
$ scp foo u111111@login.hpc.iut.ac.ir:~/subdir/baz

To copy back files from Panthera to your local machine, you just need to reverse the order of the arguments:

$ scp u111111@login.hpc.iut.ac.ir:foo local_foo

And finally, scp also support recursive copying of directories, with the -r option:

$ scp -r dir/ u111111@login.hpc.iut.ac.ir:dir/

This will copy the dir/ directory and all of its contents in your home directory on Panthera.

SFTP (Secure File Transfer Protocol)

SFTP clients are interactive file transfer programs, similar to FTP, which perform all operations over an encrypted transport.

A variety of graphical SFTP clients are available for different OSes:

• WinSCP
• SecureFX
• CyberDuck

When setting up your connection to Panthera in the above programs, use the following information:

Hostname: login.hpc.iut.ac.ir
Port:     22
Username: your_cluster_username
Password: your_password

OpenSSH also provides a command-line SFTP client, originally named sftp.

To log in to Panthera:

$ sftp u111111@login.hpc.iut.ac.ir
Connected to login.hoc.iut.ac.ir.
sftp>

For more information about using the command-line SFTP client, you can refer to this tutorial for more details and examples.

rsync

If you have complex hierarchies of files to transfer, or if you need to synchronize a set of files and directories between your local machine and Panthera, rsync will be the best tool for the job. It will efficiently transfer and synchronize files across systems, by checking the timestamp and size of files. Which means that it won't re-transfer files that have not changed since the last transfer, and will complete faster.

For instance, to transfer the whole ~/data/ folder tree from your local machine to your home directory on Panthera, you can use the following command:

$ rsync -a ~/data/ u111111@login.hpc.iut.ac.ir:data/

Note the slash (/) at the end of the directories name, which is important to instruct rsync to synchronize the whole directories.

To get more information about the transfer rate and follow its progress, you can use additional options:

$ rsync -avP ~/data/ u111111@login.hpc.iut.ac.ir:data/
sending incremental file list
./
file1
      1,755,049 100%    2.01MB/s    0:00:00 (xfr#2, to-chk=226/240)
file2
      2,543,699 100%    2.48MB/s    0:00:00 (xfr#3, to-chk=225/240)
file3
     34,930,688  19%   72.62MB/s    0:00:08

[...]

For more information about using the rsync, you can refer to this tutorial for more details and examples.

SSHFS (comming soon)

Sometimes, moving files in and out of the cluster, and maintaining two copies of each of the files you work on, both on your local machine and on Panthera, may be painful. Fortunately, Panthera offers the ability to mount any of its filesystems to your local machine, using a secure and encrypted connection.

With SSHFS, a FUSE-based filesystem implementation used to mount remote SSH-accessible filesystems, you can access your files on Panthera as if they were locally stored on your own computer.

This comes particularly handy when you need to access those files from an application that is not available on Panthera, but that you already use or can install on your local machine. Like a data processing program that you have licensed for your own computer but can't use on Panthera, a specific text editor that only runs on macOS, or any data-intensive 3D rendering software that wouldn't work comfortably enough over a forwarded X11 connection.

SSHFS is available for Linux , macOS and Windows .

SSHFS on macOS

SSHFS on macOS is known to try to automatically reconnect filesystem mounts after resuming from sleep or suspend, even without any valid credentials. As a result, it will generate a lot of failed connection attempts and likely make your IP address blacklisted on login nodes.

Make sure to unmount your SSHFS drives before putting your macOS system to sleep to avoid this situation.

The following option could also be useful to avoid some permission issues: -o defer_permissions

For instance, on a Linux machine with SSHFS installed, you could mount your Panthera home directory via a Data Transfer Node with the following commands:

$ mkdir ~/Panthera_home     #create this directory on your local system.
$ sshfs u111111@login.hpc.iut.ac.ir:./ ~/Panthera_home

And to unmount it:

$ umount ~/Panthera_home

For more information about using SSHFS on your local machine, you can refer to this tutorial for more details and examples.

Cloud storage services

Currently we do not provide any connection to external cloud services.