Several methods are available to transfer files into and from TRACE.
| Table of Contents | ||||
|---|---|---|---|---|
|
...
Paths for TRACE file space Home directory
See https://cmu-enterprise.atlassian.net/l/cp/jFDV0K6E for more information
Commands to transfer files
You can use rsync, scp, sftp to copy files to and from TRACE.
rsync
You can use the rsync command to copy files to and from TRACE. A sample rsync command to copy to a TRACE directory is
| Code Block |
|---|
rsync -rltpDvp -e 'ssh -l username' source_directory data.trace.cmu.edu:target_directory |
The rsync man page contains full command usage details.
Substitute your username for 'username'. Make sure you use the correct groupname in your target directory. By default, rsync will not copy older files with the same name in place of newer files in the target directory. It will overwrite older files.
We recommend the following rsync options -rltDvp. See the rsync man page for information on these options and other options you might want to use.
| Code Block |
|---|
-rltDvp |
We also recommend the option
| Code Block |
|---|
-oMACS=umac-64@openssh.com |
If you use this option, your transfer will use a faster data validation algorithm.
You may want to put your rsync command in a loop to insure that it completes. A sample loop is
| Code Block |
|---|
RC=1 |
...
n=0 |
...
while [[ $RC -ne 0 && $n -lt 20 ]] do |
...
rsync source-file target-file |
...
RC = $? |
...
let n = n + 1 |
...
sleep 10 |
...
done |
This loop will try your rsync command 20 times. If it succeeds it will exit. If an rsync invocation is unsuccessful the system will try again and pick up where it left off. It will copy only those files that have not already been transferred. You can put this loop, with your rsync command, into a batch script and run it with sbatch.
scp
To use scp for a file transfer you must specify a source and destination for your transfer. The format for either source or destination is
username@machine
| Code Block |
|---|
username@machine-name:path/filename |
For transfers involving TRACE-HPC, username is your TRACE username. The machine-name should be given as data.trace.cmu.edu. This is the name for a high-speed data connector at TRACEthe TRACE data transfer node. We recommend using it for all file transfers using scp involving TRACE. Using it prevents file transfers from disrupting interactive use on TRACE's login nodes.
File transfers using scp must specify full paths for TRACE file systems. See Paths for TRACE file spaces for details https://cmu-enterprise.atlassian.net/l/cp/jFDV0K6E for more information.
sftp
To use sftp, first connect to the remote machine:sftp username@machine
| Code Block |
|---|
sftp username@machine-name |
When TRACE When TRACE is the remote machine, use your TRACE userid as username. The TRACE machine-name should be specified as data.trace.cmu.edu. This is the name for a high-speed data connector at TRACE. We recommend using it for all file transfers using sftp involving TRACE. Using it prevents file transfers from disrupting interactive use on TRACE's login nodes.
You will be prompted for your password on the remote machine. If TRACE is the remote machine, enter your TRACE password.
You can then enter sftp subcommands, like put to copy a file from the local system to the remote system, or get to copy a file from the remote system to the local system.
To copy files into TRACE, you must either cd to the proper directory or use full pathnames in your file transfer commands. See Paths for TRACE file spaces for detailsSee https://cmu-enterprise.atlassian.net/l/cp/jFDV0K6E for more information.
Graphical SSH client
If you are using a graphical SSH client, configure it to connect to data.trace.cmu.edu the TRACE data transfer node on port 22/TCP. Login using your TRACE Portal username and password.