- Data servers move bytes. The metadata server records create, open,
and close.
- A million small files stall the copy. Pack them first.
- Home is for code, scratch is for the heavy copy, project is for
long-lived group data.
- The node disk is only for the inside of a job.
-
--sort=name keeps the tar bytes stable from one run to
the next.
-
mksquashfs run run.sqfs -comp zstd packs the directory
into one mountable file.
- On 20,000 small files, tar then
cp is fastest because
it moves one file.
-
sha256sum -c prints src.tar: OK when the
copied archive matches.
-
-a keeps links, times, and permissions.
--partial keeps a broken file for the next run.
-
--info=progress2 is one progress line.
--checksum reads every byte.
- A trailing slash on the source copies the contents.
-
timeout 2 with --bwlimit=2000 leaves a
short part/big.bin. The next rsync finishes
it.
-
cmp prints nothing and exits 0 when the bytes
match.
-
xargs -P 4 and fpsync -n 4 run several
rsync processes.
- Several streams help large files. Pack a million tiny files
first.
-
dwalk counts, dcp -p copies,
dsync matches the destination to the source,
drm --dryrun lists a delete, and dtar -c
writes a pax archive.
-
dcp.slurm asks for the rome partition, 2
nodes, and 8 tasks per node.
- EESSI 2023.06 and 2025.06 do not provide mpiFileUtils. The course
cluster has it installed, and that install is not an EESSI module.
-
scp starts again from the first file.
rsync -a --partial continues.
- Globus uses endpoints you registered (
SRC,
DEST). iRODS iCommands on Snellius are the 2023 modules
iRODS-iCommands/4.3.0.
-
/archive/$USER is on login and staging nodes.
daget stages tape. Compute nodes do not mount
/archive.
-
archive.surf.nl is the Data Archive login host.
snellius.surf.nl is the SSH host.
- Research Drive is an rclone WebDAV remote named
RD in
the SURF example.
-
$TMPDIR is the node disk. The job removes it on the way
out.
- The copy back is a line in the same script, before the script
exits.
-
stage.slurm requests rome,
--constraint=scratch-node, one task, and ten minutes.
- Pack, checksum, scratch,
rsync -a --partial, large-file
streams, stage out, and daget before a compute node. That
is the checklist.
