Content from Why copying hurts

Last updated on 2026-10-28 | Edit this page

Estimated time: 10 minutes

Overview

Questions

  • Why does a directory of small files copy slowly?
  • Which directory is for the heavy copy, and which one exists only inside a job?

Objectives

  • Separate the work of the data servers from the work of the metadata server.
  • Put code in home, the heavy copy on scratch, long-lived group data on project, and node-disk files only inside a job.

A million small files

A parallel file system splits the work. Data servers move bytes. A metadata server records each create, open, and close. Copying a million small files asks that server for a million operations, and the data servers wait. Pack the files first.

Sam’s case is the same shape at a smaller count. Thousands of output files, copied one by one, stall on metadata. The next episode packs 20,000 of them and times the difference.

Where data lives

Keep code and notes in home. Do the heavy copy on scratch. Keep the group’s long-lived data on the project file system. Use the node disk only inside a job. It is the fastest, and it disappears when the job ends.

The paths differ by machine.

Place Path What it is for
Home /home/$USER Shared with the compute nodes
Project /project Shared
Scratch /scratch Shared with the manager, not a separate parallel scratch
Node disk $TMPDIR This job only. Removed when the job ends

There is no /scratch-shared, no /scratch-node, and no /archive.

Key Points
  • Data servers move bytes. The metadata server records create, open, and close.
  • A million small files stall the copy. Pack them first.
  • Home is for code, scratch is for the heavy copy, project is for long-lived group data.
  • The node disk is only for the inside of a job.

Content from Pack, then checksum

Last updated on 2026-10-28 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • Why is one archive faster to copy than the directory it came from?
  • What does sha256sum -c print when the archive still matches?

Objectives

  • Build a tar archive with --sort=name and a squashfs image with mksquashfs -comp zstd.
  • Time cp -r, tar then cp, and rsync -a on 20,000 small files.
  • Check the archive after the copy and read src.tar: OK.

One archive, then a checksum

Turn a directory of small files into one archive, then record a checksum. Check it again after the copy. A squashfs image is one file you can mount read-only.

GNU tar, sha256sum, and mksquashfs are the tools. --sort=name makes the tar bytes stable from one run to the next.

BASH 

tar --sort=name -cf run.tar run/
sha256sum run.tar > run.tar.sha256
sha256sum -c run.tar.sha256
mksquashfs run run.sqfs -comp zstd

sha256sum -c reads the .sha256 file and prints the archive name and OK when the bytes match. The hands-on below uses src.tar, and that check prints src.tar: OK.

Write the order first

Use bash. time prints real, user, and sys. Write down real.

The deck’s sequence is episodes/files/predict.sh. Run it from an empty directory on scratch, so the 20,000 files stay off home.

BASH 

mkdir -p src out
yes data | head -n 20000 | split -l 1 -a 5 -d - src/f
ls src | wc -l
{ time cp -r src out/by-cp; }
tar --sort=name -cf src.tar src
{ time cp src.tar out/; }
{ time rsync -a src/ out/by-rsync/; }
sha256sum src.tar > src.tar.sha256
cp src.tar.sha256 out/
( cd out && sha256sum -c src.tar.sha256 )

yes data | head -n 20000 | split -l 1 -a 5 -d - src/f writes 20,000 files. Each file is one line. The suffixes from -a 5 -d run from f00000 through f19999. ls src | wc -l prints 20000.

cp -r creates 20,000 files at the destination. rsync -a does the same. tar --sort=name -cf src.tar src builds one archive, and cp src.tar moves that one file. The metadata server does one create for the tar copy, and 20,000 creates for each of the other two.

Challenge

Predict the order

Before you run the script, write the order of the three real times. Which copy is fastest, and why?

Tar then cp is fastest because it moves one file. cp -r and rsync -a each create 20,000 files.

A reference run on a local ext4 disk, 26 October 2026, matched that order. Record your own seconds. The deck does not list the three real times from that run. After the copy, sha256sum -c prints src.tar: OK.

Challenge

Time it and check the archive

Change to an empty directory on scratch, then run bash on episodes/files/predict.sh from this lesson. Write down the three real times, and write down the sha256sum -c line.

The file count line is 20000. The real time for cp src.tar is the smallest of the three, because that copy is one file. cp -r and rsync -a each create 20,000 files, so they take longer. The check line is:

src.tar: OK

Your seconds will differ from the ext4 run on 26 October 2026. The order is the result to compare.

Key Points
  • --sort=name keeps the tar bytes stable from one run to the next.
  • mksquashfs run run.sqfs -comp zstd packs the directory into one mountable file.
  • On 20,000 small files, tar then cp is fastest because it moves one file.
  • sha256sum -c prints src.tar: OK when the copied archive matches.

Content from rsync

Last updated on 2026-10-28 | Edit this page

Estimated time: 25 minutes

Overview

Questions

  • Which flags let a stopped copy continue?
  • What does cmp print when the two files match?

Objectives

  • Copy with rsync -a --partial and read one progress line from --info=progress2.
  • Use a trailing slash on the source so the contents arrive, with no extra directory level.
  • Stop one rsync with timeout, run it again, and check the bytes with cmp.

One careful copy

-a keeps links, times, and permissions. --partial leaves a broken file in place so the next run continues. --info=progress2 is one progress line for the whole transfer. --checksum reads every byte when size and time are not enough. A trailing slash on the source copies the contents, and it does not add an extra directory level.

Resume by running the same command again:

BASH 

rsync -a --partial \
  --info=progress2 \
  run/ /scratch-shared/$USER/run/

When size and time are not enough, add --checksum:

BASH 

rsync -a --checksum --partial \
  run/ /scratch-shared/$USER/run/

run/ with the slash fills /scratch-shared/$USER/run/ with the contents of run. The same source without the slash copies the directory as well, so the files sit at run/run/.

Catch it midway

Predict a short part/big.bin after timeout, then equal sizes. --bwlimit slows the first copy so two seconds is enough to stop it. --partial keeps the broken file, and the next rsync continues.

cmp exits 0 when the bytes match, and it prints nothing.

dd with bs=1M count=48 writes 48 MiB, which is 50331648 bytes. That is the size of big.bin. The partial file is shorter. On the ext4 example from 26 October 2026 it was 4128768 bytes of 50331648. Your byte count after two seconds can differ. The check is that the first ls shows a short file, the second rsync finishes the copy, and cmp is silent.

BASH 

dd if=/dev/zero of=big.bin bs=1M count=48
mkdir -p part
timeout 2 rsync -a --partial \
  --bwlimit=2000 big.bin part/
ls -l big.bin part/big.bin
rsync -a --partial big.bin part/
cmp big.bin part/big.bin

timeout stops the first rsync. That line is the interrupted copy. Continue with ls.

Challenge

Predict the two sizes

You run the block above. After timeout, how does part/big.bin compare with big.bin? After the second rsync, what does cmp print, and what is its exit status?

After timeout, part/big.bin is shorter than big.bin. big.bin is 50331648 bytes. On the ext4 run of 26 October 2026 the partial file was 4128768 bytes. After the second rsync the sizes match. cmp big.bin part/big.bin prints nothing and exits 0.

Challenge

Resume it

In an empty directory, run the six commands from this episode. Record the two sizes from the first ls -l, then run cmp and record its exit status with echo $?.

The first size column for part/big.bin is smaller than 50331648. The ext4 reference is 4128768 bytes. The second rsync makes the sizes equal. cmp prints nothing. echo $? prints 0.

Key Points
  • -a keeps links, times, and permissions. --partial keeps a broken file for the next run.
  • --info=progress2 is one progress line. --checksum reads every byte.
  • A trailing slash on the source copies the contents.
  • timeout 2 with --bwlimit=2000 leaves a short part/big.bin. The next rsync finishes it.
  • cmp prints nothing and exits 0 when the bytes match.

Content from Several streams

Last updated on 2026-10-28 | Edit this page

Estimated time: 25 minutes

This episode is optional.

Overview

Questions

  • When do four copy streams help?
  • What does each mpiFileUtils program do, and where does the deck’s job script run it?

Objectives

  • Run four rsync processes with xargs -P 4 or fpsync -n 4 on large shards.
  • Name dwalk, dcp, dsync, drm, and dtar, and match each one to the job it does.
  • Read dcp.slurm: partition rome, 2 nodes, 8 tasks per node.

Four streams, for large files

Four streams help when each shard is a large file. A million tiny files still need a pack first. xargs -P and fpsync (from the fpart package) both run several rsync processes.

--exclude skips build products. The destination tree must already exist for the xargs form, because each call writes one shard.

BASH 

printf '%s\0' shard-*/ |
  xargs -0 -P 4 -I{} \
  rsync -a --partial {}/ \
  /scratch-shared/$USER/copy/{}/

fpsync is a separate command from the exclude example. The exclude line is an ordinary rsync.

BASH 

fpsync -n 4 -o '-a --partial' \
  /scratch-shared/$USER/run/ \
  /scratch-shared/$USER/copy/
rsync -a --partial --exclude '*.o' \
  --exclude '.git/' run/ dest/

mpiFileUtils inside a Slurm job

These are MPI programs from the mpiFileUtils suite. srun starts one of them across the tasks in the job. The options below are from the project manual.

  • dwalk -v counts files and bytes.
  • dcp -p copies and keeps permissions, group, and timestamps.
  • dsync makes the destination match the source.
  • drm --dryrun lists what a recursive delete would remove.
  • dtar -c writes a pax archive.

BASH 

srun dwalk -v "$SRC"
srun dcp -p "$SRC" "$DST"
srun dsync -v "$SRC" "$DST"
srun drm --dryrun "$OLD"
srun dtar -c -f run.tar run/

The manual is https://mpifileutils.readthedocs.io/.

EESSI 2023.06 and 2025.06 do not provide mpiFileUtils. Snellius loads a site module or builds it with EasyBuild. The course cluster has mpiFileUtils installed, and that install is not an EESSI module. The tab has the commands for the machine in your welcome note.

mpiFileUtils is installed on this SURF Research Cloud workspace. It is not an EESSI module. EESSI 2023.06 and 2025.06 do not provide it.

BASH 

sbatch dcp.slurm

BASH 

#!/bin/bash
#SBATCH --job-name=dcp
#SBATCH --partition=rome
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --output=dcp-%j.out
set -eu
SRC=/scratch/$USER/run
DST=/scratch/$USER/run-copy
srun dwalk -v "$SRC"
srun dcp -p "$SRC" "$DST"

dwalk counts files. dcp -p keeps permissions, group, and timestamps. dsync makes the destination match. drm --dryrun lists a delete. dtar -c writes a pax archive.

Two Rome nodes

The published script walks the source and copies it. The slide text also says the job walks the copy. The script block on the slide, which is episodes/files/dcp.slurm, ends at dcp.

Snellius uses the rome partition for these CPU nodes. Submit with sbatch dcp.slurm from a directory on scratch, so the heavy tree stays off home.

BASH 

#!/bin/bash
#SBATCH --job-name=dcp
#SBATCH --partition=rome
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --output=dcp-%j.out
set -eu
SRC=/scratch-shared/$USER/run
DST=/scratch-shared/$USER/run-copy
srun dwalk -v "$SRC"
srun dcp -p "$SRC" "$DST"

That is 2 nodes and 8 tasks per node, and a one-hour limit. The partition line in the file is the published Snellius line. The tab is the request for the machine in your welcome note. The scripts stay as published.

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition.

BASH 

#SBATCH --partition=rome

Shrink --ntasks and --cpus-per-task on the sbatch line when sinfo shows small nodes. The scripts stay as published.

Challenge

Large files, or a million small ones

You have four disks’ worth of bandwidth and a directory of a million tiny files. What do you do before you open four rsync streams?

Pack the tiny files first. Four streams help when each shard is a large file. They leave the metadata server with a million creates if you point them at a million tiny files.

Challenge

Submit the published copy

episodes/files/dcp.slurm is the job. Which partition, node count, and tasks per node does the file request? Then read the tabs before you change a line for the machine in your welcome note.

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition.

BASH 

#SBATCH --partition=rome

Shrink --ntasks and --cpus-per-task on the sbatch line when sinfo shows small nodes. The scripts stay as published.

mpiFileUtils is installed on this SURF Research Cloud workspace. It is not an EESSI module. EESSI 2023.06 and 2025.06 do not provide it.

BASH 

sbatch dcp.slurm

BASH 

#!/bin/bash
#SBATCH --job-name=dcp
#SBATCH --partition=rome
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --time=01:00:00
#SBATCH --output=dcp-%j.out
set -eu
SRC=/scratch/$USER/run
DST=/scratch/$USER/run-copy
srun dwalk -v "$SRC"
srun dcp -p "$SRC" "$DST"

dwalk counts files. dcp -p keeps permissions, group, and timestamps. dsync makes the destination match. drm --dryrun lists a delete. dtar -c writes a pax archive.

The file requests partition rome, --nodes=2, and --ntasks-per-node=8. Submit it from scratch with sbatch dcp.slurm.

EESSI 2023.06 and 2025.06 do not provide mpiFileUtils. On the course cluster it is installed, and it is not an EESSI module. The partition tab is the request for this machine. Leave the script’s partition line as published.

Key Points
  • xargs -P 4 and fpsync -n 4 run several rsync processes.
  • Several streams help large files. Pack a million tiny files first.
  • dwalk counts, dcp -p copies, dsync matches the destination to the source, drm --dryrun lists a delete, and dtar -c writes a pax archive.
  • dcp.slurm asks for the rome partition, 2 nodes, and 8 tasks per node.
  • EESSI 2023.06 and 2025.06 do not provide mpiFileUtils. The course cluster has it installed, and that install is not an EESSI module.

Content from From outside the cluster

Last updated on 2026-10-28 | Edit this page

Estimated time: 15 minutes

Overview

Questions

  • What does scp do when the transfer stops, and what does rsync do?
  • Where is /archive/$USER mounted, and what has to happen before a compute node can read that tree?

Objectives

  • Continue a partial copy with rsync, and recognise that scp starts over.
  • Name Globus, iRODS iCommands, daget, and an rclone WebDAV remote for Research Drive.
  • Keep the Data Archive on login and staging nodes, which are the nodes that mount /archive.

From your own computer

scp starts again from the first file if the transfer stops. rsync continues a partial file. Globus moves between endpoints you registered. On Snellius, the iRODS iCommands below are the documented 2023 modules. The Data Archive is /archive/$USER on login and staging nodes only. daget stages tape, and compute nodes do not mount /archive. Research Drive works as an rclone WebDAV remote. SURF documents the setup.

SRC and DEST are Globus endpoints you registered. RD is the rclone remote in the SURF Research Drive example.

BASH 

scp -r run/ USER@snellius.surf.nl:/scratch-shared/$USER/run
rsync -a --partial --info=progress2 run/ \
  USER@snellius.surf.nl:/scratch-shared/$USER/run/
globus transfer SRC:run/ DEST:run/
module load 2023
module load iRODS-iCommands/4.3.0
iput -r run
daget /archive/$USER/run
srun -t 1:00:00 -p staging cp -r /archive/$USER/run /scratch-shared/$USER/
rclone copy run/ RD:run/

snellius.surf.nl is the SSH host. archive.surf.nl is the Data Archive login host.

The srun line is one hour on the staging partition. It copies from /archive/$USER/run onto /scratch-shared/$USER/. Run daget on a login or staging node first. A compute node has no /archive mount, so a job there cannot open that path.

The SURF pages, by topic:

  • Connecting to the system, host snellius.surf.nl.
  • Data Archive from Snellius: daget, /archive, partition staging, login host archive.surf.nl.
  • How to use Rclone with Research Drive, remote RD.
  • Snellius filesystems: $TMPDIR and scratch-node, used in the next episode.
Challenge

The transfer stops halfway

You started scp -r run/ USER@snellius.surf.nl:/scratch-shared/$USER/run and the connection dropped. You still have a partial tree on scratch. Which command continues that partial copy, and which command starts again from the first file?

rsync -a --partial --info=progress2 continues a partial file. Run the same command again. scp starts again from the first file if the transfer stops.

Key Points
  • scp starts again from the first file. rsync -a --partial continues.
  • Globus uses endpoints you registered (SRC, DEST). iRODS iCommands on Snellius are the 2023 modules iRODS-iCommands/4.3.0.
  • /archive/$USER is on login and staging nodes. daget stages tape. Compute nodes do not mount /archive.
  • archive.surf.nl is the Data Archive login host. snellius.surf.nl is the SSH host.
  • Research Drive is an rclone WebDAV remote named RD in the SURF example.

Content from Stage in, stage out

Last updated on 2026-10-28 | Edit this page

Estimated time: 30 minutes

Overview

Questions

  • Why does the copy back have to be inside the job script?
  • Which seven checks does the deck list before you call a copy finished?

Objectives

  • Stage inputs into $TMPDIR and copy results back to scratch before the job ends.
  • Submit stage.slurm: partition rome, constraint scratch-node, ten minutes, one task.
  • Apply the deck’s seven-point checklist.

Around the job

Copy the inputs from scratch to $TMPDIR, which is the node disk. Write outputs there too. Copy the results back to scratch before the job ends: the node disk is removed for you. Then delete the scratch tree you no longer need.

BASH 

rsync -a --partial \
  /scratch-shared/$USER/run/ $TMPDIR/run/
rsync -a --partial \
  $TMPDIR/out/ /scratch-shared/$USER/out/

After the results are back:

BASH 

rm -rf /scratch-shared/$USER/run
srun drm --dryrun "$OLD"
srun drm "$OLD"

Use srun drm instead of rm -rf when the tree is large enough to stall one process. Run drm --dryrun first. $OLD is that tree.

Callout

The copy back is inside the script. Once the job ends, the node disk is gone, and a second job has nothing to copy.

The paths and the scratch constraint differ by machine. Read these before you edit a path into the script.

BASH 

#SBATCH --partition=rome
rsync -a --partial /project/$USER/src/ "$TMPDIR/src/"
rsync -a --partial "$TMPDIR/src/" /project/$USER/out/

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition. Do not add --constraint=scratch-node.

There is no --constraint=scratch-node. $TMPDIR is already on the node’s own disk. Drop the constraint and write to $TMPDIR.

A ten-minute job

Submit from scratch with sbatch stage.slurm. The file is episodes/files/stage.slurm. --constraint=scratch-node selects a node where $TMPDIR is the node disk. That disk is removed when the job ends, so the script copies the tree back to scratch before it exits.

BASH 

#!/bin/bash
#SBATCH --job-name=stage
#SBATCH --partition=rome
#SBATCH --constraint=scratch-node
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:10:00
#SBATCH --output=stage-%j.out
set -eu
rsync -a --partial \
  /scratch-shared/$USER/src/ "$TMPDIR/src/"
rsync -a --partial \
  "$TMPDIR/src/" /scratch-shared/$USER/out/

The published script requests partition rome, one node, and one task, for ten minutes. The second rsync is the copy back, and it is inside the script. It copies the staged src tree to /scratch-shared/$USER/out/.

The script already asks for one task. When a published script in the room asks for more cores than the node has, the override goes on the sbatch command line.

On a two-core node, pass the smaller request on the command line. Do not edit the script.

BASH 

sbatch --cpus-per-task=2 pi.sh
sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
iend=8 wdir=TMPDIR sbatch run.sh

Check the core count with sinfo first.

Checklist

  1. Pack small files with tar or mksquashfs before a wide copy.
  2. Checksum the archive before the move and again after it.
  3. Put the heavy copy on scratch, not in home.
  4. Use rsync -a --partial, and run the same command again to resume.
  5. Open several streams only for large files.
  6. Stage into $TMPDIR, copy results out, then delete the scratch copy.
  7. The Data Archive is not mounted on compute nodes. Run daget on a login or staging node first.

Point 6 is the one this script exists for. The copy out happens before the script exits.

Challenge

Where the copy back goes

A job writes result.txt into $TMPDIR and exits. A second script, submitted afterwards, copies $TMPDIR/result.txt to scratch. What is missing, and which checklist point is it?

Then write the two rsync lines for the machine in your welcome note. The tab has the paths. Keep the copy back in the same script.

BASH 

#SBATCH --partition=rome
rsync -a --partial /project/$USER/src/ "$TMPDIR/src/"
rsync -a --partial "$TMPDIR/src/" /project/$USER/out/

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition. Do not add --constraint=scratch-node.

There is no --constraint=scratch-node. $TMPDIR is already on the node’s own disk. Drop the constraint and write to $TMPDIR.

The node disk is removed when the job ends. The second script starts too late. Checklist point 6 says to stage into $TMPDIR, copy results out, then delete the scratch copy. Both rsync commands belong in the first script, in the order the tab shows: into $TMPDIR, then back to the shared path, before the script exits.

stage.slurm does that with the second rsync of "$TMPDIR/src/" onto /scratch-shared/$USER/out/. On a machine whose tab uses other paths, use those paths, and still keep the copy back inside the script. Add --constraint=scratch-node only when that tab says the node disk is selected that way.

Challenge

Seven checks

A collaborator copies 20,000 small files with four rsync streams into home, skips the checksum, and reads /archive/$USER from a compute node. Which checklist points did they miss?

Point 1: pack the small files before a wide copy. Point 3: the heavy copy belongs on scratch. Point 2: checksum before the move and again after it. Point 5: several streams are for large files. Point 7: compute nodes do not mount the Data Archive, so daget on a login or staging node comes first.

Key Points
  • $TMPDIR is the node disk. The job removes it on the way out.
  • The copy back is a line in the same script, before the script exits.
  • stage.slurm requests rome, --constraint=scratch-node, one task, and ten minutes.
  • Pack, checksum, scratch, rsync -a --partial, large-file streams, stage out, and daget before a compute node. That is the checklist.