Summary and Schedule

This lesson moves data on a parallel file system. You pack small files, checksum the archive, resume a copy with rsync, and stage a tree onto the node disk for the length of a job.

One episode, on several streams and mpiFileUtils, is optional.

Prerequisite

• You can open a terminal and run tar, rsync, and sha256sum
• You have a directory for a heavy copy. On a cluster, use scratch
• For the stage script, you can submit a Slurm job

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Why copying hurts	Why does a directory of small files copy slowly? Which directory is for the heavy copy, and which one exists only inside a job?
Duration: 00h 10m	2. Pack, then checksum	Why is one archive faster to copy than the directory it came from? What does sha256sum -c print when the archive still matches?
Duration: 00h 35m	3. rsync	Which flags let a stopped copy continue? What does cmp print when the two files match?
Duration: 01h 00m	4. Several streams	When do four copy streams help? What does each mpiFileUtils program do, and where does the deck’s job script run it?
Duration: 01h 25m	5. From outside the cluster	What does scp do when the transfer stops, and what does rsync do? Where is /archive/$USER mounted, and what has to happen before a compute node can read that tree?
Duration: 01h 40m	6. Stage in, stage out	Why does the copy back have to be inside the job script? Which seven checks does the deck list before you call a copy finished?
Duration: 02h 10m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

You need a terminal with bash, tar, rsync, sha256sum, split, dd, timeout, and cmp.

Run the 20,000-file exercise from an empty directory on scratch. Home is the wrong place for that copy.

The stage exercise needs Slurm. The script in episodes/files/stage.slurm is the published ten-minute job. Submit it from scratch.

mksquashfs is the squashfs command in the pack episode. If it is missing, the tar exercise still stands.

fpsync comes from the fpart package. The xargs form in the several-streams episode does the same kind of copy.

mpiFileUtils is optional for the exercises that use rsync. EESSI 2023.06 and 2025.06 do not provide it. It is installed on the course cluster, and that install is not an EESSI module. The episode tab has the commands for the machine in your welcome note.

The commands that leave the cluster use snellius.surf.nl as the SSH host and archive.surf.nl as the Data Archive login host.