Summary and Schedule

This lesson runs thousands of short tasks. A Slurm array is the first tool. QCG-PilotJob then runs those tasks inside one allocation. The last episode compares HyperQueue, Snakemake, and FireWorks on the same exercises. That episode is optional.

The hands-on tree is episodes/files/hands-on/. Examples 01 to 05 are the QCG module. alternatives/ is the comparison, including measurements/.

Prerequisite

• You can submit a Slurm job
• You can read a short Python script
• You are in this lesson’s episodes/files/hands-on tree when you run a script

	Setup Instructions	Download files required for the lesson
Duration: 00h 00m	1. Embarrassingly parallel	What is an embarrassingly parallel workload on a dense node? Why does one Slurm job per short command waste the scheduler?
Duration: 00h 15m	2. Slurm arrays	What is the same across every task in a Slurm array, and what can change? How do you cap how many tasks run at once, and how does each task find its index?
Duration: 00h 55m	3. QCG-PilotJob	What does QCG-PilotJob keep, while it runs many short tasks? Which lines in qcg_job.py and qcg_job.sh submit 512 hostname tasks?
Duration: 01h 40m	4. Averages with a dependency	Which version first makes the aggregation wait? Where in the job description do you read that?
Duration: 02h 15m	5. Idle time	What does the bash pipeline wait for at the end of each block? What does QCG-PilotJob do when a task ends and a core is free?
Duration: 02h 45m	6. Other tools	Where does each tool say that the aggregation waits for every average? What is measurements/sacct.txt for, and what does it leave out?
Duration: 03h 30m	Finish

The actual schedule may vary slightly depending on the topics and exercises chosen by the instructor.

Use the hands-on tree in this lesson:

episodes/files/hands-on/

Run each example from its own directory. The alternative scripts call ../../example_04 and ../../example_05, so keep that layout.

You need Slurm and the Python the job scripts load. The scripts name their own modules. The episode tabs are the load lines for the machine in your welcome note.

Examples 2 and 4 need input.tar.gz in that example’s directory. Both archives are in the tree.

On a two-core node the hands-on README uses:

BASH

sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
TIMES_FILE=times_short.txt sbatch --ntasks=2 pipeline.sh
TIMES_FILE=times_short.txt sbatch --ntasks=2 qcg_job.sh

The profile plot in example 5 uses the modules named in that example’s README:

BASH

module load 2025
module load SciPy-bundle/2025.06-gfbf-2025a
module load matplotlib/3.10.3-gfbf-2025a

The last episode is optional. HyperQueue, Snakemake, and FireWorks each have a directory under alternatives/.