Content from Embarrassingly parallel

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • What is an embarrassingly parallel workload on a dense node?
  • Why does one Slurm job per short command waste the scheduler?

Objectives

  • Describe a workload of many independent tasks that share a node badly if each task is its own job.
  • Name the three families on the deck: background processes with numactl, Slurm arrays, and a service inside one allocation.
  • Name QCG-PilotJob, HyperQueue, Snakemake, and FireWorks as the tools the last episode compares.

Many independent tasks

An embarrassingly parallel workload runs many independent small jobs at the same time, so the node stays busy.

You reach for that pattern in three situations.

  • The runs cannot be combined into a single application.
  • A large parameter sweep uses the same application, or a workflow whose pipeline steps depend on each other.
  • A step depends on software that stops scaling past some number of cores.

Nodes are getting denser: more cores and threads, more GPU cards, more memory. Applications do not necessarily scale even within a node. The deck names shared memory here: OpenMP, MPI shared memory, and pthreads.

One job per command

Sam’s thousand short commands, each with its own sbatch, give the scheduler a thousand jobs. Each job waits for an allocation, writes a job record, and starts a shell. The commands themselves are the small part.

The allocation should start the next task when a core is free. The scheduler then sees one job. Something inside that job hands work to a free core.

Many tasks inside one Slurm job. The nodes drawn are one allocation, one pilot, and many tasks.
Many tasks inside one Slurm job. The nodes drawn are one allocation, one pilot, and many tasks. The arrows read “one Slurm job” and “starts the next free task”.

Tools on the deck

The deck lists three ways to do this.

  • Linux background processes on one node, placed with numactl. Snellius supports that. It needs bash and care about where each process sits. Someone who knows the placement has full manual control.
  • Slurm arrays, for similar jobs. Snellius supports them. They have limits. The next episode teaches the limits and the control variables.
  • A service that runs many jobs inside one allocation. QCG-PilotJob is the service this course walks through. It needs Python. The documentation is https://qcg-pilotjob.readthedocs.io/en/develop/.

The last episode compares four tools on the same two exercises: QCG-PilotJob, HyperQueue, Snakemake, and FireWorks. That comparison has a measured table. This episode does not. The discussion is the cost of a thousand separate jobs, and the shape of an allocation that refills a free core.

Discussion

Where the time goes

Take a command that runs for about a second. You submit it with its own sbatch, a thousand times.

Where does the time go, beyond that one second? When the work sits inside one allocation, what should happen at the moment a core becomes free?

Key Points
  • Many independent tasks on a dense node are the embarrassingly parallel case.
  • A thousand one-task jobs spend their time in the scheduler.
  • The allocation should start the next task when a core is free.
  • Arrays come next. QCG-PilotJob is the pilot service. HyperQueue, Snakemake, and FireWorks are the other three tools in the last episode.

Content from Slurm arrays

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • What is the same across every task in a Slurm array, and what can change?
  • How do you cap how many tasks run at once, and how does each task find its index?

Objectives

  • List the deck’s limits: same options, Snellius max array size 30000, Snellius minimum 16 cores and 28 GiB on rome, and a wait for every task.
  • Use SLURM_ARRAY_TASK_ID, the % throttle, and %A / %a in output names.
  • Run example 01 and explain why each element has its own $TMPDIR.
  • Run example 02, aggregate to result.csv, and scale with --array=0-99%10 on 2 cores.

Collections of similar jobs

A Slurm job array submits and manages a collection of similar jobs. Every task has the same initial options: the same size, the same time limit, and the same script. The index is what changes.

The deck’s limits:

  • The maximum array size depends on the Slurm configuration. On Snellius it is 30000.
  • The allocation for each task cannot be smaller than the site’s minimum. On Snellius that minimum is 16 cores and 28 GiB of memory on the rome partition.
  • You wait for each task to be allocated, and you wait for each task to finish. A short task does not hand its cores to the next index. The next index is another Slurm task.

This is the shape the deck shows. sbatch --array=0-10 job.sh creates 11 tasks, indexes 0 through 10.

BASH 

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 2
#SBATCH -p rome
#SBATCH -t 1
hostname

-t 1 is one minute. -n 2 is two tasks. At one core per task that is two cores, under the 16-core minimum the deck states for Snellius on rome. The slide uses the small request to show the array syntax. The minimum still applies on Snellius.

The published scripts in the hands-on keep their own #SBATCH lines, including -p rome. The tab is the request for the machine in your welcome note.

Ways of control

Each task sees SLURM_ARRAY_TASK_ID set to its index.

A % in the array range caps how many tasks run at once.

BASH 

sbatch --array=0-10 job.sh
sbatch --array=0-10%2 job.sh

The first line creates 11 tasks. The second tells Slurm to run at most 2 of them at the same time. The command still returns one array job id.

The deck’s control script prints the index and the hostname:

BASH 

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 2
#SBATCH -p rome
#SBATCH -t 1
echo "SLURM_ARRAY_TASK_ID: $SLURM_ARRAY_TASK_ID"
hostname

Five variables are worth learning as a set.

Variable What it holds
SLURM_ARRAY_JOB_ID The first job ID of the array
SLURM_ARRAY_TASK_ID This task’s index
SLURM_ARRAY_TASK_COUNT How many tasks the array contains
SLURM_ARRAY_TASK_MAX The highest index
SLURM_ARRAY_TASK_MIN The lowest index

Output names can use two replacements. %A becomes SLURM_ARRAY_JOB_ID. %a becomes SLURM_ARRAY_TASK_ID.

BASH 

#SBATCH --output slurm-%A_%a.out
Challenge

Count and cap

sbatch --array=0-10 job.sh and sbatch --array=0-10%2 job.sh are the deck’s pair. How many tasks does the first create? How many run at once in the second?

--array=0-10 creates 11 tasks, because the indexes run from 0 through 10. --array=0-10%2 is the same 11 tasks, and Slurm runs at most 2 of them at the same time.

Example 1: hostname and TMPDIR

episodes/files/hands-on/example_01/array_job.sh submits 12 tasks, indexes 0 through 11, at most 3 at once. Each task is 1 node and 1 task, partition rome, ten minutes (-t 10).

BASH 

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p rome
#SBATCH -t 10
#SBATCH --array 0-11%3

echo "Running on "
hostname
echo "TMPDIR $TMPDIR"

echo "SLURM_ARRAY_JOB_ID $SLURM_ARRAY_JOB_ID"
echo "SLURM_ARRAY_TASK_ID $SLURM_ARRAY_TASK_ID"
echo "SLURM_ARRAY_TASK_COUNT $SLURM_ARRAY_TASK_COUNT"
echo "SLURM_ARRAY_TASK_MAX $SLURM_ARRAY_TASK_MAX"
echo "SLURM_ARRAY_TASK_MIN $SLURM_ARRAY_TASK_MIN"

The file in the tree has a non-breaking space in three of those echo strings. The variables are the same five from the table.

Each array element is its own job, so each element has its own node-local directory. The TMPDIR lines differ for that reason. Hostnames differ when two elements run on different nodes. They can repeat when two elements share a node. The directory still differs, because the path is per job.

$TMPDIR on your machine:

$TMPDIR in a job is /localscratch/$USER.$SLURM_JOB_ID.0 on the node’s own disk. The cluster deletes it when the job ends. There is one login node, so compare the login node’s /tmp with a job’s $TMPDIR.

Challenge

Why the directories differ

Example 1 prints hostname and TMPDIR twelve times. Which of those two values is a new path for every element, and why?

$TMPDIR in a job is /localscratch/$USER.$SLURM_JOB_ID.0 on the node’s own disk. The cluster deletes it when the job ends. There is one login node, so compare the login node’s /tmp with a job’s $TMPDIR.

TMPDIR is a new path for every element. Each array task is its own job, and the job’s node-local directory is removed when that task ends. The tab says where that directory lives on this machine.

Hostnames match when two tasks share a node, and they differ when Slurm places the tasks on different nodes. Twelve output files is the count of indexes 0 through 11. %3 caps the running set at 3. It does not change the count.

Example 2: one column, then result.csv

Example 2 averages one column of each CSV and gathers the averages into result.csv. The files are the NOAA Global Historical Climatology Network daily set: https://www.ncei.noaa.gov/products/land-based-station/global-historical-climatology-network-daily.

From episodes/files/hands-on/example_02:

BASH 

./prepare_input.sh
sbatch array_job.sh
./aggregate.sh <job_id>

prepare_input.sh extracts input.tar.gz and copies each file from input/ into input_mod/ with a numeric prefix, starting at 0. The array script selects input_mod/${SLURM_ARRAY_TASK_ID}_*.

The published header is 1000 tasks, indexes 0 through 999, at most 10 at once, one task each, partition rome, ten minutes:

BASH 

#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p rome
#SBATCH -t 10
#SBATCH --array 0-999%10

Inside the task the script loads module load 2025 and module load Python/3.13.1-GCCcore-14.2.0, copies input_mod and average.py into $TMPDIR, and runs python3 average.py on that one file. average.py prints one line, dataset_name,mean.

aggregate.sh takes the job id sbatch printed. It reads slurm-<job id>_*.out, keeps the average line, and sorts them into result.csv. The deck’s --output slurm-%A_%a.out is the same shape of name. The result of the exercise is result.csv.

The deck asks for the total execution time and the system billing units. sacct is the tool. Example 4’s README works one job in full (job 16235860, CPUTimeRAW 7232, 2.0 SBU). Use that conversion when you have a CPUTimeRAW column. On a machine that does not bill in system billing units, that column is still core-seconds, and core-hours are CPUTimeRAW / 3600.

On 2 cores the hands-on README runs 100 of the 1000 files:

BASH 

sbatch --array=0-99%10 array_job.sh

Leave the script’s --array 0-999%10 line as published. The smaller range is an override on the command line.

On a two-core node, pass the smaller request on the command line. Do not edit the script.

BASH 

sbatch --cpus-per-task=2 pi.sh
sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
iend=8 wdir=TMPDIR sbatch run.sh

Check the core count with sinfo first.

QCG is next

This episode stops at arrays. You still wait for each task to be allocated and to finish. The next episode is QCG-PilotJob, which keeps one allocation and starts the next task itself. The load line for that tool:

BASH 

module load 2024
module load QCG-PilotJob

module load 2024 selects the EESSI 2025.06 stack on this cluster and is the load line that comes before QCG-PilotJob.

Challenge

A hundred files on two cores

You are in example_02. prepare_input.sh has already built input_mod/. The node has two cores. What do you pass to sbatch, and what file do you have after aggregate.sh?

On a two-core node, pass the smaller request on the command line. Do not edit the script.

BASH 

sbatch --cpus-per-task=2 pi.sh
sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
iend=8 wdir=TMPDIR sbatch run.sh

Check the core count with sinfo first.

BASH 

sbatch --array=0-99%10 array_job.sh
./aggregate.sh <job_id>

The override selects indexes 0 through 99, at most 10 at once, which is 100 of the 1000 files. The script in the tree still says 0-999%10. After every selected task has finished, aggregate.sh writes result.csv.

Key Points
  • An array is a collection of similar jobs. Every task shares the initial options.
  • On Snellius the deck’s maximum array size is 30000, and the minimum allocation on rome is 16 cores and 28 GiB.
  • --array=0-10 is 11 tasks. --array=0-10%2 runs at most 2 at once.
  • SLURM_ARRAY_TASK_ID is the index. %A and %a fill in the job id and the index in a file name.
  • Example 1’s $TMPDIR values differ because each element has its own node-local directory.
  • Example 2’s aggregate writes result.csv. On 2 cores: sbatch --array=0-99%10 array_job.sh.

Content from QCG-PilotJob

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • What does QCG-PilotJob keep, while it runs many short tasks?
  • Which lines in qcg_job.py and qcg_job.sh submit 512 hostname tasks?

Objectives

  • Describe QCG-PilotJob as a manager inside one Slurm allocation.
  • Read the Slurm script, the Python script, the job-description fields, and the execution models.
  • Predict the two lines qcg-three.py prints.
  • Run example 3: 512 tasks, two full rome nodes, the default model, 256 tasks on each node.

A manager inside the allocation

QCG-PilotJob submits jobs inside one large allocation, including an allocation that spans nodes. It does not take a fresh small allocation for each task the way an array does. It places each job onto the cores that job asked for, so the allocation stays full. Failed or timed-out jobs can be restarted. You can also set dependencies from results.

The costs are concrete.

  • You estimate the total time of all the jobs combined, because the Slurm job has one time limit.
  • Job and service state files appear in the submission directory.
  • A Singularity container that embeds an MPI program gets harder to launch.
  • When something fails, the log files may say little.

The path is three files. The Slurm script describes the resources, loads QCG-PilotJob, and runs the Python script. The Python script builds a manager and the job list. The executable is the program those jobs run.

Documentation: https://qcg-pilotjob.readthedocs.io/en/develop/.

The Slurm script

Load QCG-PilotJob from the 2024 environment. The slide says the 2025 stack has no build of it, and it tells you to load the modules in the batch script. The same slide says Slurm exports the submission environment by default. The hands-on README says that on Snellius the login environment is not transferred to the compute node. The command in both places is the same: the module load lines go in the batch script.

The deck’s smallest script is one node, 16 cores, partition rome, ten minutes, then:

BASH 

module load 2024
module load QCG-PilotJob/0.14.1-gfbf-2024a
python test.py

Example 3’s episodes/files/hands-on/example_03/qcg_job.sh is the script you submit for the 512 tasks. These are the lines in the file:

BASH 

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=128
#SBATCH --time=00:05:00
#SBATCH --partition=rome

# QCG-PilotJob is installed in the 2024 stack only (toolchain gfbf-2024a)
module load 2024
module load QCG-PilotJob/0.14.1-gfbf-2024a

python3 qcg_job.py

Two nodes, 128 tasks per node, five minutes, partition rome. The comment in the file is the toolchain note. Leave those lines as published.

BASH 

module load 2024
module load QCG-PilotJob

module load 2024 selects the EESSI 2025.06 stack on this cluster and is the load line that comes before QCG-PilotJob.

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition.

BASH 

#SBATCH --partition=rome

Shrink --ntasks and --cpus-per-task on the sbatch line when sinfo shows small nodes. The scripts stay as published.

The nodes in this SURF Research Cloud workspace can be small. A two-core override is the usual request when sinfo shows two cores:

BASH 

#SBATCH --cpus-per-task=2

Check sinfo before you copy that number. CPU jobs still request --partition=rome.

The hands-on README’s example for a small node, after you have checked sinfo -o '%n %c', is:

BASH 

sbatch --nodes=3 --ntasks-per-node=2 qcg_job.sh

That override is for this example’s full-node request. The 16-core and 28 GiB minimum from the array episode is the deck’s Snellius floor on rome. Example 3 asks for a full node of 128 cores, which is above that floor. The cpu-minimum tab is the floor on the machine in your welcome note.

The Python script

The deck’s minimal script creates a manager, prints the resources, waits, and finishes:

PYTHON 

from qcg.pilotjob.api.manager import LocalManager
manager = LocalManager()
print('available resources: ', manager.resources())
manager.wait4all()
manager.finish()

The slide’s example output is for the 16-core script (#SBATCH -n 16):

available resources: {'total_nodes': 1, 'total_cores': 16, 'used_cores': 0, 'free_cores': 16}

That dict is the slide’s example for 16 cores. Example 3 does not print resources(), and its allocation is the two-node script above.

Working with the manager has three steps. Create it. Wait until the jobs you submitted have finished (wait4all, or wait4 on a list of ids). Call finish(), which stops the background thread the manager runs in.

To run several programs you create a Jobs object, add descriptions, and submit that object:

PYTHON 

from qcg.pilotjob.api.job import Jobs
jobs = Jobs()
jobs.add(...)
jobs_ids = manager.submit(jobs)

Fields and execution models

The deck’s main fields:

  • name, the job name
  • exec, the path to the executable
  • args, the arguments
  • script, bash to run when you are not pointing at an executable
  • stdin, stdout, stderr, the three files
  • modules, modules to load before that job starts
  • numCores, how many cores
  • numNodes, how many nodes
  • iteration, how the description repeats
  • model, how the processes start

Execution models, from the same deck:

  • default launches a single process in the allocation it was given
  • threads is for OpenMP on one node
  • openmpi starts the processes with mpirun
  • intelmpi is the Intel MPI form
  • srunmpi starts the processes with srun

More on the models: https://qcg-pilotjob.readthedocs.io/en/develop/execution_models.html.

The deck’s executable example uses script and iteration instead of exec. Four iterations, one description:

PYTHON 

from qcg.pilotjob.api.manager import LocalManager
from qcg.pilotjob.api.job import Jobs

manager = LocalManager()
jobs = Jobs()
jobs.add(script='echo "job ${it} executed at `date` @ `hostname`"',
         name='test_job',
         stdout='job.out.${it}',
         iteration=4)
job_names = manager.submit(jobs)
print('submited job names: ', str(job_names))
job_status = manager.status(job_names)
print('job status: ', job_status)
manager.wait4all()
manager.finish()

The print text on the slide is submited, with that spelling. The date and the hostname in job.out.${it} depend on the node and the clock. This episode does not ask you to predict them. The next challenge is a different file, with three tasks and a fixed line.

Challenge

Predict the printed line

Read episodes/files/exercises/qcg-three.py. The loop is for i in range(3). Each task runs /usr/bin/printf with the argument task-<i>\n and writes out_<i>.txt. The script then does this:

PYTHON 

print("qcg-finished")
print(open("out_0.txt", encoding="utf-8").read().strip())

What two lines does the script print?

qcg-finished
task-0

printf receives the format task-0\n for the first task, so out_0.txt holds task-0 and a newline. strip() removes the newline. The script prints out_0.txt only. task-1 and task-2 stay in out_1.txt and out_2.txt. The task names in the description are job_0, job_1, and job_2.

The script changes to a temporary directory before it submits, so those files are not left in the lesson tree. The two lines above are what the source prints. They do not depend on a hostname.

Example 3: 512 hostname tasks

The task is 512 independent short jobs on two nodes of the rome partition. The hands-on text says each node in that partition has 128 cores. Use hostname as the executable and default as the model.

episodes/files/hands-on/example_03/qcg_job.py is the description. The lines that create the tasks:

PYTHON 

# create 512 independent jobs
for job in range(512):
    # define a job
    jobs.add(name="job_{}".format(job),
             exec='hostname',
             stdout='output/job.{}.out'.format(job),
             stderr='output/job.{}.err'.format(job),
             model='default',
             numCores={ "exact" : 1},
             iteration=1)

numCores={ "exact" : 1} is one core. model='default' is one process. iteration=1 runs the description once per name. The names are job_0 through job_511.

The script then submits and waits. These are the print lines in the file:

PYTHON 

print("-- submit jobs")
manager.submit(jobs)
print("-- wait for all jobs")
manager.wait4all()
manager.finish()
print("-- finished")

Two nodes of 128 cores give 256 cores. Each task takes one core, and there are 512 tasks, so each node runs 256 tasks. Each task writes one line, the hostname, into output/job.<n>.out. Counted across those files, each node name appears in 256 lines.

The hands-on README says the state of each job is in nl-agent-{hostname}.log. The log page is https://qcg-pilotjob.readthedocs.io/en/develop/logs.html. This lesson has no sample of that log, so the states to read are the ones in the file you get after the run.

The Python creates output/ (os.makedirs("output", exist_ok=True)). Later examples remove .qcgpjm-service* and .*:0.hostfile. Those are the service files the cleanup script knows about. The slide also says state files appear in the submission directory.

Challenge

256 lines, and the files you submit

From example_03, name the node count, the tasks per node, and the number of hostname tasks. How many of those tasks run on each node?

Submit the published qcg_job.sh. If sinfo shows fewer cores than 128, use the README’s override and the tabs. Leave the script body as it is.

BASH 

module load 2024
module load QCG-PilotJob

module load 2024 selects the EESSI 2025.06 stack on this cluster and is the load line that comes before QCG-PilotJob.

The course cluster is a SURF Research Cloud workspace. CPU jobs request the rome partition.

BASH 

#SBATCH --partition=rome

Shrink --ntasks and --cpus-per-task on the sbatch line when sinfo shows small nodes. The scripts stay as published.

The nodes in this SURF Research Cloud workspace can be small. A two-core override is the usual request when sinfo shows two cores:

BASH 

#SBATCH --cpus-per-task=2

Check sinfo before you copy that number. CPU jobs still request --partition=rome.

qcg_job.sh requests --nodes=2, --ntasks-per-node=128, --partition=rome, and --time=00:05:00. qcg_job.py adds 512 jobs with exec='hostname', model='default', and numCores={ "exact" : 1}.

256 tasks run on each node: 512 tasks, two nodes, one core each, 128 cores on the node the hands-on describes. Each output file is one hostname line, so each node name accounts for 256 lines.

The small-node override from the hands-on README is sbatch --nodes=3 --ntasks-per-node=2 qcg_job.sh, after sinfo. The tabs are the load line, the partition, and the core floor for this machine. The module lines and the partition line inside the script stay as published.

A finished Python script prints:

-- submit jobs
-- wait for all jobs
-- finished

The hostnames are the contents of output/job.*.out, not those three lines.

Key Points
  • QCG-PilotJob is a manager inside one Slurm allocation. The Slurm script loads it and runs the Python script.
  • The published example 3 script loads 2024 and QCG-PilotJob/0.14.1-gfbf-2024a, and asks for 2 nodes at 128 tasks per node on rome.
  • qcg-three.py prints qcg-finished and then task-0.
  • Example 3 runs 512 hostname tasks, model='default', one core each. That is 256 tasks on each of the two nodes.
  • Job states are in nl-agent-{hostname}.log.

Content from Averages with a dependency

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • Which version first makes the aggregation wait?
  • Where in the job description do you read that?

Objectives

  • Walk qcg_job_v0.py through qcg_job_v4.py in order.
  • Point at after=job_names in qcg_job_v1.py as the first wait.
  • Run one version at a time, with QCG_MAX_INPUTS=100 and --ntasks=2 on 2 cores.
  • Convert the README’s CPUTimeRAW of 7232 seconds to 2.0 SBU.

Same averages, one allocation

Example 4 repeats example 2 with QCG-PilotJob. One average per CSV, then one file result.csv. The inputs are the same NOAA daily files. Extract them with tar -xvf input.tar.gz in episodes/files/hands-on/example_04.

qcg_job.sh copies the submission directory to $TMPDIR, runs one Python file, and copies result.csv back to results/. It also archives the working directory to results/log.<job id>.tar.gz, excluding input and result.

The script as copied runs v1. The other versions are comments:

BASH 

# execute jobs
# python3 qcg_job_v0.py
python3 qcg_job_v1.py
# python3 qcg_job_v2.py
# python3 qcg_job_v3.py
# python3 qcg_job_v4.py

Change which line is active, one version at a time. The Slurm header is one node, 16 tasks, 20 minutes, partition rome.

Every version reads QCG_MAX_INPUTS. 0, which is the default in the Python, selects every CSV. Any other nonnegative value keeps that many files from the sorted list. On 2 cores the hands-on README uses 100 files:

BASH 

QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh

The #SBATCH --ntasks=16 line stays in the file. --ntasks=2 is the override.

v0

qcg_job_v0.py builds one Jobs object. It adds an average job per CSV (exec='python3', arguments average.py and the file, iteration=1). It then adds the aggregate in the same object:

PYTHON 

jobs.add(name="aggregate",
         script='cat average_*.csv | sort',
         stdout='result.csv',
         stderr='aggregate.err',
        )

There is no after. The aggregate is submitted together with the averages. Nothing in the description tells the manager to wait.

v1

qcg_job_v1.py is the same list, and the aggregate gains one field:

PYTHON 

jobs.add(name="aggregate",
         script='cat average_*.csv | sort',
         stdout='result.csv',
         stderr='aggregate.err',
         after=job_names
        )

job_names is the list of average names. after=job_names is what makes the aggregation wait. This is the first version that has the field.

v2

qcg_job_v2.py splits the work into jobs_comp and jobs_agg. It submits the averages, then waits on the ids the submit call returned:

PYTHON 

job_comp_ids = manager.submit(jobs_comp)
manager.wait4(job_comp_ids)

The comment above that call says you could use manager.wait4all() instead. The aggregate is a second submit, and it still sets after=job_names. The wait is now also in the Python, before that second submit. The field that first expressed the wait is still the one v1 added.

v3

qcg_job_v3.py uses one Jobs object. It submits the averages, calls manager.wait4all(), then jobs.clear(), then adds the aggregate with after=job_names and submits that. Same wait, a third arrangement. The after field is still there.

v4

qcg_job_v4.py is the version the README calls final. It waits in Python and checks each average before it submits anything for the aggregation:

PYTHON 

def succeeded(manager, ids):
    states = manager.status(ids)['jobs']
    return bool(ids) and all(states[name]['data']['status'] == 'SUCCEED' for name in ids)

PYTHON 

ids = manager.submit(jobs)
manager.wait4(ids)
if not succeeded(manager, ids):
    raise RuntimeError('An average task failed; aggregation cannot run')

The success state in that check is SUCCEED. The script writes the selected output names to selected.json under a qcg-averages-* directory. It then submits aggregate.py with that directory as the only argument. There is no after field on this aggregate. The wait already happened, and a failed average raises before the aggregate is submitted.

aggregate.py reads selected.json, requires one finite mean for each selected file, and writes sorted rows to stdout. The aggregate description sets stdout='result.csv'. qcg_job.sh copies that file to results/. The qcg-averages-* directory lives under the job’s $TMPDIR, so it is gone after the job. The archive in results/log.<job id>.tar.gz is where the README says selected.json remains.

Challenge

Which version first waits?

Open qcg_job_v0.py and qcg_job_v1.py. Which version first makes the aggregation wait, and which field says so?

v1. The aggregate description in qcg_job_v1.py is the first one that sets the field:

PYTHON 

         after=job_names

v0’s jobs.add for aggregate has script, stdout, and stderr, and no after. v2 and v3 also set after=job_names. v4 waits with manager.wait4 and the SUCCEED check, and its aggregate has no after. The first version is v1.

Challenge

One version, 100 files

qcg_job.sh currently calls qcg_job_v1.py. On a 2-core node, what command runs 100 files? What do you change to run v0, and what do you change back before v4?

BASH 

QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh

To see v0, comment python3 qcg_job_v1.py and uncomment python3 qcg_job_v0.py. Restore a single active line before you move to v2, v3, or v4. QCG_MAX_INPUTS=100 keeps the first 100 sorted names. 0 would select every CSV.

The README’s accounting example, for a different job, is:

sacct -X --format="JobID%24,Timelimit,AllocCPUS,CPUTime,CPUTimeRAW" -j 16235860
                   JobID  Timelimit  AllocCPUS    CPUTime CPUTimeRAW
------------------------ ---------- ---------- ---------- ----------
                16235860   00:20:00         16   02:00:32       7232
1 SBU / CPU hour * 7232 CPU seconds / 3600 seconds per hour = 2.0

That 2.0 is the README’s figure for job 16235860. Your job id, and your CPUTimeRAW, come from your own sacct.

Key Points
  • v0 submits the aggregate with no after, so the description does not make it wait.
  • v1 is the first version that waits. The field is after=job_names.
  • v2 waits with manager.wait4(job_comp_ids), v3 with wait4all and jobs.clear(). Both still set after.
  • v4 checks status SUCCEED, writes selected.json, and only then submits aggregate.py.
  • On 2 cores: QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh.

Content from Idle time

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • What does the bash pipeline wait for at the end of each block?
  • What does QCG-PilotJob do when a task ends and a core is free?

Objectives

  • Compare pipeline.sh and qcg_job.sh on times_short.txt.
  • Read the block size off SLURM_NTASKS, and the wait at the end of each block.
  • Plot both outputs with plot_profile.py.

Two ways through the same sleeps

Example 5 runs the same list of sleeps two ways. sleep.py sleeps for the number of seconds you pass and prints one line: TASK, the CPU id, the start time, and the end time.

The README’s example of that line, from python sleep.py 7, is:

TASK,54,1763290002.014343,1763290009.014470

The CPU number and the timestamps are from that example. The two times differ by about 7 seconds, which is the argument.

pipeline.sh is a Slurm job of 1 node and 16 tasks, one hour, partition rome, output pipeline.%j.out. qcg_job.sh is 1 node and 16 tasks, 40 minutes, partition rome, output qcg.%j.out. Both default to times.txt. TIMES_FILE selects another file.

times.txt has 160 lines. times_short.txt is the first 32 of those lines, unchanged. They sum to 338 seconds. The longest of the 32 is 56 and the shortest is 2.

On 2 cores the hands-on README runs both with that short file, so both use the same number of slots:

BASH 

TIMES_FILE=times_short.txt sbatch --ntasks=2 pipeline.sh
TIMES_FILE=times_short.txt sbatch --ntasks=2 qcg_job.sh

The comment in pipeline.sh says the block size is the number of tasks Slurm gave the job, 16 on the #SBATCH line, so the comparison with QCG stays fair. --ntasks=2 changes that number for both commands. Leave the #SBATCH --ntasks=16 lines in the files.

The README’s warning on the lengths: times_short.txt keeps the original task lengths. Shorter tasks would make QCG’s start-up time per task count for more than the idle time it saves, and the comparison would turn around.

The pipeline waits for the slowest task in the block

pipeline.sh reads the times, then sends the sleeps in blocks. This is the loop:

BASH 

times=${TIMES_FILE:-times.txt}
block=${SLURM_NTASKS:-16}
mapfile -t nums < "$times"

for ((i=0; i<${#nums[@]}; i+=block)); do
    for num in "${nums[@]:i:block}"; do
        python sleep.py "$num" &
    done
    wait
done

wait returns when every background sleep in that block has finished. With --ntasks=2 the lines stay in order, so 56 shares a block with the 4 that precedes it in times_short.txt. The 4-second sleep finishes, and wait holds the next block until the 56-second sleep ends.

QCG fills slots as tasks end. qcg_job.py adds every line of TIMES_FILE as its own job (exec='python3', arguments sleep.py and the seconds, iteration=1) and then calls manager.wait4all(). The tasks share the allocation. When one finishes, the manager can start another on that core. There is no wait between blocks, because there are no blocks.

qcg_job.sh records the wall clock around that Python, then concatenates the task files:

BASH 

START=$( date +%s )
python qcg_job.py
END=$( date +%s )

echo START,$START
cat $TMPDIR/qcg_*.out
echo END,$END

pipeline.sh prints START before the loop and END after it. The TASK lines from sleep.py fall between them in the Slurm output.

The plot

plot_profile.py reads START, END, and every TASK line. The lanes are the CPU number each task reported at the start. From example_05, after both jobs finish:

BASH 

module load 2025
module load SciPy-bundle/2025.06-gfbf-2025a
module load matplotlib/3.10.3-gfbf-2025a

python plot_profile.py pipeline.<SLURM_JOB_ID>.out
python plot_profile.py qcg.<SLURM_JOB_ID>.out

The output names follow the scripts: pipeline.%j.out and qcg.%j.out. The README says you can open the PNG files in the Open OnDemand file browser, or copy them to your own computer.

The measured spans and idle percentages for this pair are in the next episode. They come from one run. This episode is the two scripts.

Challenge

Who waits for 56?

times_short.txt contains 56. On --ntasks=2, that value shares a block with the 4 written just before it. Which script waits for 56 before it starts the next pair, and which line does the waiting?

pipeline.sh. The inner loop starts one sleep.py per value in the block, in the background. The bare wait after that loop returns only when every sleep in the block has finished. In the block that holds 4 and 56, that means waiting for 56.

qcg_job.py submits the whole file and calls manager.wait4all() once. A finished task frees a core, and the manager can start the next sleep on it. The 2-second task does not hold the allocation until 56 finishes.

Challenge

Same slots

You have a 2-core node. Write the two sbatch lines that compare the pipeline and QCG on the short file. What must you leave unchanged in the sleep times, and why?

BASH 

TIMES_FILE=times_short.txt sbatch --ntasks=2 pipeline.sh
TIMES_FILE=times_short.txt sbatch --ntasks=2 qcg_job.sh

times_short.txt stays as the first 32 lines of times.txt. The hands-on README says shorter tasks would make QCG’s start-up time per task count for more than the idle time it saves, and the comparison would turn around. --ntasks=2 sets the pipeline’s block size and QCG’s slot count to the same number, because the block size is SLURM_NTASKS.

Key Points
  • pipeline.sh runs sleeps in blocks of SLURM_NTASKS and waits for the whole block.
  • The slowest task in a block holds the next block. A finished short task in that block does not start new work.
  • QCG submits every line and fills a free core when a task ends.
  • times_short.txt is the first 32 of 160 lines, original lengths, summing to 338 seconds.
  • On 2 cores, both scripts take TIMES_FILE=times_short.txt and --ntasks=2.

Content from Other tools

Last updated on 2026-10-28 | Edit this page

Overview

Questions

  • Where does each tool say that the aggregation waits for every average?
  • What is measurements/sacct.txt for, and what does it leave out?

Objectives

  • Run the average and profile exercises with HyperQueue, Snakemake, and FireWorks.
  • Read the HyperQueue --wait, the Snakemake aggregate input, and the FireWorks parents.
  • Use 16 tasks for HyperQueue and Snakemake, and one worker for the file-backed FireWorks example.
  • Read the 25 October 2026 table as one run, and use sacct.txt for the Slurm accounting of that run.

The same two exercises

episodes/files/hands-on/alternatives/ repeats example 4 and example 5.

  • Example 4: one average per CSV, then an aggregation that must wait for every average.
  • Example 5: sleeps of different lengths, so idle time can be computed from each task’s start and end.

HyperQueue and Snakemake request 16 tasks. The file-backed FireWorks example requests one task, for its one worker. The average scripts default to 100 files. QCG_MAX_INPUTS=0 selects every file. Run each tool from its own directory. Two tools writing one result.csv in one directory will collide.

On 2 cores the README scales HyperQueue and Snakemake, and leaves FireWorks at one task:

BASH 

QCG_MAX_INPUTS=100 sbatch --ntasks=2 average.sh
TIMES_FILE=../../example_05/times_short.txt sbatch --ntasks=2 profile.sh

FireWorks keeps --ntasks=1. For the short profile: TIMES_FILE=../../example_05/times_short.txt sbatch profile.sh.

The load line for each tool is the tab. The scripts also name a version.

BASH 

module load 2025
module load HyperQueue/0.24.0

BASH 

module load 2025
module load Snakemake

BASH 

module load 2025
module load FireWorks

If module spider FireWorks finds nothing, use a virtual environment in your home directory.

On a two-core node, pass the smaller request on the command line. Do not edit the script.

BASH 

sbatch --cpus-per-task=2 pi.sh
sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
iend=8 wdir=TMPDIR sbatch run.sh

Check the core count with sinfo first.

HyperQueue

A server and one worker run inside the allocation. The average script lists the CSV paths in files.txt and submits one task per line.

The dependency is hq submit --wait. The flag returns when every task has finished. The aggregation is the next line. From alternatives/hyperqueue/average.sh:

BASH 

hq submit --each-line files.txt --cpus=1 --wait \
    --stdout "$averages/%{TASK_ID}.csv" --stderr "$averages/%{TASK_ID}.err" \
    -- bash -c 'python3 ../../example_04/average.py "$HQ_ENTRY"'

python3 ../aggregate.py "$averages"/*.csv > result.csv

The script loads module load 2025, Python/3.13.1-GCCcore-14.2.0, and HyperQueue/0.24.0. The alternatives README says module spider HyperQueue in the Snellius 2025 stack lists 0.19.0, 0.20.0, and 0.24.0, checked on 2026-10-25. The header asks for --ntasks=16.

profile.sh uses the same --each-line and --wait pattern on the times file, and prints START and END around that submit.

Snakemake

snakemake --cores "$SLURM_NTASKS" runs the rules on the cores of the allocation. The average script’s header also asks for --ntasks=16.

The dependency is the input of the aggregate rule in alternatives/snakemake/Snakefile:

PYTHON 

rule aggregate:
    input:
        expand("averages/{name}.csv", name=NAMES),
    output:
        "result.csv",
    shell:
        "cat {input:q} | sort > {output:q}"

Snakemake starts that rule when every file in the input list exists. Snakefile.profile has one sleep rule instance per line of the times file.

The alternatives README gives the module as snakemake/9.22.0-foss-2025b. It also says module spider snakemake finds nothing in the Snellius 2025 stack, and the script then activates $HOME/venvs/snakemake. The install line in that README is pip install snakemake after module load 2025 and Python/3.13.1-GCCcore-14.2.0.

FireWorks

FireWorks stores the workflow in a LaunchPad. Here the LaunchPad is a file under $TMPDIR (MONGOMOCK_SERVERSTORE_FILE). There is no MongoDB server. rlaunch rapidfire starts one worker, which runs each Firework that is ready until none remain.

A file LaunchPad allows one writer. The comment in average.sh says several rlaunch workers lose Firework states. The aggregate then never runs. Parallel workers need a MongoDB server. The header is one node and one task:

BASH 

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:20:00
#SBATCH --partition=rome

profile.sh uses the same --ntasks=1. The tasks run one after another, because there is one worker.

The dependency is parents=averages on the aggregate Firework in alternatives/fireworks/workflow.py:

PYTHON 

command = shlex.join([sys.executable, os.path.join(HERE, "..", "aggregate.py"),
                      "--manifest", manifest])
aggregate = Firework(ScriptTask.from_str(f"{command} > {shlex.quote(os.path.join(HERE, 'result.csv'))}"),
                     name="aggregate", parents=averages)
lp.add_wf(Workflow(averages + [aggregate], name="average"))

parents=averages is the link. FireWorks runs that Firework after every average Firework has finished. The workflow writes selected.json and calls aggregate.py --manifest. HyperQueue still calls aggregate.py with the output paths as arguments.

The module line in the script is FireWorks/2.1.4-gfbf-2025a. The README says the Snellius 2025 stack has no FireWorks module, and the fallback is $HOME/venvs/fireworks with pip install "fireworks[mongomock]".

On Snellius the one worker took 9 s for 100 files and 212 s for all 1000 files, on its one core. Snakemake took 15 s for the 1000 files on 16 cores, about 240 core-seconds. Per core those costs are close. The file LaunchPad still allows one worker, so FireWorks in this example cannot take more cores.

average.sh uses 100 of the 1000 files unless QCG_MAX_INPUTS says otherwise.

One measured run

The slides and alternatives/README.md report one run on 25 October 2026: two-core nodes, each job alone, --nodes=1 --ntasks=2 (FireWorks --ntasks=1), the 32 tasks of times_short.txt at their original lengths, 338 task-seconds in all. The slide says this is one run, and it is not a general result. A 16-core Snellius allocation was not measured for this table.

Idle is 1 - busy / (slots * span). Span is END - START. Busy is the sum of the task durations. The lanes in plot_profile.py are the CPU number a task reported when it started. Tasks are not pinned, so overlapping bars can leave white space in a lane. That white space is not the idle figure. The idle figure comes from the start and end times.

Tool Span Idle Where it sits
bash pipeline 267 s 37% each block of 2 tasks waits for its longer task
QCG-PilotJob 181 s 7% 1 s before the first task, 3 s after the last, short gaps
HyperQueue 172 s 2% 1 s of start-up, 0.1 s between tasks
Snakemake 250 s 32% 77 s of start-up before the first task; the tasks themselves spanned 171 s with 0.1 s gaps
FireWorks 344 s 2% of its one slot one worker, so the tasks run one after another

An earlier QCG run that loaded a module inside every task showed 23% idle. The examples pass no modules=, so a task inherits the job’s environment and does not pay for a module load.

Averages of 100 files, same nodes:

Tool Time
HyperQueue 11 s
QCG-PilotJob, qcg_job_v4.py 18 s
FireWorks, one worker 25 s
Snakemake 67 s

QCG’s 7% is a second at the start and three at the end, plus short gaps. HyperQueue’s 2% is the next task starting when a slot frees. Snakemake’s gaps between tasks are the same 0.1 s. The 32% is the 77 s before the first task. FireWorks at 344 s is the one worker, task after task.

The bash pipeline’s 37% is the block of 2: the block waits for its longer task. That is the same wait as in pipeline.sh.

What sacct.txt is for

episodes/files/hands-on/alternatives/measurements/sacct.txt is the Slurm listing of that run. measurements/README.md says measure.sh submitted the nine jobs and sacct.txt lists them. The format in measure.sh is:

BASH 

sacct -X -n -j "$ids" -o JobID,JobName%20,NodeList,AllocCPUS,Elapsed,State,ExitCode

The file has those nine lines:

18158                 pipeline.sh     coursenode1          2   00:04:35  COMPLETED      0:0
18159                  qcg_job.sh     coursenode2          2   00:03:13  COMPLETED      0:0
18160                  profile.sh     coursenode3          2   00:03:04  COMPLETED      0:0
18161                  average.sh     coursenode3          2   00:00:11  COMPLETED      0:0
18162                  profile.sh     coursenode2          2   00:04:31  COMPLETED      0:0
18163                  average.sh     coursenode3          2   00:01:07  COMPLETED      0:0
18164                  profile.sh     coursenode3          1   00:05:57  COMPLETED      0:0
18165                  average.sh     coursenode3          1   00:00:25  COMPLETED      0:0
18166                  qcg_job.sh     coursenode1          2   00:00:18  COMPLETED      0:0

The output files next to it name the tool: pipeline.18158.out, qcg.18159.out, hq.18160.out, hq-average.18161.out, snakemake.18162.out, snakemake-average.18163.out, fw.18164.out, fw-average.18165.out, and slurm-18166.out for QCG-PilotJob v4 averages.

Elapsed is the scheduler column. The span in the idle table is END - START inside the job output. On the average jobs the two agree with the table (11 s, 67 s, 25 s, 18 s). On the profile jobs they differ. Job 18158’s Elapsed is 00:04:35 and the pipeline span is 267 s. Use Elapsed when you want the accounting, and the span when you want the idle formula.

sacct.txt has no CPUTimeRAW column. The 2.0 SBU figure in example 4 is job 16235860, not one of these nine. Core-hours for a new run use the example 4 format, CPUTimeRAW / 3600.

How the deck chooses

  • QCG-PilotJob and HyperQueue: many tasks inside one allocation. QCG-PilotJob is driven from Python. HyperQueue is driven from the command line, and it has a Python API as well.
  • Snakemake: rules on files. A rerun builds what changed.
  • FireWorks: a database of workflows, with states and reruns. Without a MongoDB server the file LaunchPad allows one worker.
Challenge

Where the dependency is written

For the average exercise, name the line or field that makes the aggregation wait, in HyperQueue, in Snakemake, and in FireWorks.

HyperQueue: hq submit --each-line files.txt --cpus=1 --wait in hyperqueue/average.sh. The aggregate line is the python3 ../aggregate.py command after that submit returns.

Snakemake: the input of rule aggregate in Snakefile, expand("averages/{name}.csv", name=NAMES).

FireWorks: parents=averages on the aggregate Firework in fireworks/workflow.py, in the lp.add_wf(Workflow(averages + [aggregate], name="average")) call.

QCG’s first form of the same idea is after=job_names in qcg_job_v1.py.

Challenge

Scale the optional run

You are on a 2-core node and you want the short profile from all three tools. What does FireWorks keep that the other two override? Which file lists the nine measured jobs?

On a two-core node, pass the smaller request on the command line. Do not edit the script.

BASH 

sbatch --cpus-per-task=2 pi.sh
sbatch --array=0-99%10 array_job.sh
QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh
iend=8 wdir=TMPDIR sbatch run.sh

Check the core count with sinfo first.

BASH 

module load 2025
module load HyperQueue/0.24.0

BASH 

module load 2025
module load Snakemake

BASH 

module load 2025
module load FireWorks

If module spider FireWorks finds nothing, use a virtual environment in your home directory.

HyperQueue and Snakemake take --ntasks=2 on the command line. Their scripts still say --ntasks=16. FireWorks keeps --ntasks=1, because the file LaunchPad has one worker.

BASH 

TIMES_FILE=../../example_05/times_short.txt sbatch --ntasks=2 profile.sh
TIMES_FILE=../../example_05/times_short.txt sbatch profile.sh

The second line is the FireWorks directory. The first is HyperQueue or Snakemake, from that tool’s directory.

episodes/files/hands-on/alternatives/measurements/sacct.txt lists the nine jobs from 25 October 2026: job id, name, node, allocated CPUs, Elapsed, state, and exit code. The idle percentages are computed from START, END, and TASK lines, not from the Elapsed column.

Key Points
  • HyperQueue waits with hq submit --wait. The aggregate command is the next line.
  • Snakemake waits because the aggregate rule’s input lists every average file.
  • FireWorks waits with parents=averages. The file LaunchPad has one worker and the script asks for one task.
  • HyperQueue and Snakemake scripts request 16 tasks. On 2 cores, override with --ntasks=2.
  • The 25 October 2026 table is one run: pipeline 267 s and 37% idle, QCG 181 s and 7%, HyperQueue 172 s and 2%, Snakemake 250 s and 32%, FireWorks 344 s and 2% of one slot.
  • measurements/sacct.txt is the Slurm accounting for that run. It has Elapsed, and it has no CPUTimeRAW.