Many small tasks

Last updated on 2026-10-26 | Edit this page

Overview

Questions

  • Why is one Slurm job per tiny task a poor plan?
  • What does a pilot job keep hold of, while it runs the tasks?

Objectives

  • Explain the cost of submitting thousands of separate Slurm jobs.
  • Run three tasks inside one QCG-PilotJob LocalManager on the course stack.
  • Describe a HyperQueue server as a process that fills one Slurm allocation with tasks.
  • Read a three-rule Snakemake file as a graph of files, not as a queue.
  • Say what FireWorks adds when tasks must be tracked in a database.

The queue is not free

A one-second task with its own sbatch still waits in the queue, still writes a job record, and still starts a shell. A few thousand of those tasks can spend more time starting than working. A job array, from the Slurm episode, is the first improvement: one submission, many tasks, and Slurm still tracks each task.

A pilot job goes further. You ask Slurm for a block of cores once. A small manager sits inside that allocation and starts the short tasks itself. The queue saw one job.

Many tasks inside one Slurm job. The nodes drawn are one allocation, pilot, and many tasks.
Many tasks inside one Slurm job. The nodes drawn are one allocation, pilot, and many tasks.

QCG-PilotJob on the course cluster

QCG-PilotJob is the pilot manager this course installs. It is a site module. It is not in the EESSI stack. module spider QCG-PilotJob inside EESSI 2025.06 says it cannot be found, and that is expected.

On the course cluster:

BASH 

module purge
module load 2024
module load QCG-PilotJob
python3 episodes/files/exercises/qcg-three.py

module load 2024 is the course alias for EESSI 2025.06. Load it before QCG-PilotJob. The script submits three tasks that each run printf, waits until they finish, and prints:

qcg-finished
task-0

The Python is short. A LocalManager runs the tasks on the machine where you started it. Jobs collects the descriptions. Each task asks for exactly one core.

PYTHON 

manager = LocalManager()
jobs = Jobs()
jobs.add(name="job_0", exec="/usr/bin/printf", args=["task-0\\n"],
         stdout="out_0.txt", numCores={"exact": 1})
manager.submit(jobs)
manager.wait4all()
manager.finish()

On a cluster you normally put that manager inside one sbatch script, so the cores come from the allocation instead of from the login node. For three printf tasks the login node is an acceptable place to see the tool work. A real parameter sweep belongs in the job.

qcg-pilotjob --help-h prints the command-line help if you want the non-Python entry point. The exercises use the Python API because that is what the course stack is there to support.

Callout

If module load QCG-PilotJob fails, you are probably on the EESSI container or on a machine without the site module. Skip the run. The HyperQueue and Snakemake sections are still readable, and the QCG script is still in the lesson files for later.

HyperQueue

HyperQueue is the tool to reach for when the pilot should be a small server plus workers, and the workers are Slurm jobs. You start hq server start, you submit tasks to that server, and hq alloc add slurm asks Slurm for workers. The server fills the workers. You do not write one batch script per task.

The idea you should keep: the Slurm job is the bucket of resources. HyperQueue is who pours tasks into the bucket. You do not need to memorise every flag today.

Snakemake

Snakemake starts from files. A rule names an output, the inputs it needs, and the command that builds the output. Snakemake runs a rule when the output is missing or older than the input. Three rules that feed each other are a pipeline:

PYTHON 

rule all:
    input: "summary.txt"

rule step_one:
    output: "part.txt"
    shell: "printf 'part\\n' > part.txt"

rule summary:
    input: "part.txt"
    output: "summary.txt"
    shell: "cp part.txt summary.txt"

That is a different question from QCG. QCG asks how to run many copies of a command. Snakemake asks which files are out of date. People combine them: Snakemake decides the graph, and a cluster profile or a pilot runs the commands.

FireWorks

FireWorks stores tasks in a database, with dependencies, and workers pull the tasks. You use it when many people submit tasks, when a task must be retried, or when you need a record of what ran. It is more machinery than a three-task LocalManager. If your whole problem is "run this command on these fifty inputs today", stay with the array or with QCG.

Which tool answers which question

Tool When you reach for it
QCG-PilotJob a Python script should launch tasks inside one allocation
HyperQueue a small server should fill Slurm workers
Snakemake the work is a graph of files
Slurm array every task is the same batch script with a different index
FireWorks tasks must be stored and shared through a database

A faded load

BASH 

module purge
module load ________
module load ________

On the course cluster the two names are 2024 and QCG-PilotJob, in that order.

Challenge

Predict the printed line

qcg-three.py runs three tasks. The last two lines of the program print a fixed word and then the contents of out_0.txt. What are those two lines?

qcg-finished
task-0

Task 0 writes task-0 into out_0.txt. The other two tasks write out_1.txt and out_2.txt. The script only prints the first file, so you do not see task-1 in the program output.

Challenge

Run the three tasks

On the course cluster, from the lesson directory, after the two module load commands:

BASH 

python3 episodes/files/exercises/qcg-three.py

If you cannot load the module, do not install a random copy into your home directory during class. Read the script instead and write down the three task names.

A successful run ends with qcg-finished and task-0. The working directory is a temporary directory the script creates, so you will not see out_0.txt in the lesson tree afterwards. The three task names inside the script are job_0, job_1, and job_2.

If Python says No module named qcg, the site module is not loaded. module purge and then module load 2024 before module load QCG-PilotJob is the sequence that works on the course cluster.

Key Points
  • Thousands of one-task Slurm jobs spend their time starting, not working.
  • A pilot job holds one allocation and starts the short tasks inside it.
  • On the course cluster, module load 2024 and then module load QCG-PilotJob.
  • QCG-PilotJob is not an EESSI module. module spider inside EESSI will not find it.
  • HyperQueue fills a Slurm allocation, Snakemake tracks files, and FireWorks tracks tasks in a database.