Embarrassingly parallel

  • Many independent tasks on a dense node are the embarrassingly parallel case.
  • A thousand one-task jobs spend their time in the scheduler.
  • The allocation should start the next task when a core is free.
  • Arrays come next. QCG-PilotJob is the pilot service. HyperQueue, Snakemake, and FireWorks are the other three tools in the last episode.

Slurm arrays

  • An array is a collection of similar jobs. Every task shares the initial options.
  • On Snellius the deck’s maximum array size is 30000, and the minimum allocation on rome is 16 cores and 28 GiB.
  • --array=0-10 is 11 tasks. --array=0-10%2 runs at most 2 at once.
  • SLURM_ARRAY_TASK_ID is the index. %A and %a fill in the job id and the index in a file name.
  • Example 1’s $TMPDIR values differ because each element has its own node-local directory.
  • Example 2’s aggregate writes result.csv. On 2 cores: sbatch --array=0-99%10 array_job.sh.

QCG-PilotJob

  • QCG-PilotJob is a manager inside one Slurm allocation. The Slurm script loads it and runs the Python script.
  • The published example 3 script loads 2024 and QCG-PilotJob/0.14.1-gfbf-2024a, and asks for 2 nodes at 128 tasks per node on rome.
  • qcg-three.py prints qcg-finished and then task-0.
  • Example 3 runs 512 hostname tasks, model='default', one core each. That is 256 tasks on each of the two nodes.
  • Job states are in nl-agent-{hostname}.log.

Averages with a dependency

  • v0 submits the aggregate with no after, so the description does not make it wait.
  • v1 is the first version that waits. The field is after=job_names.
  • v2 waits with manager.wait4(job_comp_ids), v3 with wait4all and jobs.clear(). Both still set after.
  • v4 checks status SUCCEED, writes selected.json, and only then submits aggregate.py.
  • On 2 cores: QCG_MAX_INPUTS=100 sbatch --ntasks=2 qcg_job.sh.

Idle time

  • pipeline.sh runs sleeps in blocks of SLURM_NTASKS and waits for the whole block.
  • The slowest task in a block holds the next block. A finished short task in that block does not start new work.
  • QCG submits every line and fills a free core when a task ends.
  • times_short.txt is the first 32 of 160 lines, original lengths, summing to 338 seconds.
  • On 2 cores, both scripts take TIMES_FILE=times_short.txt and --ntasks=2.

Other tools

  • HyperQueue waits with hq submit --wait. The aggregate command is the next line.
  • Snakemake waits because the aggregate rule’s input lists every average file.
  • FireWorks waits with parents=averages. The file LaunchPad has one worker and the script asks for one task.
  • HyperQueue and Snakemake scripts request 16 tasks. On 2 cores, override with --ntasks=2.
  • The 25 October 2026 table is one run: pipeline 267 s and 37% idle, QCG 181 s and 7%, HyperQueue 172 s and 2%, Snakemake 250 s and 32%, FireWorks 344 s and 2% of one slot.
  • measurements/sacct.txt is the Slurm accounting for that run. It has Elapsed, and it has no CPUTimeRAW.