Content from Nodes, tasks, and threads

Last updated on 2026-10-27 | Edit this page

Estimated time: 40 minutes

Overview

Questions

  • Which Slurm option is a node, a task, a core, and a thread?
  • How many hostname lines does one node print when four tasks run there?

Objectives

  • Map nodes, tasks, cores, and threads to --nodes, --ntasks, --ntasks-per-node, and --cpus-per-task.
  • Load 2025 and foss/2025a inside a job and run mpicc -v.
  • Predict the hostname lines for 2 nodes, 8 tasks, and 4 tasks per node.
  • Use the course-cluster override --nodes=3 --ntasks=6 --ntasks-per-node=2 when the nodes are small.

Four words, four options

A node is one computer. A task is one MPI rank. A core is one hardware processor reserved for that rank. A thread is one OpenMP thread inside the rank. Slurm does not count threads. It counts cores, and you point the threads at those cores.

You mean Option
How many computers --nodes
How many MPI ranks in the job --ntasks
How many of those ranks on each computer --ntasks-per-node
How many cores for each rank --cpus-per-task

--cpus-per-task is the shared-memory side. The hybrid scripts set OMP_NUM_THREADS from SLURM_CPUS_PER_TASK. One core gives the rank one thread. Sixteen threads need --cpus-per-task=16 on a node that has the cores.

sbatch reserves the allocation and runs your shell script. It does not launch the ranks. srun launches them. squeue shows the queue, scancel removes a job, and sinfo shows the nodes.

What the published scripts ask for

The scripts under episodes/files/hands-on/ are the Snellius requests. The sequential scripts ask for one task and --cpus-per-task=16. The hybrid scripts ask for --ntasks=128 and --cpus-per-task=1 on one node. Keep those lines on Snellius.

The smallest allocation and the partition name are in the tabs. Do not edit the scripts to change them. Options on the sbatch command line override the #SBATCH lines.

On the course cluster the hostname exercise is --nodes=3 --ntasks=6 --ntasks-per-node=2. That is two tasks on each of three nodes.

Load the compiler inside the job

The stack module differs by machine. Use the tab, then load the compiler toolchain the scripts use.

Challenge

Load foss/2025a in a job

Write load_modules.sh and submit it with sbatch load_modules.sh.

BASH 

#!/bin/bash
#SBATCH -J load_modules
#SBATCH -o load_modules.out
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH -t 01:00:00

module load 2025
module load foss/2025a

echo "Loaded modules:"
module list

echo "Checking compiler version:"
mpicc -v

After the job finishes, cat load_modules.out. What shows that the compiler module is loaded?

The file lists the loaded modules, including foss/2025a, and then the version text from mpicc -v. A login-node module list is not that file. The job has its own environment. Add the partition line from the partition tab if the job waits with an invalid partition.

Where hostname runs

hostname prints the name of the computer the task is on. One task on one node prints one line. The sample script spreads eight tasks across two nodes, four per node.

BASH 

#!/bin/bash
#SBATCH -J name
#SBATCH -o name.out
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH -t 01:00:00

srun hostname
Challenge

Four lines per host

The script asks for 2 nodes, 8 tasks, and 4 tasks per node. How many lines in name.out name each host? How many lines are there in total?

Four lines per host. Each node runs four tasks, and each task runs hostname once, so that host name appears four times. Two hosts give eight lines. The lines from the two hosts can be mixed in the file. The names are compute nodes.

On the course cluster, submit the same script with --nodes=3 --ntasks=6 --ntasks-per-node=2. Three hosts, two tasks on each, so two lines per host and six lines in total.

Callout

Ethernet and InfiniBand

Snellius connects nodes with InfiniBand. The course machines connect them with Ethernet. A job that spans nodes scales more weakly on the course machines than the same job on Snellius.

Key Points
  • --nodes, --ntasks, --ntasks-per-node, and --cpus-per-task are the node, the ranks, the ranks per node, and the cores (threads) per rank.
  • sbatch reserves resources. srun starts the tasks.
  • module load 2025 and module load foss/2025a belong in the job. mpicc -v shows the compiler the job loaded.
  • Eight tasks with four per node print four hostname lines for each of two hosts.
  • The course-cluster hostname override is --nodes=3 --ntasks=6 --ntasks-per-node=2.

Content from AXPY

Last updated on 2026-10-27 | Edit this page

Estimated time: 40 minutes

Overview

Questions

  • Why does AXPY speed up and then flatten?
  • What does the hybrid program print for y[0] and for the local length?

Objectives

  • Run the sequential AXPY in hands-on/ex1 and the hybrid AXPY in hands-on/ex2.
  • Match y[0] to 2 + 100 * 2 * 1.
  • Time 2, 4, and 8 tasks at a fixed vector length.
  • On two-core nodes, submit with sbatch --cpus-per-task=1 submit-axpy.sh 10000000 and sbatch --ntasks=2 submit-axpy-hybrid.sh 10000000.

Pointwise, and limited by bandwidth

AXPY is y = a * x + y. Each element is y[i] = a * x[i] + y[i]. No element waits on another, and the loop sends no messages. The work is one multiply and one add per element, and the bytes moved are the vectors. The run hits memory bandwidth early, so adding tasks stops shortening the time. That flattening is the result you are looking for. This episode does not give you a filled speedup table. You record the times.

Work in episodes/files/hands-on/ex1 and ex2. N is the global length. Options you pass to sbatch override the #SBATCH lines.

Sequential AXPY

hands-on/ex1/axpy.c fills x with 1 and y with 2, sets a to 2, and calls the loop 100 times. The script compiles and runs it like this:

BASH 

gcc -O2 axpy.c -o axpy.out

BASH 

srun ./axpy.out $N

Submit from the ex1 directory:

BASH 

sbatch submit-axpy.sh 10000000

The published script asks for --cpus-per-task=16. On a two-core node:

BASH 

sbatch --cpus-per-task=1 submit-axpy.sh 10000000

A recorded run for N = 10000000 printed this. Your time will differ. y[0] will not.

Length of the arrays: 10000000
y[0] = 202.000000
Time taken for axpy execution: 0.963870 seconds

y starts at 2. Each of the 100 calls adds a * x[i] = 2 * 1. So y[0] = 2 + 100 * 2 * 1 = 202.

Challenge

Predict y[0], then time one run

Before you submit, write the value of y[0] for N = 10000000. Submit the sequential script. Which lines are fixed by the program, and which line is your measurement?

y[0] is 202. The length line is the N you passed. The time line is your measurement. A second submit can print a different time. It should print the same y[0].

Hybrid AXPY

hands-on/ex2/axpy-hybrid.c splits N across MPI ranks. If N divides evenly, each rank holds N / ntasks elements. Rank 0 prints that local length. Every rank fills its own arrays with the same 1 and 2, so y[0] on rank 0 is still 202. There is no halo and no reduction.

The OpenMP loop is the outer element loop on that rank:

C 

#pragma omp parallel for
for (size_t i = 0; i < local_n; i++) {
    y[i] = a * x[i] + y[i];
}

The script compiles and launches with:

BASH 

mpicc -O2 -fopenmp axpy-hybrid.c -o axpy-hybrid.out

BASH 

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

BASH 

srun ./axpy-hybrid.out $N

The published script is --nodes=1, --ntasks=128, --cpus-per-task=1. A recorded run at N = 10000000 with those 128 tasks printed:

Global length of the arrays: 10000000
Local length of the arrays: 78125
y[0] = 202.000000
Time taken for axpy execution: 0.009485 seconds

10000000 / 128 = 78125. That local length checks the split. Your time will differ. These two samples are separate recorded runs. Keep them out of a speedup series.

Hyb-T-C means T MPI tasks and C OpenMP threads. sbatch --ntasks=1 --cpus-per-task=16 submit-axpy-hybrid.sh 10000000 is Hyb-1-16 on a machine that can give you 16 cores.

Callout

Pin threads when a task has more than one

The published ex2 script does not pin threads. When you give a task more than one thread, pin them so the operating system does not move them during the timed loop. Set this before srun:

BASH 

export OMP_PLACES=cores
export OMP_PROC_BIND=close
srun --cpu-bind=cores ./axpy-hybrid.out $N

OMP_PLACES=cores gives each thread its own core. OMP_PROC_BIND=close keeps that task’s threads on neighbouring cores. --cpu-bind=cores keeps each rank on the cores Slurm gave it.

Challenge

Time 2, 4, and 8 tasks

From ex2, at N = 10000000, record the time line for 2, 4, and 8 tasks, one thread per task:

BASH 

sbatch --ntasks=2 --cpus-per-task=1 submit-axpy-hybrid.sh 10000000
sbatch --ntasks=4 --cpus-per-task=1 submit-axpy-hybrid.sh 10000000
sbatch --ntasks=8 --cpus-per-task=1 submit-axpy-hybrid.sh 10000000

On a two-core node the hybrid line that fits on one node is:

BASH 

sbatch --ntasks=2 submit-axpy-hybrid.sh 10000000

Four and eight tasks need more cores than that node has. Use the course tab from the previous episode for the runs that fit, and do not edit the script.

What do you write down, and what shape do you expect from a bandwidth limit?

Write down the Time taken for axpy execution line, the task count, and y[0]. y[0] stays 202. The local length is 10000000 divided by the task count when the split is even: 5000000, 2500000, and 1250000.

The times should drop and then flatten. AXPY moves a lot of bytes per arithmetic operation, so extra tasks stop helping once the memory path is full. A time that stays flat when you double the tasks is the bandwidth limit. y[0] is the correctness check, and it stays 202.

On Snellius the published script’s 128 tasks are one more column on the sheet. Still record 2, 4, and 8.

A sheet for a full node

On Snellius you can also fill this sheet. Leave the cells empty until your jobs print a time. Hyb-256-1 and Hyb-2-128 need 256 cores, so they need --nodes=2. The script asks for one node.

BASH 

sbatch --nodes=2 --ntasks=256 --cpus-per-task=1 submit-axpy-hybrid.sh <N>
sbatch --nodes=2 --ntasks=2 --cpus-per-task=128 submit-axpy-hybrid.sh <N>

The last row, N = 10000000000, needs about 160 GB for x and y. Add --exclusive so the job gets the memory of a full node.

Vector size Sequential Hyb-16-1 Hyb-128-1 Hyb-256-1 Hyb-1-16 Hyb-1-128 Hyb-2-128
1,000,000
10,000,000
100,000,000
1,000,000,000
10,000,000,000

On the course cluster, do not use those wide columns. The course tab lists Hyb-2-1, Hyb-1-2, Hyb-3-2, and Hyb-6-1, and it leaves out the two largest sizes.

After a job finishes you can read how the allocation was used:

BASH 

seff JOBID
sacct -j JOBID --format=JobID,AllocCPUS,Elapsed,TotalCPU,MaxRSS,State

seff reports CPU time used, divided by allocated cores times elapsed time. The whole job is in that figure, including compilation. Compare two configurations. Do not treat one short run’s percentage as the kernel’s speedup.

Key Points
  • AXPY is pointwise. The hybrid code has no reduction. Bandwidth limits it, so speedup flattens early.
  • Sequential compile and run: gcc -O2 axpy.c -o axpy.out and srun ./axpy.out $N.
  • Hybrid compile and run: mpicc -O2 -fopenmp axpy-hybrid.c -o axpy-hybrid.out, export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK, and srun ./axpy-hybrid.out $N.
  • y[0] is 202 for this initialisation and 100 repetitions. The local length is the global length split across tasks.
  • Two-core submissions: sbatch --cpus-per-task=1 submit-axpy.sh 10000000 and sbatch --ntasks=2 submit-axpy-hybrid.sh 10000000.

Content from Dot product

Last updated on 2026-10-27 | Edit this page

Estimated time: 40 minutes

Overview

Questions

  • Where is the reduction in the hybrid dot product?
  • Why can the last digits differ when the printed value matches?

Objectives

  • Run the sequential dot product in hands-on/ex3 and the hybrid program in hands-on/ex4.
  • Find the OpenMP reduction(+:) clause and the MPI reduction.
  • Time 2, 4, and 8 tasks at a fixed length.
  • On the course cluster, submit sbatch --nodes=3 --ntasks=6 submit-dot-hybrid.sh 30000000.

A pointwise product, then a sum

The dot product is the sum of x[i] * y[i]. Each product is independent. The sum is not. Someone has to add the partial sums. That add is the reduction.

In hands-on/ex3/dot.c the vectors are sin(i) and 0.5 * cos(i). The program adds the dot product into result over 100 repetitions. The script compiles and runs:

BASH 

gcc -O2 dot.c -o dot.out -lm

BASH 

srun ./dot.out $N

BASH 

sbatch submit-dot.sh 10000000

On a two-core node:

BASH 

sbatch --cpus-per-task=1 submit-dot.sh 10000000

A recorded run for N = 10000000 printed:

Length of the arrays: 10000000
Dot product result: 12.3778
Time taken for dot product execution: 1.21664 seconds

Your time will differ. The result line is the value to compare with the hybrid run.

Two reductions

hands-on/ex4/dot-hybrid.c gives each rank a slice of the global index, stored in offset, so sin and cos see the same i as the sequential loop. The live OpenMP loop is:

C 

#pragma omp parallel for reduction(+:local_sum)
for (size_t i = 0; i < local_n; i++) {
    local_sum += x[i] * y[i];
}

reduction(+:local_sum) gives each thread a private sum and adds those private sums at the end of the loop. The file also contains a commented loop that splits the iterations by hand. The live function is the reduction clause.

Each rank then needs a global sum. The call in the timed loop is MPI_Allreduce with MPI_SUM:

C 

MPI_Allreduce(&local_result, &global_result, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
result += global_result;

MPI_Reduce would leave the sum on one rank. MPI_Allreduce is that reduction with the sum delivered to every rank. This file uses MPI_Allreduce because every rank adds global_result into result. Rank 0 prints result. The collective sits inside the 100 repetitions, so the printed time includes it.

The script compiles and launches with:

BASH 

mpicc -O2 -fopenmp dot-hybrid.c -o dot-hybrid.out -lm

BASH 

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

BASH 

srun ./dot-hybrid.out $N

A recorded run at N = 10000000 with the script’s 128 tasks printed:

Global length of the arrays: 10000000
Local length of the arrays: 78125
Global dot product result: 12.3778
Time taken for dot product execution: 0.039711 seconds

The result matches the sequential run to the digits that were printed. The order of the additions is different, so later digits can differ. As with AXPY, do not turn these two sample times into a speedup table.

When a task has more than one thread, pin them the same way as in the AXPY episode: OMP_PLACES=cores, OMP_PROC_BIND=close, and srun --cpu-bind=cores.

Challenge

Name the reductions, then time the tasks

In dot-hybrid.c, which OpenMP clause combines the thread sums, and which MPI call combines the rank sums?

Then, from ex4, record the time at N = 10000000 for 2, 4, and 8 tasks:

BASH 

sbatch --ntasks=2 --cpus-per-task=1 submit-dot-hybrid.sh 10000000
sbatch --ntasks=4 --cpus-per-task=1 submit-dot-hybrid.sh 10000000
sbatch --ntasks=8 --cpus-per-task=1 submit-dot-hybrid.sh 10000000

On the course cluster the run that matches the override table is:

BASH 

sbatch --nodes=3 --ntasks=6 submit-dot-hybrid.sh 30000000

What local length should rank 0 print for that course command? What result digits should match the sequential run?

The OpenMP clause is reduction(+:local_sum). The MPI call is MPI_Allreduce with MPI_SUM. That is a reduction. MPI_Reduce is the form that keeps the sum on one rank. This file uses the all-ranks form.

The course command runs 6 tasks and N = 30000000. 30000000 / 6 = 5000000, and 6 divides 30000000, so rank 0 prints local length 5000000.

Compare Global dot product result with a sequential run at the same N. The printed digits should match. Digits past what %g showed can move, because the sum is associated differently. The time includes 100 global reductions, so this kernel has a communication cost that AXPY does not. Extra tasks still help the products, and the reduction grows with the task count. Record the times. Do not expect them to halve forever.

Four and eight tasks on one two-core node do not fit. The six-task line above is the course override. It uses three nodes. Cross-node scaling on Ethernet is weaker than on InfiniBand. Keep a course three-node time in the course column.

The same sheet, still empty

On Snellius the recording sheet has the same columns as AXPY. Fill it with your times. Hyb-256-1 and Hyb-2-128 need --nodes=2:

BASH 

sbatch --nodes=2 --ntasks=256 --cpus-per-task=1 submit-dot-hybrid.sh <N>
sbatch --nodes=2 --ntasks=2 --cpus-per-task=128 submit-dot-hybrid.sh <N>

The last row needs about 160 GB for x and y. Add --exclusive for that row.

Vector size Sequential Hyb-16-1 Hyb-128-1 Hyb-256-1 Hyb-1-16 Hyb-1-128 Hyb-2-128
1,000,000
10,000,000
100,000,000
1,000,000,000
10,000,000,000

On the course cluster, use Hyb-2-1, Hyb-1-2, Hyb-3-2, and Hyb-6-1, and leave out the two largest sizes. The six-task command in the challenge is Hyb-6-1 at N = 30000000.

Key Points
  • The products are independent. The sum is a reduction.
  • OpenMP uses reduction(+:local_sum). MPI uses MPI_Allreduce with MPI_SUM, the all-ranks form of MPI_Reduce.
  • Compile and run with mpicc -O2 -fopenmp dot-hybrid.c -o dot-hybrid.out -lm, export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK, and srun ./dot-hybrid.out $N.
  • The printed result matches the sequential digits. Later digits can differ because the add order differs.
  • The course command is sbatch --nodes=3 --ntasks=6 submit-dot-hybrid.sh 30000000.

Content from Matrix multiplication

Last updated on 2026-10-27 | Edit this page

Estimated time: 40 minutes

Overview

Questions

  • What has to stay the same when you parallelise the outer loop?
  • Which compile flag makes the pragma real?

Objectives

  • Run hands-on/ex5 at sizes 500, 1000, and 2000.
  • Read Matrix checksum: and see that the same size prints the same line.
  • Parallelise the outer loop with OpenMP without changing that checksum.

One product, then a sum of the entries

hands-on/ex5/matmul.c computes C = A * B for square n by n matrices. For each i and j it dots row i of A with column j of B. It does this 100 times. Each repetition overwrites C. The checksum runs after that loop, so it sums the last product. The 100 repetitions overwrite C.

A is filled with 1 and B with 2. Each entry of C is 2 * n. The checksum adds every entry, so it is 2 n^3. It depends only on n. Two runs with the same n print the same Matrix checksum: line. The printf is Matrix checksum: %.17g.

The script compiles and runs:

BASH 

gcc -O2 matmul.c -o matmul.out

BASH 

srun ./matmul.out $N

The published request is one task and --cpus-per-task=16. On a two-core node, size 500 is:

BASH 

sbatch --cpus-per-task=1 submit-matmul.sh 500
Challenge

Same size, same checksum

From ex5, run sizes 500, 1000, and 2000. Run 500 twice.

BASH 

sbatch submit-matmul.sh 500
sbatch submit-matmul.sh 500
sbatch submit-matmul.sh 1000
sbatch submit-matmul.sh 2000

On a two-core node start with sbatch --cpus-per-task=1 submit-matmul.sh 500.

What is Matrix checksum: for n = 500, and what must the second run print?

2 * 500^3 = 250000000. The line is Matrix checksum: 250000000. The second run with n = 500 prints that same line. n = 1000 prints 2000000000. n = 2000 prints 16000000000.

The time line is your measurement. The checksum is the correctness check. The script’s time limit is one hour. If a larger size is still running at that limit, the job ends. That does not change the checksum of a run that finished.

Parallelise the outer loop

There is no hybrid matrix program in the hands-on set. The open exercise is to parallelise matmul with OpenMP.

The outer loop is the i loop. Row i of C is only written by that iteration. A and B are only read. Threads that each own a different i do not write the same entry. The j and k loops stay inside the thread, so each entry is still a serial dot product. The checksum loop stays serial, after the parallel loop, so it adds the entries in the same order.

The finished file is episodes/files/solutions/matmul-omp.c. Try the change yourself before you read it.

Challenge

OpenMP on the i loop

Copy matmul.c and add OpenMP on the outer loop only. Build with -fopenmp. The published gcc line has no OpenMP flag, so a pragma in the source does nothing until you change the compile line:

BASH 

gcc -O2 -fopenmp matmul-omp.c -o matmul.out

Set the thread count from the allocation, as the hybrid scripts do:

BASH 

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

Run the same n as the sequential checksum. What must Matrix checksum: do?

One core and 16 threads do not run 16 ways at once. --cpus-per-task is the number of cores. Match it to the thread count on a node that has the cores. The checksum check itself does not need 16 cores.

The pragma is on the outer loop:

C 

#pragma omp parallel for
for (size_t i = 0; i < n; i++) {

That is the change in episodes/files/solutions/matmul-omp.c, plus #include <omp.h>. The printf is still Matrix checksum: %.17g.

For a given n, the checksum line matches the sequential run. For n = 500 that is 250000000. A different checksum means two iterations wrote the same C[i][j], or the inner dot product changed. The time line in this file comes from clock, which reports CPU time. Use the checksum to decide if the parallel loop is right.

Key Points
  • C[i][j] is 2 * n for these inputs, and Matrix checksum: is 2 n^3.
  • The same n prints the same checksum. The printf format is %.17g.
  • The sequential compile and run lines are gcc -O2 matmul.c -o matmul.out and srun ./matmul.out $N.
  • The course size-500 line is sbatch --cpus-per-task=1 submit-matmul.sh 500.
  • Parallelise the outer i loop with #pragma omp parallel for, compile with -fopenmp, and leave the checksum unchanged.