How to evaluate a run

  • The life cycle is run, evaluate, change, and run again.
  • time -p prints real, user, and sys.
  • real is the wall-clock seconds you waited.
  • Start with one short timed run before you reach for a profiler.

What limits a run

  • Look at I/O, communication, computation, and memory footprint separately.
  • I/O: time the read and the write. On the deck’s NVMe table, binary read is 2666.4 MB/s and ASCII read is 29.7 MB/s.
  • Communication: speedup falls away from ideal as processes are added. Cost depends on message size, topology, and synchronization.
  • Compute: at fixed total work the ideal speedup is P on P processors.
  • Memory: the footprint can cause cache misses and can force a larger node. valgrind --tool=massif measures it.

Strong and weak scaling

  • Strong scaling fixes the problem size. Weak scaling grows it with the processes.
  • Amdahl: 1 / (s + p/N). Gustafson: s + p*N.
  • Speedup is T_s / T_N. Efficiency is T_s / (N * T_N).
  • 8.0 / (8 * 1.6) = 0.625 on the worked 8-core row.
  • For AXPY, fixed n at 1, 2, 4, and 8 tasks is strong scaling. n scaled by the task count is weak scaling, and that series stays flatter.

The roofline

  • Arithmetic intensity is FLOP/byte. The ridge is where the bandwidth roof meets the peak roof.
  • Left of the ridge is memory-bound. Right of the ridge is compute-bound.
  • An add is 1 FLOP and a multiply is 1 FLOP, so N of each is 2N FLOP. Dense matrix multiplication is about 2 N^3 FLOP, which O(N^3) does not state.
  • The deck’s uProf sample is AMDuProfPcm roofline -X -o res.csv, and 6.7257 / 2.6767 = 2.5 FLOP/byte.
  • The course tab is perf and PAPI, not uProf.

Energy

  • #SBATCH --ear=on and #SBATCH --ear-policy=monitoring turn on EAR monitoring.
  • eacct -j JOBID prints the report. Read POWER (W) and ENERGY (J).
  • The deck’s sample lists 253.62 W and 14366 J.
  • This episode is optional, and it is the first to cut. The course tab says to skip the live job.

Slurm resources

  • 4 MPI ranks and 8 threads each is --ntasks=4 and --cpus-per-task=8. --nodes=1 when the node can hold 32 cores.
  • SLURM_NTASKS is 4 and SLURM_CPUS_PER_TASK is 8 when those options are set.
  • Without --cpus-per-task, Slurm tries to allocate one processor per task.
  • --exclusive does not share nodes. --sockets-per-node and --gpus-per-node constrain sockets and GPUs. --contiguous asks for a contiguous set of nodes.
  • seff JOBID is CPU time used, divided by allocated cores times elapsed time. Billing units are the accounting tab.