Why use a Cluster?

  • High-Performance Computing (HPC) typically involves connecting to very large computing systems elsewhere in the world.
  • These other systems can be used to do work that would either be impossible or much slower on smaller systems.
  • HPC resources are shared by multiple users.
  • The standard method of interacting with such systems is via a command line interface.

Connecting Securely to an HPC Cluster

  • HPC clusters are shared computational resources that users access remotely.
  • Secure Shell (SSH) provides encrypted communication between your local computer and an HPC cluster.
  • SSH public-private key pairs provide a secure method for authenticating to remote systems.
  • A private key remains on your local computer and should be protected with a strong passphrase.
  • An SSH agent can temporarily retain a private-key identity so that its passphrase does not need to be entered on every connection.
  • Users typically connect to an HPC system through a login node, where they authenticate, manage files, configure their environment, and prepare their work.

Working on a remote HPC system

  • An HPC system is a set of networked machines.
  • HPC systems typically provide login nodes and a set of worker nodes.
  • The resources found on independent (worker) nodes can vary in volume and type (amount of RAM, processor architecture, availability of network-mounted filesystems, etc.).
  • Files saved on one node are typically accessible from all nodes via a shared filesystem.

Scheduler Fundamentals

  • The scheduler handles how compute resources are shared between users.
  • A job is just a shell script.
  • Request slightly more resources than you will need.

Environment Variables

  • Shell variables are by default treated as strings
  • Variables are assigned using “=” and recalled using the variable’s name prefixed by “$”
  • Use “export” to make an variable available to other programs
  • The PATH variable defines the shell’s search path

Accessing software via Modules

  • Load software with module load softwareName.
  • Unload software with module unload
  • The module system handles software versioning and package conflicts for you automatically.

Transferring files with remote computers

  • wget -O and curl -o download a file from the internet.
  • scp and rsync transfer files to and from your computer.
  • Users can install custom software on their accounts.
  • The system can help meet software dependencies.

Running a parallel job

  • Parallel programming allows applications to take advantage of parallel hardware.
  • The queuing system facilitates executing parallel tasks.
  • Performance improvements from parallel execution do not scale linearly.

Parallelising with Job Arrays

  • Parallel programming allows applications to take advantage of parallel hardware.
  • The queuing system facilitates executing parallel tasks.
  • Parallel computing allows applications to distribute the workload over several CPUs or nodes
  • There are multiple parallelization strategies that are generally supported by resource managers.
  • Array parallel jobs are suitable for independent runs of the same executable with varying inputs or outputs.

Using resources effectively

  • Accurate job scripts help the queuing system efficiently allocate shared resources.

Using shared resources responsibly

  • Be careful how you use the login node.
  • Your data on the system is your responsibility.
  • Plan and test large data transfers.
  • It is often best to convert many files to a single archive file before transferring.

Accounting on Snellius

Python and OpenMP on a node