OpenMPI on the cluster

OpenMPI implements MPI for parallel computing across processes and nodes. This guide uses a hybrid MPI/OpenMP approximation of pi to demonstrate compilation, allocation, submission, and validation.

Submit parallel calculations through Slurm

Login nodes are for editing, script preparation, and brief checks.

Example files

Public examples:

/hpcfs/fpublic/script_example/openmpitest/

Copy them to your home directory:

cp -r /hpcfs/fpublic/script_example/openmpitest ~/openmpitest
cd ~/openmpitest

Alternatively, create a new working directory:

mkdir -p ~/openmpitest
cd ~/openmpitest

Test program

Save as openmpitest.c:

#include <mpi.h>
#include <omp.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    const long int n = 10000000;
    int rank, size;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    const long int base = n / size;
    const long int remainder = n % size;
    const long int local_n = base + (rank < remainder ? 1 : 0);
    const long int first = rank * base + (rank < remainder ? rank : remainder);
    const long int last = first + local_n;
    const double h = 1.0 / (double)n;
    double local_pi = 0.0;
    double pi = 0.0;

#pragma omp parallel for reduction(+ : local_pi)
    for (long int i = first; i < last; ++i) {
        const double x = (i + 0.5) * h;
        local_pi += 4.0 / (1.0 + x * x);
    }

    MPI_Reduce(&local_pi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0,
               MPI_COMM_WORLD);

    if (rank == 0) {
        printf("MPI processes: %d, OpenMP threads/process: %d\n",
               size, omp_get_max_threads());
        printf("Approximation of pi: %.13f\n", pi * h);
    }

    MPI_Finalize();
    return 0;
}

MPI processes divide the integration interval, OpenMP threads compute each process's share, and MPI_Reduce combines the results.

Slurm script

Save as openmpitest.slurm:

#!/bin/bash
#SBATCH --job-name=openmpitest
#SBATCH --partition=qcpu_18i
#SBATCH --nodes=2
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=00:10:00
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

set -euo pipefail

module purge
module load gcc
module load mpi/openmpi-4.1.6

export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}"
export OMP_PLACES=cores
export OMP_PROC_BIND=close

mpicc -O3 -fopenmp openmpitest.c -o openmpitest
mpirun -np "${SLURM_NTASKS}" ./openmpitest

Option	Value	Meaning
--nodes	2	Two compute nodes
--ntasks	4	Four MPI processes total
--ntasks-per-node	2	Two MPI processes per node
--cpus-per-task	8	Eight CPUs/OpenMP threads per process
--mem	32G	32 GB per node

Each node uses 2 × 8 = 16 CPU cores. Keep resource requests consistent with the process/thread counts. For pure MPI, set --cpus-per-task=1 and adjust task counts as needed.

Submit and inspect

sbatch openmpitest.slurm

Example response:

Submitted batch job 280135

Inspect that job (replace the example ID with your actual ID) or all your jobs:

squeue -j 280135

squeue -u "$USER"

Read its output:

cat openmpitest-280135.out

Expected output is similar to:

MPI processes: 4, OpenMP threads/process: 8
Approximation of pi: 3.1415926535898

Demonstration:

Environment and resource checks

module list
which mpicc
which mpirun
mpicc --version
mpirun --version

Inspect allocated nodes:

scontrol show job 280135

Inspect accounting information:

sacct -j 280135 --format=JobID,State,Elapsed,AllocCPUS,MaxRSS,ExitCode

Troubleshooting

If mpicc is missing, load compatible GCC and OpenMPI modules; use module avail mpi to find the actual installed name. For failures or missing output, inspect the error log (作业号 means job ID), compilation errors, and the working directory containing both source and script:

cat openmpitest-作业号.err

Do not exceed --ntasks with mpirun -np; the example uses SLURM_NTASKS. Keep OMP_NUM_THREADS at or below SLURM_CPUS_PER_TASK. For multi-node startup failures, verify partition support and inspect network/MPI initialization errors. Use the same compatible MPI installation for compilation and execution.