[QE-users] [QE-GPU] High GPU oversubscription detected

[Yin-Ying Ting]

Dear Quantum Espresso Experts,

I am encountering an unexpected message regarding GPU oversubscription when running pw.x with GPU on an HPC system. The system has 4 GPU cores per node. I did not receive any warnings when running calculations with 1 node and 1 GPU (ntasks=1, npools=1) or with 1 node and 2 GPUs (ntasks=2, npools=2). However, upon running with 1 node and 4 GPUs (ntasks=4, npools=4), I receive the following message: "Message from routine print_cuda_info: High GPU oversubscription detected. Are you sure this is what you want?"

Here's the SLURM batch script I used:

#!/bin/bash -x
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --output=TbHf.%j
#SBATCH --error=mpi-err.%j
#SBATCH --gres=gpu:4

export OMP_NUM_THREADS=1

srun ~/qe-7.2-gpu/bin/pw.x -nk 4 -nb 1 -nt 1 -nd 1 < inp_pwscf > out_pwscf


Based on the environment.f90 file, this message is triggered when nproc > ndev * nnode * 2. If I understand correctly, I have nproc (Number of parallel processe)=4, ndev(Number of GPU Devices per Node) =4 and nnode (Number of Nodes)=1. This condition seems to be false (4 > 8). Despite this, the message still appears. All 4 GPUs were active during the run.

Could you please help me understand why this message is appearing under these conditions? Any insights or suggestions to address this would be greatly appreciated.

Thank you in advance for your help!


PhD student in Forschungszentrum Jülich

Yin-Ying Ting


--

Forschungszentrum Jülich GmbH
Institute of Energy and Climate Research
Theory and Computation of Energy Materials (IEK-13)
E-mail: y.ting at fz-juelich.de<mailto:y.ting at fz-juelich.de>


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Jülich GmbH
52425 Jülich
Sitz der Gesellschaft: Jülich
Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Stefan Müller
Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende),
Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20231127/d77f57e1/attachment.html>