[QE-users] [QE 7.0/GPU] Build configured with Autotools fails

Mon May 23 17:55:49 CEST 2022

# Summary
QE 7.0/GPU compilation with Autotools fails with "Can't find include file fftw3.f03 (fft_scalar.FFTW3.f90: 40)"

# Version
qe-7.0-ReleasePack.tgz

# Environment
## Hardware
1. 2xAMD EPYC 7452
2. 4xNVIDIA A100
3. 512 GB RAM
## Software
1. OS: Rocky Linux release 8.5 (Green Obsidian)
2. NVHPC 22.3
3. OpenMPI 4.1.3 built with NVHPC 22.3
4. CUDA 11.3.1 with Driver 470.82.01
5. libxc 5.1.5
6. Autotools 20210128

# Steps to reproduce
## Configured with:
`./configure --prefix=$HOME/install/qe-7.0 --enable-openmp --with-scalapack=no --enable-cuda-env-check=no --with-cuda=$EBROOTNVHPC/Linux_x86_64/22.3/cuda/11.0 --with-cuda-runtime=11.0 --with-cuda-cc=80`
## Prebuild options
`cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export FCPP='cpp -E' && `
## make options
`make all epw`

# Observed behavior
Compilation fails with:
```
make[1]: Entering directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_param.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_support.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c stick_base.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_types.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c scatter_mod.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scatter_2d.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_buffers.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scatter_gpu.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scatter_2d_gpu.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_ggen.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scalar.DFTI.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scalar.ESSL.f90
NVFORTRAN-W-0006-Input file empty (fft_scalar.ESSL.f90)
NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fftw_interfaces.f90
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scalar.FFTW.f90
NVFORTRAN-W-0006-Input file empty (fft_scalar.FFTW.f90)
NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c fft_scalar.FFTW3.f90
NVFORTRAN-F-0906-Can't find include file fftw3.f03 (fft_scalar.FFTW3.f90: 40)
NVFORTRAN/x86-64 Linux 22.3-0: compilation aborted
make[1]: *** [../make.inc:16: fft_scalar.FFTW3.o] Error 2
make[1]: Leaving directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
make: *** [Makefile:196: libfft] Error 1
```

# Questions
1. What do I do wrong?
2. Why configure didn't fail if FFTW is mandatory?
3. Is it possible to configure the build system to use cuFFT instead?

Dr. rer. nat. Robert Mijaković | HPC System Software Architect

LuxProvide
3, Op der Poukewiss | L-7795 Bissen
Grand-Duchy of Luxembourg
M (+352) 691 396 474
robert.mijakovic at lxp.lu<mailto:robert.mijakovic at lxp.lu> | www.luxprovide.lu<http://www.luxprovide.lu/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20220523/6c1529c7/attachment.html>