<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="en-DE" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="en-DE"># Summary<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">QE 7.0/GPU compilation with Autotools fails with "Can't find include file fftw3.f03 (fft_scalar.FFTW3.f90: 40)"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"># Version<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">qe-7.0-ReleasePack.tgz<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"># Environment<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">## Hardware<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">1. 2xAMD EPYC 7452<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">2. 4xNVIDIA A100<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">3. 512 GB RAM<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">## Software<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">1. OS: Rocky Linux release 8.5 (Green Obsidian)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">2. NVHPC 22.3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">3. OpenMPI 4.1.3 built with NVHPC 22.3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">4. CUDA 11.3.1 with Driver 470.82.01<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">5. libxc 5.1.5<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">6. Autotools 20210128<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"># Steps to reproduce<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">## Configured with:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">`./configure --prefix=$HOME/install/qe-7.0 --enable-openmp --with-scalapack=no --enable-cuda-env-check=no --with-cuda=$EBROOTNVHPC/Linux_x86_64/22.3/cuda/11.0 --with-cuda-runtime=11.0 --with-cuda-cc=80`<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">## Prebuild options<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">`cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export FCPP='cpp -E' && `<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">## make options<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">`make all epw`<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"># Observed behavior<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">Compilation fails with:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">make[1]: Entering directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_param.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_support.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c stick_base.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_types.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c scatter_mod.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scatter_2d.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_buffers.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scatter_gpu.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scatter_2d_gpu.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_ggen.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scalar.DFTI.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scalar.ESSL.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN-W-0006-Input file empty (fft_scalar.ESSL.f90)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fftw_interfaces.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scalar.FFTW.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN-W-0006-Input file empty (fft_scalar.FFTW.f90)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
-acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c fft_scalar.FFTW3.f90<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN-F-0906-Can't find include file fftw3.f03 (fft_scalar.FFTW3.f90: 40)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">NVFORTRAN/x86-64 Linux 22.3-0: compilation aborted<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">make[1]: *** [../make.inc:16: fft_scalar.FFTW3.o] Error 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">make[1]: Leaving directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">make: *** [Makefile:196: libfft] Error 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">```<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"># Questions<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE">1. What do I do wrong?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2</span><span lang="en-DE">. Why configure didn't fail if FFTW is mandatory?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">3</span><span lang="en-DE">. Is it possible to configure
</span><span lang="HR">the build system </span><span lang="en-DE">to use cuFFT</span><span lang="EN-US"> instead</span><span lang="en-DE">?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="LB-LU" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000">Dr. rer. nat. Robert Mijaković </span><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00B0F0;mso-fareast-language:#2000">|</span><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"> HPC
System Software Architect</span><span lang="en-DE" style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"><br>
</span><b><span lang="LB-LU" style="font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">Lux</span></b><b><span lang="LB-LU" style="font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000">Provide</span></b><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"><br>
</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">3, Op der Poukewiss</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000"> |</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"> L-7795
Bissen</span><span lang="en-DE" style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">Grand-Duchy of Luxembourg<br>
</span><span lang="LB-LU" style="font-size:9.0pt;color:#595959;mso-fareast-language:#2000">M (+352) 691 396 474<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:#2000"><a href="mailto:robert.mijakovic@lxp.lu"><span lang="LB-LU" style="color:blue">robert.mijakovic@lxp.lu</span></a></span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000"> |<span style="background:white"> </span></span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0563C1;background:white;mso-fareast-language:#2000"><a href="http://www.luxprovide.lu/" target="_blank"><span style="color:blue">www.luxprovide.lu</span></a></span><span lang="LB-LU" style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="en-DE"><o:p> </o:p></span></p>
</div>
</body>
</html>