<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-2">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Robert,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think it’s an issue with PGI in NVHPC 22.3. Please use NVHPC 21.9 instead (PGI compiler).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Normally the configure script should catch it and prevent using NVHPC 22.3, unfortunately the check has been introduced only after QE 7.0.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best,<o:p></o:p></p>
<p class="MsoNormal">Louis<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> users <users-bounces@lists.quantum-espresso.org>
<b>On Behalf Of </b>Robert MIJAKOVIC<br>
<b>Sent:</b> Monday, May 23, 2022 5:23 PM<br>
<b>To:</b> users@lists.quantum-espresso.org<br>
<b>Subject:</b> [QE-users] [QE 7.0/GPU] Run built with CMake fails with cuFFT error<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121">Some people who received this message don't often get email from
<a href="mailto:robert.mijakovic@lxp.lu">robert.mijakovic@lxp.lu</a>. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a><o:p></o:p></span></p>
</div>
</td>
<td width="75" style="width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;align:left">
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="1" cellspacing="5" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:black">
</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"># Summary<o:p></o:p></p>
<p class="MsoNormal">QE 7.0/GPU compiled with CMake fails on our system in "routine fft_scalar_cuFFT: cft_1z_gpu (8)".<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Version<o:p></o:p></p>
<p class="MsoNormal">qe-7.0-ReleasePack.tgz<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Environment<o:p></o:p></p>
<p class="MsoNormal">## Hardware<o:p></o:p></p>
<p class="MsoNormal">1. 2xAMD EPYC 7452<o:p></o:p></p>
<p class="MsoNormal">2. 4xNVIDIA A100<o:p></o:p></p>
<p class="MsoNormal">3. 512 GB RAM<o:p></o:p></p>
<p class="MsoNormal">## Software<o:p></o:p></p>
<p class="MsoNormal">1. OS: Rocky Linux release 8.5 (Green Obsidian)<o:p></o:p></p>
<p class="MsoNormal">2. NVHPC 22.3<o:p></o:p></p>
<p class="MsoNormal">3. OpenMPI 4.1.3 built with NVHPC 22.3<o:p></o:p></p>
<p class="MsoNormal">4. CUDA 11.3.1 with Driver 470.82.01<o:p></o:p></p>
<p class="MsoNormal">5. libxc 5.1.5<o:p></o:p></p>
<p class="MsoNormal">6. CMake 3.20.1<o:p></o:p></p>
<p class="MsoNormal">7. M4 1.4.19<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Steps to reproduce<o:p></o:p></p>
<p class="MsoNormal">## Configured with:<o:p></o:p></p>
<p class="MsoNormal">`-DQE_ENABLE_CUDA=1 -DQE_FFTW_VENDOR=Internal -DQE_ENABLE_LIBXC=1 -DQE_ENABLE_OPENMP=1 `<o:p></o:p></p>
<p class="MsoNormal">## Prebuild options<o:p></o:p></p>
<p class="MsoNormal">`cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export FCPP='cpp -E' && `<o:p></o:p></p>
<p class="MsoNormal">## make options<o:p></o:p></p>
<p class="MsoNormal">`make all epw`<o:p></o:p></p>
<p class="MsoNormal">## Execute<o:p></o:p></p>
<p class="MsoNormal">srun <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">## Input files<o:p></o:p></p>
<p class="MsoNormal"><span lang="HR">QEF AUSURF112 benchmark<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Observed behaviour<o:p></o:p></p>
<p class="MsoNormal">When example is started it fails with:<o:p></o:p></p>
<p class="MsoNormal">```<o:p></o:p></p>
<p class="MsoNormal">%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%<o:p></o:p></p>
<p class="MsoNormal">Error in routine fft_scalar_cuFFT: cft_1z_gpu (8):<o:p></o:p></p>
<p class="MsoNormal">cufftPlanMany failed<o:p></o:p></p>
<p class="MsoNormal">```<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Questions<o:p></o:p></p>
<p class="MsoNormal">1. What do I do wrong?<o:p></o:p></p>
<p class="MsoNormal">2. Why is there no option to set FFTW_VENDOR to cuFFT?<o:p></o:p></p>
<p class="MsoNormal">3. Why it got linked against cuFFT if FFTW_VENDOR is set to Internal?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white"><span lang="LB-LU" style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000">Dr. rer. nat. Robert Mijakoviæ </span><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#00B0F0;mso-fareast-language:#2000">|</span><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"> HPC
System Software Architect</span><span style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"><br>
</span><b><span lang="LB-LU" style="font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">Lux</span></b><b><span lang="LB-LU" style="font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000">Provide</span></b><span lang="LB-LU" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"><br>
</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">3, Op der Poukewiss</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000"> |</span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000"> L-7795
Bissen</span><span style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#414141;mso-fareast-language:#2000">Grand-Duchy of Luxembourg<br>
</span><span lang="LB-LU" style="font-size:9.0pt;color:#595959;mso-fareast-language:#2000">M (+352) 691 396 474<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:#2000"><a href="mailto:robert.mijakovic@lxp.lu"><span lang="LB-LU" style="color:blue">robert.mijakovic@lxp.lu</span></a></span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#41BEF0;mso-fareast-language:#2000"> |<span style="background:white"> </span></span><span lang="LB-LU" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0563C1;background:white;mso-fareast-language:#2000"><a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.luxprovide.lu%2F&data=05%7C01%7Clstuber%40nvidia.com%7C92950777c584476ead5508da3cd03d33%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637889163050914322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ofAyQvxQ6OinQiA5HgsBEO6nQfPosHKHdTnXt17ld4Y%3D&reserved=0" target="_blank"><span style="color:blue">www.luxprovide.lu</span></a></span><span lang="LB-LU" style="mso-fareast-language:#2000"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>