<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hi Cameron,</p>
<p><br>
</p>
<p> I am no expert on QE at all, but I can maybe suggest a few things. It seems QE 5.1 is an old version, I have only used 6.0 so I am unfamiliar with some of the output, particularly the report on memory usage. It could be worth using QE 6.0 at least or
6.1 which is out now to be on par with other users?</p>
<p><br>
</p>
<p>As for 'exit signal 9', if this is the OS (linux) then this is SIGKILL which may have been issued due to running out of memory amongst possibly other reasons. Again I am unfamiliar with this version of QE, but ecutwfc seems particularly high compared to
what I've seen on this forum, and perusing the 'recommended ecutwfc' in pseudopotential files, I don't know if you need this though? QE 6.0 pwscf tells you the total estimated memory per process, which has been very accurate on 4/8/16/32 cpu scf calculations
I have done so far.</p>
<p><br>
</p>
<p>Kindest Regards,</p>
<p>Louis</p>
<p>PhD student, University of Leeds, IMP</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> pw_forum-bounces@pwscf.org <pw_forum-bounces@pwscf.org> on behalf of Cameron Foss <cjfoss@umass.edu><br>
<b>Sent:</b> 14 March 2017 22:55:19<br>
<b>To:</b> pw_forum@pwscf.org<br>
<b>Subject:</b> [Pw_forum] 62 atom interface relaxation</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hello all,
<div><br>
</div>
<div>I am trying to relax a supercell of an graphene/MoS2 interface and have been running into an exit signal 9 error. I have a 62 atom representation of the interface, which consists of a 5x2 supercell of graphene and a 4x2 supercell of MoS2. Below are the
submission script, vc-relax input file, and output file.<br>
</div>
<div><br>
</div>
<div>From what I understand, the exit signal 9 is a sort of load balancing error (?). I have worked with much smaller systems (generally unit cells with less than 10 atoms) with relatively good success using 16 MPI processes. However, I have no experience with
a simulation of this scale. I have tried increasing to 32 or 40 MPI processes but only to get the same exit 9 signal. I should note that, in the past I have only made use of PW parallelization and haven't used parallelization over images or k-points as I
suspect may be necessary for this simulation. </div>
<div><br>
</div>
<div>My first question is that, is my above assessment accurate in that the exit signal 9 is (in this case) a load balancing issue and that it can be resolved with the appropriate resources for PW, images, and k-point parallelization?</div>
<div><br>
</div>
<div>And second, I only have access to say 200 or so cpu cores (equating to about 200 MPI processes), is that a reasonable range of resources that can handle a simulation of this size? I apologize for the rather naive question but I have very little experience
with simulations of this size and am unfamiliar with how one should scale resources accordingly. Perhaps a better question would be how should one digest the 'Parallelization Info' in the output file in order to get a feel for the best distribution of resources?
(perhaps a good resource material would suffice for this question but what I've found on the topic has been difficult to apply to this case).</div>
<div><br>
</div>
<div><b>% submission line : bsub -n 40 < ./myscript</b></div>
<div><b>% submission script</b></div>
<div>
<div>#!/bin/sh<br>
</div>
<div>#BSUB -J GrMoS2PHlda</div>
<div>#BSUB -o grmos2-lda.out</div>
<div>#BSUB -e grmos2-lda.err</div>
<div>#BSUB -q long</div>
<div>#BSUB -W 96:00</div>
<div>#BSUB -R select[ncpus=20]</div>
<div>#BSUB -R "span[ptile=10]"</div>
<div>#BSUB -R rusage[mem=10000]</div>
<div>#BSUB -L /bin/sh</div>
<div><br>
</div>
<div>export OMP_NUM_THREADS=1</div>
<div>module load gcc/4.7.4</div>
<div>module load mvapich2/2.0a</div>
<div><br>
</div>
<div>mpirun -n 32 ~/espresso-par/espresso-5.1.2/bin/pw.x <<a href="http://gr_mos2.vc-relax.reduced.in">gr_mos2.vc-relax.reduced.in</a>> gr_mos2.vc-relax.out</div>
</div>
<div><br>
</div>
<div>%%%%%%%%%%%</div>
<div><b>pw input file</b></div>
<div>
<div> 1 &control</div>
<div> 2 calculation='vc-relax',</div>
<div> 3 restart_mode='from_scratch',</div>
<div> 4 outdir='/project/uma_zlatan_aksamija/out_grmos2',</div>
<div> 5 prefix='GrMoS2-LDA3'</div>
<div> 6 pseudo_dir = '/home/cf79a/espresso-par/espresso-5.1.2/pseudo',</div>
<div> 7 /</div>
<div> 8 &system</div>
<div> 9 ibrav = 0,</div>
<div> 10 celldm(1) =4.6477,</div>
<div> 11 !celldm(3)=2,</div>
<div> 12 nat= 62,</div>
<div> 13 ntyp= 3,</div>
<div> 14 ecutwfc = 250.0</div>
<div> 15 /</div>
<div> 16 &electrons</div>
<div> 17 mixing_beta = 0.7</div>
<div> 18 conv_thr = 1.0d-8</div>
<div> 19 /</div>
<div> 20 &ions</div>
<div> 21 ion_dynamics='bfgs',</div>
<div> 22 /</div>
<div> 23 &cell</div>
<div> 24 cell_dynamics='bfgs',</div>
<div> 25 /</div>
<div> 26 ATOMIC_SPECIES</div>
<div> 27 C 12.01 C.pw-mt_fhi.UPF</div>
<div> 28 Mo 95.94 Mo.pw-mt_fhi.UPF</div>
<div> 29 S 32.065 S.pw-mt_fhi.UPF</div>
<div> 30 ATOMIC_POSITIONS {alat}</div>
<div> 31 C 0.00000000000 0.00000000000 0.63039440618</div>
<div> 32 C 0.50000000000 0.28867281200 0.63039440618</div>
<div> 33 C 1.00000000000 0.00000000000 0.63039440618</div>
<div> 34 C 1.50000000000 0.28867281200 0.63039440618</div>
</div>
<div>
<div> 35 C 0.50000000000 0.86602500000 0.63039440618</div>
<div> 36 C 2.00000000000 0.00000000000 0.63039440618</div>
<div> 37 C 2.50000000000 0.28867281200 0.63039440618</div>
<div> 38 C 1.50000000000 0.86602500000 0.63039440618</div>
<div> 39 C 0.00000000000 1.15469781200 0.63039440618</div>
<div> 40 C 1.00000000000 1.15469781200 0.63039440618</div>
<div> 41 C 2.00000000000 1.15469781200 0.63039440618</div>
<div> 42 C 0.00000000000 1.73205000000 0.63039440618</div>
<div> 43 C 1.00000000000 1.73205000000 0.63039440618</div>
<div> 44 C 1.50000000000 2.02072321400 0.63039440618</div>
<div> 45 C 1.00000000000 2.88674861800 0.63039440618</div>
<div> 46 C 0.50000000000 3.75277402100 0.63039440618</div>
<div> 47 C 0.50000000000 2.59807540400 0.63039440618</div>
<div> 48 C 0.00000000000 3.46410080800 0.63039440618</div>
<div> 49 C 0.50000000000 2.02072321400 0.63039440618</div>
<div> 50 C 0.00000000000 2.88674861800 0.63039440618</div>
<div> 51 C -0.50000000000 3.75277402200 0.63039440618</div>
<div> 52 C -0.50000000000 2.59807540400 0.63039440618</div>
<div> 53 C -1.00000000000 3.46410080800 0.63039440618</div>
<div> 54 C -0.50000000000 2.02072321400 0.63039440618</div>
<div> 55 C -1.00000000000 2.88674861800 0.63039440618</div>
<div> 56 C -1.50000000000 3.75277402100 0.63039440618</div>
<div> 57 C -0.50000000000 0.86602500000 0.63039440618</div>
<div> 58 C -1.00000000000 1.73205000000 0.63039440618</div>
<div> 59 C -1.50000000000 2.59807540000 0.63039440618</div>
<div> 60 C -2.00000000000 3.46410081000 0.63039440618</div>
<div> 61 S 2.00000000000 2.17500376000 0.00000000000</div>
<div> 62 S 2.00000000000 2.17500376000 1.26078881236</div>
<div> 63 S 1.37213049000 3.26250564000 0.00000000000</div>
</div>
<div>
<div> 64 S 1.37213049000 3.26250564000 1.26078881236</div>
<div> 65 S 3.25573901037 0.00000000000 0.00000000000</div>
<div> 66 S 3.25573901037 0.00000000000 1.26078881236</div>
<div> 67 S 2.62786950519 1.08750188350 0.00000000000</div>
<div> 68 S 2.62786950519 1.08750188350 1.26078881236</div>
<div> 69 S 4.51147802074 0.00000000000 0.00000000000</div>
<div> 70 S 4.51147802074 0.00000000000 1.26078881236</div>
<div> 71 S 3.88360851556 1.08750188350 0.00000000000</div>
<div> 72 S 3.88360851556 1.08750188350 1.26078881236</div>
<div> 73 S 5.76721702400 0.00000000000 0.00000000000</div>
<div> 74 S 5.76721702400 0.00000000000 1.26078881236</div>
<div> 75 S 5.13934752100 1.08750188350 0.00000000000</div>
<div> 76 S 5.13934752100 1.08750188350 1.26078881236</div>
<div> 77 S 3.25573901000 2.17500376000 0.00000000000</div>
<div> 78 S 3.25573901000 2.17500376000 1.26078881236</div>
<div> 79 S 2.62786950000 3.26250564000 0.00000000000</div>
<div> 80 S 2.62786950000 3.26250564000 1.26078881236</div>
<div> 81 S 4.51147801000 2.17500376000 0.00000000000</div>
<div> 82 S 4.51147801000 2.17500376000 1.26078881236</div>
<div> 83 S 3.88360850000 3.26250564000 0.00000000000</div>
<div> 84 S 3.88360850000 3.26250564000 1.26078881236</div>
<div> 85 Mo 3.88360851555 0.36249771126 0.63039440618</div>
<div> 86 Mo 5.13934752593 0.36249771126 0.63039440618</div>
<div> 87 Mo 3.25573901037 1.44999959476 0.63039440618</div>
<div> 88 Mo 4.51147802074 1.44999959476 0.63039440618</div>
<div> 89 Mo 2.62786950000 2.53750147400 0.63039440618</div>
<div> 90 Mo 3.88360850500 2.53750147400 0.63039440618</div>
<div> 91 Mo 2.00000000000 3.62500335400 0.63039440618</div>
<div> 92 Mo 3.25573899500 3.62500335400 0.63039440618</div>
<div> 93 K_POINTS AUTOMATIC</div>
<div> 94 27 27 1 1 1 1</div>
<div> 95 CELL_PARAMETERS alat</div>
</div>
<div>
<div> 96 6.5 0.00000000000 0.0</div>
<div> 97 -2.5 4.330127018900 0.0</div>
<div> 98 0.0 0.000000000000 6.0</div>
</div>
<div><b>%%%%%%%%%%%%%</b></div>
<div><b><br>
</b></div>
<div><b>And the output file</b>:</div>
<div>
<div> 1</div>
<div> 2 Program PWSCF v.5.1.2 starts on 10Mar2017 at 18:29:19</div>
<div> 3</div>
<div> 4 This program is part of the open-source Quantum ESPRESSO suite</div>
<div> 5 for quantum simulation of materials; please cite</div>
<div> 6 "P. Giannozzi et al., J. Phys.:Condens. Matter 21 395502 (2009);</div>
<div> 7 URL <a href="http://www.quantum-espresso.org">http://www.quantum-espresso.org</a>",</div>
<div> 8 in publications or presentations arising from this work. More details at</div>
<div> 9 <a href="http://www.quantum-espresso.org/quote">http://www.quantum-espresso.org/quote</a></div>
<div> 10</div>
<div> 11 Parallel version (MPI & OpenMP), running on 32 processor cores</div>
<div> 12 Number of MPI processes: 32</div>
<div> 13 Threads/MPI process: 1</div>
<div> 14 R & G space division: proc/nbgrp/npool/nimage = 32</div>
<div> 15 Waiting for input...</div>
<div> 16 Reading input from standard input</div>
<div> 17</div>
<div> 18 Current dimensions of program PWSCF are:</div>
<div> 19 Max number of different atomic species (ntypx) = 10</div>
<div> 20 Max number of k-points (npk) = 400000</div>
<div> 21 Max angular momentum in pseudopotentials (lmaxx) = 3</div>
<div> 22 file C.pw-mt_fhi.UPF: wavefunction(s) 4f renormalized</div>
<div> 23 file Mo.pw-mt_fhi.UPF: wavefunction(s) 4f renormalized</div>
<div> 24 file S.pw-mt_fhi.UPF: wavefunction(s) 4f renormalized</div>
<div> 25</div>
<div> 26 Subspace diagonalization in iterative solution of the eigenvalue problem:</div>
<div> 27 a serial algorithm will be used</div>
<div> 28</div>
<div> 29</div>
<div> 30 Parallelization info</div>
<div> 31 --------------------</div>
<div> 32 sticks: dense smooth PW G-vecs: dense smooth PW</div>
<div> 33 Min 1511 1511 391 282925 282925 37169</div>
<div> 34 Max 1512 1512 392 282930 282930 37173</div>
</div>
<div>
<div> 35 Sum 48359 48359 12513 9053647 9053647 1189491</div>
<div> 36</div>
<div> 37</div>
<div> 38</div>
<div> 39 bravais-lattice index = 0</div>
<div> 40 lattice parameter (alat) = 4.6477 a.u.</div>
<div> 41 unit-cell volume = 16954.2860 (a.u.)^3</div>
<div> 42 number of atoms/cell = 62</div>
<div> 43 number of atomic types = 3</div>
<div> 44 number of electrons = 312.00</div>
<div> 45 number of Kohn-Sham states= 156</div>
<div> 46 kinetic-energy cutoff = 250.0000 Ry</div>
<div> 47 charge density cutoff = 1000.0000 Ry</div>
<div> 48 convergence threshold = 1.0E-08</div>
<div> 49 mixing beta = 0.7000</div>
<div> 50 number of iterations used = 8 plain mixing</div>
<div> 51 Exchange-correlation = SLA-PW ( 1 4 0 0 0 0)</div>
<div> 52 nstep = 50</div>
<div> 53</div>
<div> 54</div>
<div> 55 celldm(1)= 4.647700 celldm(2)= 0.000000 celldm(3)= 0.000000</div>
<div> 56 celldm(4)= 0.000000 celldm(5)= 0.000000 celldm(6)= 0.000000</div>
<div> 57</div>
<div> 58 crystal axes: (cart. coord. in units of alat)</div>
<div> 59 a(1) = ( 6.500000 0.000000 0.000000 )</div>
<div> 60 a(2) = ( -2.500000 4.330127 0.000000 )</div>
<div> 61 a(3) = ( 0.000000 0.000000 6.000000 )</div>
<div> 62</div>
<div> 63 reciprocal axes: (cart. coord. in units 2 pi/alat)</div>
<div> 64 b(1) = ( 0.153846 0.088823 -0.000000 )</div>
<div> 65 b(2) = ( 0.000000 0.230940 0.000000 )</div>
<div> 66 b(3) = ( 0.000000 -0.000000 0.166667 )</div>
<div> 67</div>
<div> 68</div>
</div>
<div>
<div> 69 PseudoPot. # 1 for C read from file:</div>
<div> 70 /home/cf79a/espresso-par/espresso-5.1.2/pseudo/C.pw-mt_fhi.UPF</div>
<div> 71 MD5 check sum: 13ff76dfb29e90984447383659675a25</div>
<div> 72 Pseudo is Norm-conserving, Zval = 4.0</div>
<div> 73 Generated using FHI98PP, converted with fhi2upf.x v.5.0.1</div>
<div> 74 Using radial grid of 461 points, 3 beta functions with:</div>
<div> 75 l(1) = 0</div>
<div> 76 l(2) = 1</div>
<div> 77 l(3) = 3</div>
<div> 78</div>
<div> 79 PseudoPot. # 2 for Mo read from file:</div>
<div> 80 /home/cf79a/espresso-par/espresso-5.1.2/pseudo/Mo.pw-mt_fhi.UPF</div>
<div> 81 MD5 check sum: a86585854cc5d78a2dd10559b26ab9a5</div>
<div> 82 Pseudo is Norm-conserving, Zval = 6.0</div>
<div> 83 Generated using FHI98PP, converted with fhi2upf.x v.5.0.1</div>
<div> 84 Using radial grid of 541 points, 3 beta functions with:</div>
<div> 85 l(1) = 0</div>
<div> 86 l(2) = 2</div>
<div> 87 l(3) = 3</div>
<div> 88</div>
<div> 89 PseudoPot. # 3 for S read from file:</div>
<div> 90 /home/cf79a/espresso-par/espresso-5.1.2/pseudo/S.pw-mt_fhi.UPF</div>
<div> 91 MD5 check sum: d54d28406084323b3c894f38c8f5e215</div>
<div> 92 Pseudo is Norm-conserving, Zval = 6.0</div>
<div> 93 Generated using FHI98PP, converted with fhi2upf.x v.5.0.1</div>
<div> 94 Using radial grid of 501 points, 3 beta functions with:</div>
<div> 95 l(1) = 0</div>
<div> 96 l(2) = 1</div>
<div> 97 l(3) = 3</div>
<div> 98</div>
<div> 99 atomic species valence mass pseudopotential</div>
<div>100 C 4.00 12.01000 C( 1.00)</div>
<div>101 Mo 6.00 95.94000 Mo( 1.00)</div>
<div>102 S 6.00 32.06500 S( 1.00)</div>
</div>
<div>
<div>103</div>
<div>104 No symmetry found</div>
<div>105 (note: 1 additional sym.ops. were found but ignored</div>
<div>106 their fractional translations are incommensurate with FFT grid)</div>
<div>107</div>
<div>108</div>
<div>109 Cartesian axes</div>
<div>110</div>
<div>111 site n. atom positions (alat units)</div>
<div>112 1 C tau( 1) = ( 0.0000000 0.0000000 0.6303944 )</div>
<div>
<div>113 2 C tau( 2) = ( 0.5000000 0.2886728 0.6303944 )</div>
<div>114 3 C tau( 3) = ( 1.0000000 0.0000000 0.6303944 )</div>
</div>
<div><b>................skipped for sake of redundancy</b><br>
</div>
</div>
<div>
<div>172 61 Mo tau( 61) = ( 2.0000000 3.6250034 0.6303944 )</div>
<div>173 62 Mo tau( 62) = ( 3.2557390 3.6250034 0.6303944 )</div>
<div>174</div>
<div>175 number of k points= 365</div>
<div>176</div>
<div>177 Number of k-points >= 100: set verbosity='high' to print them.</div>
<div>178</div>
<div>179 Dense grid: 9053647 G-vectors FFT dimensions: ( 320, 240, 288)</div>
<div>180</div>
<div>181 Largest allocated arrays est. size (Mb) dimensions</div>
<div>182 Kohn-Sham Wavefunctions 84.31 Mb ( 35420, 156)</div>
<div>183 NL pseudopotentials 377.25 Mb ( 35420, 698)</div>
<div>184 Each V/rho on FFT grid 10.55 Mb ( 691200)</div>
<div>185 Each G-vector array 2.16 Mb ( 282930)</div>
<div>186 G-vector shells 2.16 Mb ( 282930)</div>
<div>187 Largest temporary arrays est. size (Mb) dimensions</div>
<div>188 Auxiliary wavefunctions 337.25 Mb ( 35420, 624)</div>
<div>189 Each subspace H/S matrix 5.94 Mb ( 624, 624)</div>
<div>190 Each <psi_i|beta_j> matrix 1.66 Mb ( 698, 156)</div>
<div>191 Arrays for rho mixing 84.38 Mb ( 691200, 8)</div>
<div>192</div>
<div>193 Initial potential from superposition of free atoms</div>
<div>194 Check: negative starting charge= -0.137438</div>
<div>195</div>
<div>196 starting charge 311.98520, renormalised to 312.00000</div>
<div>197</div>
<div>198 negative rho (up, down): 1.374E-01 0.000E+00</div>
<div>199 Starting wfc are 992 randomized atomic wfcs</div>
<div>200</div>
<div>201 ===================================================================================</div>
<div>202 = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</div>
</div>
<div>
<div>203 = EXIT CODE: 9</div>
<div>204 = CLEANING UP REMAINING PROCESSES</div>
<div>205 = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</div>
<div>206 ===================================================================================</div>
</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Cameron </div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</body>
</html>