<div dir="ltr"><div>Is kunit=1 ?<br>Is your nkf and nkf_tot the same as nks nks_tot provided by QE environment?<br></div><div>Could you print the nbase for each processor and see if they are the numbers expected?<br></div><div><br></div>Ye<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div>
<br><div class="gmail_quote">2016-10-30 18:59 GMT-05:00 Vahid Askarpour <span dir="ltr"><<a href="mailto:vh261281@dal.ca" target="_blank">vh261281@dal.ca</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>I allocate an array that needs to be calculated among pools using:</div>
<div><br>
</div>
<div>ALLOCATE(A(3,nbnd,nkf)), where nkf is the number of k points per pool and initialized as:</div>
<div>A(:,:,:)=0</div>
<div><br>
</div>
<div>Once array A is calculated, I allocate array A_tot as:</div>
<div><br>
</div>
<div>ALLOCATE(A_tot(3,nbnd,nkf_tot)<wbr>), where nkf_tot is the total number of k points. and initialized as:</div>
<div>A_tot(:,:,:)=0</div>
<div><br>
</div>
<div>Then I call the poolgather1 as below:</div>
<div><br>
</div>
<div>#if defined(__MPI)</div>
<div>CALL poolgather1(3,nbnd,nkf_tot,<wbr>nkf,A,A_tot)</div>
<div>#else</div>
<div>A_tot=A</div>
<div>#endif</div>
<div><br>
</div>
<div>I have tried both allocating A_tot outside and inside the #if defined section.</div>
<div><br>
</div>
<div>Poolgather1 is similar to other EPW poolgather and QE poolcollect routines and is given below:</div>
<div><br>
</div>
<div>subroutine poolgather1 (nsize1,nsize2,nkstot,nks, f_in, f_out)<br>
!---------------------------<wbr>------------------------------<wbr>-----------<br>
USE kinds, ONLY : DP<br>
USE mp_global, ONLY : my_pool_id, inter_pool_comm, npool,kunit<br>
USE mp, ONLY : mp_barrier, mp_bcast,mp_sum<br>
USE mp_world, ONLY : mpime<br>
implicit none<br>
!<br>
INTEGER, INTENT (in) :: nsize1<br>
!! first dimension of vectors f_in and f_out<br>
INTEGER, INTENT (in) :: nsize2<br>
!! second dimension of vectors f_in and f_out<br>
INTEGER, INTENT (in) :: nks<br>
!! number of k-points per pool<br>
INTEGER, INTENT (in) :: nkstot<br>
!! total number of k-points <br>
REAL (KIND=DP), INTENT (in) :: f_in(nsize1,nsize2,nks)<br>
! input ( only for k-points of mypool )<br>
REAL (KIND=DP), INTENT (out) :: f_out(nsize1,nsize2,nkstot)<br>
! output ( contains values for all k-point )<br>
!<br>
#if defined(__MPI)<br>
INTEGER :: rest, nbase<br>
! the rest of the integer division nkstot / npool<br>
! the position in the original list<br>
</div>
<div> rest = nkstot / kunit - ( nkstot / kunit / npool ) * npool <br>
!<br>
nbase = nks * my_pool_id<br>
!<br>
IF ( ( my_pool_id + 1 ) > rest ) nbase = nbase + rest * kunit<br>
!<br>
f_out = 0.d0<br>
f_out(1:nsize1,1:nsize2,(<wbr>nbase+1):(nbase+nks)) = f_in(1:nsize1,1:nsize2,1:nks)<br>
!<br>
! ... reduce across the pools<br>
! <br>
CALL mp_sum(f_out,inter_pool_comm)<br>
!<br>
#else<br>
f_out(:,:,:) = f_in(:,:,:)<br>
!<br>
#endif<br>
!<br>
end subroutine poolgather1</div>
<div><br>
</div>
<div>For my test case, nsize1=3, nsize2=4, nkstot=47.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Vahid</div><div><div class="h5">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Oct 30, 2016, at 8:19 PM, Ye Luo <<a href="mailto:xw111luoye@gmail.com" target="_blank">xw111luoye@gmail.com</a>> wrote:</div>
<br class="m_-7142281832352810826Apple-interchange-newline">
<div>
<div dir="ltr">
<div>
<div>mp_sum_rt is interfaced with mp_sum for 3d array. When you mp_sum a 3d array, it will be automatically called.<br>
<div>USE mp, ONLY : mp_sum_rt is not necessary.<br>
</div>
<div>If you want to check if mp_sum_rt is implicitly called via interface, you can print something before and after your mp_sum call and also print something in mp_sum_rt. You should find the expected printing behaviour.<br>
</div>
<br>
if you compare the mp_sum_rt with the 2d and 4d implementation mp_sum_rm and mp_sum_r4d, you should find they flat the dimensionality and call the same 1-d routine reduce_base_real.<br>
<br>
</div>
As I asked earlier, how did you allocate your 3-D arrays input and output? Are you allocating sufficient size for the output, did you deallocate it by mistake? How large is output array size, if it exceeds 2^32, you might hit the 32bit integer bug of the variable
msglen.<br>
<br>
</div>
Ye<br>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_-7142281832352810826gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div>
</div>
</div>
<br>
<div class="gmail_quote">2016-10-30 14:25 GMT-05:00 Vahid Askarpour <span dir="ltr">
<<a href="mailto:vh261281@dal.ca" target="_blank">vh261281@dal.ca</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Hi Ye,</div>
<div><br>
</div>
<div>The changes I am making are part of the EPW/QE code and so the parallelization depends on the QE MPI routines. I have no problem using mp_sum for a 2-D array such as</div>
<div>energy(iband,ik). I can use mp_sum to add the contributions from various nodes and print the total in agreement with the output from the serial run. As far as the allocation goes, I use the same 2-D approach to my 3-D array.</div>
<div><br>
</div>
<div>The problem arises when I use the 3-D array with mp_sum. I noticed that in the Modules/mp.f90 file, there is a subroutine as follows:</div>
<div><br>
</div>
<div> SUBROUTINE mp_sum_rt( msg, gid )<br>
IMPLICIT NONE<br>
REAL (DP), INTENT (INOUT) :: msg(:,:,:)<br>
INTEGER, INTENT(IN) :: gid<br>
#if defined(__MPI)<br>
INTEGER :: msglen<br>
msglen = size(msg)<br>
CALL reduce_base_real( msglen, msg, gid, -1 )<br>
#endif<br>
END SUBROUTINE mp_sum_rt<br>
<br>
</div>
<div><br>
</div>
<div>Here msg has three dimensions. There are other subroutines for 1, 2, and 4 dimensions. Somehow, if I can get EPW to use the above subroutine, it might work. Adding a statement like:</div>
<div><br>
</div>
<div>USE mp, ONLY : mp_sum_rt</div>
<div><br>
</div>
<div>at the beginning of the file does not work as the compilation fails because EPW does not see the subroutine. Interestingly, it does see mp_sum which is in the same mp.f90 file.</div>
<div><br>
</div>
<div>So is there a way that I can get EPW to see mp_sum_rt? This might solve my problem. </div>
<div><br>
</div>
<div>Since I know that mp_sum works with 2-D arrays, one option is to rewrite my files in 2-D, one array for each x,y,z direction. I am trying to avoid this.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Vahid</div>
<div>
<div class="m_-7142281832352810826h5">
<div><br>
</div>
<br>
<div>
<blockquote type="cite">
<div>On Oct 30, 2016, at 3:59 PM, Ye Luo <<a href="mailto:xw111luoye@gmail.com" target="_blank">xw111luoye@gmail.com</a>> wrote:</div>
<br class="m_-7142281832352810826m_-7519985385342902231Apple-interchange-newline">
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div>
<div>
<div>Hi Vahid,<br>
</div>
segfault in mp_sum doesn't necessarily mean a problem there. Probably you wrote something to output array but not in a valid place before mp_sum.<br>
</div>
Try to check your allocation of output and the copy make sure they are correct.<br>
<br>
</div>
Ye<br>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br clear="all">
<div>
<div class="m_-7142281832352810826m_-7519985385342902231gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div>
</div>
</div>
<br>
<div class="gmail_quote">2016-10-28 14:33 GMT-05:00 Vahid Askarpour<span class="m_-7142281832352810826m_-7519985385342902231Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:vh261281@dal.ca" target="_blank">vh261281@dal.ca</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Hi Ye,</div>
<div><br>
</div>
<div>Thank you for your suggestion. I tried it and when I ran the code, it seg-faulted. I put flags in the code to see where the segmentation faults occurs. It happens as the code calls mp_sum. It seems that mp_sum may not</div>
<div>be able to handle this reduction.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Vahid</div>
<div>
<div class="m_-7142281832352810826m_-7519985385342902231h5"><br>
<div>
<blockquote type="cite">
<div>On Oct 28, 2016, at 2:51 PM, Ye Luo <<a href="mailto:xw111luoye@gmail.com" target="_blank">xw111luoye@gmail.com</a>> wrote:</div>
<br class="m_-7142281832352810826m_-7519985385342902231m_2808508241249506849Apple-interchange-newline">
<div>
<div dir="ltr">
<div>
<div>In Fortran, whatever-D array is 1-D array. mp_sum should be fine.<br>
</div>
I saw something strange in your code that you were not copying the right things as you expected.<br>
How about the following?<br>
output(1:3,1:nbnds,(k_pool*poo<wbr>l_id+1:k_pool*pool_id+k_pool))<wbr>=input(1:3,1:nbnds,1:k_pool)<br>
</div>
<br>
Ye<br>
<div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_-7142281832352810826m_-7519985385342902231m_2808508241249506849gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div>
</div>
</div>
<br>
<div class="gmail_quote">2016-10-28 12:29 GMT-05:00 Vahid Askarpour<span class="m_-7142281832352810826m_-7519985385342902231Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:vh261281@dal.ca" target="_blank">vh261281@dal.ca</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Dear QE Users,<br>
<br>
I am working on some modifications to the QE-6.0 code using symmetry. When I try to combine a 3-D array scattered across nodes, I use the following:<br>
<br>
output(3,nbnds,(k_pool*pool_id<wbr>+1:k_pool*pool_id+k_pool))=inp<wbr>ut(3,nbnds,1:k_pool)<br>
<br>
Here, nbnds is the number of bands, k_pool is the number of k points/pool, and pool_id is the id of the pool. Here I am assuming the the number of k points is divisible by the number of pools.<br>
<br>
Then I call mp_sum(output,inter_pool_comm) to put all the segments of input across the nodes into one output file.<br>
<br>
When I run the modified QE code in parallel, the output file is different from the serial run.<br>
<br>
Does the QE's mp_sum allow the above operation for a three-D array?<br>
<br>
Any hints or suggestions would be greatly appreciated.<br>
<br>
Vahid<br>
<span class="m_-7142281832352810826m_-7519985385342902231m_2808508241249506849HOEnZb"><font color="#888888"><br>
Vahid Askarpour<br>
Department of Physics and Atmospheric Science<br>
Dalhousie University,<br>
Halifax, NS, Canada<br>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a><br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a><br>
</blockquote>
</div>
<br>
</div>
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">______________________________<wbr>_________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">Pw_forum
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="mailto:Pw_forum@pwscf.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">Pw_forum@pwscf.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="http://pwscf.org/mailman/listinfo/pw_forum" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a><br>
</blockquote>
</div>
<br>
</div>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a></div>
</blockquote>
</div>
<br>
</div></div></div>
<br>______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br></blockquote></div><br></div>