[Pw_forum] [Pw_Forum] Parallel execution and configuration issues

[nicola varini]

Dear Konrad, as Alex suggested the MPI only configuration might be the 
best in your case.
However, if you would like to run on bigger machines or newer hardware 
like KNL processor it's important to understand how OpenMP performs as well.
To this end, by using the AiiDA software(http://www.aiida.net/), I 
created a workflow that do a series
of benchmark and compare the performance of  different combination 
MPI+OpenMP by splitting
the communicators following a certain logic.
At the moment this is not present in the AiiDA release, however if there 
is any interest I am more than
happy to share.

Nicola


On 20.09.2016 00:25, Konrad Gruszka wrote:
> Dear Stefano and Axel,
>
> Thank you very much for your answers. In fact, the 'silly' mistake was
> in confusing OpenMP with OpenMPI! (just like Axel suggested) - now it
> seems so clear.
> I'll now stick with OpenMPI or Hydra only (both did work fine within one
> machine) and just try to configure it properly so it can 'see' both hosts.
>
> Sometimes just few words from experts can do real magic.
>
> Konrad.
>
> W dniu 2016-09-19 o 20:17, Axel Kohlmeyer pisze:
>> On Mon, Sep 19, 2016 at 6:01 AM, Konrad Gruszka <kgruszka at wip.pcz.pl> wrote:
>>> Dear colleagues,
>>>
>>> I'm Writing to our community to clarify once and for all issues with
>>> parallel pwscf execution.
>>>
>>> By lecture of PWSCF manual, I know that there are several levels of
>>> parallelization, including FFTW and more. At the same time, I'm quite sure
>>> that most of beginner QE users do not have complicated clusters, but are
>>> rather starting with decent multicore PC's.
>>>
>>> There are two main parallel mechanisms (known to me) namely:  MPI and
>>> OpenMP. Because I'm using Ubuntu, there is free Hydra MPI software (mpich)
>> as far as i know, ubuntu ships with *two* MPI package OpenMPI (not
>> OpenMP) and MPICH and OpenMPI is the default MPI package.
>> OpenMP is not a specific package, but a compiler feature (and included in GCC).
>>
>>> and also free OpenMP. From what I know (please correct me if this is wrong)
>>> MPI software is used for parallelization within one multicore PC and OpenMP
>>> is used to make parallelization between several PC's.
>> that is incorrect. OpenMP is *restricted* to multi-core CPUs, because
>> it is based on multi-threading.
>> MPI (i.e. either OpenMPI or MPICH, but not both at the same time) can
>> be used for intra-node or inter-node or a combination of both.
>> in many cases, using only MPI is the most effective parallelization
>> approach. only when you have many cores per node and communication
>> contention become a performance limiting factor, you will run faster
>> with MPI+OpenMP
>>
>>
>>> So my reasoning is: because I have 2 multicore PCs (namely PC1: 6 cores 2
>>> threads, and PC2: 4 cores 2 threads, giving in overall PC1: 12 and PC2 8
>>> 'cores') on both PC1 and PC2 I've installed hydra MPI (for parallelization
>>> within multicores) and on both there are OpenMP for communication between
>>> PC1 and PC2 to act as 'cluster'.
>> you seem to be confusing OpenMPI with OpenMP. for your system, an
>> MPI-only configuration (using the default MPI, i.e. OpenMPI) should
>> work well. no need to worry about the additional complications from
>> OpenMP until you have a larger, more potent machine.
>>
>>> I've configured OpenMP so both machines sees one another and when I'm
>>> starting some pw.x job using openmp on both machines i see that processors
>>> are 100% busy. Problem is that in produced output files I can see several
>>> repeats of pwscf output (just like several separate pwscf processes write
>>> into one file at the same time).
>> that is an indication of compiling executable without MPI support or
>> with support for an MPI library *different* from the one used when
>> launching the executable.
>>
>>> The QE was configured and compiled with --enable-openmp on both machines and
>>> when I simply run pw.x on PC1 it writes:
>>>
>>>     "Parallel version (MPI & OpenMP), running on      12 processor cores"
>>>
>>> I'm running job with this command:
>>>
>>>    mpirun.openmp -np 20 pw.x -in job.in > job.out
>>>
>>> and as said before, both machines are occupied with this job (tested using
>>> htop).
>>>
>>> Looking for help in this topic I often get to the truth 'please contact your
>>> system administrator', but sadly I am the administrator!
>> then look for somebody with experience in linux clusters. you
>> definitely want somebody local to help you, that can look over your
>> shoulder while you are doing things and explain stuff. this is really
>> tricky to do over e-mail. these days it should not be difficult to
>> find somebody, as using linux in parallel is ubiquitous in most
>> computational science groups.
>>
>> there also is a *gazillion* of material available online including
>> talk slides from workshops and courses and online courses and
>> self-study material. HPC university has an aggregator for such
>> material at:
>> http://hpcuniversity.org/trainingMaterials/
>>
>> axel.
>>
>>> Could some one with a similar hardware configuration could comment here how
>>> to achieve properly working cluster?
>>>
>>> Sincerely yours
>>>
>>> Konrad Gruszka
>>>
>>>
>>> _______________________________________________
>>> Pw_forum mailing list
>>> Pw_forum at pwscf.org
>>> http://pwscf.org/mailman/listinfo/pw_forum
>>

-- 
Nicola Varini, PhD

Scientific IT and Application Support (SCITAS)
Theory and simulation of materials (THEOS)
CE 0 813 (Bâtiment CE)
Station 1
CH-1015 Lausanne
http://scitas.epfl.ch

Nicola Varini