[Q-e-developers] Summary of the QE developers meeting

[O. Baris Malcioglu]

Dear Paolo,
I've got some humble suggestions from my experiences accumulated
during my time working as a developer in SISSA, these are my personal
thoughts, and I hope you accept them in the same goodwill as they have
been presented.

First of all, it should be noted that, in the current situation, it is
quite hard, at least in some cases almost impossible I might say,  to
start a new project tightly integrated to PWSCF (i.e. that uses core
routines), without actually being in SISSA and being in close contact
with the developers of routines.

The problem stems from how the Q-E is organised, I get the impression
that people are expected to threat Q-E as a source of code-snippets
(in binary form or otherwise).

This has many major disadvantages, the most important two is:
1) you never know the snippet will be changed in a consecutive update or not
2) it takes considerable effort to understand what a snippet does, due
to lack of a general naming convention, and poor to none
documentation of the algorithm of the snippets does not help as well.

Since I was asked explicitly to work on a version that is always
compatible with CVS by my contractor, I've lost a huge amount of time
"fighting" with 20+ other developers. It was not a good experience l
am afraid, and pointless work to say the least. Statistically
speaking, I was debugging practically each other day trying to find
out who has done what that prevents TDDFPT code from working, until I
traced out all the binaries that contain snippets resembling the
TDDFPT code has (which provided not to be a straightforward task
considering the lack of development documentations), reducing
redundancy, and hoping the changes do not alter the behavior of the
snippet drastically. This hope might or might not be true in each cvs
update, but at least, I had more time to do other things.

My suggestion regarding this problem is to employ something similar
what software specialists say a "standart development kit" (SDK) to
Q-E. SDK approach in a distributed development environment has many
advantages, and when finely planned, can make an environment it is
designed for "the winner". This is most recently exampled by the core
strategies of Apple that made them almost dominate the mobile software
market until google came with a similar if not better one.

A SDK, can be seen as a contract between the "core developers" and
"external developers". In my opinion, the most important points in
this pseudo-contract is:
1) It provides a set of very well documented routines that are
designed to do what they do in the most efficient manner possible
2) It guarantees that these routines will always do the documented
thing, in the documented manner, using documented interfaces
(variables, environment constants, etc.).
3) There are naming conventions
4) An external developer can not mess with core routines
5) A core developer can not break external developers' programs as
long as SDK is maintained.

In my opinion, an initial point of starting a SDK might be to create
routines for (super-)operator algebra of physics. This can reside in a
separate directory, as a different project, collecting relevant
snippets in Q-E trees, presenting them in a well designed manner, with
good documentation. The SDK hierarchy can be designed in layers on top
of this project.

I think this will make the life of new developers considerably easy,
and will certainly increase the popularity of Q-E in general.

Another point I do still suffer from is that, due to many different
architectures and compilers, I never know if a particular system will
be happy with what I do in the code. I think a solution to this should
be more physical, i.e. a computing farm that contains all the relevant
machines, so that compilation can automatically be checked. Doesn't
SISSA have access to machines that are retired that can be donated for
this purpose?

I hope this text will be of help and best wishes,
O. Baris Malcioglu

2011/1/26 Paolo Giannozzi <paolo.giannozzi at uniud.it>:
> Summary of Quantum ESPRESSO (QE) Developers' Meeting
> Trieste 12/1/2011
>
> Participants:
> 1- Paolo Giannozzi
> 2- Layla Martin-Samos
> 3- Stefano de Gironcoli
> 4- Riccardo Sabatini
> 5- Andrea Dal Corso
> 6- Tone Kokalj
> 7- Nicola Varini
> 8- Giovanni Bussi
> 9- Sandro Massidda
> 10 Marco Monni
> 11 Emine Kukucbenli
> 12 Alessandro Laio
> 13 Feliciano Giustino
> 14 Ralph Genbauer
> 15 Nicola Marzari
> 16 Stefano Fabris
> 17 Andrea Ferretti
> 18 Simone Piccinin
>
> (Missing: SB, CC, DC, LP. FM, MCal)
>
> PG introduces the main topics that will be discussed:
> 1) Establishment of a QE Foundation
> 2) QE modularization and packaging
> 3) Various Verification and Validation initiatives
> 4) The Moka builder and transition to XML input and output
>
> ----------------------------------------------------------
> 1) QE Foundation
> RS explains the goals of the foundation:
> - keep QE domain names and trademarks (name, symbol, etc)
> - become the recipient of donations from users (if any) and
>   from other institutions (e.g. for QE tutorials)
> - apply for EU grants, spend the available money (if any)
>   without the constraints and the bureaucracy affecting
>   universities and public research centers.
> Many serious open-source projects have an associated foundation
> or the like. The consulting company in Udine we have contacted
> (Conecta: services related to open-source software) is very
> positive on this and advises UK as a convenient place, since
> establishing and maintaining it is quick, simple, relatively
> cheap. Once the foundation is established, we can apply for
> "charity" status, yielding fiscal advantages (but it must be
> no-profit and stricty limited to its stated goal). The most
> immediate task is to write down the statute.
> During the discussion, NM suggests to specify a method by
> which current members have ways to participate during the
> admission of new members.
>
> ----------------------------------------------------------
>
> 2) QE MODULARIZATION AND PACKAGING (Layla):
> i. Current status: packages, plugins, libraries, etc.
> - External/independent libraries (with their own development
>   environment, release time ...) like lapack, blas, iotk are in a
>   .tar.gz form inside extlibs. They have been downloaded from their
>   "official site" in their last release. The installation procedure
>   is controlled by QE configure and Makefiles.
> - External/independent "post-processing like" packages (WanT, YAMBO,
>   Wannier90) are in a .tar.gz form inside dir archive. They have been
>   downloaded from their official web site (no more cvs replicas as it
>   was the case for Wannier90 in the past). The installation procedure
>   is controlled by QE configure and Makefiles that, if available, run
>   the package configure passing the already identified arguments such
>   as the lapack path, compiler,.... For reaching the same portability
>   as pw, ph .. however some effort has to be done in ensuring a better
>   arguments compatibility between configure's.
> - The question of distributing packages that need some PW/PH/CPV
>   routines is still an open issue, mainly due to the fact that QE core
>   routines are still in daily movement! suggestions on how to maintain
>   compatibility, to simplify synchronization etc are really welcome.
>
> ii. Current and future work on modularization and directory
>    (re-)organization
> - We are working on how to manage plug-ins as PLUMED (for
>   metadynamics) and the call to DFT engine from other codes as for
>   the NEB case. We are slowly reaching "convergence" and in (very)
>   few weeks we will be able to propose a solution that is "portable"
>   and minimizes maintainance and synchronization operations.
> - For version 5.0 we propose to change the directory structure
>   moving to something more "distribution like": each module/dir in
>   espresso will have the following structure
>                         DIR
>                              src
>                              doc
>                              examples
>                              tests
>                              bin
>   see GWW and TDDFPT module that already follow a similar structure.
>   This will simplify the download of a customized release and simplify
>   the search of examples and doc.
>
> iii. Web interface for "intelligent" QE download
> - We are working in a web interface for giving the users the
>   possibility of downloading a customized release. We are also
>   thinking of changing completely the web page, as it should
>   reflect much better the distribution character of ESPRESSO.
>
> iv.  binary distributions?
> - volunteers sought!
>
> v. a maintainer for each package?
> - It would simplify the coordination of ESPRESSO (improving the
>   communication with users too), if we could define a "coordinator"
>   for each package. Meetings (or at least skype meetings) among
>   coordinators should be regular and relatively frequent.
>
> - Following the proposal of N. Marzari, we need to find a way of being
>   all informed about current and future developments, who is doing
> what,
>   money .... Suggestions are welcome.
>   [ see http://www.quantum-espresso.org/wiki/index.php/Projects
>     for a first attempt to collect info on what is going on ]
>   However in the meanwhile we invite all of you to subscribe to the
>   developers mailing list: http://qe-forge.org/mail/?group_id=10
>   and start to send information about current and future developments,
>   open positions, whatever you think could be relevant for improving
> QE.
>
> ----------------------------------------------------------
>
> 3) Verification and Validation (V&V), Pseudopotentials (PP)
> The state of V&V in DFT/PW-PP codes is rather questionable wrt what
> has been achieved in other fields. There are a number of V&V
> initiatives already started or starting soon:
> - Cecam (PG will follow this)
> - A pre-proposal for a EU project (submitted); one of the participants
>   (V. Blum) will visit Trieste soon (SdG is following this aspect)
> - The "IOM startup" project (submitted): this is aimed at verifying
>   the quality of PP by comparing with all-electron calculations,
>   to be performed by the IOM unit in Cagliari. NM reports on more
>   work he and collaborators are currently doing on PP/AE comparison.
> ----------------------------------------------------------
>
> 4) Moka builder, new XML I/O
> RS present the status of the Moka structure builder and job submitter.
> He also explains the need for a more structured QE input and output,
> based on XML, so as to streamline production of input data, job
> submission and retrieval of info from output. In the following
> discussion, TK argues that we should not get rid of the possibility
> to produce input by simply editing a file with vi and that XML has
> a heavy and baroque syntax. He suggests considering a "simplified"
> input of the form "keyword followed by value" instead of using the
> full <markup option='value'> ... </markup> syntax.
>
> [ note by PG: later I heard other unfavorable opinions on the XML
>   input. I wonder if it isn't wiser to postpone the new XML input
>   and finish first the XML output. This could also be an opportunity
>   to make some serious and much-needed cleaning and documentation work
>   on the XML output, and to provide more tools to extract data from
>   XML output: something like the pw_export tools by AF, that
>   unfortunately was never really integrated with the rest of the
>   distribution, so it becomes quickly outdated ]
> Pania:~/Work/Espresso$
>
> ---
> Paolo Giannozzi, Dept of Chemistry&Physics&Environment,
> Univ. Udine, via delle Scienze 208, 33100 Udine, Italy
> Phone +39-0432-558216, fax +39-0432-558222
>
>
>
>
> _______________________________________________
> Q-e-developers mailing list
> Q-e-developers at qe-forge.org
> http://qe-forge.org/mailman/listinfo/q-e-developers
>


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dr. O. Baris Malcioglu,
University of Liege,
Bât. B5 Physique de la matière condensée
allée du 6 Août 17
4000 Liège 1
Belgique
Tel:+32 4366 3612
Fax: +32 4366 3629
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=