<div dir="ltr"><div><div><div><div><div>Hi Thomas,<br><br></div>I just checked my local QE build with gcc 4.8.4 and gdb 7.7.1. And I can see the line info.<br>Here is the configuration in my make.sys and only -g is needed.<br><br>CFLAGS = -O3 $(DFLAGS) $(IFLAGS)<br>F90FLAGS = $(FFLAGS) -x f95-cpp-input $(FDFLAGS) $(IFLAGS) $(MODFLAGS)<br>FFLAGS = -O3 -g<br><br></div>I'd like you to check two things:<br></div>1) How you modified the makefile to add your mbdvdw module source file. Are all the flags passed to it when it is compiled?<br></div>You can check this remove the .o and .mod file associated to the module and rerun make. Check if -g appears in the compilation.<br></div><div>2) Try another computer.. I have no better ideas..<br><br></div><div>Ye<br></div><div><div><div><div><div><div><br>My call tree:<br>#0 simpson (mesh=<optimized out>, func=..., rab=..., asum=0) at simpsn.f90:39<br>#1 0x000000000057f345 in atomic_rho (rhoa=..., nspina=1) at atomic_rho.f90:95<br>#2 0x00000000005052a4 in potinit () at potinit.f90:123<br>#3 0x00000000004b64e3 in init_run () at init_run.f90:99<br>#4 0x0000000000407219 in run_pwscf (exit_status=0) at run_pwscf.f90:72<br>#5 0x00000000004070a5 in pwscf () at pwscf.f90:30<br>#6 main (argc=argc@entry=3, argv=0x7fffffffdf98) at pwscf.f90:14<br>#7 0x00007ffff69b6ec5 in __libc_start_main (main=0x407050 <main>, argc=3, argv=0x7fffffffdbd8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdbc8)<br> at libc-start.c:287<br>#8 0x00000000004070ef in _start ()<br><br></div></div></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div>
<br><div class="gmail_quote">2016-05-05 13:39 GMT-05:00 Thomas Markovich <span dir="ltr"><<a href="mailto:thomasmarkovich@gmail.com" target="_blank">thomasmarkovich@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ye,<div><br></div><div>Thank you for your response! I just ran GDB to see if it was an issue with valgrind alone, but I'm getting similar behaviour:</div><div><br></div><div><div>(gdb) where</div><div>#0 0x0000000100267d24 in __mbdvdw_module_MOD_mbdvdw_compute_tdip ()</div><div>#1 0x000000010026a392 in __mbdvdw_module_MOD_mbdvdw_compute_tlr ()</div><div>#2 0x000000010026ac7f in __mbdvdw_module_MOD_mbdvdw_compute_tlr_complex ()</div><div>#3 0x000000010026c425 in __mbdvdw_module_MOD_mbdvdw_tgg_complex ()</div><div>#4 0x0000000100288e0a in __mbdvdw_module_MOD_mbdvdw_check_quantity_dh ()</div><div>#5 0x0000000000000000 in ?? ()</div></div><div><br></div><div><div>(gdb) info symbol 0x0000000100288e0a</div><div>__mbdvdw_module_MOD_mbdvdw_check_quantity_dh + 4475 in section .text of /Users/tmarkovich/Dropbox (Aspuru-Guzik Lab)/Projects/MBD/mbd_espresso/PW/src/pw.x</div><div>(gdb) info symbol 0x0000000100267d24</div><div>__mbdvdw_module_MOD_mbdvdw_compute_tdip + 62 in section .text of /Users/tmarkovich/Dropbox (Aspuru-Guzik Lab)/Projects/MBD/mbd_espresso/PW/src/pw.x</div></div><div><br></div><div>Note, this was done with GDB 7.11. I have pasted my make.sys file below.</div><div><br></div><div>Best,</div><div>Thomas</div><div><br></div><div><div># make.sys. Generated from <a href="http://make.sys.in" target="_blank">make.sys.in</a> by configure.</div><div><br></div><div># compilation rules</div><div><br></div><div>.SUFFIXES :</div><div>.SUFFIXES : .o .c .f .f90</div><div><br></div><div># most fortran compilers can directly preprocess c-like directives: use</div><div># <span style="white-space:pre-wrap"> </span>$(MPIF90) $(F90FLAGS) -c $<</div><div># if explicit preprocessing by the C preprocessor is needed, use:</div><div># <span style="white-space:pre-wrap"> </span>$(CPP) $(CPPFLAGS) $< -o $*.F90</div><div>#<span style="white-space:pre-wrap"> </span>$(MPIF90) $(F90FLAGS) -c $*.F90 -o $*.o</div><div># remember the tabulator in the first column !!!</div><div><br></div><div>.f90.o:</div><div><span style="white-space:pre-wrap"> </span>$(MPIF90) $(F90FLAGS) -c $<</div><div><br></div><div># .f.o and .c.o: do not modify</div><div><br></div><div>.f.o:</div><div><span style="white-space:pre-wrap"> </span>$(F77) $(FFLAGS) -c $<</div><div><br></div><div>.c.o:</div><div><span style="white-space:pre-wrap"> </span>$(CC) $(CFLAGS) -c $<</div><div><br></div><div><br></div><div><br></div><div># topdir for linking espresso libs with plugins</div><div>TOPDIR = /Users/tmarkovich/Dropbox (Aspuru-Guzik Lab)/Projects/MBD/mbd_espresso</div><div><br></div><div># DFLAGS = precompilation options (possible arguments to -D and -U)</div><div># used by the C compiler and preprocessor</div><div># FDFLAGS = as DFLAGS, for the f90 compiler</div><div># See include/defs.h.README for a list of options and their meaning</div><div># With the exception of IBM xlf, FDFLAGS = $(DFLAGS)</div><div># For IBM xlf, FDFLAGS is the same as DFLAGS with separating commas</div><div><br></div><div># MANUAL_DFLAGS = additional precompilation option(s), if desired</div><div># You may use this instead of tweaking DFLAGS and FDFLAGS</div><div># BEWARE: will not work for IBM xlf! Manually edit FDFLAGS</div><div>MANUAL_DFLAGS =</div><div>DFLAGS = -D__GFORTRAN -D__STD_F95 -D__FFTW -D__MPI -D__PARA -D__SCALAPACK -D__OPENMP $(MANUAL_DFLAGS)</div><div>FDFLAGS = $(DFLAGS) $(MANUAL_DFLAGS)</div><div><br></div><div># IFLAGS = how to locate directories where files to be included are</div><div># In most cases, IFLAGS = -I../include</div><div><br></div><div>IFLAGS = -I../include</div><div><br></div><div># MOD_FLAGS = flag used by f90 compiler to locate modules</div><div># Each Makefile defines the list of needed modules in MODFLAGS</div><div><br></div><div>MOD_FLAG = -I</div><div><br></div><div># Compilers: fortran-90, fortran-77, C</div><div># If a parallel compilation is desired, MPIF90 should be a fortran-90</div><div># compiler that produces executables for parallel execution using MPI</div><div># (such as for instance mpif90, mpf90, mpxlf90,...);</div><div># otherwise, an ordinary fortran-90 compiler (f90, g95, xlf90, ifort,...)</div><div># If you have a parallel machine but no suitable candidate for MPIF90,</div><div># try to specify the directory containing "mpif.h" in IFLAGS</div><div># and to specify the location of MPI libraries in MPI_LIBS</div><div><br></div><div>MPIF90 = mpif90</div><div>#F90 = gfortran</div><div>CC = gcc-5</div><div>F77 = gfortran</div><div><br></div><div># C preprocessor and preprocessing flags - for explicit preprocessing,</div><div># if needed (see the compilation rules above)</div><div># preprocessing flags must include DFLAGS and IFLAGS</div><div><br></div><div>CPP = cpp-5 -ansi -C</div><div>CPPFLAGS = $(DFLAGS) $(IFLAGS)</div><div><br></div><div># compiler flags: C, F90, F77</div><div># C flags must include DFLAGS and IFLAGS</div><div># F90 flags must include MODFLAGS, IFLAGS, and FDFLAGS with appropriate syntax</div><span class=""><div><br></div><div>CFLAGS = -Og -g $(DFLAGS) $(IFLAGS)</div></span><div>F90FLAGS = $(FFLAGS) -x f95-cpp-input -fopenmp $(FDFLAGS) $(IFLAGS) $(MODFLAGS)</div><div>FFLAGS = -Og -g -fopenmp -Wall -Wextra -Warray-temporaries -Wconversion -fbacktrace -ffree-line-length-0 -finit-real=nan -ffpe-trap=zero,invalid,zero,overflow</div><div><br></div><div># compiler flags without optimization for fortran-77</div><div># the latter is NEEDED to properly compile dlamch.f, used by lapack</div><div><br></div><div>FFLAGS_NOOPT = -O0 -g</div><div><br></div><div># compiler flag needed by some compilers when the main is not fortran</div><div># Currently used for Yambo</div><div><br></div><div>FFLAGS_NOMAIN =</div><div><br></div><div># Linker, linker-specific flags (if any)</div><div># Typically LD coincides with F90 or MPIF90, LD_LIBS is empty</div><div><br></div><div>LD = mpif90</div><span class=""><div>LDFLAGS = -g -pthread -fopenmp</div></span><div>LD_LIBS =</div><div><br></div><div># External Libraries (if any) : blas, lapack, fft, MPI</div><div><br></div><div># If you have nothing better, use the local copy :</div><div># BLAS_LIBS = /your/path/to/espresso/BLAS/blas.a</div><div># BLAS_LIBS_SWITCH = internal</div><div><br></div><div>BLAS_LIBS = -lblas</div><div>BLAS_LIBS_SWITCH = external</div><div><br></div><div># If you have nothing better, use the local copy :</div><div># LAPACK_LIBS = /your/path/to/espresso/lapack-3.2/lapack.a</div><div># LAPACK_LIBS_SWITCH = internal</div><div># For IBM machines with essl (-D__ESSL): load essl BEFORE lapack !</div><div># remember that LAPACK_LIBS precedes BLAS_LIBS in loading order</div><div><br></div><div>LAPACK_LIBS = -llapack -lblas</div><div>LAPACK_LIBS_SWITCH = external</div><div><br></div><div>ELPA_LIBS_SWITCH = disabled</div><div>SCALAPACK_LIBS = -lscalapack</div><div><br></div><div># nothing needed here if the the internal copy of FFTW is compiled</div><div># (needs -D__FFTW in DFLAGS)</div><div><br></div><div>FFT_LIBS =</div><div><br></div><div># For parallel execution, the correct path to MPI libraries must</div><div># be specified in MPI_LIBS (except for IBM if you use mpxlf)</div><div><br></div><div>MPI_LIBS =</div><div><br></div><div># IBM-specific: MASS libraries, if available and if -D__MASS is defined in FDFLAGS</div><div><br></div><div>MASS_LIBS =</div><div><br></div><div># ar command and flags - for most architectures: AR = ar, ARFLAGS = ruv</div><div><br></div><div>AR = ar</div><div>ARFLAGS = ruv</div><div><br></div><div># ranlib command. If ranlib is not needed (it isn't in most cases) use</div><div># RANLIB = echo</div><div><br></div><div>RANLIB = ranlib -c</div><div><br></div><div># all internal and external libraries - do not modify</div><div><br></div><div>FLIB_TARGETS = all</div><div><br></div><div>LIBOBJS = ../flib/ptools.a ../flib/flib.a ../clib/clib.a ../iotk/src/libiotk.a</div><div>LIBS = $(SCALAPACK_LIBS) $(LAPACK_LIBS) $(FFT_LIBS) $(BLAS_LIBS) $(MPI_LIBS) $(MASS_LIBS) $(LD_LIBS)</div><div><br></div><div># wget or curl - useful to download from network</div><div>WGET = wget -O</div></div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 5, 2016 at 2:29 PM, Ye Luo <span dir="ltr"><<a href="mailto:xw111luoye@gmail.com" target="_blank">xw111luoye@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi Thomas,<br><br></div>Sorry about that I didn't catch the exact point you asked.<br></div>I just want to add a general comment.<br></div>In my experience, -g should be sufficient for your purpose.<br></div><div>All the profiling or debugging tools I use, including DDT, gdb, HPCtoolkit, vtune only requires the code compiled with -g and can provide information about call stack and line specific details.<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br></div><div>Ye<br></div></font></span></div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div>
<br></span><div><div><div class="gmail_quote">2016-05-05 12:40 GMT-05:00 Thomas Markovich <span dir="ltr"><<a href="mailto:thomasmarkovich@gmail.com" target="_blank">thomasmarkovich@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ye,<div><br></div><div>Thank you for your comment! <span style="font-size:13px"> __mbdvdw_module_MOD_mbdvdw_</span><span style="font-size:13px">tgg_complex is a subroutine, not an array. It appears that the invalid write is happening somewhere inside of tgg_complex, presumably the variable is stored at </span><span style="font-size:13px">0x103131248. I was hoping that when compiled with debug symbols, valgrind would be able to tell me the line number of tgg_complex.</span></div><div><span style="font-size:13px"><br></span></div><div><span style="font-size:13px">Best,</span></div><div><span style="font-size:13px">Thomas</span></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 5, 2016 at 1:31 PM, Ye Luo <span dir="ltr"><<a href="mailto:xw111luoye@gmail.com" target="_blank">xw111luoye@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>It doesn't seem to be a libc bug.<br></div>The call stack shows that in the subroutine check_quantity_dh of<span></span> your module mbdvdw, <br>the code failed in writing something into tgg_complex which seems to be a variable belongs to your module.<br></div><div>Is this variable dynamically allocatable and not initialized?<br></div><div><br></div>Ye<br></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div><div><div>
<br><div class="gmail_quote">2016-05-05 12:14 GMT-05:00 Hsin-Yu Ko <span dir="ltr"><<a href="mailto:hsinyu@princeton.edu" target="_blank">hsinyu@princeton.edu</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thomas,<br>
<br>
That is interesting. What you are seeing seems to be a libc bug [1]. I<br>
have encountered something similar last month. I fixed the problem on my<br>
machine by recompiling glibc with debug features enabled (I am not sure<br>
how useful gentoo documentation is but I put the reference here just in<br>
case [2]). I think removing -pg may be a quick fix according to [1].<br>
<br>
Best,<br>
Hsin-Yu<br>
<br>
[1] <a href="http://valgrind-users.narkive.com/MPnV7HOw/gcc-pg-valgrind-errors" rel="noreferrer" target="_blank">http://valgrind-users.narkive.com/MPnV7HOw/gcc-pg-valgrind-errors</a><br>
[2] <a href="https://wiki.gentoo.org/wiki/Debugging" rel="noreferrer" target="_blank">https://wiki.gentoo.org/wiki/Debugging</a><br>
<span><br>
On 05/05/2016 12:34 PM, Thomas Markovich wrote:<br>
> Hsin-Yu,<br>
><br>
> Thank you for the suggestion!<br>
><br>
> I had -g in LDFLAGS:<br>
> LDFLAGS = -g -pthread -fopenmp<br>
><br>
> but nothing equivalent in CFLAGS, which was defined as:<br>
> CFLAGS = -O3 $(DFLAGS) $(IFLAGS)<br>
><br>
> I have since gone ahead and changed CFLAGS to<br>
> CFLAGS = -Og -g $(DFLAGS) $(IFLAGS)<br>
><br>
> The resulting fortran compile statements look something like:<br>
> mpif90 -Og -g -pg -fopenmp -Wall -Wextra -Warray-temporaries<br>
> -Wconversion -fbacktrace -ffree-line-length-0 -finit-real=nan<br>
> -ffpe-trap=zero,invalid,zero,overflow -x f95-cpp-input -fopenmp<br>
> -D__GFORTRAN -D__STD_F95 -D__FFTW -D__MPI -D__PARA -D__SCALAPACK<br>
> -D__OPENMP -I../include -I../iotk/src -I../ELPA/src -I. -c mbdvdw.f90<br>
><br>
> and reran valgrind. It gave the following output:<br>
><br>
> ==30486== Invalid write of size 8<br>
> ==30486== at 0x1002735ED: __mbdvdw_module_MOD_mbdvdw_tgg_complex (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x100290F58:<br>
> __mbdvdw_module_MOD_mbdvdw_check_quantity_dh (in /Users/tmarkovich/bin/pw.x)<br>
</span>> ==30486== Address 0x1037989f0 is 0 bytes after a block of size 1,728<br>
> alloc'd<br>
<span>> ==30486== at 0x10092B4AB: malloc (in<br>
> /usr/local/Cellar/valgrind/HEAD/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)<br>
> ==30486== by 0x10028FEE2:<br>
> __mbdvdw_module_MOD_mbdvdw_check_quantity_dh (in /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x1001D4567: v_of_rho_ (in /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x10007C0BE: electrons_scf_ (in /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x10007D385: electrons_ (in /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x10018B30B: run_pwscf_ (in /Users/tmarkovich/bin/pw.x)<br>
> ==30486== by 0x100001157: MAIN__ (pwscf.f90:30)<br>
> ==30486== by 0x1004EC496: main (pwscf.f90:14)<br>
><br>
> This appears to not have changed much.<br>
><br>
> Best,<br>
> Thomas<br>
><br>
> On Thu, May 5, 2016 at 10:15 AM, Hsin-Yu Ko <<a href="mailto:hsinyu@princeton.edu" target="_blank">hsinyu@princeton.edu</a><br>
</span><div><div>> <mailto:<a href="mailto:hsinyu@princeton.edu" target="_blank">hsinyu@princeton.edu</a>>> wrote:<br>
><br>
> Hi Thomas,<br>
><br>
> Did you put -g in CFLAGS and LDFLAGS? Valgrind seems to recognize some<br>
> lines inside MAIN__ while failing to find the linked ones.<br>
><br>
> Best,<br>
> Hsin-Yu<br>
><br>
> On 05/05/2016 09:48 AM, Thomas Markovich wrote:<br>
> > Hi,<br>
> ><br>
> > I'm preparing to push my module that implements the Many Body<br>
> Dispersion<br>
> > van der Waals correction, and all associated forces. As a last<br>
> thing, I<br>
> > ran my code through valgrind, and it seems to have popped up a<br>
> couple of<br>
> > remaining things that I would like to fix before release[1].<br>
> > Unfortunately, the valgrind output below is less than clear on where<br>
> > exactly the error is, and it doesn't give any important line numbers.<br>
> > Beyond this, addr2line gives thoroughly unhelpful output:<br>
> > ▶ gaddr2line -e pw.x 0x100520518<br>
> > ??:0.<br>
> ><br>
> > I have compiled QE given the following flags with gfortran 4.9:<br>
> > FFLAGS = -Og -g -pg -fopenmp -fbacktrace -fcheck=all<br>
> > -finit-real=nan -ffpe-trap=zero,invalid,zero,overflow<br>
> ><br>
> > Is there any way to compile QE such that it generates all the<br>
> debugging<br>
> > symbols, so that I can get more readable and informative output from<br>
> > valgrind? I thought all I needed was the -g flag, but it appears<br>
> that I<br>
> > might need more?<br>
> ><br>
> > Best,<br>
> > Thomas Markovich<br>
> ><br>
> > [1]<br>
> > ==10233== Invalid write of size 8<br>
> > ==10233== at 0x100520518:<br>
> __mbdvdw_module_MOD_mbdvdw_tgg_complex (in<br>
> > /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x100555A88:<br>
> > __mbdvdw_module_MOD_mbdvdw_check_quantity_dh (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== Address 0x103131248 is 8 bytes after a block of size 1,728<br>
> > alloc'd<br>
> > ==10233== at 0x1011814AB: malloc (in<br>
> ><br>
> /usr/local/Cellar/valgrind/HEAD/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)<br>
> > ==10233== by 0x1005547B0:<br>
> > __mbdvdw_module_MOD_mbdvdw_check_quantity_dh (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x1003D56E4: v_of_rho_ (in /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x1000F540B: electrons_scf_ (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x1000F6E18: electrons_ (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x10032AA28: run_pwscf_ (in<br>
> /Users/tmarkovich/bin/pw.x)<br>
> > ==10233== by 0x1000010BB: MAIN__ (pwscf.f90:30)<br>
> > ==10233== by 0x100B67C1F: main (pwscf.f90:14)<br>
> _______________________________________________<br>
> Q-e-developers mailing list<br>
</div></div>> <a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a> <mailto:<a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a>><br>
> <a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
<div><div>><br>
><br>
_______________________________________________<br>
Q-e-developers mailing list<br>
<a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a><br>
<a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
</div></div></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Q-e-developers mailing list<br>
<a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a><br>
<a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Q-e-developers mailing list<br>
<a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a><br>
<a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Q-e-developers mailing list<br>
<a href="mailto:Q-e-developers@qe-forge.org" target="_blank">Q-e-developers@qe-forge.org</a><br>
<a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Q-e-developers mailing list<br>
<a href="mailto:Q-e-developers@qe-forge.org">Q-e-developers@qe-forge.org</a><br>
<a href="http://qe-forge.org/mailman/listinfo/q-e-developers" rel="noreferrer" target="_blank">http://qe-forge.org/mailman/listinfo/q-e-developers</a><br>
<br></blockquote></div><br></div></div>