<div dir="ltr">Download the CUDA</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 25, 2016 at 11:00 AM, <span dir="ltr"><<a href="mailto:pw_forum-request@pwscf.org" target="_blank">pw_forum-request@pwscf.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send Pw_forum mailing list submissions to<br>
<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:pw_forum-request@pwscf.org">pw_forum-request@pwscf.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:pw_forum-owner@pwscf.org">pw_forum-owner@pwscf.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of Pw_forum digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. controlling output (Murray Daw)<br>
2. Re: error messege igcx (Paolo Giannozzi)<br>
3. Re: controlling output (Paolo Giannozzi)<br>
4. Tutorial: Install QE-GPU binaries in latest ubuntu systems<br>
(v. 16.04), Nvidia Cuda 7.5 , Intel MKL and Intel MPI software<br>
with NVIDIA cards hardware (Josue Itsman Clavijo Penagos)<br>
5. Re: Tutorial: Install QE-GPU binaries in latest ubuntu<br>
systems (v. 16.04), Nvidia Cuda 7.5 , Intel MKL and Intel MPI<br>
software with NVIDIA cards hardware (Rolly Ng)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Wed, 24 Aug 2016 13:39:18 +0000<br>
From: Murray Daw <<a href="mailto:daw@clemson.edu">daw@clemson.edu</a>><br>
Subject: [Pw_forum] controlling output<br>
To: "<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>" <<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>><br>
Message-ID: <<a href="mailto:BA67C13A-155E-4B92-B1A4-50F11A8C370F@clemson.edu">BA67C13A-155E-4B92-B1A4-<wbr>50F11A8C370F@clemson.edu</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I am writing a driver that will call the PWSCF functions numerous times (~10,000).<br>
It is producing more output than I need.<br>
Is there any way to reduce the amount of output?<br>
I have verbosity set to ?low? already.<br>
I don?t want to turn it off completely, just lower it.<br>
<br>
Thanks.<br>
<br>
Best,<br>
Murray Daw<br>
<br>
______________________________<wbr>______________________<br>
MURRAY S. DAW<br>
R. A. Bowen Professor of Physics<br>
Dept of Physics & Astronomy<br>
Clemson University<br>
202A Kinard Labs<br>
Clemson, SC 29634-0978<br>
Phone: (864)656-6702<br>
FAX: (864)656-0805<br>
e-mail: <a href="mailto:daw@clemson.edu">daw@clemson.edu</a><br>
website <a href="http://myweb.clemson.edu/~daw" rel="noreferrer" target="_blank">http://myweb.clemson.edu/~daw</a><br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 24 Aug 2016 15:44:37 +0200<br>
From: Paolo Giannozzi <<a href="mailto:p.giannozzi@gmail.com">p.giannozzi@gmail.com</a>><br>
Subject: Re: [Pw_forum] error messege igcx<br>
To: PWSCF Forum <<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>><br>
Message-ID:<br>
<CAPMgbCsSaA5yFZVKeJ7yZRh+<wbr>m6kGV6UXS3AFMKn=<a href="mailto:1Pcw4jHkBw@mail.gmail.com">1Pcw4jHkBw@<wbr>mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
If you are using option "input_dft", check that he value you provided is<br>
correct. If not: check the DFT label in the header of pseudopotential files<br>
(for UPF files, search for 'Exchange_Correlation')<br>
<br>
Paolo<br>
<br>
On Wed, Aug 24, 2016 at 11:49 AM, mohammadreza hosseini <<br>
<a href="mailto:mhr.hosseini@modares.ac.ir">mhr.hosseini@modares.ac.ir</a>> wrote:<br>
<br>
> Dear all<br>
><br>
> I am studding electronic and magnetic properties of SnO2 clusters. I have<br>
> performed relax calculations and now i am doing SCF job using espresso<br>
> v.5.1. In the scf calculations i get this error :<br>
><br>
><br>
> Error in routine set_dft_from_name (1):<br>
> conflicting values for igcx<br>
><br>
> I have searched different webs in particular espresso forum.<br>
> Is it possible for you to help me what i should do?<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Pw_forum mailing list<br>
> <a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
> <a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br>
><br>
<br>
<br>
<br>
--<br>
Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>
Phone <a href="tel:%2B39-0432-558216" value="+390432558216">+39-0432-558216</a>, fax <a href="tel:%2B39-0432-558222" value="+390432558222">+39-0432-558222</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://pwscf.org/pipermail/pw_forum/attachments/20160824/da3d53f9/attachment-0001.html" rel="noreferrer" target="_blank">http://pwscf.org/pipermail/pw_<wbr>forum/attachments/20160824/<wbr>da3d53f9/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 24 Aug 2016 15:45:39 +0200<br>
From: Paolo Giannozzi <<a href="mailto:p.giannozzi@gmail.com">p.giannozzi@gmail.com</a>><br>
Subject: Re: [Pw_forum] controlling output<br>
To: PWSCF Forum <<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>><br>
Message-ID:<br>
<CAPMgbCuBQ9w=<a href="mailto:ZKCE3WauypSQ2bAXcjy91q-umVGBRxyfkVFXgA@mail.gmail.com">ZKCE3WauypSQ2bAX<wbr>cjy91q-umVGBRxyfkVFXgA@mail.<wbr>gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
See variable "printout" in electrons.f90. Paolo<br>
<br>
On Wed, Aug 24, 2016 at 3:39 PM, Murray Daw <<a href="mailto:daw@clemson.edu">daw@clemson.edu</a>> wrote:<br>
<br>
> I am writing a driver that will call the PWSCF functions numerous times<br>
> (~10,000).<br>
> It is producing more output than I need.<br>
> Is there any way to reduce the amount of output?<br>
> I have verbosity set to ?low? already.<br>
> I don?t want to turn it off completely, just lower it.<br>
><br>
> Thanks.<br>
><br>
> Best,<br>
> Murray Daw<br>
><br>
> ______________________________<wbr>______________________<br>
> MURRAY S. DAW<br>
> R. A. Bowen Professor of Physics<br>
> Dept of Physics & Astronomy<br>
> Clemson University<br>
> 202A Kinard Labs<br>
> Clemson, SC 29634-0978<br>
> Phone: (864)656-6702<br>
> FAX: (864)656-0805<br>
> e-mail: <a href="mailto:daw@clemson.edu">daw@clemson.edu</a><br>
> website <a href="http://myweb.clemson.edu/~daw" rel="noreferrer" target="_blank">http://myweb.clemson.edu/~daw</a><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Pw_forum mailing list<br>
> <a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
> <a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br>
<br>
<br>
<br>
<br>
--<br>
Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>
Phone <a href="tel:%2B39-0432-558216" value="+390432558216">+39-0432-558216</a>, fax <a href="tel:%2B39-0432-558222" value="+390432558222">+39-0432-558222</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://pwscf.org/pipermail/pw_forum/attachments/20160824/6ed27313/attachment-0001.html" rel="noreferrer" target="_blank">http://pwscf.org/pipermail/pw_<wbr>forum/attachments/20160824/<wbr>6ed27313/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Wed, 24 Aug 2016 15:21:10 -0500<br>
From: Josue Itsman Clavijo Penagos <<a href="mailto:jiclavijop@unal.edu.co">jiclavijop@unal.edu.co</a>><br>
Subject: [Pw_forum] Tutorial: Install QE-GPU binaries in latest ubuntu<br>
systems (v. 16.04), Nvidia Cuda 7.5 , Intel MKL and Intel MPI software<br>
with NVIDIA cards hardware<br>
To: PWSCF Forum <<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>><br>
Message-ID:<br>
<<a href="mailto:CAJgMqa%2BH%2B09SSrJcHU1wTcQWzn_Sk0Jj9Mm%2BpGR-%2BZOYUqk%2Bww@mail.gmail.com">CAJgMqa+H+09SSrJcHU1wTcQWzn_<wbr>Sk0Jj9Mm+pGR-+ZOYUqk+ww@mail.<wbr>gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Dear all,<br>
<br>
I?m posting the following rather introductory, not professional-level<br>
tutorial for Installing QE-GPU binaries In recent ubuntu systems using<br>
Nvidia Cuda, Intel MKL, Intel MPI software and NVIDIA GPU kepler-model<br>
cards hardware, since i think this might be useful for any of you fellow<br>
scientists struggling to get working their quantum espresso GPU-enabled<br>
installations.<br>
<br>
I, by no means, do pretend to offer an in-depth description of every and<br>
all given steps, since i?m no a technology expert; in fact, much of this<br>
are referred to mail discussions, Urls and documentation readings found out<br>
inside the same used installation packages.<br>
<br>
*The only purpose of this post* is to give some useful advice and, mainly,<br>
to unify and share to whom may be interested all the available information<br>
I?ve found in order to get, to the present extent of my technology skills,<br>
a more-or-less complete and comprehensive tutorial containing all what it?s<br>
needed to get success about QE-GPU installation and usage.<br>
<br>
A big *thank you* to the following scholars for all of your highly valuable<br>
advice and key assistance to get this done:<br>
<br>
<br>
*Ari Paavo Seitsonen*<br>
*Claudio Quarti*<br>
and all of you<br>
*PW_FORUM FELLOWS*<br>
<br>
<br>
All trademarks, copyrights and author ownerships over texts, codes and<br>
original information links *are respected and are the exclusive property of<br>
their rightfully legitimate owners*.<br>
<br>
I apologize for any typos or vocabulary/redaction errors; I?m not a native<br>
english speaker.<br>
<br>
<br>
All said above, in the attached file I share what I did to get a flawless<br>
compilation.<br>
<br>
*Best regards, *<br>
<br>
Josu? Clavijo, Dr. Sc. in Chemistry<br>
Assistant Professor<br>
Universidad Nacional de Colombia<br>
Science College<br>
Chemistry Department<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://pwscf.org/pipermail/pw_forum/attachments/20160824/e461b163/attachment-0001.html" rel="noreferrer" target="_blank">http://pwscf.org/pipermail/pw_<wbr>forum/attachments/20160824/<wbr>e461b163/attachment-0001.html</a><br>
-------------- next part --------------<br>
<br>
I?m posting the following rather introductory, not professional-level tutorial for Installing QE-GPU binaries In ubuntu systems using Nvidia Cuda, Intel MKL, Intel MPI software and NVIDIA cards hardware since i think this would be useful for any of you fellow scientists struggling to get working their quantum espresso GPU-enabled installations.<br>
<br>
I, by no means, do pretend to offer an in-depth description of every and all given steps, since i?m no a technology expert; in fact, much of this are referred to Urls and documentation found out inside the same used installation packages.<br>
<br>
The only purpose of this post is to give some useful advice and, mainly, to unite all the available information I?ve found in order to get, to the present extent of my technology skills, a more-or-less complete and comprehensive tutorial containing all what it?s needed to get success about QE-GPU installation and usage.<br>
<br>
A BIG THANK YOU to the following scholars for all you you advice and key assistance to get this done:<br>
<br>
PW_FORUM FELLOWS<br>
Ari Paavo Seitsonen<br>
Claudio Quarti<br>
<br>
All trademarks, copyrights, author ownerships over texts, codes and original information links are respected and are the exclusive property of their rightfully legitimate owners.<br>
<br>
I apologize for any typos/missing vocabulary/redaction errors; I?m not a native english speaker.<br>
<br>
<br>
ALL SAID ABOVE , HERE?S WHAT I DID TO GET A FLAWLESS COMPILATION FOR QE-GPU BINARIES<br>
<br>
<br>
BASIC HARDWARE/SOFTWARE SYSTEM SETUP EXAMPLE:<br>
<br>
-> UBUNTU MATE 16.04 LTS (MANY People posting in the Web recommends MATE or another Ubuntu ?Flavor? instead original ubuntu distribution, due to the Unity graphic desktop interface suffers of crash issues using Nvidia Cards very often)<br>
<br>
(NOTE: Why using a graphic interface? Mainly due to get able to use PW-Gui for QE binary executables, Virtual NanoLab Interface, XCrysden and another auxiliary and useful crystallography tools such as Vesta, and also to be able to use remote manager solutions such as Teamviewer.)<br>
<br>
-> GPU Card 1: NVIDIA QUADRO K620<br>
-> GPU Card 2: NVIDIA TESLA K20C<br>
-> CUDA 7.5<br>
-> NVIDIA DRIVER 364<br>
-> QUANTUM ESPRESSO 5.4.0<br>
-> QE-GPU 5.4.0<br>
<br>
<br>
TESTED pw-gpu.x and ph-gpu.x performance using Barium Titanate and Methyl Ammonium Lead Iodide perovskite unit cells (from *.cif crystallographic files, exported to QE format using the free Virtual Nanolab GUI and edited with the right path for PPs? and outdir folders, and some chosen prefix .) - not so very extensive benchmarks, only proof-of-working tests -:<br>
<br>
pw-gpu.x works in SCF with full-relativistic PPs and non collinear spin-orbit settings, Relax and VC-Relax modes including force and stress minimization in vc-relax and relax modes. The nscf mode was not tested, but I see no reason this mode would crash)<br>
<br>
ph-gpu.x calculates Raman and IR spectra using a fully relaxed cell in gamma-only mode, following tutorial examples in the quantum espresso packages;<br>
<br>
Various automatic k-points grids and e_cutoffs used. Also, the ./pw_cutoff.sh tests for optimal e_cutoff for PPS used also works: directions posted in <a href="http://larrucea.eu/checking-optimum-cutoff-qe/" rel="noreferrer" target="_blank">http://larrucea.eu/checking-<wbr>optimum-cutoff-qe/</a> .<br>
<br>
Finally, the nvidia-smi test shows the usage of the Tesla GPU for calculations.<br>
<br>
<br>
I?m open to every comments, suggestions and corrections and they all are already welcome.<br>
<br>
<br>
******************************<wbr>******************************<wbr>******************************<wbr>******************************<wbr>******************************<wbr>*****<br>
<br>
<br>
<br>
********* ---- MAJOR TUTORIAL UPDATE : AUG 24th, 2016 ---- **********<br>
<br>
############ PRELIMINARY REMARKS: #################<br>
<br>
1 - IF YOU?RE BEHIND A PROXY, after a fresh ubuntu installation, do this to get internet connection (for apt-get), setting the following apt config via terminal:<br>
<br>
$ sudo pluma /etc/apt/apt.conf<br>
<br>
Paste into that file, and save-and-close after:<br>
<br>
Acquire::http::proxy "http://username:password@<wbr>proxy-name:8080/";<br>
Acquire::https::proxy "http://username:password@<wbr>proxy-name:8080/";<br>
<br>
2 - Setup Internet browser proxy config, to download QE and QE-GPU packages:<br>
<br>
Set automatic proxy settings with something like the URL: <a href="http://proxy-name:8080/proxy.pac" rel="noreferrer" target="_blank">http://proxy-name:8080/proxy.<wbr>pac</a> (consult your web proxy administrator, if needed)<br>
<br>
3 - Install COMPILERS AND Libraries<br>
<br>
GCC<br>
G++<br>
GFORTRAN<br>
Intel MKL<br>
Intel MPI<br>
<br>
-> FFTW?s are read from MKL libraries.<br>
<br>
OPEN MPI (Follow instructions in the *.tar.gz package downloaded from <a href="https://www.open-mpi.org" rel="noreferrer" target="_blank">https://www.open-mpi.org</a>) (It?s entirely OPTIONAL if you?re gonna use INTEL MPI)<br>
<br>
############ END OF PRELIMINARY REMARKS: #################<br>
<br>
<br>
############ INSTALLING NVIDIA DRIVER AND CUDA 7.5 SECTION: #################<br>
<br>
LATEST WORKING CONFIGURATION FOR FUTURE QE-GPU ./configure step:<br>
<br>
INSTALL THE NVIDIA DRIVER 364<br>
(As stated at <a href="http://askubuntu.com/questions/760934/graphics-issues-after-installing-ubuntu-16-04-with-nvidia-graphics" rel="noreferrer" target="_blank">http://askubuntu.com/<wbr>questions/760934/graphics-<wbr>issues-after-installing-<wbr>ubuntu-16-04-with-nvidia-<wbr>graphics</a>)<br>
<br>
You need the PPA APT link for Nvidia-364 driver. First, include the graphics-drivers debian repositories in Ubuntu software origins. Go to<br>
<br>
<a href="https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa" rel="noreferrer" target="_blank">https://launchpad.net/~<wbr>graphics-drivers/+archive/<wbr>ubuntu/ppa</a><br>
<br>
<br>
And copy the PPA link according to your ubuntu distribution.<br>
<br>
Adde this link in software origins repositories listing, either by terminal , running<br>
<br>
$ sudo add-apt-repository ppa:graphics-drivers/ppa and then sudo apt-get update<br>
<br>
or by the same software origins tab via the software updates window.<br>
<br>
<br>
Then install the driver typing in terminal<br>
<br>
$ sudo apt-get install nvidia-364<br>
<br>
or by choosing the Nvidia-364 driver in additional drivers tab in the the software updates window.<br>
<br>
It may happen that, when using a former Nvidia driver, you already are experiencing too many graphic glitches, freezing, blinking, or even you are not able to log in anymore. To fix this, follow the directions below:<br>
<br>
Log into your account in the TTY.<br>
Run $ sudo apt-get purge nvidia-*<br>
Run $ sudo add-apt-repository ppa:graphics-drivers/ppa and then sudo apt-get update.<br>
Run $ sudo apt-get install nvidia-364.<br>
Reboot and your graphics issue should be fixed.<br>
<br>
If you are unable to enter a TTY (just a black screen and a blinking cursor):<br>
Reboot into GRUB.<br>
Highlight the Ubuntu option and press e.<br>
Add nouveau.modeset=0 to the end of the line beginning with the word ?linux?.<br>
Press F10 to boot.<br>
Follow the instructions above to install the install nvidia-364 driver.<br>
<br>
INSTALL CUDA-7.5:<br>
<br>
Type the code given below, as suggested in <a href="https://www.pugetsystems.com/labs/hpc/NVIDIA-CUDA-with-Ubuntu-16-04-beta-on-a-laptop-if-you-just-cannot-wait-775/" rel="noreferrer" target="_blank">https://www.pugetsystems.com/<wbr>labs/hpc/NVIDIA-CUDA-with-<wbr>Ubuntu-16-04-beta-on-a-laptop-<wbr>if-you-just-cannot-wait-775/</a> :<br>
<br>
sudo apt-get install ca-certificates-java default-jre default-jre-headless fonts-dejavu-extra freeglut3 freeglut3-dev java-common libatk-wrapper-java libatk-wrapper-java-jni libdrm-dev libgl1-mesa-dev libglu1-mesa-dev libgnomevfs2-0 libgnomevfs2-common libice-dev libpthread-stubs0-dev libsctp1 libsm-dev libx11-dev libx11-doc libx11-xcb-dev libxau-dev libxcb-dri2-0-dev libxcb-dri3-dev libxcb-glx0-dev libxcb-present-dev libxcb-randr0-dev libxcb-render0-dev libxcb-shape0-dev libxcb-sync-dev libxcb-xfixes0-dev libxcb1-dev libxdamage-dev libxdmcp-dev libxext-dev libxfixes-dev libxi-dev libxmu-dev libxmu-headers libxshmfence-dev libxt-dev libxxf86vm-dev lksctp-tools mesa-common-dev x11proto-core-dev x11proto-damage-de x11proto-dri2-dev x11proto-fixes-dev x11proto-gl-dev x11proto-input-dev x11proto-kb-dev x11proto-xext-dev x11proto-xf86vidmode-dev xorg-sgml-doctools xtrans-dev libgles2-mesa-dev nvidia-modprobe build-essential<br>
<br>
(FORM THE LINK: You can "sudo apt-get install" the above list and that should get most or all of the dependencies. There is a possibility that there will still be missing debs. For example I added the last three entries to the list when I discovered that they were missing on my new 16.04 install after the others were installed. That's is probably because they were already installed on the 15.04 system it ran "apt-get -s install cuda" on i.e. they didn't come up as needed dependencies because they were already installed.)<br>
<br>
USING CUDA runtime Installer (That?s better than the *.deb file option, since you are able to NOT install the packed NVIDIA driver (That WUold overwrite the 364 driver) and just install the Cuda Toolkit and Samples.<br>
<br>
Download the CUDA .run file from the NVIDIA download site. I used this, <a href="http://developer.download.nvidia.com/compute/cuda/7.5/Prod/local_installers/cuda_7.5.18_linux.run" rel="noreferrer" target="_blank">http://developer.download.<wbr>nvidia.com/compute/cuda/7.5/<wbr>Prod/local_installers/cuda_7.<wbr>5.18_linux.run</a><br>
<br>
$ chmod 755 cuda_7.5.18_linux.run<br>
<br>
$ sudo ./cuda_7.5.18_linux.run --override<br>
<br>
NOTICE: The "--override" is needed so you don't get the fatal error saying: Toolkit: Installation Failed. Using unsupported Compiler, that prompts the installer when finds that GCC is a >4.9 version, and coda seems to be incompatible with that.<br>
<br>
Be sure to NOT install the NVIDIA driver that is in the .run file since you already have a more up to date version installed, as said before.<br>
<br>
YOU SHOULD GET FINALLY:<br>
<br>
Driver: Not Selected<br>
Toolkit: Installed in /usr/local/cuda-7.5<br>
Samples: Installed in /usr/local/cuda-7.5<br>
<br>
3 - To check if CUDA are working and calling to the Tesla card properly, open a terminal window and go to /deviceQuery folder:<br>
<br>
$ cd /home/quantum/NVIDIA_CUDA-7.5_<wbr>Samples/1_Utilities/<wbr>deviceQuery<br>
<br>
# AND compile deviceQuery executable:<br>
<br>
$ make<br>
<br>
# Then type<br>
<br>
$ ./deviceQuery<br>
<br>
<br>
# If CUDA was properly installed and Tesla card are recognized, you should get something like:<br>
<br>
<br>
$ ./deviceQuery Starting...<br>
<br>
CUDA Device Query (Runtime API) version (CUDART static linking)<br>
<br>
Detected 2 CUDA Capable device(s)<br>
<br>
Device 0: "Tesla K20c"<br>
CUDA Driver Version / Runtime Version 6.0 / 6.0<br>
CUDA Capability Major/Minor version number: 3.5<br>
Total amount of global memory: 4800 MBytes (5032706048 bytes)<br>
(13) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores<br>
GPU Clock rate: 706 MHz (0.71 GHz)<br>
Memory Clock rate: 2600 Mhz<br>
Memory Bus Width: 320-bit<br>
L2 Cache Size: 1310720 bytes<br>
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)<br>
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers<br>
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers<br>
Total amount of constant memory: 65536 bytes<br>
Total amount of shared memory per block: 49152 bytes<br>
Total number of registers available per block: 65536<br>
Warp size: 32<br>
Maximum number of threads per multiprocessor: 2048<br>
Maximum number of threads per block: 1024<br>
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)<br>
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)<br>
Maximum memory pitch: 2147483647 bytes<br>
Texture alignment: 512 bytes<br>
Concurrent copy and kernel execution: Yes with 2 copy engine(s)<br>
Run time limit on kernels: No<br>
Integrated GPU sharing Host Memory: No<br>
Support host page-locked memory mapping: Yes<br>
Alignment requirement for Surfaces: Yes<br>
Device has ECC support: Enabled<br>
Device supports Unified Addressing (UVA): Yes<br>
Device PCI Bus ID / PCI location ID: 4 / 0<br>
Compute Mode:<br>
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) ><br>
<br>
Device 1: "Quadro K620"<br>
CUDA Driver Version / Runtime Version 6.0 / 6.0<br>
CUDA Capability Major/Minor version number: 5.0<br>
Total amount of global memory: 2048 MBytes (2147155968 bytes)<br>
( 3) Multiprocessors, (128) CUDA Cores/MP: 384 CUDA Cores<br>
GPU Clock rate: 1124 MHz (1.12 GHz)<br>
Memory Clock rate: 900 Mhz<br>
Memory Bus Width: 128-bit<br>
L2 Cache Size: 2097152 bytes<br>
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)<br>
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers<br>
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers<br>
Total amount of constant memory: 65536 bytes<br>
Total amount of shared memory per block: 49152 bytes<br>
Total number of registers available per block: 65536<br>
Warp size: 32<br>
Maximum number of threads per multiprocessor: 2048<br>
Maximum number of threads per block: 1024<br>
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)<br>
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)<br>
Maximum memory pitch: 2147483647 bytes<br>
Texture alignment: 512 bytes<br>
Concurrent copy and kernel execution: Yes with 1 copy engine(s)<br>
Run time limit on kernels: Yes<br>
Integrated GPU sharing Host Memory: No<br>
Support host page-locked memory mapping: Yes<br>
Alignment requirement for Surfaces: Yes<br>
Device has ECC support: Disabled<br>
Device supports Unified Addressing (UVA): Yes<br>
Device PCI Bus ID / PCI location ID: 3 / 0<br>
Compute Mode:<br>
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) ><br>
> Peer access from Tesla K20c (GPU0) -> Quadro K620 (GPU1) : No<br>
> Peer access from Quadro K620 (GPU1) -> Tesla K20c (GPU0) : No<br>
<br>
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 6.0, CUDA Runtime Version = 6.0, NumDevs = 2, Device0 = Tesla K20c, Device1 = Quadro K620<br>
Result = PASS<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------------------<br>
<br>
4 - It seems not necessary to install the basic versions fftw3, mpi (message passing interface standards) and MPICH libraries, however. In Software Center, search for each one, but install the -dev options, because quantum espresso and QE-GPU compilers make calls to *.dev libraries and NOT to normal libraries (e. g., call to libfftw3-mpi-dev and NOT to libfftw3-mpi3 !!!)<br>
<br>
5 - The phiGEMM package comes bundled with the QE-GPU package, then IT is NOT necessary to download the phigemm package separately.<br>
<br>
6 - in ./configure, it seems that the openmp option blocks usage of lapack and fftw3 libraries. Need some research switching ./configure parameters and flags in order to get optimal configuration and best make final builds. ?> ANSWER: USE openmp with intel MKL and MPI libraries.<br>
<br>
<br>
############ END OF INSTALLING NVIDIA DRIVER AND CUDA 7.5 SECTION #################<br>
<br>
<br>
POST-INSTALLATION CHECKS for CUDA and Tesla card proper operation:<br>
<br>
( Remember that ?caja? replaces nautilus and ?pluma? replaces gedit IN UBUNTU MATE )<br>
<br>
PLEASE take in account the following, AFTER INSTALLING CUDA-7.5<br>
<br>
edit /usr/local/cuda/include/host_<wbr>config.h and comment out line 115:<br>
<br>
$ sudo pluma /usr/local/cuda/include/host_<wbr>config.h<br>
<br>
line: 115 comment out error<br>
//#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!<br>
<br>
THAT PREVENTS FATAL ERRORS for the QE-GPU compilation steps, since Cuda 7.5 and gcc > 5+ are INCOMPATIBLE.<br>
<br>
<br>
**** PRE-QE-GPU CONFIGURATION Requirements: **********<br>
<br>
<br>
1. MAKE SURE YOU HAVE INSTALLED ALL THE REQUIRED LIBRARIES<br>
<br>
2. DECLARE some Environment variables (by pasting as bottom lines in .bashrc): (examples given, replace with the actual paths for your system)<br>
<br>
$ export PATH=/home/quantum/Descargas/<wbr>QE-5.4/espresso-5.4.0:$PATH<br>
$ export PATH=/home/quantum/Descargas/<wbr>QE-5.4/PWgui-5.4.0:$PATH<br>
$ export PHI_DGEMM_SPLIT=0.950<br>
$ export PHI_ZGEMM_SPLIT=0.950<br>
$ export PATH=/usr/local/cuda-7.5/bin:$<wbr>PATH<br>
$ export LD_LIBRARY_PATH=/usr/local/<wbr>cuda-7.5/lib64:$LD_LIBRARY_<wbr>PATH<br>
$ source /opt/intel/bin/compilervars.sh intel64<br>
$ export PATH=/opt/intel/bin:$PATH<br>
$ export LD_LIBRARY_PATH=/opt/intel/<wbr>lib/intel64:$LD_LIBRARY_PATH<br>
$ export LD_LIBRARY_PATH=/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>2.181/linux/mpi/intel64/lib:$<wbr>PATH<br>
<br>
3. Rebuild library configuration (examples given, replace with the actual paths):<br>
<br>
Open a nautilus (MATE: caja) windows AS ROOT (nautilus: graphical file explorer for debian-based linux distros with Unity desktop; Caja= Nautilus equivalent in MATE desktops flavor)<br>
<br>
Note: You may need to install gksu previously : sudo apt-get install gksu<br>
<br>
$ sudo gksu nautilus (sudo gksu caja)<br>
<br>
Go to the file /etc/ld.so.conf.d/x86_64-<wbr>linux-gnu.conf<br>
<br>
put the necessary links one per line at the end of other existing lines (example):<br>
<br>
Multiarch support<br>
/lib/x86_64-linux-gnu<br>
/usr/lib/x86_64-linux-gnu<br>
/usr/local/cuda-7.5/lib64<br>
/opt/intel/composer_xe_2015.3.<wbr>187/mkl/lib/intel64<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mkl/lib/intel64_lin<br>
/opt/intel/compilers_and_<wbr>libraries/linux/mkl/lib/<wbr>intel64_lin<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mpi/intel64/lib<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mpi/intel64/lib<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mkl/interfaces/fftw3xc<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mkl/interfaces/fftw3x_cdft<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mkl/interfaces/fftw3x_cdft/<wbr>obj_intel64_lp64<br>
/opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mkl/interfaces/fftw3xf<br>
<br>
save file and exit the nautilus window.<br>
<br>
Ready to Rebuild library configuration....<br>
<br>
In terminal:<br>
<br>
$ sudo ldconfig<br>
<br>
Check if libraries are properly installed and listed:<br>
<br>
$ ldconfig -p |less #Scroll lines with the mouse wheel, when "END" appears, press q<br>
<br>
MAKE SURE YOU SEE MKL, OPENMP, CUDA AND INTEL MPI libraries (libmkl?) and exit terminal.<br>
<br>
**** END OF PRE-QE-GPU CONFIGURATION Requirements **********<br>
<br>
<br>
**** AND AT LEAST, QE-GPU CONFIGURATION ****<br>
<br>
<br>
DOWNLOAD THE QUANTUM ESPRESSO AND QE-GPU PACKAGES FROM THEIR RESPECTIVE URLs.<br>
<br>
** NOTE: AN ERROR ABOUT TOO MANY DIRECTORY LEVELS COMPLAINS OFTEN. INSTALL ESPRESSO IN A ROOT DIRECTORY TO AVOID IT. (/home ) **<br>
<br>
0.1 Unpack the espresso-5.4.0 tar.gz package in /home/your-username or just in /home .<br>
<br>
0.2 Move the packages<br>
<br>
atomic-5.4.0.tar.gz<br>
GWW-5.4.0.tar.gz<br>
neb-5.4.0.tar.gz<br>
PHonon-5.4.0.tar.gz<br>
pwcond-5.4.0.tar.gz<br>
tddfpt-5.4.0.tar.gz<br>
xspectra-5.4.0.tar.gz<br>
<br>
to the ?Archives? Folder in the espresso root directory.<br>
<br>
1. Copy QE-GPU in espresso directory<br>
<br>
Move to the espresso root directory, uncompress the archive<br>
$ tar zxvf QE-GPU-<TAG-NAME>.tar.gz<br>
<br>
create a symbolic link with the name GPU<br>
$ ln -s QE-GPU-<TAG-NAME> GPU<br>
<br>
Replace <TAG-GPU> with the ACTUAL TAG name/id (example: 5.4.0 )<br>
<br>
2. Run QE-GPU configure (in terminal, from GPU dir):<br>
<br>
NOTICE: I did no use the ?with-scalapack option because this is no a cluster installation.<br>
<br>
$ cd /opt/intel/compilers_and_<wbr>libraries_2016.2.181/linux/<wbr>mkl/bin<br>
<br>
$ source mklvars.sh intel64 lp64<br>
<br>
check that INTEL MPI is running the mpirun protocol, typing in the same terminal:<br>
<br>
$ cd /opt/intel/compilers_and_<wbr>libraries_2016.3.223/linux/<wbr>mpi/intel64/bin<br>
<br>
quantum@quantum-Precision-<wbr>Tower-7810:/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>3.223/linux/mpi/intel64/bin$ source mpivars.sh release<br>
<br>
quantum@quantum-Precision-<wbr>Tower-7810:/opt/intel/<wbr>compilers_and_libraries_2016.<wbr>3.223/linux/mpi/intel64/bin$<wbr>mpirun<br>
<br>
The mpirun command above gives a lot of lines that should end in something kinda:<br>
<br>
Intel(R) MPI Library for Linux* OS, Version 5.1.3 Build 20160601 (build id: 15562)<br>
Copyright (C) 2003-2016, Intel Corporation. All rights reserved.<br>
<br>
<br>
NOW GO BACK TO THE GPU FOLDER INSIDE espresso-x.x.x folder:<br>
<br>
$ cd /home/quantum/Descargas/QE-5.<wbr>4/espresso-5.4.0/GPU<br>
<br>
./configure --enable-parallel --enable-openmp --enable-cuda ?-without-scalapack with-gpu-arch=sm_35 --with-cuda-dir=/usr/local/<wbr>cuda-7.5/bin --without-magma --with-phigemm<br>
<br>
<br>
3. The ./configure command should create new files in the espresso root folder:<br>
<br>
Make.sys<br>
<br>
makefile.gpu<br>
<br>
# Since your are in terminal inside /home/quantum/Descargas/QE-5.<wbr>4/espresso-5.4.0/GPU folder , type<br>
<br>
cd ..<br>
<br>
# To come back to cd /home/quantum/Descargas/QE-5.<wbr>4/espresso-5.4.0 (in terminal).<br>
<br>
4. ALERT: Before doing Make, edit the Make.sys:<br>
<br>
- if you are using Intel MPI, please add to DFLAGS "-DMPICH_SKIP_MPICXX" to make.sys DFLAGS<br>
ignore MPI C++ bindings.<br>
<br>
ADD to THE make.sys FILE THE FOLLOWING LD_LIBS flags:<br>
<br>
-L/usr/lib64 -lstdc++<br>
<br>
??> THE ABOVE LINE IS CRUCIAL TO AVOID ERRORS during doing make compilation, errors such as<br>
<br>
?> ERROR: Too many symbolic link levels<br>
?> stdc++ errors<br>
?> A too buggy compilation<br>
?> Compilation finishes, but the pw-gpu.x or ph-gpu.x executables does not work properly<br>
<br>
...AND DON?T FORGET to add to the NVCC line the flag -D_FORCE_INLINES in make.sys:<br>
<br>
...<br>
NVCCFLAGS = -O3 -gencode arch=compute_35,code=sm_35 -D_FORCE_INLINES<br>
<br>
<br>
to make.sys for ph-gpu.x compilation!<br>
<br>
<br>
-> NOTICE: May be too naive or not worthy to comment about, but DO NOT attempt to compile ph-gpu.x BEFORE to compile pw-gpu.x. If you do so, you?ll SURELY get a bunch of fatal errors saying something like<br>
<br>
error: modules not found.<br>
<br>
If you do not use the all-gpu option by any reason, use THE VERY SAME logical compilation order<br>
<br>
make -f Makefile.gpu pw-gpu.x<br>
make -f Makefile.gpu ph-gpu.x<br>
make -f Makefile.gpu neb-gpu.x (if needed)<br>
<br>
<br>
5. FINAL: To build pw-gpu.x, ph-gpu.x and neb-gpu.x executables:<br>
<br>
make -f Makefile.gpu all-gpu<br>
<br>
<br>
<br>
<br>
**** sMISSION COMPLETE. ****<br>
<br>
<br>
<br>
*** RAMAN: POST-PROCESSING THE *.DMAT FILES THAT ph.x / ph-gpu.x CREATES : *** (always remember to type the actual paths)<br>
<br>
create a <custom-name>.<a href="http://dm.in" rel="noreferrer" target="_blank">dm.in</a> file containing (examples given):<br>
<br>
&input fildyn='/home/quantum/PWgui-5.<wbr>4.0/dmat.Mapbi3pospress', asr='simple' /<br>
<br>
then, in a terminal, go to dynmat.x location:<br>
<br>
$ cd /home/quantum/Descargas/QE-5.<wbr>4-NO-GPU/espresso-5.4.0/bin<br>
<br>
and type<br>
<br>
$ ./dynmat.x < /home/quantum/PWgui-5.4.0/<a href="http://mapbi3pospress.dm.in" rel="noreferrer" target="_blank">mapb<wbr>i3pospress.dm.in</a> > /home/quantum/PWgui-5.4.0/<wbr><a href="http://mapbi3pospress-asr-simple.dm">mapbi3pospress-asr-simple.dm</a>.<wbr>out<br>
<br>
note that in the output file, "asr-simple" points out the acoustic sum rule used;<br>
there are three basic options available:<br>
<br>
simple<br>
crystal<br>
zero-dim<br>
<br>
You should known which is best for each material you study (ask your tutor/supervisor).<br>
<br>
The resulting file, as in the example, <a href="http://mapbi3pospress-asr-simple.dm">mapbi3pospress-asr-simple.dm</a>.<wbr>out, contains the frequencies and intensities for both IR an Raman spectra, ready to plot.<br>
<br>
<br>
**** ---- END OF TUTORIAL - MAJOR UPDATE : AUG 24TH, 2016 ---- ****<br>
<br>
<br>
THAT?S ALL , THANKS FOR READING AND COMMENTING.<br>
<br>
<br>
KIND REGARDS,<br>
<br>
Josu? Clavijo<br>
Universidad Nacional de Colombia<br>
Science College<br>
Chemistry Department<br>
<br>
******************************<wbr>******************************<wbr>******************************<wbr>******************************<wbr>******************************<wbr>*****<br>
<br>
LINKS:<br>
<br>
<a href="http://askubuntu.com/questions/760934/graphics-issues-after-installing-ubuntu-16-04-with-nvidia-graphics" rel="noreferrer" target="_blank">http://askubuntu.com/<wbr>questions/760934/graphics-<wbr>issues-after-installing-<wbr>ubuntu-16-04-with-nvidia-<wbr>graphics</a><br>
<br>
<a href="https://blog.levilentz.com/?p=312" rel="noreferrer" target="_blank">https://blog.levilentz.com/?p=<wbr>312</a><br>
<br>
<a href="https://www.pugetsystems.com/labs/hpc/NVIDIA-CUDA-with-Ubuntu-16-04-beta-on-a-laptop-if-you-just-cannot-wait-775/" rel="noreferrer" target="_blank">https://www.pugetsystems.com/<wbr>labs/hpc/NVIDIA-CUDA-with-<wbr>Ubuntu-16-04-beta-on-a-laptop-<wbr>if-you-just-cannot-wait-775/</a><br>
<br>
?> PPA APT link for the Nvidia-364 driver repository:<br>
<br>
<a href="https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa" rel="noreferrer" target="_blank">https://launchpad.net/~<wbr>graphics-drivers/+archive/<wbr>ubuntu/ppa</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Thu, 25 Aug 2016 08:07:45 +0800<br>
From: Rolly Ng <<a href="mailto:rollyng@gmail.com">rollyng@gmail.com</a>><br>
Subject: Re: [Pw_forum] Tutorial: Install QE-GPU binaries in latest<br>
ubuntu systems (v. 16.04), Nvidia Cuda 7.5 , Intel MKL and Intel MPI<br>
software with NVIDIA cards hardware<br>
To: PWSCF Forum <<a href="mailto:pw_forum@pwscf.org">pw_forum@pwscf.org</a>><br>
Message-ID: <<a href="mailto:f9981ea0-3efb-9b5f-8434-bc037479d611@gmail.com">f9981ea0-3efb-9b5f-8434-<wbr>bc037479d611@gmail.com</a>><br>
Content-Type: text/plain; charset="windows-1252"<br>
<br>
Dear Josu?,<br>
<br>
Very nice and detail steps. Thank you!<br>
<br>
I would like to add that it also works for QE ver 5.3.0 on OpenSUSE 13.2.<br>
<br>
Supported GPU including:<br>
<br>
1) Tesla C2050<br>
<br>
2) Tesla C2070/75<br>
<br>
3) Titan Z/Black/Original<br>
<br>
Regards,<br>
<br>
Rolly<br>
<br>
<br>
On 08/25/2016 04:21 AM, Josue Itsman Clavijo Penagos wrote:<br>
> Dear all,<br>
><br>
> I?m posting the following rather introductory, not professional-level<br>
> tutorial for Installing QE-GPU binaries In recent ubuntu systems using<br>
> Nvidia Cuda, Intel MKL, Intel MPI software and NVIDIA GPU kepler-model<br>
> cards hardware, since i think this might be useful for any of you<br>
> fellow scientists struggling to get working their quantum espresso<br>
> GPU-enabled installations.<br>
><br>
> I, by no means, do pretend to offer an in-depth description of every<br>
> and all given steps, since i?m no a technology expert; in fact, much<br>
> of this are referred to mail discussions, Urls and documentation<br>
> readings found out inside the same used installation packages.<br>
><br>
> *The only purpose of this post* is to give some useful advice and,<br>
> mainly, to unify and share to whom may be interested all the available<br>
> information I?ve found in order to get, to the present extent of my<br>
> technology skills, a more-or-less complete and comprehensive tutorial<br>
> containing all what it?s needed to get success about QE-GPU<br>
> installation and usage.<br>
><br>
> A big *thank you* to the following scholars for all of your highly<br>
> valuable advice and key assistance to get this done:<br>
><br>
><br>
> *Ari Paavo Seitsonen*<br>
> *Claudio Quarti*<br>
> and all of you<br>
> *PW_FORUM FELLOWS*<br>
> *<br>
> *<br>
> *<br>
> *<br>
> All trademarks, copyrights and author ownerships over texts, codes and<br>
> original information links *are respected and are the exclusive<br>
> property of their rightfully legitimate owners*.<br>
><br>
> I apologize for any typos or vocabulary/redaction errors; I?m not a<br>
> native english speaker.<br>
><br>
><br>
> All said above, in the attached file I share what I did to get a<br>
> flawless compilation.<br>
><br>
> *Best regards, *<br>
><br>
> Josu? Clavijo, Dr. Sc. in Chemistry<br>
> Assistant Professor<br>
> Universidad Nacional de Colombia<br>
> Science College<br>
> Chemistry Department<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Pw_forum mailing list<br>
> <a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
> <a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br>
<br>
--<br>
PhD. Research Fellow,<br>
Dept. of Physics & Materials Science,<br>
City University of Hong Kong<br>
Tel: +852 3442 4000<br>
Fax: +852 3442 0538<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://pwscf.org/pipermail/pw_forum/attachments/20160825/6f3a4400/attachment-0001.html" rel="noreferrer" target="_blank">http://pwscf.org/pipermail/pw_<wbr>forum/attachments/20160825/<wbr>6f3a4400/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br>
<br>
End of Pw_forum Digest, Vol 109, Issue 25<br>
******************************<wbr>***********<br>
</blockquote></div><br></div>