I am unable to build Charm++ properly on Titan with this new argument. After adding the file with “touch src/arch/common/conv-mach-perftools.h”, the perftools argument is now accepted, but now the build fails. My current module list
is as follows:
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ module list
Currently Loaded Modulefiles:
1) eswrap/1.3.3-1.020200.1278.0
6) craype-interlagos
11) hsi/5.0.2.p1
16) ugni/6.0-1.0502.10863.8.28.gem
21) dvs/2.5_0.9.0-1.0502.2188.1.113.gem
26) perftools-base/6.4.5
2) craype-network-gemini
7) lustredu/1.4
12) DefApps
17) pmi/5.0.11
22) alps/5.2.4-2.0502.9774.31.12.gem
27) perftools
3) gcc/4.9.3
8) xalt/0.7.5
13) craype-hugepages8M
18) dmapp/7.0.1-1.0502.11080.8.74.gem
23) rca/1.0.0-2.0502.60530.1.63.gem
28) cudatoolkit/7.5.18-1.0502.10743.2.1
4) craype/2.5.9
9) module_msg/0.1
14) cray-libsci/16.11.1
19) gni-headers/4.0-1.0502.10859.7.8.gem
24) atp/2.0.5
5) cray-mpich/7.5.2
10) modulator/1.2.0
15) udreg/2.3.2-1.0502.10518.2.17.gem
20) xpmem/0.1-2.0502.64982.5.3.gem
25) PrgEnv-gnu/5.2.82
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$
As per the specific Cray XK7 installation instructions in the Charm++ documentation, I issued the build command (with the perftools argument added):
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ ./build charm++ gni-crayxe-cuda perftools --with-production -j8
However, the build fails (I attached the build output). Should I not be using the latest development version (6.8.0) and instead use v6.7.1 in the manner described below? Thanks.
-Tom
From: Ted Packwood <malice AT cray.com>
Date: Thursday, July 6, 2017 at 1:47 PM
To: Phil Miller <mille121 AT illinois.edu>
Cc: "Papatheodore, Thomas L." <papatheodore AT ornl.gov>, "charm AT lists.cs.illinois.edu" <charm AT lists.cs.illinois.edu>
Subject: Re: [charm] CrayPat with Charm++
Please note the document your referenced in your first email says:
"Building for Perftools
Do not load perftools or perftools-lite when building Charm++ (perftools-base can
be safely loaded prior to building). Instead, ensure frame pointers are not omitted in
your conv-mach.sh file (this is the new default for Cray XC builds). These options are
provided in the new conv-mach.sh file submitted to the Charm++ developer"
This document was intended for users wanting to build charm++ 6.7.1 (the document
was published prior to the addition of perftools support being added into Charm++
develop). The line "ensure frame pointers are not omitted in your conv-mach.sh file"
is key.
If using v6.7.1, you can do this yourself by adding the following lines to your
conv-mach.sh file. This will support any PrgEnv on your machine (though PGI
has not been tested).
BUILD_FOR_PERFTOOLS=1 # Comment out if desired
# If building for Cray perftools, use the following.
# Do not have perftools or perftools-lite loaded during
# the build of charm++ (perftools-base is ok)
if test -n "$BUILD_FOR_PERFTOOLS"
then
CRAYPAT_FLAGS=" "
if test -n "$PGCC"
then
CRAYPAT_FLAGS=" "
elif test -n "$CCE"
then
CRAYPAT_FLAGS=" -hkeep_frame_pointer"
elif test -n "$ICPC"
then
CRAYPAT_FLAGS=" -fno-omit-frame-pointer"
else # gcc
CRAYPAT_FLAGS=" -fno-omit-frame-pointer"
fi
fi
Then change these lines in the same file:
CMK_CC="cc $CRAYPAT_FLAGS "
CMK_CXX="CC $CRAYPAT_FLAGS "
The process (v6.7.1) therefore is:
1) Add the compiler flags to your conv-mach.sh file above
2) Build charm++ as normal, *without* perftools or perftools-lite loaded
3) Load perftools or perftools-lite
4) Build and link your example program.
5) pat_build your program (if using perftools). (skip this step if using perftools-lite).
6) Run as normal.
I'll work with Phil to ensure that we get all this straightened out in 6.8.0.
If you'd like, I can work with you directly if you have any additional problems.
Ted
On 07/06/2017 11:42 AM, Phil Miller wrote:
Sorry, I messed up the path to touch. It should have been `src/arch/common/conv-mach-perftools.h`
On Thu, Jul 6, 2017 at 11:29 AM, Papatheodore, Thomas L. <papatheodore AT ornl.gov> wrote:
I created the file with “touch”, but I still get the same error:
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ touch src/arch/conv-mach-perftools.h
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ ./build charm++ gni-crayxc smp perftools --with-production -g
Error> option: perftools is not supported in this version!
Supported compilers: clang craycc gcc icc iccstatic pgcc xlc xlc64
Supported options: bigemulator bigsim causalft cuda g95 gfortran hugepages ifort lcs mlogft nolb omp ooc papi persistent pgf90 pxshm regularpages smp syncft tsan xpmem
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ ls src/arch/
common
gni
mpi-crayxc
mpi-linux-mips64
multicore
multicore-linux64
net
net-linux-ppc
netlrts-linux-arm7
pami-bluegeneq
shmem-crayxe
uth-linux
verbs-linux-x86_64
conv-mach-fix.sh
gni-crayxc
mpi-crayxe
mpi-linux-ppc
multicore-arm7
multicore-linux-ppc
net-darwin-x86_64
net-linux-x86_64
netlrts-linux-ppc
pami-linux-ppc64le
sim
uth-linux-x86_64
win
conv-mach-perftools.h
gni-crayxe
mpi-darwin-x86_64
mpi-linux-x86_64
multicore-darwin-x86_64
multicore-linux-x86_64
net-linux
netlrts
netlrts-linux-x86_64
pamilrts
sim-linux
util
win32
cuda
mpi
mpi-linux
mpi-win64
multicore-linux
multicore-win64
net-linux-amd64
netlrts-darwin-x86_64
netlrts-win-x86_64
pamilrts-bluegeneq
template
verbs
win64
gemini_gni-crayxe
mpi-bluegeneq
mpi-linux-amd64
mpi-win-x86_64
multicore-linux32
multicore-win-x86_64
net-linux-arm7
netlrts-linux
pami
shmem
uth
verbs-linux-ppc64le
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$
Please let me know if there is something else I need to do. Thanks again.
-Tom
On Thu, Jul 6, 2017 at 11:06 AM, Papatheodore, Thomas L. <papatheodore AT ornl.gov> wrote:
I grabbed the latest development version from github (git clone
https://charm.cs.illinois.edu/gerrit/charm.git) as described on the charm++ downloads page.
The CHANGES file states it is version 6.8.0:
What's new in Charm++ 6.8.0
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ git describe
v6.8.0-beta1-287-gd57c83d
OK, I see. In the future, we should label the contents of the CHANGES file tentatively until a final release is tagged and announced.
There is not a file called “conv-mach-perftools.h” on src/arch/, but there is a “conv-mach-fix.sh”.
Indeed there isn't. The intention of the command I asked you to run is to create it - our build script looks for both a header snippet and a shell snippet for build-time options.
In this case, the header snippet was missing, and so it thought the option was invalid.
On Thu, Jul 6, 2017 at 10:20 AM, Papatheodore, Thomas L. <papatheodore AT ornl.gov> wrote:
Hey Phil-
Wow! Thanks for the quick and thorough response! I’m using version 6.8.0, but when I add the “perftools” argument I get:
Uhhh... Version 6.8.0 hasn't been released yet. We announced a beta of it some months ago, but not the final release. Could you share the name of the file containing the code calling
itself Charm++ 6.8.0, and output of 'git describe' in the Charm++ directory?
[tpapathe@titan-ext2: /lustre/atlas2/csc198/proj-shared/tpapathe/charm]$ ./build charm++ gni-crayxc smp
perftools --with-production -g
Error> option: perftools is not supported in this version!
Supported compilers: clang craycc gcc icc iccstatic pgcc xlc xlc64
Supported options: bigemulator bigsim causalft cuda g95 gfortran hugepages ifort lcs mlogft nolb omp ooc
papi persistent pgf90 pxshm regularpages smp syncft tsan xpmem
Is there a version yet that supports this argument? Thanks.
This seems to be the result of a mistake on our part. In the charm++ directory, please run
touch src/arch/conv-mach-perftools.h
And then try the build. If that works, please let us know.
-Tom
What version of Charm++ did you use in your attempt(s)? As noted in the linked document, the changes in Charm++ to support CrayPat are in our development line, but have not quite
yet been released.
Our compiler wrapper 'charmc' has always called through the Cray compiler wrappers on Cray systems. If some component is missing or not working correctly, the problem lies elsewhere.
Looking through that document, there seems to be some skew introduced between the patch to Charm++ as Cray provided it, and what we ultimately integrated. On your Charm++ 'build' command
line, you'll need to add the 'perftools' option like so to get all of the necessary compiler settings:
./build charm++ gni-crayxc smp perftools --with-production -g
This difference was introduced (I believe) out of concern for a potential negative performance impact of always applying the flags in question, since they reduce the number of CPU registers
available to the compiler.
Also note the lack of 'craycc' option on that line - via Cray's CC, charmc will always use the compiler selected by the loaded PrgEnv-* module.
There may be some other discrepancies between the documentation provided by Cray and the current state of the various software components. Feel free to get back to us with further
issues. In the meanwhile, I'll have someone on the Charm++ team walk through the entire process and see what other updates might need to be made.
--
On Thu, Jul 6, 2017 at 8:47 AM, Papatheodore, Thomas L. <papatheodore AT ornl.gov> wrote:
Hello. I am having trouble running CrayPat on a Charm++ example problem on Titan (Cray XK7) at ORNL. The error message that I encounter when trying to instrument the executable
with pat_build is:
ERROR: Missing required ELF section '.note.link' from the program '/lustre/atlas2/csc198/proj-shared/tpapathe/test/charm-cuda/multicore-linux-x86_64-cuda/tests/charm++/simplearrayhello/hello'.
Load the correct 'perftools' module and rebuild the program.
This error is usually caused by not having the perftools-base and perftools modules loaded before compiling a program. However, I compiled the program with these modules loaded
and I still receive the message. I also recompiled Charm++ itself with these modules loaded, but still get the message. Is there anything special I need to do to get CrayPat to profile Charm++ applications? I found the Cray document (http://docs.cray.com/books/S-2802-10//S-2802-10.pdf),
but this does not resolve my issue. Could it be that Charm++ compiles with charmc and CrayPat requires the Cray compiler wrappers (e.g. CC, ftn, etc.)?
Any help would be appreciated. Thanks.
-Tom
|
