Glad to hear it worked. But the point about the need to use MPI build needs to be clarified:
I think what you are saying is that: since you need to communicate between nodes, you need the MPI build. But (if so) that is not necessary. Charm++ can use lower level layers than MPI,
and that can sometimes be faster. On infiniband networks, for example, you could use the vrbs build. On proprietary networks (such as Cray’s GNI or IBM’s PAMI) also, the native builds are faster.
201 N. Goodwin Avenue Ph: (217) 244-0094
From:
Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>
Reply-To: Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>
Date: Tuesday, October 4, 2016 at 5:13 AM
To: "Buch, Ronak Akshay" <rabuch2 AT illinois.edu>
Cc: "Mikida, Eric P" <mikida2 AT illinois.edu>, "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
Subject: Re: [charm] compilation of the dense LU benchmark
Thank you very much for your answer!
No, I don't plan to use offload mode. The main goal is to attempt to experiment in an multi-node environment with Intel Xeon Haswells and Intel Knights Corner CPUs, where Intel Knight
Corner are used in native mode. So I do need MPI to communicate between nodes.
If you can provide some information about your machine, we can give you more pointed advice on how best to execute in your environment.
I would be very grateful! Thank you for your time!
2016-10-04 1:00 GMT+03:00 Ronak Buch <rabuch2 AT illinois.edu>:
What sort of heterogeneous environment do you want to execute in? I assume since you're talking about MKL that you're interested in using Xeon Phis. If so, MKL should take care of the heterogeneous execution (see https://software.intel.com/sites/default/files/11MIC42_How_to_Use_MKL_Automatic_Offload_0.pdf
for details).
As for the compiler, Charm++ doesn't really have its own compiler, we just wrap another compiler under the hood. Presumably in this case you'll want to use icc, which should handle the AVX2 vectorization.
One other thing is that you seem to have built an MPI build of Charm++. In general, building on top of MPI isn't the best way to build Charm++, as it is often slower than our own native layers. If you can provide some information about
your machine, we can give you more pointed advice on how best to execute in your environment.
On Mon, Oct 3, 2016 at 4:50 PM, Mikida, Eric P <mikida2 AT illinois.edu> wrote:
Ah, that is my fault. I was compiling with an older version of Charm++. The comlib module was deprecated, and is no longer included in Charm 6.7.1. I’ve pushed another fix that removes the -module comlib from Makefile so it should build
fine now. For your second question, if it can’t find ckmulticast.h it probably means you didn’t build LIBS. Did you accidentally clean the build before building everylb?
I’m not as familiar with the heterogeneous configuration part, so I’ll make sure to hand that off to someone else in the lab.
Now compilation phase passes successfully, but I've encountered another issue during linking stage. It requires some "module comlib":
Generating dependencies for
driver.ci
Generating dependencies for
lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E
lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E
driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -c -o lu.o lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -c -o scheduler.o scheduler.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -c -o benchmark.o benchmark.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -module comlib -module CkMulticast -o charmlu lu.o scheduler.o benchmark.o -L/opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a
/opt/intel/mkl/lib/intel64/libmkl_core.a /opt/intel/mkl/lib/intel64/libmkl_sequential.a /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl
/usr/bin/ld: cannot find -lmodulecomlib
collect2: error: ld returned 1 exit status
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
My CHARM is builded with the next options:
./build LIBS mpi-linux-x86_64 mpicxx
and I can't found the module
comlib; there also no tests with it. I have charm-6.7.1.
What am I doing wrong now?
I'm tried to compile EvelyLB target for charm:
./build everylb mpi-linux-x86_64 mpicxx
where mpicxx is Intel mpi.
But the dense LU benchmark again encountered compilation issue, now with the
ckmulticast.h header file for this Charm build:
Generating dependencies for
driver.ci
Generating dependencies for
lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E
lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E
driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -c -o lu.o lu.C
In file included from manager.h:5:0,
from lu.h:1,
from lu.C:13:
messages.h:5:25: fatal error: ckmulticast.h: No such file or directory
#include <ckmulticast.h>
^
compilation terminated.
Could you, please, give me advise, how the CHARM should be configured to use the heterogeneous environment, Intel MPI (and what about avx2 features for the compiler)?
I want to try to run the dense LU benchmark in heterogeneous environment with Intel CPUs...
Thank you very much for your time!
2016-10-03 3:40 GMT+03:00 Mikida, Eric P <mikida2 AT illinois.edu>:
Hi Ekaterina,
This one was actually our fault. There was a syntax error in
lu.ci that popped up after a small change to how the SDAG language is defined. I fixed that error and pushed my change back to the repository. If you do a pull and try to compile again, it should work this time but let me know if more problems crop up.
and I've encountered problems in linking it with Intel MKL BLAS.
It's my first experience with CHARM, so sorry for the maybe obvious question.
I have next error:
Generating dependencies for
driver.ci
Generating dependencies for
lu.ci
STDIN:303:575-575: error: invalid SDAG member
};
^
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
Command /storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmxi -M -orig-file
lu.ci returned error code 1
charmc exiting...
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
In file included from benchmark.C:2:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
#error "No BLAS Header files included!"
^
Generating dependencies for scheduler.C
Generating dependencies for lu.C
In file included from lu.C:15:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
#error "No BLAS Header files included!"
^
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51 -c -o lu.o lu.C
In file included from lu.h:1:0,
from lu.C:13:
manager.h:4:26: fatal error: luUtils.decl.h: No such file or directory
#include "luUtils.decl.h"
^
compilation terminated.
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
Command mpicxx -m64 -fPIC -I/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/../include -D__CHARMC__=1 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51 -O3 -fno-stack-protector -std=c++0x -c lu.C -o lu.o
returned error code 1
charmc exiting...
make: *** [lu.o] Error 1
My
config.mk contains:
SEND_LIM = 2
MKLROOT = /opt/intel/mkl
BLAS_INC = -I${MKLROOT}/include
BLAS_LIBS = ${MKLROOT}/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group
-lpthread -lm -ldl
The include path do contains BLAS headers:
ls -la /opt/intel/mkl/include/mkl_blas.h
-rw-r--r--. 1 root root 76919 Jun 29 14:25 /opt/intel/mkl/include/mkl_blas.h
Please, could you be so kind to give me some hint, what am I doing wrong? Where should I start?
Should I define MKL usage somewhere?
--
--
|
