Charm++ parallel programming system

[Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>]

Dear Eric and all,

Thank you for your help!

I've compiled now Charm LU successfully.

I've rebuilded charm once again with incdir and libdir options instead just pointing to mpicxx.

Thank you for your help with this issue.

I'm so hope to your help with heterogeneous run and Intel Xeon optimizations!

Thank you!

Best regards,

Ekaterina

2016-10-04 13:46 GMT+03:00 Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>:

Dear Eric,

Thank you!
But i'm definitely looser...
Without the comlib module I still have some linking issues with MPI, while I've pointed the way to MPI libs...

/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -module CkMulticast  -L/opt/intel/impi/2017.0.098//lib64 -o charmlu lu.o scheduler.o benchmark.o /opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a /opt/intel/mkl/lib/intel64/libmkl_core.a /opt/intel/mkl/lib/intel64/libmkl_sequential.a /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/../lib/libck.a(mpi-interoperate.o): In function `CharmLibInit':
mpi-interoperate.C:(.text+0x4f2): undefined reference to `AMPI_Comm_dup'
mpi-interoperate.C:(.text+0x50c): undefined reference to `AMPI_Comm_size'
mpi-interoperate.C:(.text+0x526): undefined reference to `AMPI_Comm_rank'

Did you accidentally clean the build before building everylb?

O, yes, thank you. Yes, my fault. I've rebuild AMPI and LIBS only from scratch.

Best regards,

Ekaterina

 

2016-10-04 0:50 GMT+03:00 Mikida, Eric P <mikida2 AT illinois.edu>:

Ah, that is my fault. I was compiling with an older version of Charm++. The comlib module was deprecated, and is no longer included in Charm 6.7.1. I’ve pushed another fix that removes the -module comlib from Makefile so it should build fine now. For your second question, if it can’t find ckmulticast.h it probably means you didn’t build LIBS. Did you accidentally clean the build before building everylb?

I’m not as familiar with the heterogeneous configuration part, so I’ll make sure to hand that off to someone else in the lab.

Thanks

Eric

On Oct 3, 2016, at 7:54 AM, Ekaterina Tutlyaeva <xgl AT rsc-tech.ru> wrote:

Dear Eric,

Thank you for your fix!

Now compilation phase passes successfully, but I've encountered another issue during linking stage. It requires some "module comlib":

Generating dependencies for driver.ci
Generating dependencies for lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o lu.o lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o scheduler.o scheduler.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o benchmark.o benchmark.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -module comlib -module CkMulticast  -o charmlu lu.o scheduler.o benchmark.o -L/opt/intel/mkl/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group /opt/intel/mkl/lib/intel64/libmkl_intel_ilp64.a /opt/intel/mkl/lib/intel64/libmkl_core.a /opt/intel/mkl/lib/intel64/libmkl_sequential.a /opt/intel/mkl/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl
/usr/bin/ld: cannot find -lmodulecomlib
collect2: error: ld returned 1 exit status
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu

My CHARM is builded with the next options:

./build LIBS mpi-linux-x86_64 mpicxx

and I can't found the module comlib; there also no tests with it. I have charm-6.7.1.

What am I doing wrong now?

I'm tried to compile EvelyLB target for charm:

./build everylb mpi-linux-x86_64 mpicxx

where mpicxx is Intel mpi.

But the dense LU benchmark again encountered compilation issue, now with the ckmulticast.h header file for this Charm build:

Generating dependencies for driver.ci
Generating dependencies for lu.ci
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
Generating dependencies for scheduler.C
Generating dependencies for lu.C
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E luUtils.ci && touch luUtils.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E lu.ci && touch lu.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1 -E driver.ci && touch driver.ci.stamp
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -I/opt/intel/mkl/include -DLU_REVISION=a336a1b68dec67d067035a829eea046ddf118ac1  -c -o lu.o lu.C
In file included from manager.h:5:0,
                 from lu.h:1,
                 from lu.C:13:
messages.h:5:25: fatal error: ckmulticast.h: No such file or directory
 #include <ckmulticast.h>
                         ^
compilation terminated.

Could you, please, give me advise, how the CHARM should be configured to use the heterogeneous environment, Intel MPI (and what about avx2 features for the compiler)?
I want to try to run the dense LU benchmark in heterogeneous environment with Intel CPUs...

Thank you very much for your time!

best regards,

Ekaterina

2016-10-03 3:40 GMT+03:00 Mikida, Eric P <mikida2 AT illinois.edu>:

Hi Ekaterina,

This one was actually our fault. There was a syntax error in lu.ci that popped up after a small change to how the SDAG language is defined. I fixed that error and pushed my change back to the repository. If you do a pull and try to compile again, it should work this time but let me know if more problems crop up.

Thanks

Eric

On Oct 2, 2016, at 10:03 AM, Ekaterina Tutlyaeva <xgl AT rsc-tech.ru> wrote:

Dear Sir/Madam,

I'm trying to compile the dense LU benchmark: http://charmplusplus.org/miniApps/#denselu

and I've encountered problems in linking it with Intel MKL BLAS.

It's my first experience with CHARM, so sorry for the maybe obvious question.

I have next error:

Generating dependencies for driver.ci
Generating dependencies for lu.ci
STDIN:303:575-575: error: invalid SDAG member
  };
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ^
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
   Command /storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmxi -M -orig-file lu.ci returned error code 1
charmc exiting...
Generating dependencies for luUtils.ci
Generating dependencies for benchmark.C
In file included from benchmark.C:2:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
 #error "No BLAS Header files included!"
  ^
Generating dependencies for scheduler.C
Generating dependencies for lu.C
In file included from lu.C:15:0:
platformBlas.h:38:2: error: #error "No BLAS Header files included!"
 #error "No BLAS Header files included!"
  ^
/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/charmc -language charm++ -O3 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51  -c -o lu.o lu.C
In file included from lu.h:1:0,
                 from lu.C:13:
manager.h:4:26: fatal error: luUtils.decl.h: No such file or directory
 #include "luUtils.decl.h"
                          ^
compilation terminated.
Fatal Error by charmc in directory /storage/CHARM/charmlu/charmlu
   Command mpicxx -m64 -fPIC -I/storage/CHARM/charm-6.7.1/mpi-linux-x86_64-mpicxx/bin/../include -D__CHARMC__=1 -DSEND_LIM=2 -I/opt/intel/mkl/include -DLU_REVISION=7cec5b0ba2a6752d4160976d00cc3881e4e64b51 -O3 -fno-stack-protector -std=c++0x -c lu.C -o lu.o returned error code 1
charmc exiting...
make: *** [lu.o] Error 1

My config.mk contains:

SEND_LIM  = 2
MKLROOT = /opt/intel/mkl
BLAS_INC =  -I${MKLROOT}/include
BLAS_LIBS = ${MKLROOT}/lib/intel64/libmkl_scalapack_ilp64.a -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_ilp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_sequential.a ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_ilp64.a -Wl,--end-group -lpthread -lm -ldl

The include path do contains BLAS headers:

ls -la /opt/intel/mkl/include/mkl_blas.h
-rw-r--r--. 1 root root 76919 Jun 29 14:25 /opt/intel/mkl/include/mkl_blas.h

 

 

Please, could you be so kind to give me some hint, what am I doing wrong? Where should I start?

Should I define MKL usage somewhere?

 Thank you for your time!

 

__________

Best regards,

Ekaterina Tyutlyaeva

--

__________

С уважением,

Тютляева Екатерина

--

__________

С уважением,

Тютляева Екатерина

--

__________

С уважением,

Тютляева Екатерина