Charm++ parallel programming system

[Ekaterina Tutlyaeva <xgl AT rsc-tech.ru>]

Dear Ronak, Support,

I've problems with cross-compiling CHARM++ for mic. (I can't compile it on co-processer directly, because It is a cluster with SLURM manager system).

How can I cross-compile CHARM++?

My build options:

./build LIBS mpi-linux-x86_64 mpicxx mic -j16 --with-production -L/opt/software/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/lib -lmpi -I/opt/software/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/include

compiler:

which mpicxx
/opt/software/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/bin/mpicxx

The build process runs successfully, until the execution phase started:

../bin/charmc: line 192: ../bin/charmxi: cannot execute binary file
Fatal Error by charmc in directory /home3/iki5/ordi/src/charm/charm-6.7.1/mpi-linux-x86_64-mic-mpicxx/tmp
   Command ../bin/charmxi -intrinsic -orig-file ckcallback.ci returned error code 126
charmc exiting...
../bin/charmc: line 192: ../bin/charmxi: cannot execute binary file
../bin/charmc -intrinsic -optimize -production  -L/opt/software/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/lib -lmpi -I/opt/software/intel/compilers_and_libraries_2016.2.181/linux/mpi/mic/include  NodeLevelLB.ci && touch NodeLevelLB.ci.stamp
../bin/charmc: line 192: ../bin/charmxi: cannot execute binary file
Fatal Error by charmc in directory /home3/iki5/ordi/src/charm/charm-6.7.1/mpi-linux-x86_64-mic-mpicxx/tmp

Of course, it can't execute, because binaries are created for co-processor.
I've tried the -host=linux-k1om-4.7 option, but it doesn't helps.. May be there is the way to disable executing during the build process?

Thank you :)

2016-10-25 12:18 GMT+03:00 Ronak Buch <rabuch2 AT illinois.edu>:

Hello Ekaterina,

The recommended way to run Charm++ programs in a heterogeneous way across x86 host cores and Xeon Phis is using Intel MPI.

Build two MPI builds of Charm++ (for example, mpi-linux-x86_64), one for the host and one for the Xeon Phi.  For example:

./build LIBS mpi-linux-x86_64 -j16 --with-production -DMPICH_IGNORE_CXX_SEEK -DMPICH_SKIP_MPICXX
./build LIBS mpi-linux-x86_64 mic -j16 --with-production

Then, your application should be compiled in each build of Charm++.  The Xeon Phi version should then be placed on the Xeon Phi (or made accessible via a shared filesystem).

Finally, to run, use mpirun to execute across the various devices.  The necessary options may vary based on system configuration (for example, in this case, we had to copy Xeon Phi versions of the MPI dynamic libraries to the card for execution to work).

export I_MPI_MIC=enable
mpirun -n 1 -host <x86 hostname> ./hello : -n 1 -host <Xeon Phi hostname> -genv LD_LIBRARY_PATH=~/mpiLibs ~/hello

See here for more information (this is specifically for Intel MPI, but is largely applicable to Charm++ as well): https://software.intel.com/en-us/articles/how-to-run-intel-mpi-on-xeon-phi

Thanks,

Ronak

On Mon, Oct 24, 2016 at 6:36 AM, Ekaterina Tutlyaeva <xgl AT rsc-tech.ru> wrote:

Dear support,

Could you be so kind to give me some advise, how should I build the Charm for run in heterogeneous environment (Intel Xeon Phi 7120 in native mode and Intel Xeon E5-2697 v3)? Interconnect is Infiniband.

The goal is to run dense LU benchmark: http://charmplusplus.org/miniApps/#denselu in this environment.

I've tried some CHARM builds, for example verbs-linux-x86_64-icc. It works nice with
Intel Xeon E5-2697.

But whan I try to compile LU benchmark for Xeon Phi, I got the k1om incompatibility issue with charm modules, not suited for Phi.

x86_64-k1om-linux-ld: skipping incompatible /home3/iki5/ordi/src/charm/charm-6.7.1/v                    erbs-linux-x86_64-icc/bin/../lib/libmoduleCkMulticast.a when searching for -lmoduleC                    kMulticast
x86_64-k1om-linux-ld: cannot find -lmoduleCkMulticast

Thank and sorry for your time. I'm tried hard to find some information on the topic, but can't do it myself.

How could I build it all for heterogeneous support?

Thank you!!!!

Best regards,

Ekaterina

--

__________

С уважением,

Тютляева Екатерина