Charm++ parallel programming system

["Choi, Jaemin" <jchoi157 AT illinois.edu>]

Hi Pierre,

 

The Charm++ topology code currently uses socket IP addresses to distinguish between physical nodes.

The LrtsInitCpuTopo routine in src/conv-core/cputopology.C obtains the IP address by calling skt_my_ip, which is defined in src/util/sockRoutines.C.

It seems to use getifaddr if that’s available, and I don’t think it’s reading anything from SLURM.

In the for loop in cpuTopoHandler, it loops through the IP addresses received from all the PEs and constructs the hostTable, which should have one record per physical node.

I think one way of debugging this would be printing out the IP addresses in cpuTopoHandler and see if some PEs in the same physical node have different IP addresses (which seems to cause this issue of detecting 15 hosts instead of the actual 8).

 

One other thing, I don’t think ‘mpi-linux-amd64’ is a valid target when building Charm++, I believe it should be ‘mpi-linux-x86_64’.

 

Best,

 

Jaemin Choi

PhD Candidate in Computer Science

Research Assistant at the Parallel Programming Laboratory

University of Illinois Urbana-Champaign

 

From: Kale, Laxmikant V <kale AT illinois.edu>
Sent: Thursday, September 9, 2021 10:37 AM
To: charm AT lists.cs.illinois.edu
Cc: Buch, Ronak Akshay <rabuch2 AT illinois.edu>; Choi, Jaemin <jchoi157 AT illinois.edu>
Subject: Fw: question about charm++ IP address definition as part of cputopology.C

 

 

From: Carrier, Pierre <pierre.carrier AT hpe.com>
Sent: Wednesday, September 8, 2021 3:10 PM
To: Kale, Laxmikant V <kale AT illinois.edu>
Cc: McMahon, Kim <kim.mcmahon AT hpe.com>; Gilmer, Brian F <brian.gilmer AT hpe.com>; Warren, Steven <steven.warren AT hpe.com>
Subject: RE: question about charm++ IP address definition as part of cputopology.C

 

... Charm++ is built using these steps:

 

module load cray-fftw

module load craype-hugepages8M

module load gcc

module swap gcc/10.2.0 gcc/9.3.0

module load cudatoolkit/20.9_11.0

module list

export LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH

 

tar xvf tcl8.5.9-linux-x86_64.tar

cd tcl8.5.9-linux-x86_64

export TCLDIR=`pwd`

cd ..

 

tar xvf NAMD_2.14_Source.tar.gz

cd NAMD_2.14_Source/

tar xvf charm-6.10.2.tar

cd charm-6.10.2/

 

# echo $CRAY_LD_LIBRARY_PATH

# echo ${PE_MPICH_GENCOMPILERS_GNU}

# echo ${CRAY_MPICH_BASEDIR}/gnu/${PE_MPICH_GENCOMPILERS_GNU}

 

export CRAY_MPICH_GNU_BASEDIR=${CRAY_MPICH_BASEDIR}/gnu/${PE_MPICH_GENCOMPILERS_GNU}

./build charm++ mpi-linux-amd64 gcc smp --incdir=${CRAY_MPICH_GNU_BASEDIR}/include --libdir=${CRAY_MPICH_GNU_BASEDIR}/lib --with-production -j4

export CHARM_DIR=mpi-linux-amd64-smp-gcc

export ARCH=Linux-x86_64-g++

 

export ARCH=Linux-x86_64-g++

export CHARM_DIR=mpi-linux-amd64-smp-gcc

sed -i '459d' config

echo $TCLDIR

./config $ARCH --charm-arch $CHARM_DIR --with-fftw3 --fftw-prefix $FFTW_ROOT --with-memopt --with-cuda --with-tcl --tcl-prefix $TCLDIR

cd Linux-x86_64-g++

make

 

Building NAMD with UCX doesn’t work. On our internal system this recipe works and allows to run 4 GPU per node. However, on a customer’s machine, with a different SLURM configuration, then the error shown below appears.

 

Thank you.

Best regards,

Pierre

 

From: Carrier, Pierre
Sent: Wednesday, September 8, 2021 2:56 PM
To: kale AT illinois.edu
Cc: McMahon, Kim <kim.mcmahon AT hpe.com>; Gilmer, Brian F <brian.gilmer AT hpe.com>; Warren, Steven <steven.warren AT hpe.com>
Subject: question about charm++ IP address definition as part of cputopology.C

 

Hi Prof. Kale,

 

I work at HPE on some NAMD benchmarks, with others (in CC) that are currently trying to resolve the following problem. On one of our systems the number of nodes is incorrect when trying to run with 4 GPU per node. For example, I get the following output

 

Charm++> Running in SMP mode: 32 processes, 4 worker threads (PEs) + 1 comm threads per process, 128 PEs total

Charm++> Running on 15 hosts (1 sockets x 64 cores x 2 PUs = 128-way SMP)

 

...where the SLURM script uses the following syntax:

 

#SBATCH --nodes=8

...

srun --ntasks=32 --ntasks-per-node=4 –cpu-bind=none  \

             ${NAMD_PATH}/namd2 ++ppn 4 +devices 0,1,2,3 ${INPUT_PATH}/chromat100-bench.namd &> namd.log

 

Using that subdivision, I expect to be running on 8 nodes. Following that error, the output becomes:

 

FATAL ERROR: Number of devices (4) is not a multiple of number of processes (3).  Sharing devices between processes is inefficient.  Specify +ignoresharing (each process uses all visible devices) if not all devices are visible to each process, otherwise adjust number of processes to evenly divide number of devices, specify subset of devices with +devices argument (e.g., +devices 0,2), or multiply list shared devices (e.g., +devices 0,1,2,0).

 

Which is just a consequence of the fact that the number of nodes numNodes is incorrect.

 

I could trace the error down to the variable “topomsg->nodes”, which is incorrectly computed, and hostTable.size(), at the line where printTopology is called:

 

/* called on PE 0 */

static void cpuTopoHandler(void *m)

{

  _procInfo *rec;

  hostnameMsg *msg = (hostnameMsg *)m;

  int pe;

 

  if (topomsg == NULL) {

    int i;

    topomsg = (nodeTopoMsg *)CmiAlloc(sizeof(nodeTopoMsg)+CmiNumPes()*sizeof(int));

    CmiSetHandler((char *)topomsg, CpvAccess(cpuTopoRecvHandlerIdx));

    topomsg->nodes = (int *)((char*)topomsg + sizeof(nodeTopoMsg));

    for (i=0; i<CmiNumPes(); i++) topomsg->nodes[i] = -1;

  }

  CmiAssert(topomsg != NULL);

 

  msg->procs = (_procInfo*)((char*)msg + sizeof(hostnameMsg));

  CmiAssert(msg->n == CmiNumPes());

  for (int i=0; i<msg->n; i++)

  {

    _procInfo *proc = msg->procs+i;

 

/*   for debug

  skt_print_ip(str, msg->ip);

  printf("hostname: %d %s\n", msg->pe, str);

*/

    skt_ip_t & ip = proc->ip;

    pe = proc->pe;

    auto iter = hostTable.find(ip);

    if (iter != hostTable.end()) {

      rec = iter->second;

    }

    else {

      proc->nodeID = pe;           // we will compact the node ID later

      rec = proc;

      hostTable.emplace(ip, proc);

    }

    topomsg->nodes[pe] = rec->nodeID;

    rec->rank ++;

  }

 

//////////   for (int i=0; i<CmiNumPes(); i++) topomsg->nodes[i] = -1;

//////////   for (int i=0; i < 16; i++) {

//////////      topomsg->nodes[i +   0] =   0;

//////////      topomsg->nodes[i +  16] =  16;

//////////      topomsg->nodes[i +  32] =  40;

//////////      topomsg->nodes[i +  48] =  60;

//////////      topomsg->nodes[i +  64] =  76;

//////////      topomsg->nodes[i +  80] =  80;

//////////      topomsg->nodes[i +  96] = 108;

//////////      topomsg->nodes[i + 112] = 116;

//////////   }

 

  for (int i=0; i < CmiNumPes(); i++) {

     printf("DEBUG PIERRE topomsg->nodes[%d]=%d\n", i, topomsg->nodes[i]);

  }

 

  printTopology(hostTable.size());

 

  hostTable.clear();

  CmiFree(msg);

 

  CmiSyncBroadcastAllAndFree(sizeof(nodeTopoMsg)+CmiNumPes()*sizeof(int), (char *)topomsg);

}

 

The comments that I added are the values I’m supposed to have when running on a different system that is configured differently with SLURM but can run correctly.

 

Could you please direct me to someone that can explain the principles of this part of the charm++ code, in particular, what variables are read from the system (SLURM?) in order to define the proc->ip and nodeIDs? And the numNodes.

 

That part of the charm++ program was done by:

 

/** This scheme relies on using IP address to identify physical nodes

* written by Gengbin Zheng  9/2008

 

...but I believe that he is now at Intel, if LinkedIn is up-to-date.

 

Thank you for your help.

 

Best regards,

Pierre

Pierre Carrier, Ph.D.

Apps & Performance Engineering

pierre.carrier AT hpe.com  

(651)354-3570