Charm++ parallel programming system

[Marcin Mielniczuk <marmistrz.dev AT zoho.eu>]

Yes, with AMPI_INFO_LB_SYNC the migration is attempted, but the program crashes.

What exactly do you mean by a synchronization point? Do you mean that there are no asynchronous/immediate MPI calls pending or that I should manually call MPI_Barrier to make sure that all local variables on all nodes are synchronized?

I suppose that the crash may be related with ASLR. charmrun seems to make use of setarch to disable ASLR, and I know there's +isomalloc_sync, but I can't make use of the latter because I don't have a shared filesystem. I'm a little confused why there's both of them. setarch should be enough

Btw. for some reasons I receive every message in this thread twice.

Regards,
Marcin

On 03.06.2019 18:14, Sam White wrote:

Yes AMPI_Register_just_migrated will work for that. Which version of AMPI are you using, v6.9.0 or the current master branch? Can you try using the AMPI_INFO_LB_SYNC info rather than ASYNC for the call to AMPI_Migrate() and re-running? The ASYNC mode is not tested and is more experimental. If you still don't see migration happening as expected I'd be interested in seeing the output with +LBDebug.

Thank you,
Sam

On Mon, Jun 3, 2019 at 10:54 AM Marcin Mielniczuk <marmistrz.dev AT zoho.eu> wrote:

Hi Sam,

I understand that AMPI_Register_just_migrated is the proper way to do 4? If so, then I confirm that the migration only happens once, on the first call to AMPI_Migrate.

I do call AMPI_Migrate multiple times. This can be seen looking at the "trying to migrate" lines in the stdout and the following lines in the source code: https://github.com/marmistrz/heat_solver/blob/master/main.cpp#L286-L291

Regards,
Marcin

On 03.06.2019 16:15, Sam White wrote:

Hi Marcin,

This is how recommend enabling and testing dynamic load balancing in an AMPI program:
1. Insert periodic calls to AMPI_Migrate(...) with the MPI_Info for LB.
2. Link with "-memory isomalloc -module CommonLBs".
3. First run with "+balancer RotateLB +LBDebug 3".
4. Verify that multiple rounds of migrations are happening (every rank is migrating at each call to AMPI_Migrate(). The AMPI manual has info on how to print the current PE # that a rank is on.

Then you can experiment with other load balancing strategies and options. You should not make calls to AMPI_Register_pup() when using Isomalloc for migration. Isomalloc is essentially a substitute for writing explicit PUP routines at the application level.

For your issue, are you sure that you are calling AMPI_Migrate() more than once? When running with +LBDebug the LB strategy will print some info as soon as it is called each time it is called, so the output in your log file suggests that it's not being called more than once for some reason.
You may also want to run with +LBTestPeSpeed so that the LB framework takes into consideration the heterogeneity of the nodes you are running on.

Let me know if this helps or if you still see migration only happening once. We will improve the manual based on your feedback, so thanks for getting in touch with us!
-Sam

On Mon, Jun 3, 2019 at 8:45 AM Marcin Mielniczuk <marmistrz.dev AT zoho.eu> wrote:

Hi,

I'm evaluating possible options for distributed computing in LAN
networks and came across AMPI. Currently I'm trying to get some hands-on
experience.

I have a toy project to test on, which I have ported to AMPI. [1]
According to the AMPI documentation, no manual PUP routines should be
needed when using isomalloc, so I have only added an AMPI_Migrate call
and an AMPI_Register_just_migrated handler. I'm not sure if it's
correct, because all the AMPI examples seem to have manual PUP routines,
even if using isomalloc.

I'm trying to verify if the processes are actually being migrated.
It appears that the just_migrated handler is never called, moreover,
when running with +LBDebug, the only CharmLB logs refer to the first
call of AMPI_Migrate. It looks like the LB doesn't even consider the
migration later on.

While for GreedyLB the load balancer may have decided that the load
imbalance isn't large enough, I have also tried RotateLB, which should
always migrate and it appears not to. The behavior persists even if I
add extra artificial CPU load on one of the nodes, which should cause a
large load imbalance.

All in all, it looks like AMPI doesn't migrate any process even when run
with a load balancer.

My command line is: 

          ./charmrun +p12 -hostfile hostfile --mca btl_tcp_if_include
<LAN subnet> ./heat_solver --size 14400 --steps 200 --noresults +vp60
+balancer RotateLB +LBDebug 100

My setup is: 2 computers in a common LAN (Ethernet) network, without a
shared file system. One node has 4 CPUs, the other has 8 CPUs. The CPUs
differ between the nodes: one is Intel Core i7-6700 (4GHz), the other
AMD Ryzen 7 1700 Eight-Core (3GHz)

I have attached execution logs. Is the lack of migration just a
programming error on my side or is it an AMPI bug?

Regards,
Marcin

[1] https://github.com/marmistrz/heat_solver