Charm++ parallel programming system

[Marcin Mielniczuk <marmistrz.dev AT zoho.eu>]

Hi,

I'm evaluating possible options for distributed computing in LAN
networks and came across AMPI. Currently I'm trying to get some hands-on
experience.

I have a toy project to test on, which I have ported to AMPI. [1]
According to the AMPI documentation, no manual PUP routines should be
needed when using isomalloc, so I have only added an AMPI_Migrate call
and an AMPI_Register_just_migrated handler. I'm not sure if it's
correct, because all the AMPI examples seem to have manual PUP routines,
even if using isomalloc.

I'm trying to verify if the processes are actually being migrated.
It appears that the just_migrated handler is never called, moreover,
when running with +LBDebug, the only CharmLB logs refer to the first
call of AMPI_Migrate. It looks like the LB doesn't even consider the
migration later on.

While for GreedyLB the load balancer may have decided that the load
imbalance isn't large enough, I have also tried RotateLB, which should
always migrate and it appears not to. The behavior persists even if I
add extra artificial CPU load on one of the nodes, which should cause a
large load imbalance.

All in all, it looks like AMPI doesn't migrate any process even when run
with a load balancer.

My command line is: 

          ./charmrun +p12 -hostfile hostfile --mca btl_tcp_if_include
<LAN subnet> ./heat_solver --size 14400 --steps 200 --noresults +vp60
+balancer RotateLB +LBDebug 100

My setup is: 2 computers in a common LAN (Ethernet) network, without a
shared file system. One node has 4 CPUs, the other has 8 CPUs. The CPUs
differ between the nodes: one is Intel Core i7-6700 (4GHz), the other
AMD Ryzen 7 1700 Eight-Core (3GHz)

I have attached execution logs. Is the lack of migration just a
programming error on my side or is it an AMPI bug?

Regards,
Marcin

[1] https://github.com/marmistrz/heat_solver

Running on 12 processors:  -hostfile hostfile --mca btl_tcp_if_include 10.30.8.0/22 ./heat_solver --size 14400 --steps 200 --noresults +vp60 +balancer GreedyLB +LBDebug 100 
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 12  -hostfile hostfile --mca btl_tcp_if_include 10.30.8.0/22 ./heat_solver --size 14400 --steps 200 --noresults +vp60 +balancer GreedyLB +LBDebug 100 
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: MPI_THREAD_SINGLE (desired: MPI_THREAD_SINGLE)
Charm++> Running in non-SMP mode: 12 processes (PEs)
Converse/Charm++ Commit ID: v6.9.0-0-gc3d50efa1
CharmLB> Verbose level 100, load balancing period: 0.5 seconds
CharmLB> Topology torus_nd_5 alpha: 3.500000e-05s beta: 8.500000e-09s.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 2 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.003 seconds.
CharmLB> GreedyLB created.
WARNING: results won't be written after the computation, as requested
[11:37:17.818138999] Progress: 0.0%. Computing step 0/200
Trying to migrate...
[11:37:18.786743571] Progress: 0.5%. Computing step 1/200
[11:37:18.997139395] Progress: 1.0%. Computing step 2/200
[11:37:19.201535668] Progress: 1.5%. Computing step 3/200

CharmLB> GreedyLB: PE [0] step 0 starting at 1.026157 Memory: 3.057358 MB
CharmLB> GreedyLB: PE [0] strategy starting at 1.565459
[0] In GreedyLB strategy
[0] Obj 52 migrating from 11 to 1
[0] Obj 27 migrating from 4 to 2
[0] Obj 49 migrating from 6 to 3
[0] Obj 54 migrating from 11 to 9
[0] Obj 58 migrating from 9 to 10
[0] Obj 59 migrating from 9 to 7
[0] Obj 21 migrating from 5 to 6
[0] Obj 34 migrating from 10 to 8
[0] Obj 46 migrating from 6 to 11
[0] Obj 44 migrating from 7 to 4
[0] Obj 47 migrating from 6 to 0
[0] Obj 23 migrating from 5 to 7
[0] Obj 41 migrating from 7 to 6
[0] Obj 31 migrating from 10 to 8
[0] Obj 22 migrating from 5 to 11
[0] Obj 36 migrating from 8 to 5
[0] Obj 38 migrating from 8 to 10
[0] Obj 20 migrating from 5 to 9
[0] Obj 39 migrating from 8 to 3
[0] Obj 43 migrating from 7 to 4
[0] Obj 48 migrating from 6 to 2
[0] Obj 35 migrating from 8 to 0
[0] Obj 33 migrating from 10 to 1
[0] Obj 37 migrating from 8 to 7
[0] Obj 57 migrating from 9 to 8
[0] Obj 32 migrating from 10 to 11
[0] Obj 55 migrating from 9 to 5
[0] Obj 42 migrating from 7 to 10
[0] Obj 53 migrating from 11 to 9
[0] Obj 29 migrating from 4 to 3
[0] Obj 50 migrating from 11 to 4
[0] Obj 30 migrating from 10 to 2
[0] Obj 40 migrating from 7 to 0
[0] Obj 28 migrating from 4 to 1
[0] Obj 56 migrating from 9 to 6
[0] Obj 26 migrating from 4 to 7
[0] Obj 14 migrating from 0 to 8
[0] Obj 25 migrating from 4 to 5
[0] Obj 1 migrating from 1 to 10
[0] Obj 13 migrating from 0 to 9
[0] Obj 0 migrating from 1 to 4
[0] Obj 5 migrating from 3 to 2
[0] Obj 16 migrating from 2 to 0
[0] Obj 7 migrating from 3 to 1
[0] Obj 17 migrating from 2 to 10
[0] Obj 2 migrating from 1 to 9
[0] Obj 15 migrating from 2 to 4
[0] Obj 10 migrating from 0 to 3
[0] Obj 12 migrating from 0 to 5
[0] Obj 19 migrating from 2 to 0
[0] Obj 8 migrating from 3 to 8
[0] Obj 9 migrating from 3 to 1
[0] Obj 4 migrating from 1 to 11
[0] Obj 3 migrating from 1 to 7
[0] Obj 11 migrating from 0 to 6
[0] 55 objects migrating.
CharmLB> Min obj: 0.059703  Max obj: 0.203260
CharmLB> PE speed:
1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 
CharmLB> PE Load:
0.636837 (0.001509)  0.641881 (0.014704)  0.651006 (0.001295)  0.645432 (0.001968)  0.644674 (0.002016)  0.653661 (0.001187)  0.664633 (0.002375)  0.652219 (0.043765)  0.646743 (0.002252)  0.665344 (0.003775)  0.651027 (0.003207)  0.666429 (0.002703)  
CharmLB> Obj Map:
4 10 9 7 11 2 3 1 8 1 3 6 5 9 8 4 0 10 2 0 9 6 11 7 5 5 7 2 1 3 2 8 11 1 8 0 5 7 10 3 0 6 10 4 4 6 11 0 2 3 4 11 1 9 9 5 6 8 10 7 
CharmLB> GreedyLB: PE [0] Memory: LBManager: 41 KB CentralLB: 6 KB
CharmLB> GreedyLB: PE [0] #Objects migrating: 55, LBMigrateMsg size: 0.00 MB
CharmLB> GreedyLB: PE [0] strategy finished at 1.565690 duration 0.000231 s
[11:37:19.419804616] Progress: 2.0%. Computing step 4/200
[11:37:19.654758785] Progress: 2.5%. Computing step 5/200
[0] Starting ReceiveMigration step 0 at 1.918553
[11:37:19.886269435] Progress: 3.0%. Computing step 6/200
[11:37:20.120222946] Progress: 3.5%. Computing step 7/200
[11:37:20.351789772] Progress: 4.0%. Computing step 8/200
[11:37:20.583751549] Progress: 4.5%. Computing step 9/200
[11:37:20.725902891] Progress: 5.0%. Computing step 10/200
Trying to migrate...
[11:37:20.956813194] Progress: 5.5%. Computing step 11/200
[11:37:21.233825823] Progress: 6.0%. Computing step 12/200
[11:37:21.329767774] Progress: 6.5%. Computing step 13/200
[11:37:21.561870196] Progress: 7.0%. Computing step 14/200
[11:37:21.794901239] Progress: 7.5%. Computing step 15/200
[11:37:22.025637669] Progress: 8.0%. Computing step 16/200
[11:37:22.257816414] Progress: 8.5%. Computing step 17/200
[11:37:22.490966703] Progress: 9.0%. Computing step 18/200
[11:37:22.721492204] Progress: 9.5%. Computing step 19/200
[11:37:22.955047314] Progress: 10.0%. Computing step 20/200
Trying to migrate...
[11:37:23.188350585] Progress: 10.5%. Computing step 21/200
[11:37:23.554900671] Progress: 11.0%. Computing step 22/200
[11:37:23.659356674] Progress: 11.5%. Computing step 23/200
[11:37:23.886203940] Progress: 12.0%. Computing step 24/200
[11:37:24.118714588] Progress: 12.5%. Computing step 25/200
[11:37:24.350450826] Progress: 13.0%. Computing step 26/200
[11:37:24.582762579] Progress: 13.5%. Computing step 27/200
[11:37:24.815921989] Progress: 14.0%. Computing step 28/200
[11:37:25.050791451] Progress: 14.5%. Computing step 29/200
[11:37:25.281931674] Progress: 15.0%. Computing step 30/200
Trying to migrate...
[11:37:25.515920902] Progress: 15.5%. Computing step 31/200
[11:37:25.839541575] Progress: 16.0%. Computing step 32/200
[11:37:25.937636299] Progress: 16.5%. Computing step 33/200
[11:37:26.168929883] Progress: 17.0%. Computing step 34/200
[11:37:26.401296157] Progress: 17.5%. Computing step 35/200
[11:37:26.633821062] Progress: 18.0%. Computing step 36/200
[11:37:26.867658686] Progress: 18.5%. Computing step 37/200
[11:37:27.100898275] Progress: 19.0%. Computing step 38/200
[11:37:27.333053757] Progress: 19.5%. Computing step 39/200
[11:37:27.566115803] Progress: 20.0%. Computing step 40/200
Trying to migrate...
[11:37:27.801728389] Progress: 20.5%. Computing step 41/200
[11:37:28.168310943] Progress: 21.0%. Computing step 42/200
[11:37:28.277796165] Progress: 21.5%. Computing step 43/200
[11:37:28.497440335] Progress: 22.0%. Computing step 44/200
[11:37:28.682660016] Progress: 22.5%. Computing step 45/200
[11:37:28.915642949] Progress: 23.0%. Computing step 46/200
[11:37:29.148840055] Progress: 23.5%. Computing step 47/200
[11:37:29.381474919] Progress: 24.0%. Computing step 48/200
[11:37:29.620903159] Progress: 24.5%. Computing step 49/200
[11:37:29.845313289] Progress: 25.0%. Computing step 50/200
Trying to migrate...
[11:37:30.081664889] Progress: 25.5%. Computing step 51/200
[11:37:30.448126815] Progress: 26.0%. Computing step 52/200
[11:37:30.595394082] Progress: 26.5%. Computing step 53/200
[11:37:30.778278083] Progress: 27.0%. Computing step 54/200
[11:37:31.011280500] Progress: 27.5%. Computing step 55/200
[11:37:31.241987647] Progress: 28.0%. Computing step 56/200
[11:37:31.474221409] Progress: 28.5%. Computing step 57/200
[11:37:31.713748664] Progress: 29.0%. Computing step 58/200
[11:37:31.944408929] Progress: 29.5%. Computing step 59/200
[11:37:32.174946123] Progress: 30.0%. Computing step 60/200
Trying to migrate...
[11:37:32.406367007] Progress: 30.5%. Computing step 61/200
[11:37:32.781153790] Progress: 31.0%. Computing step 62/200
[11:37:32.921884295] Progress: 31.5%. Computing step 63/200
[11:37:33.103177824] Progress: 32.0%. Computing step 64/200
[11:37:33.334977193] Progress: 32.5%. Computing step 65/200
[11:37:33.576483161] Progress: 33.0%. Computing step 66/200
[11:37:33.803712298] Progress: 33.5%. Computing step 67/200
[11:37:34.038519116] Progress: 34.0%. Computing step 68/200
[11:37:34.267181561] Progress: 34.5%. Computing step 69/200
[11:37:34.500864563] Progress: 35.0%. Computing step 70/200
Trying to migrate...
[11:37:34.733222256] Progress: 35.5%. Computing step 71/200
[11:37:35.101203674] Progress: 36.0%. Computing step 72/200
[11:37:35.241793028] Progress: 36.5%. Computing step 73/200
[11:37:35.428287569] Progress: 37.0%. Computing step 74/200
[11:37:35.660743793] Progress: 37.5%. Computing step 75/200
[11:37:35.894346570] Progress: 38.0%. Computing step 76/200
[11:37:36.127174548] Progress: 38.5%. Computing step 77/200
[11:37:36.358739770] Progress: 39.0%. Computing step 78/200
[11:37:36.591342052] Progress: 39.5%. Computing step 79/200
[11:37:36.823845470] Progress: 40.0%. Computing step 80/200
Trying to migrate...
[11:37:37.058345505] Progress: 40.5%. Computing step 81/200
[11:37:37.424794351] Progress: 41.0%. Computing step 82/200
[11:37:37.566598041] Progress: 41.5%. Computing step 83/200
[11:37:37.754289898] Progress: 42.0%. Computing step 84/200
[11:37:37.987101887] Progress: 42.5%. Computing step 85/200
[11:37:38.219939808] Progress: 43.0%. Computing step 86/200
[11:37:38.452503859] Progress: 43.5%. Computing step 87/200
[11:37:38.685284067] Progress: 44.0%. Computing step 88/200
[11:37:38.918326905] Progress: 44.5%. Computing step 89/200
[11:37:39.151056444] Progress: 45.0%. Computing step 90/200
Trying to migrate...
[11:37:39.383939509] Progress: 45.5%. Computing step 91/200
[11:37:39.753843644] Progress: 46.0%. Computing step 92/200
[11:37:39.896235970] Progress: 46.5%. Computing step 93/200
[11:37:40.082828144] Progress: 47.0%. Computing step 94/200
[11:37:40.315605133] Progress: 47.5%. Computing step 95/200
[11:37:40.548010842] Progress: 48.0%. Computing step 96/200
[11:37:40.781386534] Progress: 48.5%. Computing step 97/200
[11:37:41.013848309] Progress: 49.0%. Computing step 98/200
[11:37:41.246664086] Progress: 49.5%. Computing step 99/200
[11:37:41.478593521] Progress: 50.0%. Computing step 100/200
Trying to migrate...
[11:37:41.711556961] Progress: 50.5%. Computing step 101/200
[11:37:42.080747897] Progress: 51.0%. Computing step 102/200
[11:37:42.224817673] Progress: 51.5%. Computing step 103/200
[11:37:42.409746881] Progress: 52.0%. Computing step 104/200
[11:37:42.642601554] Progress: 52.5%. Computing step 105/200
[11:37:42.877336997] Progress: 53.0%. Computing step 106/200
[11:37:43.154832469] Progress: 53.5%. Computing step 107/200
[11:37:43.386733414] Progress: 54.0%. Computing step 108/200
[11:37:43.620469454] Progress: 54.5%. Computing step 109/200
[11:37:43.850764560] Progress: 55.0%. Computing step 110/200
Trying to migrate...
[11:37:44.083869509] Progress: 55.5%. Computing step 111/200
[11:37:44.408412036] Progress: 56.0%. Computing step 112/200
[11:37:44.548509302] Progress: 56.5%. Computing step 113/200
[11:37:44.736277041] Progress: 57.0%. Computing step 114/200
[11:37:44.968919146] Progress: 57.5%. Computing step 115/200
[11:37:45.202089887] Progress: 58.0%. Computing step 116/200
[11:37:45.434894043] Progress: 58.5%. Computing step 117/200
[11:37:45.667319712] Progress: 59.0%. Computing step 118/200
[11:37:45.900333750] Progress: 59.5%. Computing step 119/200
[11:37:46.133299239] Progress: 60.0%. Computing step 120/200
Trying to migrate...
[11:37:46.365867287] Progress: 60.5%. Computing step 121/200
[11:37:46.734969565] Progress: 61.0%. Computing step 122/200
[11:37:46.877130827] Progress: 61.5%. Computing step 123/200
[11:37:47.066431027] Progress: 62.0%. Computing step 124/200
[11:37:47.299323809] Progress: 62.5%. Computing step 125/200
[11:37:47.528563030] Progress: 63.0%. Computing step 126/200
[11:37:47.761791305] Progress: 63.5%. Computing step 127/200
[11:37:47.994579864] Progress: 64.0%. Computing step 128/200
[11:37:48.226718666] Progress: 64.5%. Computing step 129/200
[11:37:48.458563428] Progress: 65.0%. Computing step 130/200
Trying to migrate...
[11:37:48.692090032] Progress: 65.5%. Computing step 131/200
[11:37:49.061029616] Progress: 66.0%. Computing step 132/200
[11:37:49.203412577] Progress: 66.5%. Computing step 133/200
[11:37:49.390114714] Progress: 67.0%. Computing step 134/200
[11:37:49.622897068] Progress: 67.5%. Computing step 135/200
[11:37:49.855949957] Progress: 68.0%. Computing step 136/200
[11:37:50.088993038] Progress: 68.5%. Computing step 137/200
[11:37:50.321464068] Progress: 69.0%. Computing step 138/200
[11:37:50.554168069] Progress: 69.5%. Computing step 139/200
[11:37:50.788336294] Progress: 70.0%. Computing step 140/200
Trying to migrate...
[11:37:51.022423518] Progress: 70.5%. Computing step 141/200
[11:37:51.389530717] Progress: 71.0%. Computing step 142/200
[11:37:51.530662426] Progress: 71.5%. Computing step 143/200
[11:37:51.716840748] Progress: 72.0%. Computing step 144/200
[11:37:51.949298236] Progress: 72.5%. Computing step 145/200
[11:37:52.181131136] Progress: 73.0%. Computing step 146/200
[11:37:52.412927627] Progress: 73.5%. Computing step 147/200
[11:37:52.646777948] Progress: 74.0%. Computing step 148/200
[11:37:52.878878302] Progress: 74.5%. Computing step 149/200
[11:37:53.112146633] Progress: 75.0%. Computing step 150/200
Trying to migrate...
[11:37:53.342298566] Progress: 75.5%. Computing step 151/200
[11:37:53.710477551] Progress: 76.0%. Computing step 152/200
[11:37:53.852560968] Progress: 76.5%. Computing step 153/200
[11:37:54.038687227] Progress: 77.0%. Computing step 154/200
[11:37:54.272957661] Progress: 77.5%. Computing step 155/200
[11:37:54.503619723] Progress: 78.0%. Computing step 156/200
[11:37:54.737735645] Progress: 78.5%. Computing step 157/200
[11:37:54.968291047] Progress: 79.0%. Computing step 158/200
[11:37:55.205889734] Progress: 79.5%. Computing step 159/200
[11:37:55.431073461] Progress: 80.0%. Computing step 160/200
Trying to migrate...
[11:37:55.663821448] Progress: 80.5%. Computing step 161/200
[11:37:56.032351491] Progress: 81.0%. Computing step 162/200
[11:37:56.174331465] Progress: 81.5%. Computing step 163/200
[11:37:56.360133129] Progress: 82.0%. Computing step 164/200
[11:37:56.592676347] Progress: 82.5%. Computing step 165/200
[11:37:56.823608432] Progress: 83.0%. Computing step 166/200
[11:37:57.055884496] Progress: 83.5%. Computing step 167/200
[11:37:57.289819054] Progress: 84.0%. Computing step 168/200
[11:37:57.521453883] Progress: 84.5%. Computing step 169/200
[11:37:57.755308293] Progress: 85.0%. Computing step 170/200
Trying to migrate...
[11:37:57.988325467] Progress: 85.5%. Computing step 171/200
[11:37:58.358051249] Progress: 86.0%. Computing step 172/200
[11:37:58.499037662] Progress: 86.5%. Computing step 173/200
[11:37:58.685725112] Progress: 87.0%. Computing step 174/200
[11:37:58.925842307] Progress: 87.5%. Computing step 175/200
[11:37:59.150527958] Progress: 88.0%. Computing step 176/200
[11:37:59.383897164] Progress: 88.5%. Computing step 177/200
[11:37:59.616515964] Progress: 89.0%. Computing step 178/200
[11:37:59.849502779] Progress: 89.5%. Computing step 179/200
[11:38:00.082309552] Progress: 90.0%. Computing step 180/200
Trying to migrate...
[11:38:00.317240631] Progress: 90.5%. Computing step 181/200
[11:38:00.686200873] Progress: 91.0%. Computing step 182/200
[11:38:00.828144265] Progress: 91.5%. Computing step 183/200
[11:38:01.015058014] Progress: 92.0%. Computing step 184/200
[11:38:01.247108338] Progress: 92.5%. Computing step 185/200
[11:38:01.478759487] Progress: 93.0%. Computing step 186/200
[11:38:01.757024801] Progress: 93.5%. Computing step 187/200
[11:38:01.943370196] Progress: 94.0%. Computing step 188/200
[11:38:02.268111347] Progress: 94.5%. Computing step 189/200
[11:38:02.499922741] Progress: 95.0%. Computing step 190/200
numtasks: 60, rank: 10, nx: 14400, time: 44.880491
Trying to migrate...
[11:38:02.732657516] Progress: 95.5%. Computing step 191/200
numtasks: 60, rank: 9, nx: 14400, time: 45.064061
numtasks: 60, rank: 11, nx: 14400, time: 45.230414
[11:38:03.054982538] Progress: 96.0%. Computing step 192/200
numtasks: 60, rank: 12, nx: 14400, time: 45.370057
numtasks: 60, rank: 8, nx: 14400, time: 45.383045
[11:38:03.198814653] Progress: 96.5%. Computing step 193/200
numtasks: 60, rank: 13, nx: 14400, time: 45.501097
numtasks: 60, rank: 7, nx: 14400, time: 45.516395
[11:38:03.427758490] Progress: 97.0%. Computing step 194/200
numtasks: 60, rank: 6, nx: 14400, time: 45.730776
numtasks: 60, rank: 14, nx: 14400, time: 45.761020
[11:38:03.661120027] Progress: 97.5%. Computing step 195/200
numtasks: 60, rank: 5, nx: 14400, time: 45.952832
numtasks: 60, rank: 15, nx: 14400, time: 45.981920
[11:38:03.893337221] Progress: 98.0%. Computing step 196/200
numtasks: 60, rank: 4, nx: 14400, time: 46.172097
numtasks: 60, rank: 16, nx: 14400, time: 46.201414
[11:38:04.125404226] Progress: 98.5%. Computing step 197/200
numtasks: 60, rank: 3, nx: 14400, time: 46.392573
numtasks: 60, rank: 17, nx: 14400, time: 46.421587
[11:38:04.357441097] Progress: 99.0%. Computing step 198/200
numtasks: 60, rank: 2, nx: 14400, time: 46.613988
numtasks: 60, rank: 18, nx: 14400, time: 46.642937
numtasks: 60, rank: 35, nx: 14400, time: 46.711412
numtasks: 60, rank: 34, nx: 14400, time: 46.788056
numtasks: 60, rank: 51, nx: 14400, time: 46.799586
[11:38:04.590495429] Progress: 99.5%. Computing step 199/200
numtasks: 60, rank: 1, nx: 14400, time: 46.836264
numtasks: 60, rank: 19, nx: 14400, time: 46.864321
numtasks: 60, rank: 33, nx: 14400, time: 46.879532
numtasks: 60, rank: 50, nx: 14400, time: 46.891028
numtasks: 60, rank: 36, nx: 14400, time: 46.895123
numtasks: 60, rank: 52, nx: 14400, time: 46.981729
numtasks: 60, rank: 49, nx: 14400, time: 46.991423
numtasks: 60, rank: 32, nx: 14400, time: 47.016482
numtasks: 60, rank: 20, nx: 14400, time: 47.034317
numtasks: 60, rank: 37, nx: 14400, time: 47.044002
numtasks: 60, rank: 0, nx: 14400, time: 47.056400
Skipped writing the results, as requested
numtasks: 60, rank: 53, nx: 14400, time: 47.070826
numtasks: 60, rank: 48, nx: 14400, time: 47.078804
numtasks: 60, rank: 31, nx: 14400, time: 47.098272
numtasks: 60, rank: 21, nx: 14400, time: 47.111471
numtasks: 60, rank: 38, nx: 14400, time: 47.159946
numtasks: 60, rank: 47, nx: 14400, time: 47.176425
numtasks: 60, rank: 22, nx: 14400, time: 47.184492
numtasks: 60, rank: 54, nx: 14400, time: 47.192521
numtasks: 60, rank: 59, nx: 14400, time: 47.192848
numtasks: 60, rank: 23, nx: 14400, time: 47.218246
numtasks: 60, rank: 58, nx: 14400, time: 47.226581
numtasks: 60, rank: 30, nx: 14400, time: 47.229021
numtasks: 60, rank: 27, nx: 14400, time: 47.229296
numtasks: 60, rank: 46, nx: 14400, time: 47.243386
numtasks: 60, rank: 24, nx: 14400, time: 47.251908
numtasks: 60, rank: 26, nx: 14400, time: 47.256278
numtasks: 60, rank: 57, nx: 14400, time: 47.258221
numtasks: 60, rank: 39, nx: 14400, time: 47.261858
numtasks: 60, rank: 28, nx: 14400, time: 47.276943
numtasks: 60, rank: 56, nx: 14400, time: 47.279009
numtasks: 60, rank: 25, nx: 14400, time: 47.296906
numtasks: 60, rank: 55, nx: 14400, time: 47.298961
numtasks: 60, rank: 29, nx: 14400, time: 47.316032
numtasks: 60, rank: 40, nx: 14400, time: 47.321815
numtasks: 60, rank: 45, nx: 14400, time: 47.322154
numtasks: 60, rank: 41, nx: 14400, time: 47.335140
numtasks: 60, rank: 42, nx: 14400, time: 47.348337
numtasks: 60, rank: 43, nx: 14400, time: 47.361555
numtasks: 60, rank: 44, nx: 14400, time: 47.374671
[Partition 0][Node 0] End of program