Charm++ parallel programming system

[James Bordner <jobordner AT gmail.com>]

I've run into a scaling issue on Blue Waters.  In my chare array, chares are associated with nodes in an octree, and do periodic synchronizations with adjacent neighbors.  The problem is, as discovered using projections, the time in between entry method calls on neighbors increases as core counts increase.  This overhead seems to be very regular, and does not seem to depend on the total number of chares in the chare array, just the core count.   Although I've run on up to 110K FP cores, the scaling curve starts to drop noticeably at about 32K FP cores.

I compiled Charm 6.7.1 with gni-crayxe and--with-production (non-SMP mode). I ran similar tests a couple years ago with Charm++ 6.5.1 in SMP mode and didn't see this problem, though I only went to 32K FP cores at the time so it might not have been noticeable yet.

Any ideas what might be causing this scaling problem or how to get around it?

Thanks!

James