Charm++ parallel programming system

["Kale, Laxmikant V" <kale AT illinois.edu>]

[sorry this overlaps with Phil’s answer somewhat, but it may be useful to some of you].

 

The types (whether same or not) don’t matter for this issue. Even overlapping reductions are ok : i.e. you can contribute into the second reduction before the first reduction results are delivered. You don’t need the “bound arrays” escape for these.  The only scenario when you need bound arrays is if different members of a chare array are contributing into 2 (or more) reduction operations in different orders. E.g. Chare array with index I contributes into a sum reduction with callback X, then contributes into a max reduction with callback Y, whereas another array element J contributes the max reduction going to callback Y first, before the contribution to sum going to X.

 

(This happens in case of ChaNGa with a liveViz visualization because the requests for visualization frame arrives from an outside user asynchronously with the applications own flow: so some array elements may have contributed to the application’s K’th reduction while others have only contributed to K-1’th, when they receive a request to contribute their visualization input into another reduction)

 

I used different callbacks (X and Y) and different reductions (sum and max) to make the example clear. But even if the reductions are of the same type, but logically different, application logic has to make sure they are contributed in the same order or else use bound arrays. E.g. If I want to contribute kinetic energy into a reduction, and potential energy into another reduction, from the same chare array, in every iteration: I have to make sure every chare contributes them in the same order (say, kinetic first then potential) to avoid having to use bound arrays.

 

So, in your example, you have to ensure that the “some member function” and “some OTHER member function” execute in the same order on all chares (if you cannot or don’t want to ensure ordering between them, use a bound array for the second reduction).

 

To answer your later question: that sequencing reductions may be hard in general : in most applications, unless there is external asynchrony, you can ensure sequencing without too much trouble. A way of fixing this issue is to allow some sort of unique handle in “contribute” that must be matched, but that may add overhead and/or make the common case more cumbersome to write. But we will explore this in future.

 

 

Laxmikant (Sanjay) Kale         http://charm.cs.uiuc.edu

Professor, Computer Science     kale AT illinois.edu

201 N. Goodwin Avenue           Ph:  (217) 244-0094

Urbana, IL  61801-2302 

 

 

From: Jozsef Bakosi <jbakosi AT gmail.com>
Reply-To: Jozsef Bakosi <jbakosi AT gmail.com>
Date: Wednesday, July 20, 2016 at 10:45 AM
To: Laxmikant Kale <kale AT illinois.edu>
Cc: Phil Miller <mille121 AT illinois.edu>, Tom Quinn <trq AT astro.washington.edu>, "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
Subject: Re: [charm] Run multiple custom reducers at (potentially) the same time

 

I'm not sure if I understand correctly, let me be more specific starting from the example in the user manual 16.2 but modified a bit to be closer to how I'm using it.

 

  /*initnode*/ void registerSumTwoShorts(void) {

    sumTwoShortsType=CkReduction::addReducer(sumTwoShorts);

  }

 

  /* reductiontarget */ void fn1( CkReductionMsg* msg ) { /* 1st final aggregated result here */ }

  /* reductiontarget */ void fn2( CkReductionMsg* msg ) { /* 2nd final aggregated result here */ }

 

  // In some member function, contribute data to a customized reduction:

  short data[2]=...;

  std::pair< int, std::unique_ptr<char[]> > stream = serialize( data );  // returns serialized data as length and char stream

  CkCallback cb( fn1(nullptr), thisProxy );

  contribute( stream.first, stream.second.get(), sumTwoShortsType, cb );

 

  // In some OTHER member function, contribute data to the ANOTHER customized reduction but of the same type:

  short otherdata[2]=...;

  std::pair< int, std::unique_ptr<char[]> > stream = serialize( otherdata );  // returns serialized otherdata as length and char stream

  CkCallback cb( fn2(nullptr), thisProxy );

  contribute( stream.first, stream.second.get(), sumTwoShortsType, cb );  // 'sumTwoShortsType' okay or need 'sumTwoShortsType2'?

 

So my question is: Is it okay to use a single sumTwoShortsType for both reductions which may overlap, i.e., happening at the same time?

 

I'm thinking that since I have two separate reduction targets, fn1() and fn2(), for the two reductions, I should be fine. However, the two reductions do happen on the same chare array/group and collect different data (of the same custom type). I have a feeling that this should be okay, but even if not, I could just register a second custom reducer, e.g., sumTwoShortsType2, and use it in the second contribute() call, which would make the two reductions have nothing to do with each other. Correct?

 

On Tue, Jul 19, 2016 at 3:30 PM, Kale, Laxmikant V <kale AT illinois.edu> wrote:

So, Jozsef, in case you are using those reductions in a way that the sequencing cannot be guaranteed to be the same (i.e. some chare array elements will contribute to reduction A before B,  while others contribute B before A), you should learn how to use the “shadow arrays” (called bound arrays in the manual : http://charm.cs.illinois.edu/manuals/html/charm++/13.html ) to achieve this.

Your phrase “at the same time” makes me think you may need it.  A bound array is bound to another chare array of the same shape, so that the corresponding elements (i.e. those with the same index) are guaranteed to be on the same processor. One of your reductions will contribute via the bound array.

 

Let us know if you have trouble with this.

 

Laxmikant (Sanjay) Kale         http://charm.cs.uiuc.edu

Professor, Computer Science     kale AT illinois.edu

201 N. Goodwin Avenue           Ph:  (217) 244-0094

Urbana, IL  61801-2302 

 

 

From: Phil Miller <mille121 AT illinois.edu>
Reply-To: Phil Miller <mille121 AT illinois.edu>
Date: Tuesday, July 19, 2016 at 2:57 PM
To: Tom Quinn <trq AT astro.washington.edu>
Cc: Jozsef Bakosi <jbakosi AT gmail.com>, "charm AT cs.uiuc.edu" <charm AT cs.uiuc.edu>
Subject: Re: [charm] Run multiple custom reducers at (potentially) the same time

 

On Tue, Jul 19, 2016 at 2:48 PM, Tom Quinn <trq AT astro.washington.edu> wrote:

Are you sure this works if the reducers are running on the same chare array?  In ChaNGa we had to set up "shadow arrays" to have two reductions running concurrently.

 

There is a potential concern if elements may end up making contribution calls concurrently, and in potentially different orders. Every element in an array, group, or section must contribute to the same sequence of reductions, just like MPI programs must call the same sequence of collections on a given communicator. The result is otherwise supposed to lead to assertion failures in the runtime, but may have undefined behavior.

 

On Tue, 19 Jul 2016, Phil Miller wrote:

On Tue, Jul 19, 2016 at 9:08 AM, Jozsef Bakosi <jbakosi AT gmail.com> wrote:
      I have been successfully using custom reducers, described in
      16.2 of the user manual. Now I'm wondering if it is safe to use
      more than one reducer, utilizing the same reducer type,
      registered in the initnode routine, at the same time, i.e.,
      potentially doing two (or more) reductions running at the same
      time but whose end-points, of course, point to different
      reduction entry method member functions. Or do I have to create
      multiple reducers as the manual example would require a
      sumTwoShortsType2?


The registration of custom reducers is just to associate a common global
index to t he function pointer that may differ across processes. So go right
ahead, and it should work fine.

The only case I can think of where multiple concurrent uses of a reducer
could be a problem is if the reducer implementation contained `static`
variables, which would seem quite strange.