Charm++ parallel programming system

[Jozsef Bakosi <jbakosi AT lanl.gov>]

Hi Phil and Eric,

I finally got around to reproduce this problem with the attached minimal
example. This mimics what I'm trying to do: from a driver chare, create an 
empty
array and a group and from the group use insert() to add array entries. Note
insert.C:34. If the driver chare is created on PE 0 all is well in both serial
and parallel. However, change the Driver chare's PE to, e.g., 1, and run on 
two
PEs and the following assert is generated:

<pre><code class="c">
[1] Assertion "CkMyPe() == 0" failed in file ckarray.C line 780.
</code></pre>

I don't know if this is a behavior that would be incompatible with
design/usability requirements, but this example does reproduce a behavior that
was previously correct. Also, even if this is important for you guys to keep 
it
this this way, i.e., array::ckNew() must always be called from PE 0, it is 
okay,
because it appears that there is a simple fix: ensure that the driver is 
always
created on PE 0.

I also posted this on gerrit.

Thanks for looking into this.
Jozsef

On 03.06.2017 08:57, Jozsef Bakosi wrote:
>    ---------- Forwarded message ----------
>    From: Phil Miller 
> <[1]mille121 AT illinois.edu>
>    Date: Thu, Feb 9, 2017 at 3:03 PM
>    Subject: Re: [charm] insertInitial
>    To: "Mikida, Eric P" 
> <[2]mikida2 AT illinois.edu>
>    Cc: Jozsef Bakosi 
> <[3]jbakosi AT gmail.com>,
>  
> "[4]charm AT cs.uiuc.edu"
>    
> <[5]charm AT cs.uiuc.edu>
>    Hi Jozsef,
>    Are you still having issues with that assertion failure? If so, could
>    you please send us a full stack trace of its occurrence? I'm looking at
>    the code, and don't see how it could be triggering on your intended
>    usage.
>    Updates posted to [6]https://charm.cs.illinois.edu/redmine/issues/1397
>    would be helpful.
>    Phil
>    On Wed, Feb 1, 2017 at 9:34 PM, Mikida, Eric P
>    
> <[7]mikida2 AT illinois.edu>
>  wrote:
> 
>    Hello Jozsef,
>    Sorry for the delayed response.
>    For your first question on converting an int to an array index, you did
>    the correct thing.
>    For the second question, this appears to be a bug on our end. What is
>    currently restricted to being done on PE 0 is the call to
>    CProxy_SomeArray::ckNew(). But insert calls should work from anywhere.
>    I’ll see if I can track down the source of the bug.
>    In the meantime, depending on the sparsity patterns you have, you may
>    want to look into our sparse bulk creation in the manual section
>    13.2.1. [8]http://charm.cs.illinois.edu/manuals/html/charm++/
>    manual-1p.html
>    It is currently fairly limited to a very basic sparsity pattern, but
>    the intent was to allow for easy bulk creation of arrays that have some
>    sort of describable sparsity pattern so you don’t have to fall back to
>    just individual insertions, so depending on your needs it may be a good
>    reason for us to expand this API.
>    Thanks
>    Eric
> 
>    On Jan 26, 2017, at 9:17 AM, Jozsef Bakosi 
> <[9]jbakosi AT gmail.com>
>    wrote:
> 
>    Hello again,
>    Digging into this a little further, I noticed that the same(?) problem
>    may also appear when chare array elements are created (inserted) on all
>    PEs into a previously created chare array that is empty. For example
>    when each of a number of chare group element (residing on different
>    PEs) fires up multiple array elements, by using insert(), the same
>    assert is triggered in ckarray.C:780. I traced this back to these two
>    commits:
>    356327636f "Assert that chare arrays are only created on PE 0 to
>    support 64-bit ID construction" which really enables this and the one
>    one that puts it in as comments:
>    d71fcf194f "Add asynchronous array construction API for post-startup
>    use cases
>    When a chare array is created using this API, the callback function
>    receives a message, m, that contains the array id (m->aid).
>    This is necessary to provide positive array manager group IDs.
>    In future, when 64 bit ID is adopted, we want the group IDs to be
>    limited to a smaller compact range, hence the need to create them on
>    PE0. This API will enable us to force that requirement."
>    So it seems like that with my current use case this is not necessarily
>    true since CkCreateArray() is apparently called on a PE that is not
>    PE0: I do create my array (with CkNew) from a single chare (i.e., not
>    an array or a group) as well as a group from which later I use insert
>    to create new array elements. Note that I did this originally, because
>    I wanted to create a potentially large number of array elements in a
>    distributed/parallel fashion, i.e., spawning them from group elements
>    residing on different PEs as a tree.
>    Does this mean that in the future this is not really a good design?
>    What are ways of solving this problem? Is that assert really necessary?
>    Thanks,
>    Jozsef
>    On Tue, Jan 24, 2017 at 1:14 PM, Jozsef Bakosi 
> <[10]jbakosi AT gmail.com>
>    wrote:
> 
>    Hi folks,
>    It used to be CkArrMgr::insertInitial( int, ... ) which now takes const
>    CkArrayIndex &idx as its first argument. I see in the online
>    documentation that I could pass CkArrayIndex2D(x,y). I used to pass an
>    int 'e'. What is the simplest conversion to CkArrayIndex? Looking at
>    the constructors, I have tried CkArrayIndex(e) and CkArrayIndex1D(e),
>    but I trigger this assertion with multiple PEs:
>    [2] Assertion "CkMyPe() == 0" failed in file ckarray.C line 780.
>    How should I convert using an int to a CkArrayIndex passed to
>    insertInitial()?
>    Thanks,
>    Jozsef
> 
> References
> 
>    1. 
> mailto:mille121 AT illinois.edu
>    2. 
> mailto:mikida2 AT illinois.edu
>    3. 
> mailto:jbakosi AT gmail.com
>    4. 
> mailto:charm AT cs.uiuc.edu
>    5. 
> mailto:charm AT cs.uiuc.edu
>    6. https://charm.cs.illinois.edu/redmine/issues/1397
>    7. 
> mailto:mikida2 AT illinois.edu
>    8. http://charm.cs.illinois.edu/manuals/html/charm++/manual-1p.html
>    9. 
> mailto:jbakosi AT gmail.com
>   10. 
> mailto:jbakosi AT gmail.com

mainmodule insert {

  mainchare main {
    entry main( CkArgMsg* );
    entry [reductiontarget] void finish();
  };

  chare Driver {
    entry Driver( const CProxy_main& host );
  };

  array [1D] ChArray {
    entry ChArray( const CProxy_main& host );
  };

  group ChGroup {
    entry ChGroup( const CProxy_main& host, CProxy_ChArray& worker );
  };

};
#include <cassert>

#include "insert.decl.h"

class ChArray : public CBase_ChArray {
  public:
    explicit ChArray( const CProxy_main& host );
    explicit ChArray( CkMigrateMessage* ) {}
};

class ChGroup : public CBase_ChGroup {
  public:
    explicit ChGroup( const CProxy_main& host, CProxy_ChArray& worker );
  private:
    int m_nchare;
};

class Driver : public CBase_Driver {
  public:
    explicit Driver( const CProxy_main& host );
};

class main : public CBase_main {
public:
  main(CkMigrateMessage *m) {}
  main(CkArgMsg *m);
  void finish() { CkExit(); }
};

#include "insert.h"

ChGroup::ChGroup( const CProxy_main& host, CProxy_ChArray& worker )
  : m_nchare(10)
{
  auto chunksize = m_nchare / CkNumPes();
  auto mynchare = chunksize;
  if (CkMyPe() == CkNumPes()-1) mynchare += m_nchare % CkNumPes();

  for (int c=0; c<mynchare; ++c) {
    auto arrayid = CkMyPe() * chunksize + c;
    CkPrintf( "Group on PE %d inserting worker %d of %d total\n",
               CkMyPe(), arrayid, m_nchare );
    worker[ arrayid ].insert( host, CkMyPe() );
  }
  worker.doneInserting();
}

ChArray::ChArray( const CProxy_main& host ) {
  CkPrintf( "Hello from array ctor %d on PE %d\n", thisIndex, CkMyPe() );
  CkCallback cb( CkReductionTarget( main, finish ), host );
  contribute( cb );
}

Driver::Driver( const CProxy_main& host ) {
  CkPrintf( "Driver on PE %d\n", CkMyPe() );
  CProxy_ChArray arrayproxy = CProxy_ChArray::ckNew();
  CProxy_ChGroup groupproxy = CProxy_ChGroup::ckNew( host, arrayproxy );
}

main::main(CkArgMsg *m)
{
  CProxy_Driver::ckNew( thisProxy, 1 );
  delete m;
}

#include "insert.def.h"