Skip to Content.
Sympa Menu

charm - Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init

charm AT lists.siebelschool.illinois.edu

Subject: Charm++ parallel programming system

List archive

Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init


Chronological Thread 
  • From: Fernando Stump <fernando.stump AT gmail.com>
  • To: Aaron Becker <abecker3 AT illinois.edu>
  • Cc: Charm Mailing List <charm AT cs.illinois.edu>
  • Subject: Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init
  • Date: Fri, 9 Sep 2011 15:39:46 -0500
  • List-archive: <http://lists.cs.uiuc.edu/pipermail/charm>
  • List-id: CHARM parallel programming system <charm.cs.uiuc.edu>
HI Aaron, 

This problem is similar to something similar I had in the past, in this same code.  (the e-mail about that problem is pasted at the end of this e-mail) In the past some  FEM_* functions cannot be called from inside  nested functions, just from the outside "layers" the specific function was

//FEM_Add_ghost_layer(1,1);

Now it is a MPI_*  so MPI_* calls work inside init() but not on nested functions

So in this example, I get this error.  Any guess for were to look at?

OUTPUT

.
.
.
 Driver 0 started 
Checking MPI status
Number of processors: 1
My rank: 0
My ParFUM id: 0
read_abaqus_file_on_driver My ParFUM id: 0
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: Cannot call MPI routines before AMPI is initialized.

Charm++ fatal error:
Cannot call MPI routines before AMPI is initialized.


CODE

extern "C" void
driver(void)
{
.
.


int i, myId=FEM_My_partition();

CkPrintf(" Driver %d started \n", myId);
std::cout << "Checking MPI status" << endl;;


int num_procs;
MPI_Comm_size (MPI_COMM_WORLD, &num_procs);
std::cout << "Number of processors: " << num_procs << endl;


int my_rank;
MPI_Comm_rank (MPI_COMM_WORLD, &my_rank);
std::cout << "My rank: " << my_rank << endl;


std::cout << "My ParFUM id: " << myId << endl;

.

.


yafeq::FEM<dimension> fem_driver;

fem_driver.read_abaqus_file_on_driver(path);

.

.

.

}



template<int d, template <int> class Assembly_policy, class FEM_policy, class Transience_policy>
void FEM<d, Assembly_policy, FEM_policy, Transience_policy>::read_abaqus_file_on_driver(char* filename) 
{
.
.
int myId=FEM_My_partition();
std::cout << "read_abaqus_file_on_driver My ParFUM id: " << myId << endl;

int my_rank;
MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); ///// THIS CALL DOES NOT WORK HERE //////
std::cout << " read_abaqus_file_on_driver My rank: " << my_rank << endl;

// create new mesh and store it
mesh_pointer mp = *meshes_.insert(meshes_.end(), mesh_pointer(new mesh_type(filename)));


// send request to mesh object
mp->read_abaqus_file_on_driver(filename);

.

.

}







/////////////////////////////////////////////////////// PREVIOUS MESSAGES ////////////////////////////////////////
HI Aaron, 


I don't see a reason for not working, but it is simply does not, it does not give any error, it just do not create the layers. Look the output FEM_Mesh_print(mesh) for both cases.

My only guess is that FEM_* call need to be inside 

extern "C" void
init(void)
{

}

type environment.

Does it make sense?

Thanks
Fernando



_____________________________________CALLING FROM INSIDE "read_abaqus_file_on_driver"  (NOT WORKING)_____________________________

Abaqus file read successfully on driver...
Fernando-Stumps-MacBook-Pro:yafeq fstump2$ charmrun ./pfem.out ++p 3  ++local
Charm++: scheduler running in netpoll mode.
Charm++> Running on 1 unique compute nodes (4-way SMP).
Charm++> Cpu topology info:
PE to node map: 0 0 0 
Node to PE map:
Chip #0: 0 1 2 
Charm++> cpu topology info is gathered in 0.003 seconds.
Starting INIT
init started
Calling read_abaqus_file_on_init

PATCH TEST  [C3D4]

Passing node coords to framework
Passing elements to framework
Creating Ghost Layers
Abaqus file read successfully on init...
Finished with init (Reading took 0 s)
-------------------- Chunk 0 --------------------

Length of FEM_NODE = 16
Node global numbers: * marks primary, () surrounds local-only:
*(0)  *(1)  *(2)  *(3)  *(4)  *(5)  *(6)  *(7)  *(8)  *(9)  *(10)  *(11)  *(12)  *(13)  *(14)  *(15)  

Length of FEM_ELEM+10 = 35
FEM_ELEM+10 Connectivity: 
  Entry (0)  |  (10)  (11)  (15)  (9)  
  Entry (1)  |  (9)  (15)  (12)  (13)  
  Entry (2)  |  (10)  (15)  (14)  (12)  
  Entry (3)  |  (10)  (9)  (12)  (8)  
  Entry (4)  |  (10)  (9)  (15)  (12)  
  Entry (5)  |  (10)  (14)  (15)  (6)  
  Entry (6)  |  (6)  (15)  (3)  (7)  
  Entry (7)  |  (10)  (15)  (11)  (3)  
  Entry (8)  |  (10)  (6)  (3)  (2)  
  Entry (9)  |  (10)  (6)  (15)  (3)  
  Entry (10)  |  (9)  (13)  (12)  (5)  
  Entry (11)  |  (5)  (12)  (0)  (4)  
  Entry (12)  |  (9)  (12)  (8)  (0)  
  Entry (13)  |  (9)  (5)  (0)  (1)  
  Entry (14)  |  (9)  (5)  (12)  (0)  
  Entry (15)  |  (10)  (11)  (9)  (3)  
  Entry (16)  |  (3)  (9)  (0)  (1)  
  Entry (17)  |  (10)  (9)  (8)  (0)  
  Entry (18)  |  (10)  (3)  (0)  (2)  
  Entry (19)  |  (10)  (3)  (9)  (0)  
  Entry (20)  |  (9)  (11)  (15)  (3)  
  Entry (21)  |  (3)  (15)  (5)  (7)  
  Entry (22)  |  (9)  (15)  (13)  (5)  
  Entry (23)  |  (9)  (3)  (5)  (1)  
  Entry (24)  |  (9)  (3)  (15)  (5)  
  Entry (25)  |  (12)  (13)  (15)  (5)  
  Entry (26)  |  (5)  (15)  (6)  (7)  
  Entry (27)  |  (12)  (15)  (14)  (6)  
  Entry (28)  |  (12)  (5)  (6)  (4)  
  Entry (29)  |  (12)  (5)  (15)  (6)  
  Entry (30)  |  (12)  (14)  (10)  (6)  
  Entry (31)  |  (6)  (10)  (0)  (2)  
  Entry (32)  |  (12)  (10)  (8)  (0)  
  Entry (33)  |  (12)  (6)  (0)  (4)  
  Entry (34)  |  (12)  (6)  (10)  (0)  


 Driver 0 started 
-------------------- Chunk 0 --------------------

Length of FEM_NODE = 11
Node global numbers: * marks primary, () surrounds local-only:
6  15  *3  *7  10  11  *2  9  0  1  5  

Length of FEM_ELEM+10 = 11
FEM_ELEM+10 Connectivity: 
  Entry 6  |  6  15  3  7  
  Entry 7  |  10  15  11  3  
  Entry 8  |  10  6  3  2  
  Entry 9  |  10  6  15  3  
  Entry 15  |  10  11  9  3  
  Entry 16  |  3  9  0  1  
  Entry 18  |  10  3  0  2  
  Entry 20  |  9  11  15  3  
  Entry 21  |   Driver 1 started 
-------------------- Chunk 1 --------------------

Length of FEM_NODE = 11
Node global numbers: * marks primary, () surrounds local-only:
*3  15  5  7  
  Entry 23  |  9  3  5  1  
  Entry 24  |  9  3  15  5  10  *11  *15  *9  *12  *



PATCH TEST  [C3D4]

Abaqus file read successfully on driver...
Driver finished in 0.005242  
13  *14  *6  5  7  4  

Length of FEM_ELEM+10 = 11
FEM_ELEM+10 Connectivity: 
  Entry 0  |  10  11  15  9  
  Entry 1  |  9  15  12  13  
  Entry 2  |  10  15  14  12  
  Entry 5  |  10  14  15  6  
  Entry 22  |  9  15  13  5  
  Entry 25  |  12  13  15  5  
  Entry 26  |  5  15  6  7  
  Entry 27  |   Driver 2 started 
-------------------- Chunk 2 --------------------

Length of FEM_NODE = 13
Node global numbers: * marks primary, () surrounds local-only:
10  9  12  *8  12  15  14  6  
  Entry 28  |  12  5  15  13  *5  *0  *6  4  
  Entry 29  |  12  5  15  6  4  *1  
  Entry 30  |  12  14  10  6  


3  6  2  

Length of FEM_ELEM+10 = 13
FEM_ELEM+10 Connectivity: 
  Entry 3  |  10  9  12  8  
  Entry 4  |  10  9  15  12  
  Entry 10  |  9  13  12  5  
  Entry 11  |  5  12  0  4  
  Entry 12  |  9  12  8  0  
  Entry 13  |  
PATCH TEST  [C3D4]

Abaqus file read successfully on driver...
9  5  0  1  
  Entry 14  |  9  5  12  0  
  Entry 17  |  10  9  8  0  
  Entry 19  |  10  3  9  0  
  Entry 31  |  6  10  0  2  
  Entry 32  |  12  10  8  0  
  Entry 33  |  12  6  0  4  
  Entry 34  |  12  6  10  0  



PATCH TEST  [C3D4]

Abaqus file read successfully on driver...





_____________________________________CALLING FROM INSIDE "init"  (WORKING)_____________________________



Fernando-Stumps-MacBook-Pro:yafeq fstump2$ charmrun ./pfem.out ++p 3  ++local
Charm++: scheduler running in netpoll mode.
Charm++> Running on 1 unique compute nodes (4-way SMP).
Charm++> Cpu topology info:
PE to node map: 0 0 0 
Node to PE map:
Chip #0: 0 1 2 
Charm++> cpu topology info is gathered in 0.002 seconds.
Starting INIT
init started
Calling read_abaqus_file_on_init

PATCH TEST  [C3D4]

Passing node coords to framework
Passing elements to framework
Creating Ghost Layers
Abaqus file read successfully on init...
Finished with init (Reading took 0 s)
-------------------- Chunk 0 --------------------

Length of FEM_NODE = 16
Node global numbers: * marks primary, () surrounds local-only:
*(0)  *(1)  *(2)  *(3)  *(4)  *(5)  *(6)  *(7)  *(8)  *(9)  *(10)  *(11)  *(12)  *(13)  *(14)  *(15)  

Length of FEM_ELEM+10 = 35
FEM_ELEM+10 Connectivity: 
  Entry (0)  |  (10)  (11)  (15)  (9)  
  Entry (1)  |  (9)  (15)  (12)  (13)  
  Entry (2)  |  (10)  (15)  (14)  (12)  
  Entry (3)  |  (10)  (9)  (12)  (8)  
  Entry (4)  |  (10)  (9)  (15)  (12)  
  Entry (5)  |  (10)  (14)  (15)  (6)  
  Entry (6)  |  (6)  (15)  (3)  (7)  
  Entry (7)  |  (10)  (15)  (11)  (3)  
  Entry (8)  |  (10)  (6)  (3)  (2)  
  Entry (9)  |  (10)  (6)  (15)  (3)  
  Entry (10)  |  (9)  (13)  (12)  (5)  
  Entry (11)  |  (5)  (12)  (0)  (4)  
  Entry (12)  |  (9)  (12)  (8)  (0)  
  Entry (13)  |  (9)  (5)  (0)  (1)  
  Entry (14)  |  (9)  (5)  (12)  (0)  
  Entry (15)  |  (10)  (11)  (9)  (3)  
  Entry (16)  |  (3)  (9)  (0)  (1)  
  Entry (17)  |  (10)  (9)  (8)  (0)  
  Entry (18)  |  (10)  (3)  (0)  (2)  
  Entry (19)  |  (10)  (3)  (9)  (0)  
  Entry (20)  |  (9)  (11)  (15)  (3)  
  Entry (21)  |  (3)  (15)  (5)  (7)  
  Entry (22)  |  (9)  (15)  (13)  (5)  
  Entry (23)  |  (9)  (3)  (5)  (1)  
  Entry (24)  |  (9)  (3)  (15)  (5)  
  Entry (25)  |  (12)  (13)  (15)  (5)  
  Entry (26)  |  (5)  (15)  (6)  (7)  
  Entry (27)  |  (12)  (15)  (14)  (6)  
  Entry (28)  |  (12)  (5)  (6)  (4)  
  Entry (29)  |  (12)  (5)  (15)  (6)  
  Entry (30)  |  (12)  (14)  (10)  (6)  
  Entry (31)  |  (6)  (10)  (0)  (2)  
  Entry (32)  |  (12)  (10)  (8)  (0)  
  Entry (33)  |  (12)  (6)  (0)  (4)  
  Entry (34)  |  (12)  (6)  (10)  (0)  


FEM Ghost layer 1> 70 new ghost elements, 13 new ghost nodes
 Driver 0 started 
-------------------- Chunk 0 --------------------

Length of FEM_NODE = 11 (and 5 ghosts)
Node global numbers: * marks primary, () surrounds local-only:
6  15  *3  *7  10  11  *2  9  0  1  5  

Length of FEM_ELEM+10 = 11 (and 24 ghosts)
FEM_ELEM+10 Connectivity: 
  Entry 6  |  6  15  3  7  
  Entry 7  |  10  15  11  3  
  Entry 8  |  10  6  3  2  
  Entry 9  |  10  6  15  3  
  Entry 15  |  10  11  9  3  
  Entry 16  |  3  9  0  1  
  Entry 18  |  10  3  0  2  
  Entry 20  |  9  11  15  3  
  Entry 21  |  3  15  5  7  
  Entry 23  |  9  3  5  1  
  Entry 24  |  9  3  15  5  



PATCH TEST  [C3D4]

Abaqus file read successfully on driver...
Driver finished in 0.006951  
 Driver 2 started 
-------------------- Chunk 2 --------------------

Length of FEM_NODE = 13 (and 3 ghosts)
Node global numbers: * marks primary, () surrounds local-only:
10  9  12  *8  15  13  *5  *0  *4  *1  3  6  2  

Length of FEM_ELEM+10 = 13 (and 22 ghosts)
FEM_ELEM+10 Connectivity: 
  Entry 3  |  10  9  12  8  
  Entry 4  |  10  9  15  12  
  Entry 10  |  9  13  12  5  
  Entry 11  |  5  12  0  4  
  Entry 12  |  9  12  8  0  
  Entry 13  |  9  5  0  1  
  Entry 14  |  9  5  12  0  
  Entry 17  |  10  9  8  0  
  Entry 19  |  10  3  9  0  
  Entry 31  |  6  10  0  2  
  Entry 32  |  12  10  8  0  
  Entry 33  |  12  6  0  4  
  Entry 34  |  12  6  10  0  


 Driver 1 started 
-------------------- Chunk 1 --------------------

Length of FEM_NODE = 11 (and 5 ghosts)
Node global numbers: * marks primary, () surrounds local-only:
*10  *11  *15  *9  *12  *13  *14  *6  5  7  4  

Length of FEM_ELEM+10 = 11 (and 24 ghosts)
FEM_ELEM+10 Connectivity: 
  Entry 0  |  10  11  15  9  
  Entry 1  |  9  15  12  13  
  Entry 2  |  10  15  14  12  
  Entry 5  |  10  14  15  6  
  Entry 22  |  9  15  13  5  
  Entry 25  |  12  13  15  5  
  Entry 26  |  5  15  6  7  
  Entry 27  |  12  15  14  6  
  Entry 28  |  12  5  6  4  
  Entry 29  |  12  5  15  6  
  Entry 
PATCH TEST  [C3D4]

Abaqus file read successfully on driver...
30  |  12  14  10  6  



PATCH TEST  [C3D4]

Abaqus file read successfully on driver...
Fernando-Stumps-MacBook-Pro:yafeq fstump2$ 


On Apr 6, 2011, at 2:54 PM, Aaron Becker wrote:

There is no reason you can't call FEM_Add_ghost_layer and
FEM_Add_ghost_elem from another function inside init. In fact, several
working applications do it that way. What error message do you get
when you try calling them from read_abaqus_file_on_init?

Aaron

On Thu, Mar 31, 2011 at 2:14 PM, Fernando Stump
<fernando.stump AT gmail.com> wrote:
Hi,
 I would like to know if someone can explain why the call for add ghost
layer need to be done on the Init() function and cannot be  nested in
function inside Init(). Bellow it is and example how It works or not.
Thanks
Fernando
I'm using ParFUM in a C++ library called Yafeq (Yanhua and Osman are somehow
familiar with yafeq). I have my init function as:
extern "C" void
init(void)
{
CkPrintf("init started\n");
int mesh=FEM_Mesh_default_write(); // Tell framework we are writing to the
mesh
yafeq::FEM<2> fem_init;
// read abaqus file
char path[] = "../tests/input/abaqus_patch_tests/ece3sfp1_step1_yafeq.inp";
fem_init.read_abaqus_file_on_init(path);
// Add the nodal ghost layer. Not edge based.
const int trianglefaces[3] = {0,1,2};
FEM_Add_ghost_layer(1,1);
FEM_Add_ghost_elem(0,3,trianglefaces);

CkPrintf("Finished with init (Reading took %.f
s)\n",CmiWallTimer()-startTime);
FEM_Mesh_print(mesh);
}
And the class yafeq::FEM<d> has the class Mesh<d> which has the function
"read_abaqus_file_on_init" which read the mesh and provides it to parfum as
below
template <int d>
void Mesh<d>::read_abaqus_file_on_init(char* filename) {
.
.
.
.
{
//Pass Information to ParFUM
int mesh=FEM_Mesh_default_write(); // Tell framework we are writing to the
mesh

//Plain data type to be passed to ParFUM
int n_nodes=num_nodes();
double *coord= new double[d*n_nodes];

int count=0;
//loop over nodes
for (id_node_map_iterator nit = nodeMap.begin(); nit != nodeMap.end();
++nit) {
for (size_t i = 0; i < d; ++i) {
coord[count++]=(*(nit->second))[i];
}
}

CkPrintf("Passing node coords to framework\n");
//Passing the nodes to parfum
FEM_Mesh_data(mesh,        // Add nodes to the current mesh
  FEM_NODE,        // We are registering nodes
  FEM_DATA+0,      // Register the point locations which are normally
  // the first data elements for an FEM_NODE
  (double *)coord,   // The array of point locations
  0,               // 0 based indexing
  n_nodes,         // The number of points
  FEM_DOUBLE,      // Coordinates are doubles
  d);              // Points have dimension 2 (x,y)



//Plain data type to be passed to ParFUM
int n_elements=num_elements();
int *elements= new int[3*n_elements];

count=0;
//loop over elements
for (id_element_map_iterator elem_it = elementMap.begin(); elem_it !=
elementMap.end(); ++elem_it) {
for (typename element_type::node_iterator node_it =
((elem_it->second)->nodes_begin()); node_it !=
((elem_it->second)->nodes_end()); ++node_it){
int id = nodeIdMap[*node_it];
elements[count++]=id;
}
}

CkPrintf("Passing elements to framework\n");
FEM_Mesh_data(mesh,      // Add nodes to the current mesh
  FEM_ELEM+0,      // We are registering elements with type 0
// The next type of element could be registered with FEM_ELEM+1
  FEM_CONN,        // Register the connectivity table for this
// data elements for this type of FEM entity
  (int *)elements,      // The array of point locations
  0,               // 0 based indexing
  n_elements,            // The number of elements
  FEM_INDEX_0,     // We use zero 1 node numbering
  3);              // Elements have degree 3, since triangles are defined

delete[] elements;
delete[] coord;

// Setup a node-adjacent ghost layer as we need all the elements around a
node to compute
    //!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// THIS FUNCTIONS CANNOT BE CALLED HERE, THEY NEED TO BE CALLED AT THE
INIT() ELEVEL
// Add the nodal ghost layer. Not edge based.
// const int trianglefaces[3] = {0,1,2};
//FEM_Add_ghost_layer(1,1);
//FEM_Add_ghost_elem(0,3,trianglefaces);


}


_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm

_______________________________________________
ppl mailing list
ppl AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/ppl





On Sep 9, 2011, at 1:57 PM, Aaron Becker wrote:

The way you're compiling looks ok, and you don't need to call MPI_Init
at all.. Make sure driver is declared like this:

extern "C" void driver(void)

and if you have problems, try referring to the globalExample
application in the examples directory, as it does use MPI calls
directly.


Aaron

On Fri, Sep 9, 2011 at 11:52 AM, Fernando Stump
<fernando.stump AT gmail.com> wrote:
Hi,

Again "I'm using ParFUM to parallelize a C++ FEM code called yafeq" and I need to use MPI or AMPI commands  together with FEM_ commands

When I try to use MPI commands on the driver I get this error



------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: Cannot call MPI routines before AMPI is initialized.

Fatal error on PE 0> Cannot call MPI routines before AMPI is initialized.




Where should I call my MPI_Init(). I have the feeling that charmc need to be instructed to do so by using compiling flags. I'm compiling the code with cmake and the following flags

 CMAKE_CXX_COMPILER               /Users/fstump2/myLibs/charm-6.2/net-darwin-x86-gfortran-gcc/bin/charmc
 CMAKE_CXX_FLAGS                  -language ParFUM -module netfem
 CMAKE_CXX_FLAGS_DEBUG            -g -language ParFUM -module netfem

Am I missing something?

Thanks
Fernando
_______________________________________________
charm mailing list
charm AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/charm
_______________________________________________
ppl mailing list
ppl AT cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/ppl



Archive powered by MHonArc 2.6.16.

Top of Page