charm AT lists.siebelschool.illinois.edu
Subject: Charm++ parallel programming system
List archive
- From: Fernando Stump <fernando.stump AT gmail.com>
- To: Aaron Becker <abecker3 AT illinois.edu>, Charm Mailing List <charm AT cs.illinois.edu>
- Subject: Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init
- Date: Fri, 9 Sep 2011 16:37:38 -0500
- List-archive: <http://lists.cs.uiuc.edu/pipermail/charm>
- List-id: CHARM parallel programming system <charm.cs.uiuc.edu>
Hi,
Here it is another piece of information that may help, understand the issue.
The call
MPI_Comm_rank (MPI_COMM_WORLD, &my_rank);
when nested is failing inside ampi.C line 2239 (pasted below)
Any ideas why CtvAccess(ampiPtr) is returning a null pointer depending from where it is MPI_Comm_rank called?
Thanks
Fernando
ampiParent *getAmpiParent(void) {
ampiParent *p = CtvAccess(ampiPtr);
#ifndef CMK_OPTIMIZE
if (p==NULL) CkAbort("Cannot call MPI routines before AMPI is initialized.\n");
#endif
return p;
}
On Sep 9, 2011, at 3:39 PM, Fernando Stump wrote:
HI Aaron,This problem is similar to something similar I had in the past, in this same code. (the e-mail about that problem is pasted at the end of this e-mail) In the past some FEM_* functions cannot be called from inside nested functions, just from the outside "layers" the specific function was//FEM_Add_ghost_layer(1,1);Now it is a MPI_* so MPI_* calls work inside init() but not on nested functionsSo in this example, I get this error. Any guess for were to look at?OUTPUT...Driver 0 startedChecking MPI statusNumber of processors: 1My rank: 0My ParFUM id: 0read_abaqus_file_on_driver My ParFUM id: 0------------- Processor 0 Exiting: Called CmiAbort ------------Reason: Cannot call MPI routines before AMPI is initialized.Charm++ fatal error:Cannot call MPI routines before AMPI is initialized.CODEextern "C" voiddriver(void){..int i, myId=FEM_My_partition();CkPrintf(" Driver %d started \n", myId);std::cout << "Checking MPI status" << endl;;int num_procs;MPI_Comm_size (MPI_COMM_WORLD, &num_procs);std::cout << "Number of processors: " << num_procs << endl;int my_rank;MPI_Comm_rank (MPI_COMM_WORLD, &my_rank);std::cout << "My rank: " << my_rank << endl;std::cout << "My ParFUM id: " << myId << endl;..yafeq::FEM<dimension> fem_driver;fem_driver.read_abaqus_file_on_driver(path);...}template<int d, template <int> class Assembly_policy, class FEM_policy, class Transience_policy>void FEM<d, Assembly_policy, FEM_policy, Transience_policy>::read_abaqus_file_on_driver(char* filename){..int myId=FEM_My_partition();std::cout << "read_abaqus_file_on_driver My ParFUM id: " << myId << endl;int my_rank;MPI_Comm_rank (MPI_COMM_WORLD, &my_rank); ///// THIS CALL DOES NOT WORK HERE //////std::cout << " read_abaqus_file_on_driver My rank: " << my_rank << endl;// create new mesh and store itmesh_pointer mp = *meshes_.insert(meshes_.end(), mesh_pointer(new mesh_type(filename)));// send request to mesh objectmp->read_abaqus_file_on_driver(filename);..}/////////////////////////////////////////////////////// PREVIOUS MESSAGES ////////////////////////////////////////HI Aaron,I don't see a reason for not working, but it is simply does not, it does not give any error, it just do not create the layers. Look the output FEM_Mesh_print(mesh) for both cases.My only guess is that FEM_* call need to be insideextern "C" voidinit(void){}type environment.Does it make sense?ThanksFernando_____________________________________CALLING FROM INSIDE "read_abaqus_file_on_driver" (NOT WORKING)_____________________________Abaqus file read successfully on driver...Fernando-Stumps-MacBook-Pro:yafeq fstump2$ charmrun ./pfem.out ++p 3 ++localCharm++: scheduler running in netpoll mode.Charm++> Running on 1 unique compute nodes (4-way SMP).Charm++> Cpu topology info:PE to node map: 0 0 0Node to PE map:Chip #0: 0 1 2Charm++> cpu topology info is gathered in 0.003 seconds.Starting INITinit startedCalling read_abaqus_file_on_initPATCH TEST [C3D4]Passing node coords to frameworkPassing elements to frameworkCreating Ghost LayersAbaqus file read successfully on init...Finished with init (Reading took 0 s)-------------------- Chunk 0 --------------------Length of FEM_NODE = 16Node global numbers: * marks primary, () surrounds local-only:*(0) *(1) *(2) *(3) *(4) *(5) *(6) *(7) *(8) *(9) *(10) *(11) *(12) *(13) *(14) *(15)Length of FEM_ELEM+10 = 35FEM_ELEM+10 Connectivity:Entry (0) | (10) (11) (15) (9)Entry (1) | (9) (15) (12) (13)Entry (2) | (10) (15) (14) (12)Entry (3) | (10) (9) (12) (8)Entry (4) | (10) (9) (15) (12)Entry (5) | (10) (14) (15) (6)Entry (6) | (6) (15) (3) (7)Entry (7) | (10) (15) (11) (3)Entry (8) | (10) (6) (3) (2)Entry (9) | (10) (6) (15) (3)Entry (10) | (9) (13) (12) (5)Entry (11) | (5) (12) (0) (4)Entry (12) | (9) (12) (8) (0)Entry (13) | (9) (5) (0) (1)Entry (14) | (9) (5) (12) (0)Entry (15) | (10) (11) (9) (3)Entry (16) | (3) (9) (0) (1)Entry (17) | (10) (9) (8) (0)Entry (18) | (10) (3) (0) (2)Entry (19) | (10) (3) (9) (0)Entry (20) | (9) (11) (15) (3)Entry (21) | (3) (15) (5) (7)Entry (22) | (9) (15) (13) (5)Entry (23) | (9) (3) (5) (1)Entry (24) | (9) (3) (15) (5)Entry (25) | (12) (13) (15) (5)Entry (26) | (5) (15) (6) (7)Entry (27) | (12) (15) (14) (6)Entry (28) | (12) (5) (6) (4)Entry (29) | (12) (5) (15) (6)Entry (30) | (12) (14) (10) (6)Entry (31) | (6) (10) (0) (2)Entry (32) | (12) (10) (8) (0)Entry (33) | (12) (6) (0) (4)Entry (34) | (12) (6) (10) (0)Driver 0 started-------------------- Chunk 0 --------------------Length of FEM_NODE = 11Node global numbers: * marks primary, () surrounds local-only:6 15 *3 *7 10 11 *2 9 0 1 5Length of FEM_ELEM+10 = 11FEM_ELEM+10 Connectivity:Entry 6 | 6 15 3 7Entry 7 | 10 15 11 3Entry 8 | 10 6 3 2Entry 9 | 10 6 15 3Entry 15 | 10 11 9 3Entry 16 | 3 9 0 1Entry 18 | 10 3 0 2Entry 20 | 9 11 15 3Entry 21 | Driver 1 started-------------------- Chunk 1 --------------------Length of FEM_NODE = 11Node global numbers: * marks primary, () surrounds local-only:*3 15 5 7Entry 23 | 9 3 5 1Entry 24 | 9 3 15 5 10 *11 *15 *9 *12 *PATCH TEST [C3D4]Abaqus file read successfully on driver...Driver finished in 0.00524213 *14 *6 5 7 4Length of FEM_ELEM+10 = 11FEM_ELEM+10 Connectivity:Entry 0 | 10 11 15 9Entry 1 | 9 15 12 13Entry 2 | 10 15 14 12Entry 5 | 10 14 15 6Entry 22 | 9 15 13 5Entry 25 | 12 13 15 5Entry 26 | 5 15 6 7Entry 27 | Driver 2 started-------------------- Chunk 2 --------------------Length of FEM_NODE = 13Node global numbers: * marks primary, () surrounds local-only:10 9 12 *8 12 15 14 6Entry 28 | 12 5 15 13 *5 *0 *6 4Entry 29 | 12 5 15 6 4 *1Entry 30 | 12 14 10 63 6 2Length of FEM_ELEM+10 = 13FEM_ELEM+10 Connectivity:Entry 3 | 10 9 12 8Entry 4 | 10 9 15 12Entry 10 | 9 13 12 5Entry 11 | 5 12 0 4Entry 12 | 9 12 8 0Entry 13 |PATCH TEST [C3D4]Abaqus file read successfully on driver...9 5 0 1Entry 14 | 9 5 12 0Entry 17 | 10 9 8 0Entry 19 | 10 3 9 0Entry 31 | 6 10 0 2Entry 32 | 12 10 8 0Entry 33 | 12 6 0 4Entry 34 | 12 6 10 0PATCH TEST [C3D4]Abaqus file read successfully on driver..._____________________________________CALLING FROM INSIDE "init" (WORKING)_____________________________Fernando-Stumps-MacBook-Pro:yafeq fstump2$ charmrun ./pfem.out ++p 3 ++localCharm++: scheduler running in netpoll mode.Charm++> Running on 1 unique compute nodes (4-way SMP).Charm++> Cpu topology info:PE to node map: 0 0 0Node to PE map:Chip #0: 0 1 2Charm++> cpu topology info is gathered in 0.002 seconds.Starting INITinit startedCalling read_abaqus_file_on_initPATCH TEST [C3D4]Passing node coords to frameworkPassing elements to frameworkCreating Ghost LayersAbaqus file read successfully on init...Finished with init (Reading took 0 s)-------------------- Chunk 0 --------------------Length of FEM_NODE = 16Node global numbers: * marks primary, () surrounds local-only:*(0) *(1) *(2) *(3) *(4) *(5) *(6) *(7) *(8) *(9) *(10) *(11) *(12) *(13) *(14) *(15)Length of FEM_ELEM+10 = 35FEM_ELEM+10 Connectivity:Entry (0) | (10) (11) (15) (9)Entry (1) | (9) (15) (12) (13)Entry (2) | (10) (15) (14) (12)Entry (3) | (10) (9) (12) (8)Entry (4) | (10) (9) (15) (12)Entry (5) | (10) (14) (15) (6)Entry (6) | (6) (15) (3) (7)Entry (7) | (10) (15) (11) (3)Entry (8) | (10) (6) (3) (2)Entry (9) | (10) (6) (15) (3)Entry (10) | (9) (13) (12) (5)Entry (11) | (5) (12) (0) (4)Entry (12) | (9) (12) (8) (0)Entry (13) | (9) (5) (0) (1)Entry (14) | (9) (5) (12) (0)Entry (15) | (10) (11) (9) (3)Entry (16) | (3) (9) (0) (1)Entry (17) | (10) (9) (8) (0)Entry (18) | (10) (3) (0) (2)Entry (19) | (10) (3) (9) (0)Entry (20) | (9) (11) (15) (3)Entry (21) | (3) (15) (5) (7)Entry (22) | (9) (15) (13) (5)Entry (23) | (9) (3) (5) (1)Entry (24) | (9) (3) (15) (5)Entry (25) | (12) (13) (15) (5)Entry (26) | (5) (15) (6) (7)Entry (27) | (12) (15) (14) (6)Entry (28) | (12) (5) (6) (4)Entry (29) | (12) (5) (15) (6)Entry (30) | (12) (14) (10) (6)Entry (31) | (6) (10) (0) (2)Entry (32) | (12) (10) (8) (0)Entry (33) | (12) (6) (0) (4)Entry (34) | (12) (6) (10) (0)FEM Ghost layer 1> 70 new ghost elements, 13 new ghost nodesDriver 0 started-------------------- Chunk 0 --------------------Length of FEM_NODE = 11 (and 5 ghosts)Node global numbers: * marks primary, () surrounds local-only:6 15 *3 *7 10 11 *2 9 0 1 5Length of FEM_ELEM+10 = 11 (and 24 ghosts)FEM_ELEM+10 Connectivity:Entry 6 | 6 15 3 7Entry 7 | 10 15 11 3Entry 8 | 10 6 3 2Entry 9 | 10 6 15 3Entry 15 | 10 11 9 3Entry 16 | 3 9 0 1Entry 18 | 10 3 0 2Entry 20 | 9 11 15 3Entry 21 | 3 15 5 7Entry 23 | 9 3 5 1Entry 24 | 9 3 15 5PATCH TEST [C3D4]Abaqus file read successfully on driver...Driver finished in 0.006951Driver 2 started-------------------- Chunk 2 --------------------Length of FEM_NODE = 13 (and 3 ghosts)Node global numbers: * marks primary, () surrounds local-only:10 9 12 *8 15 13 *5 *0 *4 *1 3 6 2Length of FEM_ELEM+10 = 13 (and 22 ghosts)FEM_ELEM+10 Connectivity:Entry 3 | 10 9 12 8Entry 4 | 10 9 15 12Entry 10 | 9 13 12 5Entry 11 | 5 12 0 4Entry 12 | 9 12 8 0Entry 13 | 9 5 0 1Entry 14 | 9 5 12 0Entry 17 | 10 9 8 0Entry 19 | 10 3 9 0Entry 31 | 6 10 0 2Entry 32 | 12 10 8 0Entry 33 | 12 6 0 4Entry 34 | 12 6 10 0Driver 1 started-------------------- Chunk 1 --------------------Length of FEM_NODE = 11 (and 5 ghosts)Node global numbers: * marks primary, () surrounds local-only:*10 *11 *15 *9 *12 *13 *14 *6 5 7 4Length of FEM_ELEM+10 = 11 (and 24 ghosts)FEM_ELEM+10 Connectivity:Entry 0 | 10 11 15 9Entry 1 | 9 15 12 13Entry 2 | 10 15 14 12Entry 5 | 10 14 15 6Entry 22 | 9 15 13 5Entry 25 | 12 13 15 5Entry 26 | 5 15 6 7Entry 27 | 12 15 14 6Entry 28 | 12 5 6 4Entry 29 | 12 5 15 6EntryPATCH TEST [C3D4]Abaqus file read successfully on driver...30 | 12 14 10 6PATCH TEST [C3D4]Abaqus file read successfully on driver...Fernando-Stumps-MacBook-Pro:yafeq fstump2$On Apr 6, 2011, at 2:54 PM, Aaron Becker wrote:There is no reason you can't call FEM_Add_ghost_layer and
FEM_Add_ghost_elem from another function inside init. In fact, several
working applications do it that way. What error message do you get
when you try calling them from read_abaqus_file_on_init?
Aaron
On Thu, Mar 31, 2011 at 2:14 PM, Fernando Stump
<fernando.stump AT gmail.com> wrote:Hi,I would like to know if someone can explain why the call for add ghostlayer need to be done on the Init() function and cannot be nested infunction inside Init(). Bellow it is and example how It works or not.ThanksFernandoI'm using ParFUM in a C++ library called Yafeq (Yanhua and Osman are somehowfamiliar with yafeq). I have my init function as:extern "C" voidinit(void){CkPrintf("init started\n");int mesh=FEM_Mesh_default_write(); // Tell framework we are writing to themeshyafeq::FEM<2> fem_init;// read abaqus filechar path[] = "../tests/input/abaqus_patch_tests/ece3sfp1_step1_yafeq.inp";fem_init.read_abaqus_file_on_init(path);// Add the nodal ghost layer. Not edge based.const int trianglefaces[3] = {0,1,2};FEM_Add_ghost_layer(1,1);FEM_Add_ghost_elem(0,3,trianglefaces);CkPrintf("Finished with init (Reading took %.fs)\n",CmiWallTimer()-startTime);FEM_Mesh_print(mesh);}And the class yafeq::FEM<d> has the class Mesh<d> which has the function"read_abaqus_file_on_init" which read the mesh and provides it to parfum asbelowtemplate <int d>void Mesh<d>::read_abaqus_file_on_init(char* filename) {....{//Pass Information to ParFUMint mesh=FEM_Mesh_default_write(); // Tell framework we are writing to themesh//Plain data type to be passed to ParFUMint n_nodes=num_nodes();double *coord= new double[d*n_nodes];int count=0;//loop over nodesfor (id_node_map_iterator nit = nodeMap.begin(); nit != nodeMap.end();++nit) {for (size_t i = 0; i < d; ++i) {coord[count++]=(*(nit->second))[i];}}CkPrintf("Passing node coords to framework\n");//Passing the nodes to parfumFEM_Mesh_data(mesh, // Add nodes to the current meshFEM_NODE, // We are registering nodesFEM_DATA+0, // Register the point locations which are normally// the first data elements for an FEM_NODE(double *)coord, // The array of point locations0, // 0 based indexingn_nodes, // The number of pointsFEM_DOUBLE, // Coordinates are doublesd); // Points have dimension 2 (x,y)//Plain data type to be passed to ParFUMint n_elements=num_elements();int *elements= new int[3*n_elements];count=0;//loop over elementsfor (id_element_map_iterator elem_it = elementMap.begin(); elem_it !=elementMap.end(); ++elem_it) {for (typename element_type::node_iterator node_it =((elem_it->second)->nodes_begin()); node_it !=((elem_it->second)->nodes_end()); ++node_it){int id = nodeIdMap[*node_it];elements[count++]=id;}}CkPrintf("Passing elements to framework\n");FEM_Mesh_data(mesh, // Add nodes to the current meshFEM_ELEM+0, // We are registering elements with type 0// The next type of element could be registered with FEM_ELEM+1FEM_CONN, // Register the connectivity table for this// data elements for this type of FEM entity(int *)elements, // The array of point locations0, // 0 based indexingn_elements, // The number of elementsFEM_INDEX_0, // We use zero 1 node numbering3); // Elements have degree 3, since triangles are defineddelete[] elements;delete[] coord;// Setup a node-adjacent ghost layer as we need all the elements around anode to compute//!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!// THIS FUNCTIONS CANNOT BE CALLED HERE, THEY NEED TO BE CALLED AT THEINIT() ELEVEL// Add the nodal ghost layer. Not edge based.// const int trianglefaces[3] = {0,1,2};//FEM_Add_ghost_layer(1,1);//FEM_Add_ghost_elem(0,3,trianglefaces);}_______________________________________________charm mailing listcharm AT cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/charm_______________________________________________ppl mailing listppl AT cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/pplOn Sep 9, 2011, at 1:57 PM, Aaron Becker wrote:The way you're compiling looks ok, and you don't need to call MPI_Init
at all.. Make sure driver is declared like this:
extern "C" void driver(void)
and if you have problems, try referring to the globalExample
application in the examples directory, as it does use MPI calls
directly.
Aaron
On Fri, Sep 9, 2011 at 11:52 AM, Fernando Stump
<fernando.stump AT gmail.com> wrote:Hi,Again "I'm using ParFUM to parallelize a C++ FEM code called yafeq" and I need to use MPI or AMPI commands together with FEM_ commandsWhen I try to use MPI commands on the driver I get this error------------- Processor 0 Exiting: Called CmiAbort ------------Reason: Cannot call MPI routines before AMPI is initialized.Fatal error on PE 0> Cannot call MPI routines before AMPI is initialized.Where should I call my MPI_Init(). I have the feeling that charmc need to be instructed to do so by using compiling flags. I'm compiling the code with cmake and the following flagsCMAKE_CXX_COMPILER /Users/fstump2/myLibs/charm-6.2/net-darwin-x86-gfortran-gcc/bin/charmcCMAKE_CXX_FLAGS -language ParFUM -module netfemCMAKE_CXX_FLAGS_DEBUG -g -language ParFUM -module netfemAm I missing something?ThanksFernando_______________________________________________charm mailing listcharm AT cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/charm_______________________________________________ppl mailing listppl AT cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/ppl
- [charm] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Aaron Becker, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Aaron Becker, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Aaron Becker, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Fernando Stump, 09/09/2011
- Re: [charm] [ppl] Where to call MPI_Init or AMPI_Init, Aaron Becker, 09/09/2011
Archive powered by MHonArc 2.6.16.