Multi-Processor Handshaking Utility and programing model -------------------------------------------------------- Key Issue: provide a object-oriented, highly modular codes to facilitate multiple components handshaking and communication with also a plug & play convenience. 1. MCME (multiple components, multiple executables), e.g., CCM, POP, Coupler etc are separate executables. Each component model calls the shared handshaking routine. call MPH_setup ("atmosphere", CCM_World) ! on CCM component or call MPH_setup ("ocean", POP_World) ! on POP component which is implemented using gather, broadcast, etc, and scales as log(total_Processors). The component_model names are registered in "components.in" file. The order of file names are irrelevant. COMPONENT_LIST BEGIN atmosphere ocean coupler land ice END Sample codes for ccm.F, pop.F and coupler.F are given. ccm1.F is provided to illustrate why MPI_COMM_WORLD should not be used, and instead CCM_World etc should be used. 2. It will also support MCSE (Multiple Components, Single Executable), with component models run on different processors, but in a single program executable. The master program calls the following: call MPH_setup_SE( & atmosphere = ccm3_8, ! call ccm3_8() on atmosphere processors & ocean = pop2_2, ! call pop2_2() on ocean processors & coupler = cpl5_1, ! call cpl5_1() on coupler processors communicater = MyWorld) ! get proper communicator on this proc Here atmosphere, ocean, coupler, communicater are KEYWORDS. Additional keywords sea_ice, land, biosphere, io are supported. Order of names are irrelevant. Components are all optional: You may invoke "ocean=POP, sea_ice=ice3_1" only. Besides the master program (the main program), POP, CCM, etc are written as a subroutine, which can be easily converted to/from stand-alone, MCME or MCSE with 3-4 lines changes. Inside MPH_setup_SE, different subroutines, POP, CCM, etc, will be called on different processors specified in "processor.map" file, listing component names (keywords) and processor ranges: PROCESSOR_MAP BEGIN atmosphere 0 15 ocean 16 30 coupler 31 31 land 32 35 ice 36 39 END Everything below, the two modes MCME and MCSE will work identical. 3. MPI communication, global sum, etc, between local processors (processors on the same component model) are done as if there are no other component models running, and always use Proc_ID 0, 1, ..., TotProcs-1. (Here TotProcs is the numbe rof processors used for that component.) Global ProcessorIDs of the same component do not have to be contiguous. They are arbitrary. 4. MPI communication between local processors and remote processors (processors on other component models) are invoked through component names and the local id. E.g., a processor on atmosphere wants to send Process 3 on Ocean, it invokes MPI_send(..., MPH_global_id("ocean", 3), .....) which is exactly the same MPI_Send function except that destination_processor is replaced by MPH_global_id(). (Initially, we implemented it as MPI_send_int(..., "ocean", 3, .....) MPI_send_char(..., "ocean", 3, .....) MPI_send_real(..., "ocean", 3, .....) But they are much more restrictive than the original MPI routines and also a lot of repetitive codes. We thus changed to the current way. Suggestions for better ways are welcome.) 5. No explicit model names handcoded into the codes. No explicit number of models handcoded into the codes. It will work for 3-component system like atm + ocean + coupler. The same identical codes without a single line source code change will work for 7 component system like atm + ocean + land + ice + coupler + Biosphere + I/O Component names and numbers are completely flexible. 6. A communicator between any two models could be created: MPH_comm_join ("ATM", "POP", COMM_atm_pop) Then m-to-n transpose could be done on this communicator. Here ATM processors have ranks 0 - 15 and POP processors will be 16-23 (assuming ATM has 16 processors and POP has 8 processors). If you reverse ATM with POP, and call MPH_comm_join ("POP","ATM", COMM_pop_atm) then POP processors will rank 0 - 7 and ATM processors will be 8-23. Sending data from all POP processors to Coupler (1 proc) is done in two simple calls: MPH_comm_join ("Coupler","POP", COMM_Cpl_POP) MPI_gather(data, ..., COMM_Cpl_POP, .....) 7. The functionality are strictly limited to communications between different component executables. How atm communicates with coupler, through what kind of transpose, etc, can be implemented independently, but making use of global processor IDs provided throught this codes (using the Acomponent data structure in mph.F).