MPH1 and MPH2: Multi-Component Multi-Executable and Multi-Component Single Execut able

Unified Communication Module for Each Mode, Unified Component Source Codes with Compiler Directives

This is an implmentation interface of two modes with a unified MPH_all communication module included in file "mph.F". All modules also share same source codes for component models. Please read CSM_PCM_test for more details of testing different CSM and PCM execution modes. A common Makefile is still shared for different modes of applications and machine architectures.

You could get the source code as a tar file here or access the individual files as follows:

MPH.outline: brief introduction
MPH2b.design: extra explanation
CSM_PCM_test: testing different CSM and PCM executaion modes
README: this file
mph.F: a unified MPH_all communication module
Makefile: shared makefile.
master.F: driver code for single executable
pop.F: ocean model source code
ccm.F: atmosphere model source code
cpl.F: coupler model source code
ocean_one_step.F: ocean code for one step
atm_one_step.F: atmosphere code for one step
flux_exchange.F: flux exchange among ocean, atmosphere and coupler
script.CSM: run script for CSM mode
script.CSM_SE: run script for CSM_SE mode
script.PCM: run script for PCM mode
components.in: component model names input for CSM mode
tasklist: executable list for CSM mode
processors_map.in.CSM_SE: component model names and processor ranges input for CSM_SE mode
processors_map.in.PCM: component model names and processor ranges input for PCM mode
SAMPLE_OUTPUTS: sample outputs for three modes


To compile:
==========

    Note: Vince Wayland of NCAR contributes for SGI and Compaq

    The shared "Makefile" detects the machine architecture and compiles
    appropriately for IBM, SGI and Compaq. Also, depends on which mode
    of tests you want to run, you could generate different executables by
    typing "make CSM", "make CSM_SE" or "make PCM" (or "gmake ..." depends
    on your machine).

    One notice before you compile is that for NERSC IBM we need to use
    $TMPDIR stuff in CFOPTS to work around with F90 modules due to the
    GPFS file system, you may probably want to define "CFOPTS=" if your
    file system is compatible with F90 modules.

To run:
======

    After compile, you will have executables generated ("pop", "ccm", "cpl"
    for CSM mode, and "master_CSM" for CSM_SE mode and "master_PCM" for
    PCM mode).

    1) to run on NERSC and NCAR IBM SP interactively:
        a) % unsetenv MP_TASKS_PER_NODE
        b) % setenv pop_out_env pop.log
            % setenv ccm_out_env ccm.log
            % setenv cpl_out_env cpl.log
        c) make sure the following command in ONE LINE:
            for CSM mode:
            % poe -pgmmodel mpmd -cmdfile tasklist -nodes 5 -procs 9
                -stdoutmode ordered -infolevel 2 > & output &
            for CSM_SE and PCM modes :
            % poe master.xxx -nodes 5 -procs 9 -stdoutmode ordered
                -infolevel 2 > & output &
            here xxx is the mode name

        to run on IBM SP with batch script:
        % llsubmit script.xxx
          (here xxx is the mode name)

    2) to run on NERSC CRAY T3E interactively:
        a) % setenv pop_out_env pop.log
            % setenv ccm_out_env ccm.log
            % setenv cpl_out_env cpl.log
        b) we could not run CSM mode since there is no mpmd mechnism on T3E.
            for CSM_SE and PCM:
            % mpprun -n 9 master.xxx > & output&
            here xxx is the mode name

         to run on NERSC CRAY T3E with batch script:
         % cqsub run.t3e
         and the script "run.t3e" looks like:
         #!/bin/csh
         #QSUB -q debug
         #QSUB -l mpp_t=300    # Maximum residency time (for parallel jobs).
         #QSUB -l mpp_p=9      # Maximum PEs Needed (for parallel jobs).
         cd $HOME/CSM_PCM
         ja                    # Turn on Job Accounting
         setenv ccm_out_env ccm.log
         setenv pop_out_env pop.log
         setenv cpl_out_env cpl.log
         mpprun -n 9 master.xxx > output (here xxx is the mode name)
         ja -s                 # Print Job Accounting Summary

    3) to run on NCAR SGI interactively:
        a) % setenv pop_out_env pop.log
            % setenv ccm_out_env ccm.log
            % setenv cpl_out_env cpl.log
        b) for CSM mode:
            % mpirun -p "[%g]" -np 6 pop : -np 2 ccm : -np 1 cpl > output.a
            for CSM_SE and PCM modes:
            % mpirun -p "[%g]" -np 9 master.xxx > output.a
            here xxx is the mode name

        It's pretty simple to build an NQE batch script around it;
        just hard to get it run through any of the queues here.
        The tasklist file is not needed (can't be used?) on the SGI.

     4) to run on NCAR Compaq with batch script:
        You need two shell scripts, run.dec and runscript.dec.
        (Thanks to Dan Anderson & Bill Celmaster in NCAR).
        As on the SGI O2K script, a concern is to get all of the
        processes into execution at the same time.

        run.dec:
        #! /bin/csh
        prun -n9 -t runscript.dec

        for CSM mode, "runscript.dec" looks like this:
        #! /bin/csh
        #RMS_NODEID __ the node ID of the node this process is running on
        #RMS_NPROCS __ total number of prun processes spawned
        #RMS_RANK __ which prun process of the RMS_NPROCS processes

        echo "node= " $RMS_NODEID " Process number= " $RMS_RANK " of "
        $RMS_NPROCS
        if ($RMS_RANK >= 0 && $RMS_RANK <= 5) pop & # 6 procs for pop
        if ($RMS_RANK >= 6 && $RMS_RANK <= 7) ccm & # 2 procs for ocn
        if ($RMS_RANK == 8) cpl & # 1 proc for cpl
        exit

        for CSM_SE and PCM modes, "runscript.dec" looks like this:
        #! /bin/csh
        #RMS_NODEID __ the node ID of the node this process is running on
        #RMS_NPROCS __ total number of prun processes spawned
        #RMS_RANK __ which prun process of the RMS_NPROCS processes

        echo "node= " $RMS_NODEID " Process number= " $RMS_RANK " of "
        $RMS_NPROCS
        master.xxx &  (here xxx is the mode name)

To use:
======

    This code set is used to test three different execution modes:
    CSM, CSM_SE and PCM.  Proper codes used for each mode is invoked
    by proper compiler directives.  User only need to generate proper
    executable by makefile.
 
    In each mode, users need to "use MPH_all" in the application codes,
    and invoke the appropriate "MPH_setup_..." function calls.
    You could use the "MPH_help" call to get the corresponding info.
    It will also provides you the available inquiry functions for that mode.

    Each component maintains its own output in a separate file (file name
    defined by environment variable either in command line or in batch run
    script), assuming the local processor 0 of each component being
    responsible for most output, other occasional writes from all the
    components are stored in one combined standand output file.

    This is accomplished by processor rank 0 of each component call
    subroutine "MPH_redirect_output" with the model name as argument.
    IBM and SGI could do the output redirect with the help of system
    function "getenv" or "pxfgetenv". Compaq cannot do this. And T3E is
    able to get the correct output files created using "pxfgetenv",
    but only output with those "write(6,*)" could be redirected, but not
    those with "write(*,*)", since * is equal to unit 101, and permanently
    related to the non-redirectable stdout.

For more information, please go to web page:
http://www.nersc.gov/research/SCG/acpi/MPH
 

Last modified February 24, 2001.

Back to NERSC ACPI Homepage