MESSy/CLaMS: Restarts

• If the restart event is triggered, the status of the model is dumped with full precision to restart files
• At the end of a MESSy simulation restart files are written.

• Restart files can be written in a given simulation time interval. The simulation can be interrupted and restarted automatically when a given number of cycles is reached (TIMER-User-Manual, 4.4). The interval and the number of cycles can be specified in messy/nml/DEFAULTS/timer.nml, e.g.:

  IO_RERUN_EV = 1,'month','first',0,
  NO_CYCLES   = 12           ! restart cycles without break

  => Restart files are witten at the beginning of a new month and after 12 months the simulation will be interrupted and restarted automatically.

• If the job is submitted to a queue manager, it might be necessary to split the simulation into chain elements. The submodel QTIMER triggers the restart just before the maximum time reserved by the scheduler is reached (Development cycle 2 of the Modular Earth Submodel System, section 4). The queue time limit and the usable fraction can be specified in qtimer.nml, e.g.:

  &CTRL
  QTIME  =  4,0,0,  ! queue time limit (hh,mm,se); 0,0,0  to switch off
  QCLOCK = 'wall',  ! queue clock type (wall|cpu|user|sys)
  QFRAC  = 0.95     ! usable fraction of queue time limit

  => When 95% of 4 hours CPU time are reached, restart files are written and the next chain-element is started.

• If the file MSH_NO is in the working-directory, the model is started in rerun-mode. MSH_NO contains the number of the last chain-element.
  If you want run the simulation again from the beginning, remove file MSH_NO before starting the run script.

• All files needed for a rerun starting from a specific chain element are saved in the subdirectory save/NNNN of the working directory.

  • NNNN is the 4-digit number of the last complete chain element.
• The restart files of the last chain-element are linked into the working directory

• In order to start a rerun with chain element NNNN+1, the script messy/util/init_restart can be used to link the correct restart files:

    init_restart -r NNNN -c MMMM [-d dir]

  NNNN: restart number
  MMMM: cycle number

• A file END is written at the end of the simulation or if an error occurs.
  => If there are files END* in the working directory, no restart is done.

• The name of the experiment (EXP_NAME in run-script) must not contain the substring restart.
  All files *restart* are removed before linking the current restart files.

• Folgende Restart-Files werden bei einem CLaMS-Lauf erstellt:
  • restart_cccc_clams.nc:
    • dnparts
    • grid_switch
    • year_uvfirst, month_uvfirst, day_uvfirst, hour_uvfirst
    • chem. Spezies
    • JULSEC
    • LAT, LAT_OLD, LAT_OLD_MIX
    • LON, LON_OLD, LON_OLD_MIX
    • LEV, LEV_OLD
    • TEMP, TEMP_OLD
    • PRESS, PRESS_OLD
  • restart_cccc_CLAMS.nc
  • restart_cccc_winddata.nc
    • LAT, LON, LEV
    • UDT, VDT, WDT, LEVELDT
    • UFUT, VFUT, WFUT, LEVELFUT
    • PREDATA_TEMP, FUTDATA_TEMP
    • PREDATA_PRESS, FUTDATA_PRESS

• Es werden Restart-Files für jeweils bis zu 5 Zyklen erstellt (da in jeder der entsprechenden SMIL-Routinen messy_write_output aufgerufen wird!) :

  • nach traj
  • nach dissoc
  • nach chem
  • nach mix
  • nach bmix

  => Für jedes dieser Submodule läßt sich der Output (und damit auch das Restartfile) ausschalten (in zugehöriger Namelist: loutput_paketname=.false.)

• Ausgabe auf Restart-Files in MESSy-CLaMS:

  • In einzelnen Submodules (aus SMIL-Routinen) wird messy_write_output (messy_main_control_clams.f90) aufgerufen
    

  • In messy_write_output wird messy_channel_write_output (messy_main_channel_bi.f90) mit IOMODE_OUT aufgerufen
    => schreibe Channels/Channelobjekte auf Output-File

  • In messy_write_output wird messy_channel_write_output mit IOMODE_RST aufgerufen, falls l_rerun=.true.
    => schreibe Channels/Channelobjekte auf Restart-File (falls Restart-Event gesetzt)

  • program clams (clams_main.f90), innerhalb der Zeitschleife:
    

    • -> sub. messy_global_start (messy_main_control_clams.f90)
      

      • -> sub. main_timer_global_start (messy_main_timer_bi.f90)
        

        • -> setze l_rerun: (true, wenn nächster Rerun-Ausgabezeitschritt oder maximale Rechenzeit erreicht)

• Einlesen der Restart-Files in MESSy-CLaMS:

  • clams_main
    

    • -> messy_channel_read_restart (messy_main_channel_bi.f90)
      

      • -> channel_read_data (messy_main_channel_io.f90)
        

• Restart-Files in ECHAM:

  • messy/echam5/bmil/messy_main_control_e5.f90:
    

    • sub. messy_write_output
      

      • -> messy_channel_write_output(IOMODE_OUT) (messy_main_channel_bi.f90)
        

      in echam5/src/stepon.f90
      

      • -> messy_channel_write_output(IOMODE_RST)
        

    Z.B. Restart-Files einmal pro Monat:
    

    • In messy/nml/DEFAULTS/timer.nml:
      

      • IO_RERUN_EV = 1,'months','first',0,