MESSy/CLaMS: Restarts

If the restart event is triggered, the status of the model is dumped with full precision to restart files

Write restart files

• At the end of a MESSy simulation restart files are written.

• Restart files can be written in a given simulation time interval. The simulation can be interrupted and restarted automatically when a given number of cycles is reached (TIMER-User-Manual, 4.4). The interval and the number of cycles can be specified in the messy-script and are replaced in timer.nml, e.g.:
  
  timer.nml:

  IO_RERUN_EV = ${RESTART_INTERVAL},'${RESTART_UNIT}','last',0,
  NO_CYCLES   = ${NO_CYCLES},

  messy-script:

  ### INTERVAL FOR WRITING (REGULAR) RESTART FILES
  ### Note: This has only an effect, if it is not explicitely overwritten
  ###       in your timer.nml; i.e., make sure that in timer.nml 
  ###         IO_RERUN_EV = ${RESTART_INTERVAL},'${RESTART_UNIT}','last',0,
  ###       is active!
  ### RESTART_UNIT: steps, hours, days, months, years
  RESTART_INTERVAL=1
  RESTART_UNIT=months
  NO_CYCLES=12

  => Restart files are witten at the beginning of a new month and after 12 months the simulation will be interrupted and restarted automatically.

• If starttime of model is noon and restart files should be written at noon:
  

  • use unit 'hours'

    IO_RERUN_EV = 240,'hours','first',0

    or use unit 'days' and set offset to 86400 [sec]

    IO_RERUN_EV = 10,'days','first',86400

    This offset is not allowed for units 'years' and 'months'.

• If the job is submitted to a queue manager, it might be necessary to split the simulation into chain elements. The submodel QTIMER triggers the restart just before the maximum time reserved by the scheduler is reached (Development cycle 2 of the Modular Earth Submodel System, section 4). The queue time limit QWCH can be specified the messy-script and is replaced in qtimer.nml, e.g.:
  
  qtimer.nml

  &CTRL
  QTIME  =  $QWCH,0,0,  ! queue time limit (hh,mm,se); 0,0,0  to switch off
  QCLOCK = 'wall',  ! queue clock type (wall|cpu|user|sys)
  QFRAC  = 0.95     ! usable fraction of queue time limit

  messy-script:

  QWCH=4

  => When 95% of 4 hours CPU time are reached, restart files are written and the next chain-element is started.

Restart model

• If the file MSH_NO is in the working-directory, the model is started in rerun-mode. MSH_NO contains the number of the last chain-element.
  If you want run the simulation again from the beginning, remove file MSH_NO before starting the run script.

• All files needed for a rerun starting from a specific chain element are saved in the subdirectory save/NNNN of the working directory.
  

  NNNN is the 4-digit number of the last complete chain element.
  

  The restart files of the last chain-element are linked into the working directory.
  
  

  In order to start a rerun with chain element NNNN+1, the script messy/util/init_restart can be used to link the correct restart files:

  messy-dir/messy/util/init_restart -r NNNN -c MMMM [-d dir]

  NNNN: restart number (number of batch job in a job chain)
  MMMM: cycle number (number of restart within a batch job)

• Get restart and cycle number
  
  The script

  • messy-dir/messy/util/show_restarts.tcsh -c clams

  (called in working directory) lists all restart and cycle numbers and corresponding dates in the subdirectory save

• Continue a previous simulation:

  • Create a new output directory and call init_restart from within this directory:

    mkdir newdir
    cd newdir
    ~/messy-2.54.0-clams/messy/util/init_restart -d olddir -r NNNN -c CCCC

    • The executable and the messy-script are copied to the new directory
    • The save-directory with namelists is copied and the namelists for the specified rerun and cycle number are linked
    • If you do not create a new output directory:
      • check, that the namelists with the correct rerun and cycle number are linked

      • remove END files

  • Changes in messy-script:
    • Change WORKDIR to new output directory
    • Change the stop date
    • Do not change the start date !

  • If you want to modify the namelists, edit the namelist files in subdirectory nml

  • Start the messy-script in the new output directory
• Restarts after abnormal termination:

  If the model stops due to an occured error or a hardware problem, it can be restarted manually.
  In this case all restart files are located in the working directory and not saved to subdirectories save/NNNN.
  To clean up the working directory and move the restart files to subdirectories call the run-script with option '-c':

  xmessy_mmd -c

  Then the recent restart files can be linked into the working directory

  messy-dir/messy/util/init_restart -r NNNN -c CCCC

  Now the model can be restarted.

• The name of the experiment (EXP_NAME in run-script) must not contain the substring restart.
  All files *restart* are removed before linking the current restart files.