= MESSy/CLaMS: Restarts =

If the restart event is triggered, the status of the model is dumped with full precision to restart files

=== Write restart files ===

 * At the end of a MESSy simulation restart files are written.

 * Restart files can be written in a given simulation time interval. The simulation can be interrupted and restarted automatically when a given number of cycles is reached (TIMER-User-Manual, 4.4). The interval and the number of cycles can be specified in the messy-script and are replaced in '''timer.nml''', e.g.:<<BR>><<BR>>

 timer.nml:
 {{{
IO_RERUN_EV = ${RESTART_INTERVAL},'${RESTART_UNIT}','first',0,
NO_CYCLES   = ${NO_CYCLES},
}}}
 messy-script:
 {{{
### INTERVAL FOR WRITING (REGULAR) RESTART FILES
### Note: This has only an effect, if it is not explicitely overwritten
###       in your timer.nml; i.e., make sure that in timer.nml 
###         IO_RERUN_EV = ${RESTART_INTERVAL},'${RESTART_UNIT}','last',0,
###       is active!
### RESTART_UNIT: steps, hours, days, months, years
RESTART_INTERVAL=1
RESTART_UNIT=months
NO_CYCLES=12
}}}
 => Restart files are witten at the beginning of a new month and after 12 months the simulation will be interrupted and restarted automatically.


 * If starttime of model is noon and restart files should be written at noon:<<BR>>
   
  use unit 'hours'
  {{{
IO_RERUN_EV = 240,'hours','first',0
}}}
  or use unit 'days' and set offset to 86400 [sec]
  {{{
IO_RERUN_EV = 10,'days','first',86400
}}}
  This offset is not allowed for units 'years' and 'months'.



 * If the job is submitted to a queue manager, it might be necessary to split the simulation into chain elements. The submodel QTIMER triggers the restart just before the maximum time reserved by the scheduler is reached (Development cycle 2 of the Modular Earth Submodel System, section 4). The queue time limit QWCH can be specified the messy-script and is replaced  in '''qtimer.nml''', e.g.:<<BR>><<BR>>

 qtimer.nml
 {{{
&CTRL
QTIME  =  $QWCH,0,0,  ! queue time limit (hh,mm,se); 0,0,0  to switch off
QCLOCK = 'wall',  ! queue clock type (wall|cpu|user|sys)
QFRAC  = 0.95     ! usable fraction of queue time limit
}}}

 messy-script:
 {{{
QWCH=4
}}}

 => When 95% of 4 hours CPU time are reached, restart files are written and the next chain-element is started.

=== Restart model  ===

 * If the file '''MSH_NO''' is in the working-directory, the model is started in rerun-mode. MSH_NO contains the number of the last chain-element. <<BR>>
 If you want run the simulation again from the beginning, remove file MSH_NO before starting the run script. 

 * All files needed for a rerun starting from a specific chain element are saved in the subdirectory ''save/NNNN'' of the working directory.<<BR>>

 NNNN is the 4-digit number of the last complete chain element. <<BR>>

 The restart files of the last chain-element are linked into the working directory.<<BR>><<BR>>

 In order to start a rerun with chain element NNNN+1, the script '''messy/util/init_restart''' can be used to link the correct restart files:  
 {{{  
messy-dir/messy/util/init_restart -r NNNN -c MMMM [-d dir]
}}}

 NNNN: restart number (number of batch job in a job chain)  <<BR>>
 MMMM: cycle number   (number of restart within a batch job)

 * Get restart and cycle number <<BR>><<BR>>

 The script 
  {{{  
messy-dir/messy/util/show_restarts.tcsh -c clams
}}}
 (called in working directory) lists all restart and cycle numbers and corresponding dates in the subdirectory ''save''
  
  
 * Continue a previous simulation:

  * Create a new output directory and call ''init_restart'' '''from within this directory''':
  {{{
mkdir newdir
cd newdir
~/messy-2.54.0-clams/messy/util/init_restart -d olddir -r NNNN -c CCCC
}}}
    * The executable and the messy-script are copied to the new directory
    * The save-directory with namelists is copied and the namelists for the specified rerun and cycle number are linked 
    * If you do not create a new output directory:
      * check, that the namelists with the correct rerun and cycle number are linked
      * remove '''END''' files

  * Changes in messy-script:
    * Change WORKDIR to new output directory
    * Change the stop date
    * Do not change the start date !

  * If you want to modify the namelists, edit the namelist files '''in subdirectory ''nml'' '''

  * Start the messy-script in the new output directory

 * Restarts after abnormal termination:

 If the model stops due to an occured error or a hardware problem, it can be restarted manually.<<BR>>
 In this case all restart files are located in the working directory and not saved to subdirectories ''save/NNNN''.<<BR>>
 To clean up the working directory and move the restart files to subdirectories call the run-script with option '-c':
 {{{
xmessy_mmd -c
}}}

 Then the recent restart files can be linked into the working directory
 {{{
messy-dir/messy/util/init_restart -r NNNN -c CCCC
}}}
 Now the model can be restarted.

 * The name of the experiment (''EXP_NAME'' in run-script) must not contain the substring ''restart''. <<BR>>
 All files ''*restart*'' are removed before linking the current restart files.