Differences between revisions 7 and 10 (spanning 3 versions)

MESSy/CLaMS: Restarts

If the restart event is triggered, the status of the model is dumped with full precision to restart files
At the end of a MESSy simulation restart files are written.
Restart files can be written in a given simulation time interval. The simulation can be interrupted and restarted automatically when a given number of cycles is reached (TIMER-User-Manual, 4.4). The interval and the number of cycles can be specified in messy/nml/DEFAULTS/timer.nml, e.g.:
```
IO_RERUN_EV = 1,'month','first',0,
NO_CYCLES   = 12           ! restart cycles without break
```
=> Restart files are witten at the beginning of a new month and after 12 months the simulation will be interrupted and restarted automatically.

If the job is submitted to a queue manager, it might be necessary to split the simulation into chain elements. The submodel QTIMER triggers the restart just before the maximum time reserved by the scheduler is reached (Development cycle 2 of the Modular Earth Submodel System, section 4). The queue time limit and the usable fraction can be specified in qtimer.nml, e.g.:
```
&CTRL
QTIME  =  4,0,0,  ! queue time limit (hh,mm,se); 0,0,0  to switch off
QCLOCK = 'wall',  ! queue clock type (wall|cpu|user|sys)
QFRAC  = 0.95     ! usable fraction of queue time limit
```
=> When 95% of 4 hours CPU time are reached, restart files are written and the next chain-element is started.
If the file MSH_NO is in the working-directory, the model is started in rerun-mode. MSH_NO contains the number of the last chain-element.
If you want run the simulation again from the beginning, remove file MSH_NO before starting the run script.
All files needed for a rerun starting from a specific chain element are saved in the subdirectory save/NNNN of the working directory.
- NNNN is the 4-digit number of the last complete chain element.
The restart files of the last chain-element are linked into the working directory
In order to start a rerun with chain element NNNN+1, the script messy/util/init_restart can be used to link the correct restart files:
```
  init_restart -r NNNN -c MMMM [-d dir]
```
NNNN: restart number
MMMM: cycle number
The name of the experiment (EXP_NAME in run-script) must not contain the substring restart.
All files *restart* are removed before linking the current restart files.

-  ⇤ ← Revision 7 as of 2016-01-14 08:25:17 → 
  Size: 5591
  Editor: NicoleThomas
  Comment:
+   ← Revision 10 as of 2016-05-25 07:30:02 → ⇥
  Size: 2492
  Editor: NicoleThomas
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 16:
- Restart files are written when the subroutine ''messy_write_output'' is called and the restart event is triggered. <<BR>>
 If restart files would be written at midnight (e.g. at the beginning of a new month) and CLaMS-Output is only written at noon every day, in ''messy_write_output'' the restart event is never TRUE <<BR>> => No restart files are written !!!
-Line 50:
+Line 44:
- * A file '''END''' is written at the end of the simulation or if an error occurs. <<BR>>
 => If there are files ''END*'' in the working directory, no restart is done.
-Line 56:
+Line 46:
------


 * Folgende Restart-Files werden bei einem CLaMS-Lauf erstellt:

  * restart_cccc_clams.nc:
    * dnparts
    *  grid_switch
    * year_uvfirst, month_uvfirst, day_uvfirst, hour_uvfirst
    * chem. Spezies
    * JULSEC
    * LAT, LAT_OLD, LAT_OLD_MIX
    * LON, LON_OLD, LON_OLD_MIX
    * LEV, LEV_OLD
    * TEMP, TEMP_OLD
    * PRESS, PRESS_OLD
  * restart_cccc_CLAMS.nc
  * restart_cccc_winddata.nc
    * LAT, LON, LEV
    * UDT, VDT, WDT, LEVELDT
    * UFUT, VFUT, WFUT, LEVELFUT
    * PREDATA_TEMP, FUTDATA_TEMP
    * PREDATA_PRESS, FUTDATA_PRESS
    
 * Es werden Restart-Files für jeweils bis zu 5 Zyklen erstellt (da in jeder der entsprechenden SMIL-Routinen ''messy_write_output'' aufgerufen wird!) :
    * nach traj
    * nach dissoc
    * nach chem
    * nach mix
    * nach bmix
 => Für jedes dieser Submodule läßt sich der Output (und damit auch das Restartfile) ausschalten (in zugehöriger Namelist: ''loutput_paketname=.false.'')

 * Ausgabe auf Restart-Files in MESSy-CLaMS:

  *  In einzelnen Submodules (aus SMIL-Routinen) wird ''messy_write_output'' (''messy_main_control_clams.f90'') aufgerufen <<BR>>
  * In ''messy_write_output'' wird ''messy_channel_write_output'' (''messy_main_channel_bi.f90'') mit ''IOMODE_OUT'' aufgerufen  <<BR>>
  => schreibe Channels/Channelobjekte auf Output-File
  *In ''messy_write_output'' wird ''messy_channel_write_output'' mit ''IOMODE_RST'' aufgerufen, falls ''l_rerun=.true.'' <<BR>>
  => schreibe Channels/Channelobjekte auf Restart-File (falls Restart-Event gesetzt)
  * program clams (clams_main.f90), innerhalb der Zeitschleife: <<BR>>
    -> sub. messy_global_start (messy_main_control_clams.f90) <<BR>>
       -> sub. main_timer_global_start (messy_main_timer_bi.f90) <<BR>>
          -> setze l_rerun: (true, wenn nächster Rerun-Ausgabezeitschritt oder maximale Rechenzeit erreicht)

 * Einlesen der Restart-Files in MESSy-CLaMS:

  clams_main <<BR>>
    -> messy_channel_read_restart (messy_main_channel_bi.f90) <<BR>>
      -> channel_read_data (messy_main_channel_io.f90) <<BR>>

 * Restart-Files in ECHAM:

  messy/echam5/bmil/messy_main_control_e5.f90: <<BR>>
    sub. messy_write_output <<BR>>
      -> messy_channel_write_output(IOMODE_OUT) (messy_main_channel_bi.f90) <<BR>>
    in echam5/src/stepon.f90 <<BR>>
      -> messy_channel_write_output(IOMODE_RST)  <<BR>>

  Z.B. Restart-Files einmal pro Monat: <<BR>>
    In ''messy/nml/DEFAULTS/timer.nml'': <<BR>>
      IO_RERUN_EV = 1,'months','first',0,