The package packing

It is not a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly.

The used method is a lossy data compression and there is possibly a loss of accuracy.

The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values.

If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be.

The source code of the package packing is under the control of the CVS (Concurrent Version System) Utility. It is already checked out in the directory /usr/local/icg/icg1/local/packing. The programs for packing and unpacking data are compiled for 32- and 64-bit architectures. The scripts ncdf_pack.sh and ncdf_unpack.sh call the correct executable program for the used architecture and the used (portland) compiler version. These scripts are linked to ncdf_pack and ncdf_unpack in the directory /usr/local/icg/icg1/local/bin (which is included in PATH, if you use the ICG1 group profile).

Pack/Unpack data

Any NetCDF file can be packed or unpacked by the commands

ncdf_pack
ncdf_unpack

All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked.

Besides all files in one directory can be packed or unpacked by calling the scripts

ncdf_pack_dir <directory>
ncdf_unpack_dir <directory>

These scripts are also stored in the directory /usr/local/icg/icg1/local/bin.

The program ncdf_pack packs all REAL variables of the given files with the exception of the following variables (these are only copied):

• coordinate variables (a loss of accuracy is not acceptable)
• variables with the attribute DISABLE_PACKING set to "1"
• variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed)
• PV
• variables with data type INT or DOUBLE

The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".

After packing the file <filename> a new (packed) file with the name <filename>_pk is created. After unpacking the file <filename> a new (unpacked) file with the name <filename>_upk is created. The original data sets are not overwritten !!!

Use packed data in CLaMS

The program isentropic can use packed input data.

Besides it is possible to create packed output files with isentropic if in the configuration file isentropic.inp in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of isentropic.inp.

The packed isentropic files can be used in the other CLaMS packages.

It is possible to use packed UKMO/ECMWF data in:

• isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic)
• add_temp_dot (package tdot)

Packed isentropic data can be used in the following programs:

• traj, traj_add (package traj)
• sedi, pos_sedi (package sedi)
• add_dtp, add_eqlat (package isentropic)
• pos_area, pos_dyn (package pos)

Variable attributes for packed data