The package packing

It is not a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly.

The used method is a lossy data compression and there is possibly a loss of accuracy.

The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values.

If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be.

The source code of the package packing is under the control of the CVS (Concurrent Version System) Utility and can be checked out with the command

cvs co -P packing

Pack/Unpack data

Any NetCDF file can be packed or unpacked by the commands

ncdf_pack
ncdf_unpack

All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked. Every line in these configuration files consists of two filenames (input file, output file). The original data sets are not overwritten !!!

The program ncdf_pack packs all REAL variables of the given files with the exception of the following variables (these are only copied):

• coordinate variables (a loss of accuracy is not acceptable)
• variables with the attribute DISABLE_PACKING set to "1"
• variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed)
• PV, SH and O3
• variables with data type INT or DOUBLE

The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".

There are also two example scripts for packing a large number of input files: ncdf_pack.exec1 and ncdf_pack.exec2

Use packed data in CLaMS

The program isentropic can use packed input data.

Besides it is possible to create packed output files with isentropic if in the configuration file isentropic.inp in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of isentropic.inp.

The packed isentropic files can be used in the other CLaMS packages.

It is possible to use packed UKMO/ECMWF data in:

• isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic)
• add_temp_dot (package tdot)

Packed isentropic data can be used in the following programs:

• traj, traj_add (package traj)
• sedi, pos_sedi (package sedi)
• add_dtp, add_eqlat (package isentropic)
• pos_area, pos_dyn (package pos)

Variable attributes for packed data

Source Code in package packing

ncdf_pack.f90

• pack all files specified in ncdf_pack.inp

ncdf_unpack.f90

• unpack all files specified in ncdf_pack.inp

lib_ncpack.f90

• subroutine error_handler subroutine pack_all_variables
  • Read all variables from input netcdf file and write packed to output netcdf file
  subroutine unpack_all_variables
  • Read all variables from input netcdf file and write unpacked to output netcdf file

Source Code in Library utils

packing.f90

• subroutine pack_array
  • pack one-dimensional REAL array to SHORT INT
  subroutine unpack_array
  • unpack one-dimensional SHORT INT array to REAL

nc_pack_utils.f90

• function write_packed_attributes
  • write variable attributes used for packing
  function delete_packed_attributes
  • delete attributes "scale_factor", "add_offset", "UNPACK_missing_value"

    • and "UNPACK_FillValue"

  subroutine get_packing_accuracy
  • estimate the accuracy of packing
  function get_packed_var4
  • a four-dimensional variable is read and unpacked if it was packed
  function put_packed_var4
  • a four-dimensional variable is packed and written to netcdf file
  function nc_pack_variable
  • read variable from input file and write variable packed to output file, if the data type is REAL. Variables with data type INTEGER or DOUBLE are only copied.
  function nc_unpack_variable
  • read packed variable from input file and write unpacked variable. Variables of data type INTEGER or DOUBLE are only copied.