The package packing

It is not a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly.

The used method is a lossy data compression and there is possibly a loss of accuracy.

The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values.

If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be.

The source code of the package packing is under the control of the CVS (Concurrent Version System) Utility and can be checked out with the command

cvs co -P packing

Pack/Unpack data

Any NetCDF file can be packed or unpacked by the commands

ncdf_pack
ncdf_unpack

All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked. Every line in these configuration files consists of two filenames (input file, output file). The original data sets are not overwritten !!!

The program ncdf_pack packs all REAL variables of the given files with the exception of the following variables (these are only copied):

• coordinate variables (a loss of accuracy is not acceptable)
• variables with the attribute DISABLE_PACKING set to "1"
• variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed)
• PV, SH and O3
• variables with data type INT or DOUBLE

The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".

There are also two example scripts for packing a large number of input files: ncdf_pack.exec1 and ncdf_pack.exec2

Use packed data in CLaMS

The program isentropic can use packed input data.

Besides it is possible to create packed output files with isentropic if in the configuration file isentropic.inp in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of isentropic.inp.

The packed isentropic files can be used in the other CLaMS packages.

It is possible to use packed UKMO/ECMWF data in:

• isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic)
• add_temp_dot (package tdot)

Packed isentropic data can be used in the following programs:

• traj, traj_add (package traj)
• sedi, pos_sedi (package sedi)
• add_dtp, add_eqlat (package isentropic)
• pos_area, pos_dyn (package pos)

Create packed isentropic files

The program isentropic creates output files with packed data, if in the configuration file isentropic.inp in line 25 "y" is specified.

All variables are packed with the exception of the coordinate variables (time,lat,lon,theta/zeta) and the variables listed in line 25 of the configuration file.

#
#    Inputdatei fuer isentropic
#
.
.
.
#
# Variablen gepackt ausgeben (y/n), Liste mit nicht zu packenden Variablen
#
y    PV, SH
#
.
.
.

The programn t add_eqlat adds EQLAT to isentropic files. EQLAT is packed, if this is specified in line 3 of the configuration file add_eqlat.inp:

isen_U3S_DATUM12.nc                        ! netCDF-file, DATUM=yymmdd
/dat_icg1/rad/test                         ! its directory
y                                          ! write EQLAT packed (y/n)

The parameters added to isentropic files with the program theta2zeta can be packed, if this is specified in the configuration file theta2zeta.inp.

/private/icg112/theta2zeta_bsp      ! directory with zeta-files
ECZ                                 ! prefix for zeta-files
/private/icg112/theta2zeta_bsp      ! directory with theta-files
ECT                                 ! prefix for theta-files
12 01 09 2003                       ! start date (hh dd mm yyyy)
12 03 09 2003                       ! end date (hh dd mm yyyy)
24                                  ! time increment (in hours)
icg112                              ! username
2                                   ! number of parameters
PV         n                        ! list of parameters (name, write packed (y/n))
EQLAT      y

Variable attributes for packed data

Source Code

Source Code in package packing

ncdf_pack.f90

• Program ncdf_pack packs all files specified in ncdf_pack.inp

ncdf_unpack.f90

• Program ncdf_unpack unpacks all files specified in ncdf_pack.inp

lib_ncpack.f90

• Module lib_ncpack contains the following routines:
• subroutine error_handler
• subroutine pack_all_variables
  • Read all variables from input netcdf file and write packed to output netcdf file
• subroutine unpack_all_variables
  • Read all variables from input netcdf file and write unpacked to output netcdf file

Source Code in Library utils

packing.f90

• subroutine pack_array
  • pack one-dimensional REAL array to SHORT INT
• subroutine unpack_array
  • unpack one-dimensional SHORT INT array to REAL

nc_pack_utils.f90

• function write_packed_attributes
  • write variable attributes used for packing
• function delete_packed_attributes
  • delete attributes "scale_factor", "add_offset", "UNPACK_missing_value"

    • and "UNPACK_FillValue"

• subroutine get_packing_accuracy
  • estimate the accuracy of packing
• function get_packed_var4
  • a four-dimensional variable is read and unpacked if it was packed
• function put_packed_var4
  • a four-dimensional variable is packed and written to netcdf file
• function nc_pack_variable
  • read variable from input file and write variable packed to output file, if the data type is REAL. Variables with data type INTEGER or DOUBLE are only copied.
• function nc_unpack_variable
  • read packed variable from input file and write unpacked variable. Variables of data type INTEGER or DOUBLE are only copied.