Differences between revisions 1 and 2
Revision 1 as of 2008-02-05 11:50:19
Size: 4054
Editor: NicoleThomas
Comment:
Revision 2 as of 2008-02-05 12:27:49
Size: 4172
Editor: NicoleThomas
Comment:
Deletions are marked like this. Additions are marked like this.
Line 8: Line 8:
With an offset and a scale factor the original real values can be rebuild. An offset and a scale factor are used for rebuilding the original real values.
Line 14: Line 14:
The programs for packing and unpacking data are compiled for 32- and 64-bit architectures. The scripts ncdf_pack.sh and ncdf_unpack.sh call the correct executable program for the used architecture and the used compiler version. These scripts are linked to ncdf_pack and ncdf_unpack in the directory /usr/local/icg/icg1/local/bin (which is included in PATH, if you use the ICG1 group profile). The programs for packing and unpacking data are compiled for 32- and 64-bit architectures. The scripts ''ncdf_pack.sh'' and ''ncdf_unpack.sh'' call the correct executable program for the used architecture and the used (portland) compiler version. These scripts are linked to ''ncdf_pack'' and ''ncdf_unpack'' in the directory /usr/local/icg/icg1/local/bin (which is included in PATH, if you use the ICG1 group profile).
Line 24: Line 24:
All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked. All files listed in the configuration file ''ncdf_pack.inp''/''ncdf_unpack.inp'' are packed/unpacked.
Line 26: Line 26:

Besides all file in one directory can be packed or unpacked by calling the scripts:
Besides all files in one directory can be packed or unpacked by calling the scripts
Line 33: Line 32:
These scripts are also stored in /usr/local/icg/icg1/local/bin. These scripts are also stored in the directory /usr/local/icg/icg1/local/bin.
Line 36: Line 35:
The program ncdf_pack packs all REAL variables of the given files.
T
he following variables are only copied (NOT packed):
The program ''ncdf_pack'' packs all REAL variables of the given files with the exception of the following variables (these are only copied):
Line 44: Line 42:
The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED". The program ''ncdf_unpack'' unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".
Line 47: Line 45:
After packing the file <filename> a new (packed) file with the name <filename>_pk is created. After unpacking the file <filename> a new (unpacked) file with the name <filename>_upk is created. The original data set is not overwritten !!! After packing the file <filename> a new (packed) file with the name <filename>_pk is created. After unpacking the file <filename> a new (unpacked) file with the name <filename>_upk is created. The original data sets are not overwritten !!!
Line 54: Line 52:
Besides it is possible to create packed output files with ''isentropic'' if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can be specified in line 25. Besides it is possible to create packed output files with ''isentropic'' if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of ''isentropic.inp''.
Line 60: Line 58:
It is possible to use packed UKMO/ECMWF-data in: It is possible to use packed UKMO/ECMWF data in:
Line 71: Line 69:
=== Variable attributes === === Variable attributes for packed data ===

The package packing

It is not a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly.

The used method is a lossy data compression and there is possibly a loss of accuracy.

The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values.

If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be.

The source code of the package packing is under the control of the CVS (Concurrent Version System) Utility. It is already checked out in the directory /usr/local/icg/icg1/local/packing. The programs for packing and unpacking data are compiled for 32- and 64-bit architectures. The scripts ncdf_pack.sh and ncdf_unpack.sh call the correct executable program for the used architecture and the used (portland) compiler version. These scripts are linked to ncdf_pack and ncdf_unpack in the directory /usr/local/icg/icg1/local/bin (which is included in PATH, if you use the ICG1 group profile).

Pack/Unpack data

Any NetCDF file can be packed or unpacked by the commands

ncdf_pack
ncdf_unpack

All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked.

Besides all files in one directory can be packed or unpacked by calling the scripts

ncdf_pack_dir <directory>
ncdf_unpack_dir <directory>

These scripts are also stored in the directory /usr/local/icg/icg1/local/bin.

The program ncdf_pack packs all REAL variables of the given files with the exception of the following variables (these are only copied):

  • coordinate variables (a loss of accuracy is not acceptable)
  • variables with the attribute DISABLE_PACKING set to "1"
  • variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed)
  • PV
  • variables with data type INT or DOUBLE

The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".

After packing the file <filename> a new (packed) file with the name <filename>_pk is created. After unpacking the file <filename> a new (unpacked) file with the name <filename>_upk is created. The original data sets are not overwritten !!!

Use packed data in CLaMS

The program isentropic can use packed input data.

Besides it is possible to create packed output files with isentropic if in the configuration file isentropic.inp in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of isentropic.inp.

The packed isentropic files can be used in the other CLaMS packages.

It is possible to use packed UKMO/ECMWF data in:

  • isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic)
  • add_temp_dot (package tdot)

Packed isentropic data can be used in the following programs:

  • traj, traj_add (package traj)
  • sedi, pos_sedi (package sedi)
  • add_dtp, add_eqlat (package isentropic)
  • pos_area, pos_dyn (package pos)

Variable attributes for packed data

Name

Type

Description

add_offset

REAL

offset for unpacking

scale_factor

REAL

scale factor for unpacking

missing_value

INT*2

missing value for packed data

_FillValue

INT*2

FillValue for packed data

UNPACK_missing_value

REAL

missing value for original data

UNPACK_FillValue

REAL

FillValue for original data

UNPACK_valid_min

REAL

minimum valid value

UNPACK_valid_max

REAL

maximum valid value

PACKED_STATUS

STRING

PACKED|UNPACKED|NONE

DISABLE_PACKING

INT

0|1

packing (last edited 2010-04-30 09:50:43 by NicoleThomas)