#acl ClamsUserGroup:read,write,delete,revert All:read = The package packing = It is '''not''' a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly. The used method is a '''lossy data compression''' and there is possibly a loss of accuracy. The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values. If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be. The source code of the package ''packing'' is under the control of the CVS (Concurrent Version System) Utility and can be checked out with the command {{{ cvs co -P packing }}} == Pack/Unpack data == Any NetCDF file can be packed or unpacked by the commands {{{ ncdf_pack ncdf_unpack }}} All files listed in the configuration file ''ncdf_pack.inp''/''ncdf_unpack.inp'' are packed/unpacked. Every line in these configuration files consists of two filenames (input file, output file). The original data sets are not overwritten !!! The program ''ncdf_pack'' packs all REAL variables of the given files with the exception of the following variables (these are only copied): * coordinate variables (a loss of accuracy is not acceptable) * variables with the attribute DISABLE_PACKING set to "1" * variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed) * PV, SH and O3 * variables with data type INT or DOUBLE The program ''ncdf_unpack'' unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED". There are also two example scripts for packing a large number of input files: ''ncdf_pack.exec1'' and ''ncdf_pack.exec2'' == Use packed data in CLaMS == The program ''isentropic'' can use packed input data. Besides it is possible to create packed output files with ''isentropic'' if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of ''isentropic.inp''. The packed isentropic files can be used in the other CLaMS packages. It is possible to use packed UKMO/ECMWF data in: * isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic) * add_temp_dot (package tdot) Packed isentropic data can be used in the following programs: * traj, traj_add (package traj) * sedi, pos_sedi (package sedi) * add_dtp, add_eqlat (package isentropic) * pos_area, pos_dyn (package pos) == Create packed isentropic files == The program ''isentropic'' creates output files with packed data, if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables are packed with the exception of the coordinate variables (time,lat,lon,theta/zeta) and the variables listed in line 25 of the configuration file: {{{ # # Inputdatei fuer isentropic # . . . # # Variablen gepackt ausgeben (y/n), Liste mit nicht zu packenden Variablen # y PV, SH . . . }}} The program ''add_eqlat ''adds EQLAT to isentropic files. EQLAT is packed, if this is specified in line 3 of the configuration file ''add_eqlat.inp'': {{{ isen_U3S_DATUM12.nc ! netCDF-file, DATUM=yymmdd /dat_icg1/rad/test ! its directory y ! write EQLAT packed (y/n) }}} The parameters added to isentropic files with the program ''theta2zeta ''can be packed, if this is specified in the configuration file ''theta2zeta.inp'': {{{ /private/icg112/theta2zeta_bsp ! directory with zeta-files ECZ ! prefix for zeta-files /private/icg112/theta2zeta_bsp ! directory with theta-files ECT ! prefix for theta-files 12 01 09 2003 ! start date (hh dd mm yyyy) 12 03 09 2003 ! end date (hh dd mm yyyy) 24 ! time increment (in hours) icg112 ! username 2 ! number of parameters PV n ! list of parameters (name, write packed (y/n)) EQLAT y }}} == Variable attributes for packed data == ||'''Name''' ||'''Type''' ||'''Description''' || ||add_offset ||REAL ||offset for unpacking || ||scale_factor ||REAL ||scale factor for unpacking || ||missing_value ||INT*2 ||missing value for packed data || ||_!FillValue ||INT*2 ||!FillValue for packed data || ||UNPACK_missing_value ||REAL ||missing value for original data || ||UNPACK_!FillValue ||REAL ||!FillValue for original data || ||UNPACK_valid_min ||REAL ||minimum valid value || ||UNPACK_valid_max ||REAL ||maximum valid value || ||PACKED_STATUS ||STRING ||PACKED|UNPACKED|NONE || ||DISABLE_PACKING ||INT ||0|1 || == Source Code == === Source Code in package packing === ncdf_pack.f90 . Program ncdf_pack packs all files specified in ncdf_pack.inp ncdf_unpack.f90 . Program ncdf_unpack unpacks all files specified in ncdf_pack.inp lib_ncpack.f90 . Module lib_ncpack contains the following routines: * subroutine error_handler * subroutine pack_all_variables . Read all variables from input netcdf file and write packed to output netcdf file * subroutine unpack_all_variables . Read all variables from input netcdf file and write unpacked to output netcdf file === Source Code in Library utils === packing.f90 * subroutine pack_array . pack one-dimensional REAL array to SHORT INT * subroutine unpack_array . unpack one-dimensional SHORT INT array to REAL nc_pack_utils.f90 * function write_packed_attributes . write variable attributes used for packing * function delete_packed_attributes . delete attributes "scale_factor", "add_offset", "UNPACK_missing_value" . and "UNPACK_FillValue" * subroutine get_packing_accuracy . estimate the accuracy of packing * function get_packed_var4 . a four-dimensional variable is read and unpacked if it was packed * function put_packed_var4 . a four-dimensional variable is packed and written to netcdf file * function nc_pack_variable . read variable from input file and write variable packed to output file, if the data type is REAL. Variables with data type INTEGER or DOUBLE are only copied. * function nc_unpack_variable . read packed variable from input file and write unpacked variable. Variables of data type INTEGER or DOUBLE are only copied.