|
Size: 4054
Comment:
|
Size: 5150
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 8: | Line 8: |
| With an offset and a scale factor the original real values can be rebuild. | An offset and a scale factor are used for rebuilding the original real values. |
| Line 13: | Line 13: |
| The source code of the package ''packing'' is under the control of the CVS (Concurrent Version System) Utility. It is already checked out in the directory /usr/local/icg/icg1/local/packing. The programs for packing and unpacking data are compiled for 32- and 64-bit architectures. The scripts ncdf_pack.sh and ncdf_unpack.sh call the correct executable program for the used architecture and the used compiler version. These scripts are linked to ncdf_pack and ncdf_unpack in the directory /usr/local/icg/icg1/local/bin (which is included in PATH, if you use the ICG1 group profile). |
The source code of the package ''packing'' is under the control of the CVS (Concurrent Version System) Utility and can be checked out with the command {{{ cvs co -P packing }}} |
| Line 24: | Line 26: |
| All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked. | All files listed in the configuration file ''ncdf_pack.inp''/''ncdf_unpack.inp'' are packed/unpacked. Every line in these configuration files consists of two filenames (input file, output file). The original data sets are not overwritten !!! |
| Line 26: | Line 30: |
Besides all file in one directory can be packed or unpacked by calling the scripts: {{{ ncdf_pack_dir <directory> ncdf_unpack_dir <directory> }}} These scripts are also stored in /usr/local/icg/icg1/local/bin. The program ncdf_pack packs all REAL variables of the given files. The following variables are only copied (NOT packed): |
The program ''ncdf_pack'' packs all REAL variables of the given files with the exception of the following variables (these are only copied): |
| Line 41: | Line 34: |
| * PV | * PV, SH and O3 |
| Line 44: | Line 37: |
| The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED". | The program ''ncdf_unpack'' unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED". |
| Line 46: | Line 39: |
After packing the file <filename> a new (packed) file with the name <filename>_pk is created. After unpacking the file <filename> a new (unpacked) file with the name <filename>_upk is created. The original data set is not overwritten !!! |
There are also two example scripts for packing a large number of input files: ''ncdf_pack.exec1'' and ''ncdf_pack.exec2'' |
| Line 54: | Line 45: |
| Besides it is possible to create packed output files with ''isentropic'' if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can be specified in line 25. | Besides it is possible to create packed output files with ''isentropic'' if in the configuration file ''isentropic.inp'' in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of ''isentropic.inp''. |
| Line 60: | Line 51: |
| It is possible to use packed UKMO/ECMWF-data in: | It is possible to use packed UKMO/ECMWF data in: |
| Line 71: | Line 62: |
| === Variable attributes === | === Variable attributes for packed data === |
| Line 84: | Line 75: |
=== Source Code in package packing === ncdf_pack.f90 pack all files specified in ncdf_pack.inp ncdf_unpack.f90 unpack all files specified in ncdf_pack.inp lib_ncpack.f90 subroutine error_handler subroutine pack_all_variables Read all variables from input netcdf file and write packed to output netcdf file subroutine unpack_all_variables Read all variables from input netcdf file and write unpacked to output netcdf file === Source Code in Library utils === packing.f90 subroutine pack_array pack one-dimensional REAL array to SHORT INT subroutine unpack_array unpack one-dimensional SHORT INT array to REAL nc_pack_utils.f90 function write_packed_attributes write variable attributes used for packing function delete_packed_attributes delete attributes "scale_factor", "add_offset", "UNPACK_missing_value" and "UNPACK_FillValue" subroutine get_packing_accuracy estimate the accuracy of packing function get_packed_var4 a four-dimensional variable is read and unpacked if it was packed function put_packed_var4 a four-dimensional variable is packed and written to netcdf file function nc_pack_variable read variable from input file and write variable packed to output file, if the data type is REAL. Variables with data type INTEGER or DOUBLE are only copied. function nc_unpack_variable read packed variable from input file and write unpacked variable. Variables of data type INTEGER or DOUBLE are only copied. |
The package packing
It is not a lossless compression algorithm which only deletes redundant data so that the original data can be reconstructed exactly.
The used method is a lossy data compression and there is possibly a loss of accuracy.
The most significant bits of a four byte real value are stored on a two byte integer value. An offset and a scale factor are used for rebuilding the original real values.
If all values have the same order of magnitude, 5 to 7 digits of mantissa can be reconstructed exactly. The more orders of magnitude the data varies the greater the loss of accuracy after unpacking will be.
The source code of the package packing is under the control of the CVS (Concurrent Version System) Utility and can be checked out with the command
cvs co -P packing
Pack/Unpack data
Any NetCDF file can be packed or unpacked by the commands
ncdf_pack ncdf_unpack
All files listed in the configuration file ncdf_pack.inp/ncdf_unpack.inp are packed/unpacked. Every line in these configuration files consists of two filenames (input file, output file). The original data sets are not overwritten !!!
The program ncdf_pack packs all REAL variables of the given files with the exception of the following variables (these are only copied):
- coordinate variables (a loss of accuracy is not acceptable)
- variables with the attribute DISABLE_PACKING set to "1"
- variables with the attribute PACKED_STATUS set to "PACKED" (i.e. variable is already packed)
- PV, SH and O3
- variables with data type INT or DOUBLE
The program ncdf_unpack unpacks all packed variables (with attribute PACKED_STATUS set to "PACKED") in the given files. After unpacking the attribute PACKED_STATUS is set to "UNPACKED".
There are also two example scripts for packing a large number of input files: ncdf_pack.exec1 and ncdf_pack.exec2
Use packed data in CLaMS
The program isentropic can use packed input data.
Besides it is possible to create packed output files with isentropic if in the configuration file isentropic.inp in line 25 "y" is specified. All variables (except the coordinate variables) will be packed. If some variables should not be packed, these variable names can also be specified in line 25 of isentropic.inp.
The packed isentropic files can be used in the other CLaMS packages.
It is possible to use packed UKMO/ECMWF data in:
- isentropic, add_dtp_isobar, add_pv_isobar, add_theta, add_thetadot_isobar (package isentropic)
- add_temp_dot (package tdot)
Packed isentropic data can be used in the following programs:
- traj, traj_add (package traj)
- sedi, pos_sedi (package sedi)
- add_dtp, add_eqlat (package isentropic)
- pos_area, pos_dyn (package pos)
Variable attributes for packed data
Name |
Type |
Description |
add_offset |
REAL |
offset for unpacking |
scale_factor |
REAL |
scale factor for unpacking |
missing_value |
INT*2 |
missing value for packed data |
_FillValue |
INT*2 |
FillValue for packed data |
UNPACK_missing_value |
REAL |
missing value for original data |
UNPACK_FillValue |
REAL |
FillValue for original data |
UNPACK_valid_min |
REAL |
minimum valid value |
UNPACK_valid_max |
REAL |
maximum valid value |
PACKED_STATUS |
STRING |
PACKED|UNPACKED|NONE |
DISABLE_PACKING |
INT |
0|1 |
Source Code in package packing
ncdf_pack.f90
- pack all files specified in ncdf_pack.inp
ncdf_unpack.f90
- unpack all files specified in ncdf_pack.inp
lib_ncpack.f90
- subroutine error_handler subroutine pack_all_variables
- Read all variables from input netcdf file and write packed to output netcdf file
- Read all variables from input netcdf file and write unpacked to output netcdf file
Source Code in Library utils
packing.f90
- subroutine pack_array
- pack one-dimensional REAL array to SHORT INT
- unpack one-dimensional SHORT INT array to REAL
nc_pack_utils.f90
- function write_packed_attributes
- write variable attributes used for packing
- delete attributes "scale_factor", "add_offset", "UNPACK_missing_value"
and "UNPACK_FillValue"
- estimate the accuracy of packing
- a four-dimensional variable is read and unpacked if it was packed
- a four-dimensional variable is packed and written to netcdf file
- read variable from input file and write variable packed to output file, if the data type is REAL. Variables with data type INTEGER or DOUBLE are only copied.
- read packed variable from input file and write unpacked variable. Variables of data type INTEGER or DOUBLE are only copied.