ricepack: Rice compress a FITS file

Package: fitsutil

Usage

ricepack images

Parameters

images

A list of images. The input may be FITS or MEF FITS, or either of these gzip-compressed ('.gz'), or may be IRAF '.imh' images.

keep = no

Preserve the input images? By default the input files will be replaced by the corresponding tile compressed FITS files.

listonly = no

List the types and contents (FITS HDUs) of the input files?

verbose = yes

Print each operation as it takes place? Data volume and timing will also be reported.

quantization = 16

Floating point pixels will be quantized (see discussion below) to the background noise divided by the quantization parameter.

nimages

[Output] The number of images in the input list.

Description

The RICEPACK task will compress a list of images. The input may be FITS or MEF FITS, may be gzip compressed copies of either type of FITS files, or may be IRAF ".imh" images. The output file names will be compressed FITS with ".fits.fz" appended.

This task is a wrapper script for the CFITSIO fpack command (see [1]). RICEPACK implements the FITS tile compression standard[3]. This well established FITS convention has several features that make it preferable to host file compression such as gzip: 1) Rectangular image tiles are separately compressed to preserve rapid random access to the image pixels. 2) The FITS headers remain uncompressed for ease of read/write access. 3) Individual FITS image extensions (HDUs) are compressed separately.

FITS tile compression supports multiple compression algorithms. The IRAF RICEPACK task by default implements Rice compression. Compression of integer-valued images will be lossless. The Rice algorithm realizes a near optimal compression factor[2] and is much faster than alternatives like gzip. Example 2 describes how to override the choice of algorithm using the flags parameter.

The keep parameter offers the option of retaining the original input files. By default the IRAF RICEPACK task replaces the input files with the output compressed files (else the action of compressing the image list would actually consume additional diskspace). This is different than the default behavior of the CFITSIO fpack command.

The listonly and verbose control the amount of information listed. If listonly is set to yes, the input data files will remain untouched and no compressed output will be created. If verbose is set to no, the task will operate silently.

The number of images processed will be reported on output as the value of the nimages parameter.

Floating point handling

The compression of floating point data presents a notorious challenge. This is as true for astronomical data as for any other. Compression ratios are typically small for such data. At the same time, floating point data often retain unwarranted false precision. For example, if the input of a standard CCD processing pipeline is 16-bit integers, then generating 32-bit output is to claim roughly double the precision intrinsic to the raw data.

A widely adopted solution is to rescale the floating point data into an integer range more appropriate to the actual data. The RICEPACK task accomplishes this using the quantization parameter, which represents the number of levels into which the measured one-sigma background will be divided. The default value of 16 (see [2] and included references) has been shown to have a negligible effect (for typical purposes) on derived photometric and astrometric results.

Also, bear in mind that the poisson statistics of most astronomical detectors means that sampling the background noise into 16 such bins may result in oversampling the bright end of the dynamic range by a factor of several hundred.

Examples

1. Tile compress a file using the default Rice algorithm

fitsutil> ricepack file3.fits

The output file is: file3.fits.fz

2. Compress a mixed list of images:

fitsutil> ricepack *.fits,*.imh,*.fits.gz

3. Compress a file and retain the original:

fitsutil> ricepack file4.fits keep+

4. Uncompress gzipped files and recompress using Rice in one step:

fitsutil> ricepack *.gz
1.1.3 (March 2009) CFITSIO version  3.140

Wed 15:31:50 19-Aug-2009
kp1016311.fits.gz -> kp1016311.fits.fz
        ...
kp1016429.fits.gz -> kp1016429.fits.fz
Wed 15:31:58 19-Aug-2009

63 images, 0.13 seconds each, 0:00:08.0 elapsed

 input:      56.550 MB
output:      45.701 MB
 saved:      10.849 MB, 19%

relative R = 1.24

The Rice compressed files save 19% of the space (10.849 MB in this case) required for the gzip files; the relative compression ratio is 1.24 (output/input).

See also

funpack

References

[1] http://heasarc.gsfc.nasa.gov/fitsio/fpack

[2] http://arxiv.org/abs/0903.2140

[3] http://fits.gsfc.nasa.gov/registry/tilecompression.html