Change page style: 

Data Format and Reduction

The basic format of all Gemini data is explained in the data section of the Gemini website. The Example NIFS processing scripts for baseline calibration data, telluric data, and science data are very good starting points for reducing your data. See the notes at the top of each to learn how to update the scripts and run a nearly non-interactive data reduction for publicly available SV data. These scripts can also be accessed in your IRAF session by typing at the cl prompt:

ni> nifsexamples calibration

ni> nifsexamples telluric

ni> nifsexamples science

Pyraf-compatible adaptations of these scripts are available here: NIFS_Basecalib.py, NIFS_Telluric.py, NIFS_Science.py, which can be used with the following tutorial data set. This data set and an associated presentation (ppt|pdf) were used at a Gemini Data Reduction Workshop held in Tucson, July 2010.

 


Detailed information on NIFS data and reduction is displayed below. This information is the output of nifs.nifsinfo in the NIFS IRAF package and can be accessed from within your IRAF session by typing at the cl prompt:

ni> nifsinfo

NAME
nifsinfo -- Generic information on NIFS data and reduction

USAGE
nifsinfo

DESCRIPTION

The NIFS package contains tasks for processing NIFS integral-field
(IFU) spectroscopy data. The specifics of the individual tasks can
be found in their help files. This document describes the common
features of the generic tasks and gives a description of the NIFS
data format. NIFSEXAMPLES provides sample "recipes" (processing
scripts).

NIFS REDUCTION

Because some of the routines for proper reduction of infrared
spectral data already exist within the Gemini IRAF package,
reduction of NIFS data uses some tasks that are inherent in the
Gemini IRAF package for GNIRS data reduction. These GNIRS scripts
have the prefix "ns". Tasks that are NIFS-specific and will work
only with NIFS IFU data have the prefix "nf".

All of the tasks used for NIFS reduction produce logfiles of the
performed processing steps. The name of the logfile may be set in
each individual task, or at the package level by setting logfile.

The tasks add header keywords to the output images. These header
keywords contain information about the performed processing steps
and the values of the critical parameters that were used.

All NIFS images are written as multi-extension FITS (MEF) files.
Raw data have two unnamed extensions. Most of the header
information is written to the primary header unit (PHU) which is
extension [0]. The data read from the array is in extension [1].

DATA FORMAT AND PREPARATION

Processing with NFPREPARE (this task is used only for NIFS data -
see below - but other instruments have their own preparation tasks
with similar actions) will rename the data extension and add
variance and data quality extensions. By default (see NSHEADERS)
the extension names are SCI for science data, VAR for variance, and
DQ for data quality (0 = good). Generation of the data quality
plane (DQ) is important in order to fix hot and dark pixels on the
NIFS detector in subsequent steps in the data reduction process.
Various header keywords (used later) are also added in NFPREPARE.

NFPREPARE will also add an MDF file (extension MDF) describing the
NIFS image slicer pattern and how the IFU maps to the sky field.

MEF STRUCTURE OF REDUCED DATA

NIFS data reduction will cut the image slices and extract the
regions of interest from the raw data. In the case of NIFS IFU
frames, data that has been run through the first few reduction
steps will have many extensions - one extension for the SCI, VAR
and DQ planes for each of the 29 different slices in the IFU. The
second extension containing science data, for example, is denoted
by [SCI,2]. So, NIFS data reduced with variance and data quality
planes will have 87 different extensions after being NSCUT - 29
SCI, 29 VAR and 29 DQ.

The task GEMEXTN, when used with default parameters and no
arguments, will display all the extensions in a file:

> unlearn gemextn
> gemextn snN20060202S0015
snN20060202S0015[0]
snN20060202S0015[1][MDF]
snN20060202S0015[2][SCI,1]
snN20060202S0015[3][VAR,1]
snN20060202S0015[4][DQ,1]
snN20060202S0015[5][SCI,2]
...
snN20060202S0015[19][DQ,29]

This includes both the index (inside the first []) and the extension
name and version (second []). It is only necessary to use one of
these, since either uniquely identifies an extension. Name and
version are recommended because they specify the type of data
contained. There is no guarantee about the ordering of indices
apart from extension [0], the PHU, which always contains only header
information.

Additional information and ways to handle MEF files can be found in
http://www.gemini.edu/sciops/data/dataFormat.html

HEADER PARAMETERS AND INSTRUMENT SPECIFIC OPTIONS

The task NSHEADERS lists the header parameters used by the various
tasks in the NIFS package (some header values which have values
fixed by IRAF or FITS conventions are not included since they
cannot be changed).

Before reducing data from NIFS, the user should always run
> nsheaders nifs

Note that not all header parameters named in NSHEADERS need appear

in the raw data. Some are used internally by the NIFS package.
Others are used by NFPREPARE alone.

SUMMARY OF THE NIFS DATA REDUCTION TASKS

NSHEADERS - Prepare the session for NIFS data reduction (task
resides in the GNIRS package)

This task is describedin more detail above. It should be
called before all other scripts to configure the package for
processing data from NIFS.

NFACQUIRE - Construct quick-look NIFS images

Derive quick-look images of the NIFS IFU field from flip-mirror
acquisition data. This task is primarily used for acquisition
of faint targets at the telescope. The option exists to derive
offsets to center a target within the IFU field.

NFIMAGE - Construct quick-look NIFS images from dispersed data.
Derive quick-look images of the NIFS IFU field from dispersed
science exposures. This task can be used for acquisition of
bright targets at the telescope, but has also proved to be very
useful for viewing the IFU field sky region from any dispersed
data. The option exists to derive offsets to center a target
within the IFU field.

NFPREPARE - Prepare NIFS data for reduction

This task is described in more detail above. It must be called
before the other tasks listed below.

NSFLAT - Produce a normalized spectral flatfield (task resides in the
GNIRS package)

NSFLAT is used to produce a normalized spectralflatfield
image. For IFU data it is usually used after calling NSREDUCE
or NSCUT. To preserve a consistent normalization between
slices, IFU flats must also be processed subsequently with
NSSLITFUNCTION.

NSREDUCE - Process NearIR Spectral data (task resides in the GNIRS
package)

NSREDUCE is used for basic reduction of raw data - it provides a
single, unified interface to several tasks and also allows for
the subtraction of dark frames and dividing by the flat. For
NIFS reduction, NSREDUCE is used to call the NSCUT and NSAPPWAVE
routines.

Note that NSREDUCE will not call NFPREPARE.

NSSLITFUNCTION - Correct the flat field for inter-slice variations
(task resides in the GNIRS package)

NSSLITFUNCTION extends the flatfield produced by correcting the
NSFLAT normalization for inter-slice variations. The output
from this task is used as the flatfield image forfurther
reduction.

NSAPPWAVE - Determines a simple wavelength calibration (task
resides in the GNIRS package)

NSAPPWAVE provides a quick, approximate wavelength solution for
NIFS spectroscopic data. The wavelength solution is based on
the header information and (by default) information in the table
gnirs$data/nsappwave.fits (use the TTOOLS package to examine
this - TPRINT will display the contents - but if it is
necessary to add/delete entries or create a new table, this can
be done with the dedicated NSWEDIT task).

The task is for quick reduction purposes and to provide a
starting point for the wavelength calibration derived by
NSWAVELENGTH.

NSWAVELENGTH - Establish an accurate wavelength calibration (task
resides in the GNIRS package)

NSWAVELENGTH (automatically, if possible) determines the
wavelength solution for NIFS calibration lamp images, starting
with the initial value supplied by NSAPPWAVE (this is done
using the underlying IRAF tasks autoidentify and reidentify).

This task does not calibrate the data directly, but provides a
database entry that can be used by NSTRANSFORM (described
below).

NFSDIST - Establish a spatial calibration

NFSDIST uses the information in the "Ronchi" Calibration images
to calibrate the spatial dimension of the NIFS IFU field. The
Ronchi frame is a dispersed flat field image with a slit-mask
in the field so that the illumination on the IFU is in a
pattern of ~10 different slitlets that are stackedin the
y-dimension on the field. Proper alignment of the slits across
the image slicer pattern can be used for spatial rectification
of the on-sky science data. The spatial solution determined by
NFSDIST is linked to the science data in NFFITCOORDS.

NFFIXBAD - Fix Hot/Cold pixels on the NIFS detector

This routine uses the information in the Data Quality
extensions to fix hot and cold pixels in the NIFS science
fields. NFFIXBAD is a wrapper script which callsthe task
FIXPIX, using the DQ plane to define the pixels to be corrected.

NFFITCOORDS - Compute 2D dispersion and distortion maps

This routine uses as inputs the output from the NSWAVELENGTH
and NFSDIST routines. NFFITCOORDS takes the spatial and
spectral rectification information from NSWAVELENGTH and
NFSDIST and converts this into a calculation of where the data
information should map to in a final IFU dataset.

NSTRANSFORM - Spatially rectify and wavelength calibrate data

NFTRANSFORM applies the wavelength solution found by
NSWAVELENGTH and the spatial correction found by NFSDIST,
aligning all the IFU extensions consistently onto a common
coordinate system. The output of this routine is still in 2D
format, with each of the IFU slices represented by its own data
extension.

NIFCUBE - Construct 3D NIFS datacubes

NIFCUBE takes input from data output by either NFFITCOORDS or
NFTRANSFORM and converts the2D data images into data cubes
that have coordinates of x, y, lambda.

NFEXTRACT - Extract NIFS spectra

This could be used to extract a 1D spectra from IFU data and is
particularly useful for extracting the bright spectra of
telluric calibrator stars. Note that this routine only works
on data that has been run through NFTRANSFORM.

NFTELLURIC

NFTELLURIC uses input science and a 1D spectrum of a telluric
calibrator to correct atmospheric absorption features.

TYPICAL REDUCTION

For typical reductions the user will need appropriate flat fields,
arc calibration images, darks, Ronchi calibration frames and
science images.

NOTE - USE the NIFSEXAMPLES option at the IRAF prompt to obtain
example scripts for calibration, telluric and science reductions.

Use GEMTOOLS.GEMLIST to make separate lists of the files associated

with different exposure types. Tasks can then be invoked on all
the data of one type using iraf's "@" syntax. For example,
NFPREPARE might be called with inimages=@flatlist and then NSREDUCE
with inimages=n@flatlist (which prepends the letter "n" -
nsprepare.outprefix - to each filename).

MDF SHIFT DETERMINATION:

Step 0. Run NFPREPARE on a single "lamps on" flat to determine the
shift between your IFU data and the definition of the Image Slicer
position in the MDF file. The output from this step will be used
in all subsequence calls to NFPREPARE as the "shiftimage".

FLAT FIELD REDUCTION:

Step 1. Use NFPREPARE on the lamps on/lamps off flats to update the
raw data headers and attach the mask definition file (MDF) as a
binary table on all files. Note that dark frames will not have an
MDF attached by default. Instead, the appropriate MDF is added in
NSREDUCE or NSFLAT to match the data being reduced.

Step 2. Use NSREDUCE to cut the calibration (flat/arc) spectra to
the size specified by the MDF, placing different IFU slices in
separate image extensions.

Step 3. Use NSFLAT to generate a normalized flat field (for each
IFU slice or cross-dispersed order) from lamp flats. A mask (BPM)
will also be generated by thresholding - this can be used to flag
bad pixels in other data.

Step 4. Use NSSLITFUNCTION to produce the final flat.

WAVELENGTH CALIBRATION:

Step 5. Repeat steps 1 & 2 for the arcs and arc darks (use the
appropriate flat field and arc darks in the call to NSREDUCE).

Step 6. Use NSWAVELENGTH to calibrate arc data (after cutting and
optionally applying a flatfield with NSREDUCE).

SPATIAL RECTIFICATION:

Step 7. Repeat steps 1 & 2 for the Ronchi calibration frames. Use
the dark frame generated by the flat fielding steps in NSREDUCE.

STEP 8. Measure spatial mapping for the image slices with NFSDIST,
using the output from step 7 after cutting with NSCUT in NSREDUCE
(this step is required for accurate alignment of the slices).

SCIENCE DATA REDUCTION:

Step 9. If necessary, Repeat Steps 1 & 2 for the Dark frames
observed at the science exposure time.
Step 10. Repeat Step 1 for the science data. Make sure to use
NSREDUCE not only to process the object data to cut the frames from
NSCUT, but also to apply the approximate wavelength calibration
from NSAPPWAVE. Use the darks generated in Step 9 as input into
NSREDUCE.

Step 11. Run the science data through NFFIXBAD to clean bad pixels
- note that this step can be time consuming.

Step 12. Use the output from the calibration Step 6 (NSWAVELENGTH)
and Step 8 (NFSDIST) as input into the NFFITCOORDS routine to
generate the mapping for the IFU datacube.

Step 13. Use NFTRANSFORM to resample the combined object data to a
uniform wavelength scale, with consistent spatial alignment
between different slices and wavelengths.

Step 14. TELLURIC CALIBRATOR: Repeat Steps 10 through 13 on the
data for the telluric calibration star.

Step 15. TELLURIC CALIBRATOR: Extract 1-D spectra from the
telluric calibrator using the NFEXTRACT routine. If necessary,
merge several extracted 1-D spectra using the GEMCOMBINE routine in
the Gemini IRAF package (with statsec="[*]" for 1D data).

Step 16. Use the NIFS NFTRANSFORMED data and the 1D telluric
calibrator Spectra in the NFTELLURIC routine to correct out
telluric absorption features.

Step 17. Use NIFCUBE to construct a datacube from the final science data.

FINAL PRODUCT = Telluric corrected 3D science data cubes with
proper wavelength and spatial calibration.

CURRENT LIMITATIONS OF THE REDUCTION:

In the NIFS data reduction package, there is currently no way to:
-Flux calibrate the NIFS datacubes.
-Correct for instrument flexure in multiple exposures.
-Correct out photospheric absorption features in telluric
standards.
-Correct for heliocentric radial velocity variations.
-Combine dithered datasets into a single, larger cube.
-Rectify data for atmospheric refraction effects.

REDUCTION EXAMPLES

Example reduction scripts are available, see NIFSEXAMPLES.

ABBREVIATIONS USED IN THE NIFS HELPFILES

Abbreviation Explanations
---------------------------------------------------------------
NIFS Near-IR Integral Field Spectrograph
IFU Integral field unit
MDF Mask defintion file (a binary FITS table)
MEF Multi-extension FITS
PHU Primary header unit (extension [0] of a MEF file)
---------------------------------------------------------------

BUGS AND LIMITATIONS

The tasks in the NIFS package are designed to operate on MEF FITS
images that have been processed using NFPREPARE (for NIFS data)

SEE ALSO
nifsexamples, nsflat, nfprepare, nsreduce, nsappwave, nswavelength,
nffitcoords, nftransform, nfextract, nffixbad nfsdist, nifcube,
gemlist, gemcombine, gemarith.