Cleanir: removing electronic pattern
The GNAAC detector controller in NIRI and GNIRS sometimes superimposes vertical striping, horizontal banding, and quadrant offsets on the data. The "cleanir" python routine can remove many of these artifacts. Run the script with no arguments to see the full help.
In the default mode cleanir assumes that the pattern noise in a quadrant can be represented by a fixed pattern which is repeated over the entire quadrant. The default size for this pattern is 16 pixels wide and 4 pixels high (which may be changed via the -x and -y flags). The pattern is determined for each quadrant by taking the median of the pixel value distribution at each position in the pattern (which may be visualized with the -g flag). Once the median pattern has been determined for a quadrant it is replicated to cover the entire quadrant and subtracted, and the mean of the pattern is added back to preserve flux. The standard deviation of all the pixels in the quadrant is compared to that before the pattern subtraction, and if no reduction was achieved the subtraction is undone. The pattern subtraction may be forced via the -f flag, which may be necessary if the pattern is low-amplitude. This process is repeated for all four quadrants and the cleaned frame is written to c
Cleanir can also clean each row independently (via the -r flag), which can be useful for removing row structures which are sometimes seen in GNIRS data. In this mode cleanir looks for 8-pixel periodicities in each row, subtracting off the median 8-pixel pattern and adding a constant to bring the median of the row up to the median level of the quadrant. This mode should be used with caution as it may be affected by features which cover a significant fraction of a row, e.g. extended objects, sky lines in GNIRS, or residual image from the GNIRS acquisition keyhole. The -r option is not recommended for GNIRS XD flats.
The GNAAC pattern noise is often accompanied by an offset in the bias values between the four quadrants. One may want to use the cleanir -q flag to try to remove this offset. This attempts to match the iteratively determined median value of each quadrant. This method works best with sky subtraction (i.e. with the -s flag), and does not work well if there are large extended objects in the frame. By default the median is determined from the entire frame, although one can use the -c flag to only use a central portion of the image. Note that the derived quadrant offsets will be applied to the output pattern file.
Cleanir understands glob expansions, however, the entire string must be quoted or any pattern matching characters (*,?) must be escaped with a backslash to avoid being interpreted by the shell.
Options
-a : use all pixels for pattern determination
-b <badpixelmask> : specify a bad pixel mask (overrides DQ plane)
-c <frac> : use central <fraction> of image for bias adjustment [1]
-d <dir> : specify an input data directory
-f : force cleaning of all quads even if stddev does not decrease
-g # : graph results (0=none, 1=pattern, 2=offsets, 3=both)
-h <val> : ignore values above <val> in pattern determination
-m : use median instead of fitting a Gaussian
-o <file> : write output to <file> (instead of c<infile>)
-p <file> : write full-frame pattern to <file>
-q : adjust quadrant offsets
-r : row filtering (useful for GNIRS XD spectra)
-s <sky> : sky frame to help in pattern recognition
-v : verbose debugging output
-x <size> : set pattern x size in pix [16]
-y <size> : set pattern y size in pix [4]
Version
2019-May-07
GNIRS
Cleanir appears to work well on raw GNIRS XD files and sky-subtracted XD and long slit files. It does not work as well on raw long slit files. You may need to try several options to get the best cleaning of your data:
1. cleanir -q filename
2. cleanir -fq filename
3. cleanir -rq filename
The first example should always be attempted first as it is the most robust and least likely to introduce additional artifacts into your data. The second example uses the -f flag to force the pattern subtraction. Example 3 uses the -r flag for row-filtering, and should be the last resort if the first two methods are not successful at removing the noise.
To clean sky-subtracted GNIRS frames a sky-subtracted image should first be produced using gemarith to preserve the MEF structure:
gemarith filename1 - filename2 filename3
cleanir -q filename3
NIRI
Removing the pattern from NIRI spectroscopy can also be difficult because of many vertical sky lines. By default f/6 spectroscopy with the 2-pixel or blue slits (which do not fill the detector), uses the empty regions at the bottom (1-272) and top (720-1024) of the array for measuring the pattern. This is not possible for other modes of spectroscopy where the spectrum fills the detector. For these modes it is best to do sky subtraction before pattern removal. The quickest method is to pass a sky frame (or an offset frame) via the -s flag. The manual method is to generate and subtract the sky, determine and save the pattern via the -p flag, then subtract the pattern from the original image. One may use the -a flag to force using all of the pixels for the pattern determination.
Examples
- Remove pattern noise from an image using the default parameters:
INPUT cleanir.py N20100101S0001.fits
OUTPUT: cN20100101S0001.fits - Remove pattern noise, saving the clean image as "clean.fits" and the pattern as "pattern.fits":
INPUT: cleanir.py -o clean.fits -p pattern.fits N20100101S0001.fits - Remove pattern noise from all quadrants even if the standard deviation does not improve:
INPUT: cleanir.py -f N20100101S0001.fits - Remove pattern noise and adjust the quadrant offsets to a common value:
INPUT: cleanir.py -q N20100101S0001.fits - Remove pattern noise using an offset frame as a sky to help with pattern determination:
INPUT: cleanir.py -s N20100101S0002.fits N20100101S0001.fits - Remove pattern noise using a bad-pixel mask to mask out extended objects:
INPUT: cleanir.py -b N20100101S0001msk.pl N20100101S0001.fits - Remove pattern noise from GNIRS XD spectra using row filtering and fixing quadrant offsets:
INPUT: cleanir.py -rq N20100101S0001 - Clean all frames in the "data" directory:
INPUT: cleanir.py "data/*.fits"
Requirements:
Download
Cleanir is under development. Make sure you use the latest version.
Comments and suggestions are welcome and may be sent to astephens at gemini dot edu.