Online Pipeline History Log: IRAC Pipeline Items

S17.0 (Release Date: Jan 08)

Artifacts mitigation within the pipeline

Artifact-mitigated images from the BCD pipeline and their associated uncertainty images (*cbcd.fits and *cbunc.fits) are now available in the archive. These images include corrections for column pulldown and banding, induced by bright sources in the images. The corrections are empirical fits to the BCDs and may not always improve the data quality. The standard BCD files (*bcd.fits) remain available in the archive. The mosaics (post-BCD products) are now created from the *cbcd.fits images.

Muxbleed correction updated again

The muxbleed correction has been revised to include a better empirical fit.

Two-dimensional subarray images

A two-dimensional image is now generated for each subarray BCD cube. Each pixel in the 2D image (*sub2d.fits) is a robust (outlier-rejected) mean of the 64 samples of the bcd.fits cube. Two-dimensional masks, uncertainty images, and coverage maps are now also provided.

Artifacts now flagged for subarray images

The subarray imasks (*_bimsk.fits) now include masking for muxbleed, column pulldown and banding, induced by bright sources. The updated masks can be used to mitigate bright source artifacts the same way as with the full array data.

Darkdrift values written out in the subarray header

The pipeline darkdrift module reduces a "jailbar" bias effect in the IRAC images. The values used for the reduction within the pipeline are included in the header of the full array data, and now in the header of the subarray data for all the planes. This allows the user to remove the correction, if desired.

Mosaic header updated

We have added more information to the mosaic header. These changes will be explained in the upcoming revision of the IRAC Data Handbook.

S16.0 (Release Date: Apr 07)

Skydark change for 100 second HDR data

Labdark change for HDR data

Within 100s AORs, the channel 4 observations are split into two 50s frames. The second 50s frame received an incorrect non-HDR 50s labdark instead of an HDR labdark. This was a minor problem, and has now been corrected.

Muxbleed correction updated

The module that detects and corrects the muxbleed caused by bright sources has been updated. It now performs a more consistent and better correction than previously.

Artifacts flagged within the pipeline

The imasks (*_bimsk.fits) now include masking for muxbleed, column pulldown and banding induced by bright sources on images. The updated masks can be used with existing contributed software to mitigate bright source artifacts, and will be used in future versions of the SSC pipeline after mitigation algorithms have been implemented. In general, observers should not flag artifacts in mosaicking unless they have observations at various roll angles.

Pixel linearization

The handling of bad and saturated pixels has been changed - they are in most cases now left with their original values, as opposed to being set equal to NaN. The method of flagging saturation in BCD mask files was changed, and now more accurately reflects the presence of saturation.

S15.0 (Release Date: Dec 06)

Ghost images and scattered light flagged within pipeline

Pipeline versions of the ghost image and scattered light detection algorithms created by IRAC IST have been integrated into the IRAC pipeline. The modules use the location of bright sources upon the array (ghost image) or just outside the array, as found in 2MASS catalogs (scattered light) to predict possible optical ghosts and scattered light locations, and flags these pixels within the imask. The imask is an ancillary data product now available through the archive. IRAC observers should use the imask instead of the dmask when making mosaics etc.

Incorrect group ids in header to be fixed

A bug that caused a small percentage of BCDs (<0.1%) to have an unreadable header and therefore not to get pipeline-processed, has been fixed. This should significantly decrease the number of missed BCDs in large mapping programs.

S14.0 (Release Date: May 06)

Darkdrift module changes

As mentioned below, in S13 the darkdrift module was applied only to channel 3 data. This module is used to adjust the bias level in the four readouts in an array, thereby removing vertical striping in the data, the so-called "jailbar effect." After S13 reprocessing of IRAC data it was found that the jailbar effect can be triggered in channels 1, 2 and 4 as well. Therefore, the darkdrift module will again be applied to all four channels, and all the IRAC data will be reprocessed with pipeline version S14.

We have released a "jailbar corrector", which may be used to correct for the jailbar effect. It produces similar results to the darkdrift corrector module in the IRAC pipeline.

Added Keywords

1. Individual array readout noise (RONOISE)

S13.0 (Release Date: Nov 05)

'Super-boresight' pointing refinement (S13.2 and thereafter)

Previous versions of the pipeline performed pointing refinement on each IRAC channel separately. The refinement was performed by matching detected point sources to 2MASS stars and registering the astrometry to minimize the positional offset between matches. In most cases, the refinement of channels 3 and 4 is less accurate as the number of stars detected in an individual frame is less than in channels 1 and 2. 'Super-boresight' refinement corrects the astrometry for all four channels simultaneously using appropriately weighted matches from all four channels and the known orientations of the FPAs. This method can dramatically improve the pointing accuracy for channels 3+4, and it removes any positional offsets between channels. The superboresight pointing is inserted into the CRVAL1 and CRVAL2 keywords in the header, while the basic (less accurate) pointing refinement remains in RARFND and DECRFND header keywords, and the original boresight pointing solution is placed in new header keywords ORIGRA and ORIGDEC.

First-Frame effect

The interval between frames (INTRFDLY) is now maintained in a database, instead of the pipeline reading the previous image in an AOR to process the current image. This streamlines operations and handling of missing images. It is also placed in the header as a keyword.

Linearity Correction

New linearity corrections have been calculated from on-orbit tests and small changes will be made to channel 3 full array and all channel subarray data. The effect is roughly 2% at half-well, and 8% at 90% full-well in channel 3. The other channels are within specifications and the linearity corrections will not be changed for them.

Darkdrift module changes

Small drifts in the bias level of each of the four readouts in each array, particularly relative to the calibration labdarks, can produce a vertical striping called the "jailbar" effect. This is corrected in the pipeline software by applying a constant offset per readout channel (arranged in columns), derived from the median of those columns such that their arithmetic mean is zero. In other words, all readout channels are adjusted to a common additive bias level. In on-orbit tests, the mean offset and correction was found to be negligble, except in channel 3 data. Therefore, in S13 reprocessing, the darkdrift correction was only applied to channel 3 data. The derived correction values for each channel are located in the header in the keywords DRICORR1, DRICORR2, DRICORR3, and DRICORR4. The overall background term determined is DRIBKGND.

Distortion files

The subarray distortion files were found to be derived from the incorrect place on the full array and have now been updated with correct ones. This should only make a small, but noticeable difference in pixel sizes when measuring relative separations in the subarray.

Superflat

A new "super-skyflat" has been derived from the first two years of flatfield data on IRAC and will be used as the flatfield for all reprocessing and further campaigns. Uncertainties in the pixel-to-pixel responsivity calibration are only 0.5%, 0.2%, 0.2%, and 0.05% for channels 1-4, respectively.

Flux conversion

The flux conversion has been updated to reflect the derivation described in the IRAC calibration paper. The currently used numbers were from a nearly complete phase of this derivation, but different by 3% in ch 4.

Other Added Keywords

1. Median brightness of Calibration Skydark (SKYDKMED)
2. More Precise Start time of observation (SCLK_OBS)

S12.0 (Release Date: Apr 05)

Since S11.0 there has no been significant change to the IRAC pipeline affecting calibration. The majority of changes have to do with header keyword additions.

1. New observing mode: 'Stellar Mode' has multiple full-array short time exposures within channel 1 & 2 at the same time having a longer time integration in channel 3 & 4. This allows for brighter objects to be observed in the longer wavelength channels to higher signal-to-noise without saturating in the shorter wavelength observations. Available will be:

   ch1&2	ch3&4
   0.4s	2s
   2x2s	12s
   2x12s	30s

2. The median value of the frames used to create the skydark subtracted from the data will be placed in the header of the BCD: keyword SKYDKMED
3. The name of the labdark subtracted from the data will be placed in the header: keyword LBDRKFLE
4. The time of the observation (SCLK_OBS) will be computed using telemetry only to allow for a more exact timing. This keyword will be placed in the database and header. Further S13 changes will include calculating the first frame correction from this more exact timing.
5. Keywords PTGDIFFX, PTGDIFFY inserted to refer to the pointing differences in actual pixels along the X & Y axis.

S11.0 (Release Date: Nov 04)

All IRAC data have been reprocessed with version S11.0 or higher and are currently available within the archive. If you have data of an earlier version, it is beneficial to download a newer version of S11.0 or higher.

1. The EQUINOX header keyword for BCDs has been fixed.
2. Other changes to the header include the First Frame Delay and Immediate Delay (FFDLAY & IMMDLAY) times calculated from the first frame correction module are reported.
3. Previously, a DCE with a non-zero CHECKSUM from MIPL was not allowed to process through the pipeline. In S11, the CHECKSUM will now be reported within the header and the DCE processed.
4. The first frame correction has been fixed for the high-dynamic-range observations. The only remaining bug is for the intermediate frame times (12 sec) when used as part of an HDR frameset. This effect will not be noticeable except as slight background DC-level offset from frame to frame in the 12 sec data as part of 100s or 200s HDR framesets.
5. After study of the last year's worth of flat-fields and finding no noticeable change from campaign to campaign, a super-flat has been composed of the last year's worth of observations. A sub-array flat has been composed of this super flat and both have been loaded.
6. Overlap correction is now being applied in the post-BCD pipeline.
7. The mosaic image headers have been populated with more keywords.

S10.5 (Release Date: 8 Aug 04)

1. Updated the ffcorr module to use the correct delay time between frames for full array NON-HDR frames. The HDR frames will be fixed in S11.

S10.0 (Release Date: 14 Apr 04)

1. New linearity model in channel 4 (full and sub). Change from quadratic to cubic (actually updated in S9.5.2).
2. Module ffcorr set to output only one plane for interpolated correction image rather than all planes.
3. FITS Keyword: Create and populate new FITS header keyword (DS_IDENT).
4. Update to readnoise in initial noise image.
5. If bcd pixel = NaN, uncertainty pixel = 0.
6. Keyword from dark ensemble placed in BCD header (SKYDRKZB) (Skydark zodi backgrnd estimate added into header of science bcd).

S9.5 (Release Date: 20 Feb 04)

S9.5 Calibration Items for IRAC

1. Addition of two fields, hdrmode and numrepeats, to caldata tables. Requires a backfill script to transform and migrate current fallbacks and metadata to new tables.

   The hdrmode field is in current use. The numrepeats field is to facilitate use of the external repeat number in future calibration activities.

2. In S9.5 the flux conversion will be delivered in an IPAC table, for example:

   \char Comment Calibration data file for dntoflux module.
   \char INSTRUME = 'IRAC'
   \int CHNLNUM = 4
   \char fluxconv = 'Conversion factor in MJy/sr per DN/s'
   \char fluxconvunc = 'Uncertainty in fluxconv'
   |fluxconv |fluxconvunc |
   |float |float |
   0.195 0.020

3. HDR skydarks are now delineated from non-HDR. Skydarks are now aware of ch4 repeats and pipelines fetch skydarks for the correct repeat. Again, this is possible due to new fields in the caldata tables.
4. Scattered light removal module (slremove) added to science pipeline and calibration preprocessing.
5. Calibration ensemble pipelines now use fpgen to clean up the product header.
6. New pipeline to create subarray flat from full array flat.
7. Latent ensemble creates new request median and request average images.

S9.5 Pipeline Items for IRAC

1. New keywords to be added to the mosaic header: AOT_TYPE, AORLABEL, FOVID, FOVNAME, PRIMEARR, OBJECT, PAONUM, CAMPAIGN.

S9.1 (Release Date: 21 Jan 04)

S9.1 Pipeline Items for IRAC

1. Less than <0.1% of the DCEs may not have pointing reconstruction applied the data. BCDs with USEDBPHF=F indicate that the Boresight Pointing History File was not used, and the RA and DCE in the headers for these cases are based on pre-observation predictions which can be off by 5-50". Do not use such data if pointing is important.
2. Users should note that the keyword PRIMEARR is now present in both the MIPS BCD and POSTBCD headers. Data with PRIMEARR=1 correspond to the user's requested primary array observations. Scan mode data in all 3 arrays are always prime and have PRIMEARR=1. POSTBCD mosaics are also made for MIPS-24 data taken while MIPS-70 and MIPS-160 photometry data are primary; currently, a single mosaic is made of all non-prime data with PRIMEARR=2 for each AOR. This mosaic may not be scientifically useful. Data with PRIMEARR=3 (MIPS-70 and MIPS-160 data taken during MIPS-24 primary data) are not scientifically valid because proper stim flashes are not taken.

S9.0 (Release Date: 29 Dec 03)

S9.0 Data Items for IRAC

1. Some AORs have been affected by long term latents from previous observations. For the most part, observers have sufficiently dithered so that the impact is minimal, on processed and co-added data.
2. Note that the noise in the images and the sensitivity to point sources are not equal to our pre-launch predictions (e.g., as available from our website until December 19, or in the Observer's Manual versions before 4.0), although they are close. New sensitivity numbers are available in the revised Observer's Manual (version 4.0), which was available at our website starting ~December 19, 2003. For reference, the ratio of the new point source detection threshold to the pre-launch advertised value, for low background observations in 30 sec frames, is 0.69, 0.75, 1.60, and 1.31 in channels 1, 2, 3, and 4, respectively.
3. Persistent images in channel 1. When a bright source (K=13 mag or brighter) is stared at for a long time, for example, during a downlink, it will leave a persistent image in channel 1 that decays very slowly (persists for several hours or more). A persistent image mitigation strategy involving annealing the array after downlinks has been put in place for nominal operations. These anneals will erase the persistent images from the array, but do not protect against persistent images from bright object observations that can accumulate on the array before the next downlink. Science impact: left unmitigated, you will have extra, spurious sources in your image. These sources have a PSF that is wider than the actual true source PSF. Dithering helps to get rid of these spurious sources.
4. Persistent images in channel 4. These are different in nature from the channel 1 persistent images. A bright source leaves a persistent image that can last for more than a week and even through IRAC power cycles. These images keep building up on the array. However, the amplitude of the persistent images is rather low. Annealing has been found to erase also the channel 4 persistent images. Therefore, we will anneal both channels 1 and 4 simultaneously, every 12 hours (after each downlink), to erase persistent images. Again, dithering helps to get rid of these spurious images.
5. Diffuse stray light: All IRAC images contain a stray light pattern, resembling a "butterfly" in channels 1 and 2, and a "tic-tac-toe" board in channels 3 and 4. These artifacts are due to zodiacal light scattered onto the arrays, possibly reflected from a hole in the FPA covers above the channel 1 and 2 arrays, and from reflective surfaces outside the edges of channel 3 and 4 arrays. The stray light scales with zodiacal light, which is the light source for our flat fields, so the stray pattern contaminates the flats. As a result, the flat fields will aesthetically remove the stray light rather well from images but will induce systematic errors of approximately 5% in flux calibration for point sources that fall in the peak stray light location. Dithering will mitigate this effect, because it is unlikely that a dithered observation will keep a source within the stray light lobes. Diffuse stray light will be removed from both the flat fields and the science frames in a future version of the pipeline.
6. Stray light from point sources. Spot allows you to overlay stray light boxes on any image; if a bright star is placed in those boxes during an observation, a scattered light patch will appear on the array. We have found three more such boxes during testing, in channels 1 and 2. The new stray light boxes are included in Spot now and are also shown in the new Observer's Manual. Channels 3 and 4 have less stray light, and the stray light inducing regions are not the same as the ones we guessed (by analogy to channels 1 and 2) from the lab tests, so the channel 3 and 4 boxes were removed from Spot. In channels 3 and 4 the stray light arises when a star lands on a thin region just outside the array (the same region that causes the "tic-tac-toe" pattern from diffuse stray light in flat fields). A redundant observing strategy will help eliminate stray light problems. Observers covering fields with bright sources should inspect the individual images; this is required if the depth of coverage is less than 3, to identify spurious spots and rays that could be mistaken for real astronomical objects.
7. Dark spots on pick-up mirror. There is contamination on the mirror which causes a dark spot about 10 pixels wide in channels 2 & 4. This is a 15% effect. Flat fields completely correct for this feature in the data.
8. Muxbleed. We have a correction algorithm, but the coefficients need fine tuning. Furthermore, for bright sources, muxbleed does not scale linearly with source brightness, so even a sophisticated algorithm cannot accurately remove it. Some experiments at fitting the muxbleed for bright sources indicate that the decay pattern is always the same, and only the amplitude appears to be variable.
9. Banding and column pulldown. A bright source on the array will cause its column to be pulled down by a small amount. An algorithm to cosmetically correct the images for column pulldown has been developed and is being tested. This appears to be an additive effect. An analogous effect for an extremely bright source is that the entire image appears to have a different DC level from the preceding and following images. The physical origin of these effects and the probably related (and already known) banding effect is not yet understood. This work is in progress.

S8.9 (Release Date: 11 Nov 03)

S8.9 Data Items for IRAC

1. Some AORs have been affected by long term latents from previous observations. For the most part, observers have sufficiently dithered so that the impact is minimal, on processed and co-added data.
2. Note that the noise in the images and the sensitivity to point sources are not equal to our pre-launch predictions (e.g., as available from our website until December 19, or in the Observer's Manual versions before 4.0), although they are close. New sensitivity numbers are available in the revised Observer's Manual (version 4.0), which was available at our website starting ~December 19, 2003. For reference, the ratio of the new point source detection threshold to the pre-launch advertised value, for low background observations in 30 sec frames, is 0.69, 0.75, 1.60, and 1.31 in channels 1, 2, 3, and 4, respectively. The apparent modest decrease in sensitivity in channels 3 and 4 is under investigation.
3. Persistent images in channel 1. When a bright source (K=13 mag or brighter) is stared at for a long time, for example, during a downlink, it will leave a persistent image in channel 1 that decays very slowly (persists for several hours or more). A persistent image mitigation strategy involving annealing the array after downlinks has been put in place for nominal operations. These anneals will erase the persistent images from the array, but do not protect against persistent images from bright object observations that can accumulate on the array before the next downlink. Science impact: left unmitigated, you will have extra, spurious sources in your image. These sources have a PSF that is wider than the actual true source PSF. Dithering helps to get rid of these spurious sources.
4. Persistent images in channel 4. These are different in nature from the channel 1 persistent images. A bright source leaves a persistent image that can last for more than a week and even through IRAC power cycles. These images keep building up on the array. However, the amplitude of the persistent images is rather low. Annealing has been found to erase also the channel 4 persistent images. Therefore, we will anneal both channels 1 and 4 simultaneously, every 12 hours (after each downlink), to erase persistent images. Again, dithering helps to get rid of these spurious images.
5. Diffuse stray light: All IRAC images contain a stray light pattern, resembling a "butterfly" in channels 1 and 2, and a "tic-tac-toe" board in channels 3 and 4. These artifacts are due to zodiacal light scattered onto the arrays, possibly reflected from a hole in the FPA covers above the channel 1 and 2 arrays, and from reflective surfaces outside the edges of channel 3 and 4 arrays. The stray light scales with zodiacal light, which is the light source for our flat fields, so the stray pattern contaminates the flats. As a result, the flat fields will aesthetically remove the stray light rather well from images but will induce systematic errors of approximately 5% in flux calibration for point sources that fall in the peak stray light location. Dithering will mitigate this effect, because it is unlikely that a dithered observation will keep a source within the stray light lobes. Diffuse stray light will be removed from both the flat fields and the science frames in a future version of the pipeline.
6. Stray light from point sources. Spot allows you to overlay stray light boxes on any image; if a bright star is placed in those boxes during an observation, a scattered light patch will appear on the array. We have found three more such boxes during testing, in channels 1 and 2. The new stray light boxes are included in Spot now and are also shown in the new Observer's Manual. Channels 3 and 4 have less stray light, and the stray light inducing regions are not the same as the ones we guessed (by analogy to channels 1 and 2) from the lab tests, so the channel 3 and 4 boxes were removed from Spot. In channels 3 and 4 the stray light arises when a star lands on a thin region just outside the array (the same region that causes the "tic-tac-toe" pattern from diffuse stray light in flat fields). A redundant observing strategy will help eliminate stray light problems. Observers covering fields with bright sources should inspect the individual images; this is required if the depth of coverage is less than 3, to identify spurious spots and rays that could be mistaken for real astronomical objects.
7. Dark spots on pick-up mirror. There is contamination on the mirror which causes a dark spot about 10 pixels wide in channels 2 & 4. This is a 15% effect. Flat fields completely correct for this feature in the data.
8. Muxbleed. We have a correction algorithm, but the coefficients need fine tuning. Furthermore, for bright sources, muxbleed does not scale linearly with source brightness, so even a sophisticated algorithm cannot accurately remove it. Some experiments at fitting the muxbleed for bright sources indicate that the decay pattern is always the same, and only the amplitude appears to be variable.
9. Banding and column pulldown. A bright source on the array will cause its column to be pulled down by a small amount. An algorithm to cosmetically correct the images for column pulldown has been developed and is being tested. This appears to be an additive effect. An analogous effect for an extremely bright source is that the entire image appears to have a different DC level from the preceding and following images. The physical origin of these effects and the probably related (and already known) banding effect is not yet understood. This work is in progress.

S8.9 Pipeline Items for IRAC

1. Mosaics produced by the online pipeline for HDR mode data incorrectly weight the short and long frame times. For long exposures (> 12s), data is effectively taken in HDR mode, and hence the pipeline produced mosaics will not be very useful.
2. Cosmic ray rejection is not functioning well.
3. Muxbleed. We have a correction algorithm, but the coefficients need fine tuning. Furthermore, for bright sources, muxbleed does not scale linearly with source brightness, so even a sophisticated algorithm cannot accurately remove it. Some experiments at fitting the muxbleed for bright sources indicate that the decay pattern is always the same, and only the amplitude appears to be variable.