Online Pipeline History Log: MIPS Pipeline Items

S17 (Release Date: Jan 08)

MIPS24:

- bcd.log now archived
- linearized.fits now saved to sandbox in preparation for AOR spotflats and enhanced pbcds
- slow scan missing/bad flat has been fixed

MIPS-Ge:

- SED bit rejection for pbcd products changed, resulting in improved mosaics and improved spectra especially for faint sources.
- Code change for stim solution: if Stim - background < 0, value set to NaN, i.e. that stim will not be used in deriving the stim solution. Also, instead of extrapolated stim solutions (which can also lead to negative fluxes) nearest in time Stim-BKG value will be used for calibration. Not necessarily correct, but better than unphysical negative solution. Dramatically improves appearance for very bright photometry source, does not affect normal flux level AORs.
- dark and illumination correction images can now be retrieved for Ge via leopard (cal button). This was not possible up to now due to a software bug.

S16 (Release Date: Apr 07)

**NOTE** As of the end of August 2007, all the MIPS Campaigns (Nominal Operation and IOC/SV) have been reprocessed and archived with S16. This version of the software includes the latest calibration factors as presented in the calibration papers in the PASP September 2007 issue (for 24, 70 & 160um). The impact on the data is minimum, but makes everything (analysis & data) self-consistent.

Furthermore, the SED data is being reprocessed (S16.1.1) to update the header information and to include a very small change in the wavelength calibration (~0.5um).

S16.0.1 and S16.1.0 Pipeline items for MIPS

MIPS24:

A couple of bugs were found in the MIPS 24um pipeline that affected high flux density observations. Essentially, 24um point sources above 2 Jy or extended emission above 600 MJy/sr. This has been fixed in S16. Prior to these fixes the uncertainty at these levels ranges from a couple of percent [close to the mentioned thresholds] to 10-15% [at the highest reliable dynamic range of the 24um detector, e.g. ~2500 MJy/sr]. Most science programs at 24um won't be affected by these changes.

1. Modules SATMASK, DESATSLOPE, SLOPECORR and DROOPCOR were updated and corrected.
2. In other to acommodate "timing observations" where time accuracy is important, some header keywords in the BCDs are now given with more significant figures, and the Heliocentric Julian Date has been added.
   DATE_OBS Date & time at DCE start
   MJD_OBS [days] MJD at DCE start (JD-2400000.5)
   HMJD_OBS [days] Corresponding Heliocen. Mod. Julian Date
   UTCS_OBS [sec] J2000 ephem. time at DCE start
   SCLK_OBS [sec] SCLK time (since 1/1/1980) at DCE start
   SPTZR_X [km] Heliocentric J2000 x position
   SPTZR_Y [km] Heliocentric J2000 y position
   SPTZR_Z [km] Heliocentric J2000 z position
   SPTZR_VX [km/s] Heliocentric J2000 x velocity
   SPTZR_VY [km/s] Heliocentric J2000 y velocity
   SPTZR_VZ [km/s] Heliocentric J2000 z velocity
   SPTZR_LT [sec] One-way light time to Sun's center
   ET_OBS [sec] Ephemeris time (seconds past J2000 epoch)
   SAMPTIME [sec] Sample integration time
   EXPTIME [sec] Effective integration time per pixel
   COADD Ge coadd parameter
   

1. The above mentioned header keywords were also added to the Ge:Ga BCDs.

S15.0.4 (Release Date: Dec 06)

S15.4.0 Pipeline items for MIPS

MIPS24:

1. No changes under S15

1. Standard Deviation (STD) mosaics for both the filtered and unfiltered products.
2. BUNC (uncertainty) images are used to make filtered mosaic products.
3. A new FBUNC (filtered uncertainty) mosaic image is part of the products.

S14.4.0 (Release Date: Jul 06)

S14.4.0 Pipeline items for MIPS

MIPS24:

1. Modification of linearity correction algorithm.
2. Reduction in number of files archived.

1. SED PostBCD pipeline: Implemented SED coadds and extraction for online processing, previously only available offline.
2. TPM PostBCD pipeline: Implemented TPM coadds for online processing.
3. Updated PostBCD outlier rejection. Now mopex rejects with respect to the median (threshold_option=1) instead of rejecting with respect to mean (threshold_option=2).
4. The relative position of the 160um array was updated by about 4.55" in array coordinates to give consistent pointing solutions with the 70um and 24um arrays.
5. Offline BCD pipeline and filtering tools for Ge made public. The GeRT now supports all MIPS science modes and can be run on the Mac and Linux machines.
6. Fixed SED bug which affected the SED bmasks.

S13.2.0 (Release Date: Dec 05)

S13.2.0 Pipeline items for MIPS

MIPS24:

1. Turned off unneeded rowdroop module.
2. Fixed erroneous application of droop correction to first difference image

S13.0.2 (Release Date: Nov 05)

S13.0.2 Pipeline items for MIPS

General MIPS:

1. Added to the BCD headers an estimate of the background at the corresponding time & position using the SSC model (same as SPOT).
   • ZODY_EST = Zodiacal Background Estimate
   • ISM_EST = Interstellar Medium Estimate
   • CIB_EST = Cosmic IR Background Estimate
2. Added - when necessary - the WARMFLAG keyword to track future MIPS warm campaigns

1. New calibration factor (2% different - see version 2.3 of MIPS DH)
2. NaNs now set properly in the P-mask pixels and hard saturated pixels from the difference image.

1. New calibration factor for both 70 (11% change) and 160 (6% change - see version 2.3 of MIPS DH)
2. Improved calibration files for SED (IC and response function).
3. SED Post-BCD Products are now delivered. They include: On and Off source mosaics, uncertainty and coverage maps and extracted spectrum/a.
4. Improved radhit/sloper for 160um (reject reads after strong RH and updated thresholds; small updates for 70um).
5. Fixed bugs associated with the propagation of NaNs.
6. More flexibility in filtering. In particular, switched the order of column filter and high-pass time filter for 70um for default processing.
7. Making std error images for post-BCD processing (for good coverage std images are better than unc images for source extraction).
8. Updated the GeRT to allow for multi-thread processing and compatibility with Linux and Mac. (GeRT in testing now, to be released in early 2006)

S12.0 (Release Date: May 05)

S12.0 Pipeline items for MIPS

MIPS24:

1. For mips24, execute science/post-BCD threads following initial processing of dedicated flatfield data thru flatfield thread.
2. Functionality to process mips24 SUR-mode Total Power Mode (TPM) data.
3. Updates to pointing transfer to compute orthogonal PTGDIFF (X,Y) components for all MIPS arrays.
4. Updates to pointing transfer and DB schema to load an additional 12 QA MIPS diagnostics into the QA_ptg_Xfer table.
5. As part of mips24 post-BCD processing, create an unnormalized median product of all BCDs for every science AOR.
6. Updated mips24 science threads to use automated "spot" and scan-mirror-dependent flatfield picking. spot-dependent flatfield library was made offline.
7. New pipeline thread to perform pointing refinement of MIPS24 scan-legs and subsequent refinement of BCDs therein.

1. Spectral Energy Distribution (SED) pipeline modified to take additional calibration files.
2. Total Power Mode pipeline implemented.
3. Hipass filter - for filtered BCDs - enhanced with sigma errors calculated from data.
4. A FITS keyword to track cdf file versions (important to track the reprocessing history of the data).
5. Use of ensembles for scan map AORs (this groups together scan legs and allows a better stim interpolation, and therefore, calibration).

S11.0 (Release Date: Nov 04)

S11.0 Pipeline Items for MIPS

MIPS24:

1. Updated mask bits distinguishing between hard and soft saturation for MIPS-24.

1. Uniform absolute calibration for all modes and campaigns (no additional scale factors to put data on same calibration scale as needed for S10).
2. Application of the electronic-nonlinearity (1-2% effect on input DN).
3. Application of column filter for 70um point source fbcd products.
4. Online production of mosaics for the point-source filtered fbcds, i.e., make mosaics for both default BCDs (msaic.fits) and from the filtered fbcds (mfilt.fits).

S10.5 (Release Date: 8 Aug 04)

S10.5 Pipeline Items for MIPS

MIPS-Ge:

1. Two MIPS-Ge BCD products are produced, one optimized for point sources and another BCD which maintains extended source emission.

S10.0 (Release Date: 14 Apr 04)

S10.0 Data Items for MIPS

MIPS24:

1. Problem with the pointing for mips24 being off by about 1.2 arcsecond (0.5 pixel) has been fixed.
2. The 24um "read-2" effect can produce a slight gradient across the frame. S10.0 includes the "rowfluxcorr" algorithm, which is used to correct for a gradient of order a few % along rows in BCDs.
3. Outstanding problems are the presence of bright-source artifacts. An example is when there's a bright source in a BCD, there appear to be strong jailbars in the bottom half of the array (to be fixed in S12).
4. Occasional bad flat or missing scan-mirror dependent flats in the system thus affecting the BCD/mosaic quality. The latter is expected to be minimal.

1. Include a more sophisticated MIPS-Ge high-pass filter.

S10.0 Pipeline Items for MIPS

All Channels:

The units of image data have been changed from microJy/arcsec^2 to MJy/str.

S9.5 (Release Date: 20 Feb 04)

S9.5 Data Items for MIPS

MIPS24:

1. Problem with the pointing for mips24 being off by about 1.2 arcsecond (0.5 pixel) (to be fixed in S10.0).
2. Scan-mirror dependent flats not used judiciously so the mosaics can look a little blotchy.
3. The 24um "read-2" effect can produce a slight gradient across the frame (to be fixed in S10.0).

S9.5 Calibration Items for MIPS

MIPS-Ge:

1. Include a median filter for MIPS-Ge.

S9.1 (Release Date: 21 Jan 04)

S9.1 Data Items for MIPS

MIPS24:

1. The 24um "read-2" effect can produce a slight gradient across the frame. This will be corrected in the next software build.
2. The 1st DCE of every commanded sequence of observations (data with DCENUM=0) is calibrated differently than the rest of the data (extra boost frame). Best results in the short term may be obtained by ignoring all DCENUM=0 DCEs. BCD's with DCENUM=0 are omitted from the POSTBCD mosaics.
3. Saturating point sources and bright cosmic rays can produce enhanced jailbars (pattern every 4th row/column associated with the readouts). Extremely saturated sources may have a high background in the frames surrounding them due to inability to correctly account for the droop effect in regions which saturate during the first read.
4. The 24um photometry in a BCD can vary by about 2% across the array as the data is not corrected for distortions (the distortion correction is applied in the post-BCD pipeline).
5. Bright sources will leave a bright latent at the less than 1% level which fades over a timescale of seconds. Extremely saturated sources (tens of Jy) will leave latents at the several % level which may take hours to fade. These long-lived latents will be first bright, then dark as they fade away. Although we are trying not to schedule bright source photometry adjacent to faint source photometry, it is possible to see these latents from previous AORs in unusual cases.

1. Beware of the MIPS-160 light "leak" when observing objects that are bright in near-infrared.
2. MIPS-Ge data show stim latents which depend on background and appear to increase with the time since anneal. The stim latents linger longer for 70um than 160um, since the decay time constants associated with the stim latents are larger at 70um. The stim latents can also be mitigated by median filtering the data (but this is not yet implemented in the pipeline).
3. Due to a bad cable outside the instrument, there is a strong pattern noise on side-B of the 70um array.
4. Due to other cabling Items, there is one bad 4x8 pixel readout on the side-A of the 70um array and one bad 5x1 pixel readout on the 160um array.

S9.1 Calibration Items for MIPS

All Channels:

1. The data have preliminary flux calibration. We have not fully quantified the accuracy of the absolute calibration of MIPS.
2. The uncertainty images have not been validated.

1. The 24um flats depend on the scan mirror position and scan rate due to contamination ("dust spots") on the pickoff mirror. The S9.1 software uses scan-mirror dependent flats (a unique flat field for every commanded scan mirror position) in an effort to correct for this effect and should improve the flat fielding markedly compared to pre-S9.1 software. However, there is evidence that these spots may shift on the array (in the in-scan direction) somewhat randomly by approximately 1/2 24 micron pixel, which would cause a mismatch between the flat fields and the science observations. If this is the case, a dark/light pair of spots/streaks will appear in each BCD wherever a "spot" appears. Currently, there are no scan mirror dependent flats for the "parallel" 24 um data taken during Ge photometry.
2. The 24um linearity calibration is based on pre-launch solutions.

1. The Ge electronic non-linearity correction is based on pre-launch calibration.
2. MIPS-Ge data are calibrated using the stims to track the variation of the short-term response of the detections. The stim calibration does not completely remove long-term transients for the MIPS-Ge detectors. The measured "illumination correction" used to remove pixel to pixel response variations appears to vary as a function of time and background and this time variation is not taken into account in this current version of the pipeline. Starting with the BCDS, users can build separate "flats" as a function of time (e.g., one scan leg at a time) to help mitigate these effects. The variations of the "illumination correction" as a function of time can also be mitigated by median filtering the data, if users are only interested in point sources (but this is not yet implemented in the pipeline and the effects of calibration on median filtered data have yet to be quantified).
3. Flux calibration has yet to be fully derived as a function of source brightness and background. Information quantifying the flux non-linearities will be available when the archive opens, i.e., MIPS-Ge absolute calibration should be considered very preliminary at this time.

S9.1 Pipeline Items for MIPS

All Channels:

1. Less than <0.1% of the DCEs may not have pointing reconstruction applied the data. BCDs with USEDBPHF=F indicate that the Boresight Pointing History File was not used, and the RA and DCE in the headers for these cases are based on pre-observation predictions which can be off by 5-50". Do not use such data if pointing is important.
2. Users should note that the keyword PRIMEARR is now present in both the MIPS BCD and POSTBCD headers. Data with PRIMEARR=1 correspond to the user's requested primary array observations. Scan mode data in all 3 arrays are always prime and have PRIMEARR=1. POSTBCD mosaics are also made for MIPS-24 data taken while MIPS-70 and MIPS-160 photometry data are primary; currently, a single mosaic is made of all non-prime data with PRIMEARR=2 for each AOR. This mosaic may not be scientifically useful. Data with PRIMEARR=3 (MIPS-70 and MIPS-160 data taken during MIPS-24 primary data) are not scientifically valid because proper stim flashes are not taken.

1. The pixels in the bad half of the 70um array show zero flux instead of being replaced by NaNs as they should.
2. The online 160um mosaics may show bright rows/columns associated with the stim DCEs. The stim DCEs should be rejected in the coadds/mosaics, but these are not always rejected currently due to a software bug which affects the bmasks for 160um. Users could make updated mosaics offline by ignoring the stim DCEs; 160um stim DCEs can be identified from the keyword STMFL160>0.

S9.0 (Release Date: 29 Dec 03)

S9.0 Data Items for MIPS

MIPS24:

1. The 24um "read-2" effect can produce a slight gradient across the frame. This will not be fixed until S9.0 and may be somewhat mitigated with corrections to the flats in the short term.
2. The 1st DCE of every commanded sequence of observations (data with DCENUM=0) are calibrated differently than the rest of the data (extra boost frame). Best results in the short term may be obtained by ignoring all DCENUM=0 DCEs.
3. Saturating sources and bright cosmic rays can produce enhanced jailbars (pattern every 4th row/column associated with the readouts).
4. The 24um photometry in a BCD can vary by about 2% across the array as the data is not corrected for distortions (the distortion correction is applied in the post-BCD pipeline).

1. Beware of the MIPS-160 light "leak" when observing objects that are bright in near-infrared.
2. MIPS-Ge data show stim latents which depend on background and appear to increase with the time since anneal. The stim latents linger longer for 70um than 160um, since the decay time constants associated with the stim latents are larger at 70um. The stim latents can also be mitigated by median filtering the data (but this is not yet implemented in the pipeline).
3. Due to a bad cable outside the instrument, there is a strong pattern noise on side-B of the 70um array.
4. Due to other cabling Items, there is one bad 4x8 pixel readout on the side-A of the 70um array and one bad 5x1 pixel readout on the 160um array.

S9.0 Calibration Items for MIPS

All Channels:

1. The data have very preliminary flux calibration. We have not fully quantified the accuracy of the absolute calibration of MIPS.
2. The uncertainty images have not been validated.

1. Due to contamination on the pickoff mirror the 24um flats depend on the scan mirror position and scan rate. We currently do not have all the calibration data collected as a function of scan mirror and are using the non-mirror position dependent flat (default flat). Deep observations will show dark spots and will appear blotchy with the current calibration. Users with sufficient data could make their own flats from the BCDs sorting the data by the scan mirror position (CSM_PRED) for photometry mode and by scan mirror position and scan rate (SCANRATE) for scan observations.
2. The 24um linearity calibration is based on pre-launch solutions.

1. The Ge electronic non-linearity correction is based on pre-launch calibration.
2. MIPS-Ge data are calibrated using the stims to track the variation of the short-term response of the detections. The stim calibration does not completely remove long-term transients for the MIPS-Ge detectors. The measured "illumination correction" used to remove pixel to pixel response variations appears to vary as a function of time and background and this time variation is not taken into account in this current version of the pipeline. Starting with the BCDS, users can build separate "flats" as a function of time (e.g., one scan leg at a time) to help mitigate these effects. The variations of the "illumination correction" as a function of time can also be mitigated by median filtering the data, if users are only interested in point sources (but this is not yet implemented in the pipeline and the effects of calibration on median filtered data have yet to be quantified).
3. Flux calibration has yet to be fully derived as a function of source brightness and background. Information quantifying the flux non-linearities will be available when the archive opens, i.e., MIPS-Ge absolute calibration should be considered very preliminary at this time.

S9.0 Pipeline Items for MIPS

All Channels:

1. Roughly about 0.1-0.5% of the DCEs may not have pointing reconstruction applied to the data. BCDs with USEDBPHF=F indicate that the Boresight Pointing History File was not used, and the RA and DCE in the headers for these cases are based on pre-observation predictions which can be off by 5-50". Do not use these data if pointing is important.
2. Some DCEs may have pointing solutions that are completely bogus (>10deg away). A software update was applied to S9.0 to fix a known S8.9 bug, but it is not clear yet if this fixed the problem in all cases. Please notify the SSC if you find any examples of strange pointing solutions in your data.
3. The POSTBCD mosaics do not have important header information, such as OBJECT and PRIMEARR. If you plan to use the SSC mosaics, you will need to check the RA and DCE to determine which fields are associated with which mosaics. PostBCD products with proper header information will not be available until S10.0 (when the archive opens).

1. The online 160um mosaics may show bright rows/columns associated with the stim DCEs. The stim DCEs should be rejected in the coadds/mosaics, but these are not always rejected currently due to a software bug which affects the bmasks for 160um. Users could make updated mosaics offline by ignoring the stim DCEs; 160um stim DCEs can be identified from the keyword STMFL160>0.

S8.9 (Release Date: 11 Nov 03)

S8.9 Data Items for MIPS

MIPS24:

1. The 24um "read-2" effect is not yet corrected which can produce a slight gradient across the frame.
2. Saturating sources and saturating cosmic rays can produce enhanced jailbars.
3. The 24um photometry can vary by about 2% across the array for data not corrected for distortions.
4. The first several 24um DCEs in a scan map are darker (have lower levels).

1. Beware of the MIPS-160 light "leak" when observing objects that are bright in near-infrared.
2. There is no correction for Ge stim latents. The effects from stim latents increase with time since anneal.
3. There is a strong pattern noise on side-B of the 70um array.
4. There is one bad 4x8 pixel readout on the side-A of the 70um array.
5. There is one bad 5x1 pixel readout on the 160um array.
6. If you are interested in weak point sources (<100 mJy) at the MIPS-Ge wavelengths, we recommend stacking the BCDs and median filtering the data as a function of time (ignoring the stim DCEs). This mitigates the effects of stim latents which are particular bad for MIPS-70.

S8.9 Calibration Items for MIPS

All Channels:

1. The data have very preliminary flux calibration. We have not fully quantified the accuracy of the absolute calibration of MIPS.
2. The uncertainty images have not been validated.

1. Due to contamination on the pickoff mirror the 24um flats depend on scan mirror position and scan rate. In photometry mode, you will find dark spots, and for scan observations you will find dark streaks.
2. The 24um linearity calibration is based on pre-launch solutions.

1. The Ge electronic non-linearity correction is based on pre-launch calibration and is known to be somewhat inaccurate for in-flight data.
2. There is no correction for the variation of the illumination correction as a function of time.
3. There is no high-pass filter for MIPS-Ge which is useful for detecting weak point sources.
4. If you are interested in weak point sources (<100 mJy) at the MIPS-Ge wavelengths, we recommend stacking the BCDs and median filtering the data as a function of time (ignoring the stim DCEs). This mitigates the variations of the illumination correction as a function of time which are particular bad for MIPS-70.

S8.9 Pipeline Items for MIPS

All Channels:

1. The derived WCS pointing solutions are inaccurate for scan maps (nothing is wrong with the actual pointing of the telescope, software bug). Within scan legs, the offsets can vary by 5" or so and the WCS solutions can be off by 10-20" or more between scan legs.
2. Some DCEs may have pointing solutions which are completely bogus (>10deg away).