|
|
FAQs: General Data Processing |
|
Table of Contents: Q: What is the general philosophy governing SSC data processing? Q: What is the basic format for SSC processed data? Q: What are the primary data products that the SSC generates? What is a BCD? What is a post-BCD product? Q: What other products are available to investigators? Q: Where should a typical observer start their analysis? Q: What is the filenaming scheme for all these files? Q: What is the meaning of all the DQA status flags that I see in Spot/Leopard? Q: Are you planning data analysis workshops (at the SSC or the AAS)? Can I come visit the SSC to learn about data analysis? Q: How can I tell which pipeline processed my data? How can I tell if my data have been reprocessed under the most recent pipelines? Q: Where was Spitzer exactly during my observations so that I can calculate light-time corrections? Q: What time keywords should I use from the header for the exact start of my AOR? Q: How should I reference my Spitzer data in my journal articles? What acknowledgment should I use? Q: What is the general philosophy governing SSC data processing?A: The general philosophy is to do a few things, and do them well! Data processing for each of the eight pipelines (one per observing mode) is highly automated. A modular design allows components, or individual processing steps, to be easily updated. The 'atomic unit' for all processing is the Astronomical Observation Request (AOR), a user-specified description of an observation output by the Spot. The development of the pipelines has been carried out in close collaboration with the Spitzer Instrument Teams and Instrument Support Teams.
Q: What is the basic format for SSC processed data?A: SSC data is electronically accessible as FITS files (no extensions). The end-product files include complete calibration information and intermediate files for individual processing 'steps.'
Q: What are the primary data products that the SSC generates? What is a BCD? What is a post-BCD product?A: In general, the eight SSC data processing pipelines convert spacecraft engineering data into scientifically useful data. Raw data (engineering and science) are received at the ground tracking stations via telemetry and forwarded to the Spitzer Flight Operations Center at the Jet Propulsion Laboratory in Pasadena. The Flight Operations System at JPL receives telemetry packets from the Spitzer spacecraft and repackages them into FITS files containing sensor data, expressed in DN (data number) units. Instrument engineering and housekeeping data are also transferred to the SSC from the Flight Operations System. The archived raw data represent rationally organized, time-ordered data and include associated Observatory pointing data and calibration observations. Basic Calibrated Data (BCD) are two-dimensional images in FITS format, and correspond to individual 'data collection events (DCEs)' within an observation. An image is flux and/or wavelength (if appropriate) calibrated, and surface brightness measurements are expressed in physical units. In addition, flat-fielding and cosmetic restoration (e.g., cosmic-ray removal) algorithms are applied to the BCD. Spatial 'world' coordinates are derived from Observatory pointing information only. The BCDs represent the most reliable product achievable through automated processing. Extended pipeline products (also referred to as post-BCD products, previously referred to as Browse Quality Data) are also available. The exact nature of these data will likely change throughout the mission, as more sophisticated analysis tools are developed by some combination of the SSC, Instrument Teams, and the Legacy Science teams. For example, higher-level imaging products include co-addition, mosaicing, and source extraction. The description of SSC data products depends on the science instrument and observing mode. High-level descriptions of BCDs for each of the observing modes are available in the Spitzer Observer's Manual, and details appear in the corresponding Data Handbook (IRAC, IRS, MIPS). The SSC provides routine calibrations for each of the observing modes. These data enter the Spitzer public archive immediately upon processing and verification (unless a proprietary observation embargoes the release of the data). The investigator may propose to undertake special calibrations, although the required observing time to do so must be explicitly requested in the proposal. For Legacy Science projects, these special calibration data also enter the public archive immediately upon processing and verification.
Q: What other products are available to investigators?A: For each AOR, the SSC also provides calibration files, intermediate data files, quality analysis files, all with complete header information.
Q: Where should a typical observer start their analysis?A: If you were asking for pointers to documentation, this page is one-stop-shopping for investigators looking to get started working on Spitzer data -- archive access, data handbooks, software, data processing examples, even IAU naming guidelines and acknowledgments. If you were asking which data files to start with, in general, most observers will want to (and feel confident in) starting their analysis with the Basic Calibrated Data (BCD) processed by the SSC.
Q: What is the filenaming scheme for all these files?A: If you look either in the individual instrument pages (IRAC, IRS, MIPS) or on the Archives and Analysis page, there are links to pages for each instrument with the filenames and contents spelled out:
Q: What is the meaning of all the DQA status flags that I see in Spot/Leopard?A: See the page on DQA Status flag definitions.
Q: Are you planning data analysis workshops (at the SSC or the AAS)? Can I come visit the SSC to learn about data analysis?A: We hope our documentation is extensive enough to cover most of the issues with Spitzer data, and we are working continuously on improving it. Our Helpdesk is available to answer any Spitzer-specific data reduction questions. We are planning data reduction workshops at least every 6 months, and all of the presentations are continuously archived on the web (see http://ssc.spitzer.caltech.edu/ost/workshops/. As you can see from those pages, our data analysis workshops are typically week-long affairs. It is not clear to us how we would condense this for presentation at a one-day AAS workshop. We encourage you to approach the staff at the Spitzer AAS booth at any time with your specific data questions, and we'll do what we can to help you in real time. We're sorry, but we do not have the resources to support individual data reduction visits to the Spitzer Science Center. Please use our Helpdesk (help@spitzer.caltech.edu) and we will work with you to resolve your questions.
Q: How can I tell which pipeline processed my data? How can I tell if my data have been reprocessed under the most recent pipelines?A: This is a multi-step process at least at the moment.
Q: Where was Spitzer exactly during my observations so that I can calculate light-time corrections?A: The answer can be found in this file.
Q: What time keywords should I use from the header for the exact start of my AOR?A: In a nutshell, DATE_OBS is the most accurate, absolute measurement for comparing with astronomical emphemerides. SCLK_OBS (in real sec) is good for relative timing comparisons within an AOR or even a campaign. More words on this: The DATE_OBS value is a real date-time string representing the absolute UTC time at the very start of data aquisition or "array activity" for the DCE, i.e, the start of any boost (for DCENUM=0) or, the start of a reset (for DCENUMs > 0). (It has nothing to do with image timing or readouts.) UTCS_OBS is the number of real seconds past midnight, year 2000 at the _UTC_ time of the DCE start (or array activity as above). Both DATE_OBS and UTCS_OBS are computed from spacecraft telemetry clock counters (SCLK_OBS coarse and fine bit ticks) and calibrated using an atomic clock in Boulder to correct for on-board drifts.
Q: How should I reference my Spitzer data in my journal articles? What acknowledgment should I use?A: This answer comes in two parts: Go back to FAQ page.
|
help@spitzer.caltech.edu http://ssc.spitzer.caltech.edu/faq/dataproc.html
