How the Archive Works
OK, your observation's been scheduled, and according to the email you received when your observation was scheduled, as well as the online schedules, you're pretty sure that your data have been taken. What happens next? The bottom line is that it will be two-three weeks after the campaign ends before your data are available in the archive.
Recall that Spitzer is designed to operate autonomously for ~24-72 hours at a time -- this is called a PAO, or Period of Autonomous Operation. Right after your AOR finishes, you have to wait at least until the end of the PAO before your data come down to the ground. Hopefully all the data come through without errors in transmission (which is usually the case), if not then we have to wait for the next tranmission to the ground, which could be 1-3 days.
Because some instrument calibration tasks occur at the start and periodically throughout a campaign, however, before the data are ready for you to work with, your observation must make it through end-of-campaign reprocessing. If your observation occurs at the beginning of a 14-day campaign, for example, then you have a minimum 14 day wait until the end of the campaign and then the end-of-campaign (EoC) reprocessing begins. EoC reprocessing usually takes about 2 weeks to complete. See the online schedules to see how long your campaign is. Data, once it comes to the SSC, sits in the sandbox, a disk farm where all of the pipeline processing is done. After the end-of-campaign reprocessing (where the calibration data are analyzed and final calibrations created and all the campaign's observations are re-run through the pipeline using these final calibration products), the product archiver moves the data from the sandbox to the permanent archive file system. After data has been archived, observers then query the archive and retrieve their data.
Watch the pipeline reprocessing and archiving status page to find out where data from a given campaign are in this process.
When your data are available, you will receive an email from the SSC (firstname.lastname@example.org) notifying you that your data are ready for pickup via the archive. Note that the archive username and password are not the same as your program's Spot username and password!
We do have an early release data service for time-critical science. For programs with a strong scientific justification to have the data available as soon as possible after execution we have created a website to release these data earlier than the nominal process: http://irsa.ipac.caltech.edu/data/SPITZER/Early_Release/ The Early Release Data are high-level calibrated data products released as soon as possible after observation, typically 1-3 days. These data have been reduced and calibrated in the same manner as the archived data but have only been through the first pass processing. All data on the Early Release web page are immediately public. Please see http://irsa.ipac.caltech.edu/data/SPITZER/Early_Release/README for more information on the Early Release Data.
We periodically release new versions of our online pipeline. The pipeline history log in each of the instrument handbooks (IRAC, IRS, MIPS) keeps track of all of the major changes for each pipeline version.
Understandably, if a new pipeline version provides substantial improvements, we then want to reprocess the whole archive with these new pipelines. This process also takes time. Keep in mind that while we are reprocessing, the firehose of new data does not stop!
Not every pipeline version triggers a reprocessing. Watch the pipeline reprocessing and archiving status page to find out for a given campaign which pipeline version has most recently been run.
Since a new pipeline version provides improvements, you might naturally ask how you can get your own data reprocessed. Right now, we do not have the resources to support individual reprocessing requests; you need to wait until your campaign gets reprocessed with the new pipelines.