How the Archive Works
OK, your observation's been scheduled, and according to the email you received when your observation was scheduled, as well as the online schedules, you're pretty sure that your data have been taken. What happens next? The bottom line is that it will be two-three weeks after the campaign ends before your data are available in the archive.
Recall that Spitzer is designed to operate autonomously for ~24-48 hours at a time -- this is called a PAO, or Period of Autonomous Operation. Right after your AOR finishes, you have to wait at least until the end of the PAO before your data come down to the ground. Assuming all the data come through without errors in transmission (which is usually the case), it takes several hours for data from a PAO to move from the DSN through JPL to the SSC.
Because some instrument calibration tasks occur at the start and periodically throughout a campaign, however, before the data are ready for you to work with, your observation must make it through end-of-campaign reprocessing. If your observation occurs at the beginning of a 14-day campaign, for example, then you have a minimum 14 day wait until the end of the campaign and then the end-of-campaign (EoC) reprocessing begins. EoC reprocessing usually takes about 2 weeks to complete. See the online schedules to see how long your campaign is. Data, once it comes to the SSC, sits in the sandbox, a disk farm where all of the pipeline processing is done. After the end-of-campaign reprocessing (where the calibration data are analyzed and final calibrations created and all the campaign's observations are re-run through the pipeline using these final calibration products), the product archiver moves the data from the sandbox to the permanent archive file system. After data has been archived, observers then query the archive and retrieve their data.
This process takes time! Spitzer generates a lot of data, and it just takes time to transfer the data around. IRAC generates about 20 GB/day of data. It simply takes time to move this kind of data volume around. Our goal is to have data available in the archive two weeks after EoC reprocessing.
Watch the pipeline reprocessing and archiving status page to find out where data from a given campaign are in this process.
When your data are available in the archive, you will receive an email from the SSC (firstname.lastname@example.org) notifying you that your data are ready for pickup via the archive. Note that the archive username and password are not the same as your program's Spot username and password!
We periodically release new versions of our online pipeline. The pipeline history log in each of the instrument handbooks (IRAC, IRS, MIPS) keeps track of all of the major changes for each pipeline version.
Understandably, if a new pipeline version provides substantial improvements, we then want to reprocess the whole archive with these new pipelines. This process also takes time. Keep in mind that while we are reprocessing, the firehose of new data does not stop!
Not every pipeline version triggers a reprocessing. Watch the pipeline reprocessing and archiving status page to find out for a given campaign which pipeline version has most recently been run.
Since a new pipeline version provides improvements, you might naturally ask how you can get your own data reprocessed. Right now, we do not have the resources to support individual reprocessing requests; you need to wait until your campaign gets reprocessed with the new pipelines.