How the Archive Works

Recall that Spitzer is designed to operate autonomously for ~12 hours at a time -- this is called a PAO, or Period of Autonomous Operations. Right after your AOR finishes, you have to wait at least until the end of the PAO before your data come down to the ground. Assuming all the data come through without errors in transmission (which is usually the case), it takes several hours for data from a PAO to move from the DSN through JPL to the SSC.

Because some instrument calibration tasks occur at the start and end of a campaign, however, before the data are ready for you to work with, your observation must make it through end-of-campaign reprocessing. If your observation occurs at the beginning of a 14-day MIPS or IRS campaign, for example, then you have a minimum 14 day wait before this end-of-campaign (EoC) reprocessing begins, and it may take several days to finish. See the online schedules to see how long your campaign is.

Data, once it comes to the SSC, sits in the sandbox, a disk farm where all of the pipeline processing is done. After the end-of-campaign reprocessing, the product archiver moves the data from the sandbox to the permanent archive file system. After data has been archived, observers then use the Leopard tool to query the archive and retrieve their data. Finally, popular products such as Legacy data sets are transferred over to the popular products website.

This process takes time!! Spitzer generates a lot of data, and it just takes time to transfer the data around. For comparison, MIPS and IRAC both generate about 20 GB/day of data. It simply takes time to move this kind of data volume around.

Right now, the goal is to have IRS and IRAC data available in the archive two weeks after EoC reprocessing, and MIPS data available no more than three weeks after reprocessing. Eventually, IRS and IRAC should take no more than 10 days, and MIPS up to 2 weeks. In practice, though we are approaching these goals, we have not quite met them for MIPS and IRS. Things should improve with every pipeline version.

Watch the pipeline reprocessing and archiving status page to find out where data from a given campaign are in this process. When your data are available in the archive, you will receive an email from the SSC (help@spitzer.caltech.edu) notifying you that your data are ready for pickup. If it is the first observation from your program, you will first receive an email with your username and password so that you can access your data via Leopard. Note that your Leopard password is not the same as your program's Spot password!

We deploy new versions of our online pipeline software about every six months, Fall and Spring. For example, S15 comes out in Fall 2006, and S16 is anticipated for Spring 2007. This online history keeps track of all of the major changes for each pipeline version.

Understandably, if a new pipeline version provides substantial improvements, we then want to reprocess the whole archive with these new pipelines. This process also takes time. For IRS, the entire mission so far can be reprocessed in a few weeks. For IRAC and MIPS, it takes much longer. Keep in mind that while we are reprocessing, the firehose of new data does not stop!

Not every pipeline version triggers a reprocessing, and not every instrument need be reprocessed at the same time. Watch the pipeline reprocessing and archiving status page to find out for a given campaign which pipeline version has most recently been run.

Since a new pipeline version provides improvements, you might naturally ask how you can get your own data reprocessed. Right now, we do not have the resources to support individual reprocessing requests; you need to wait until your campaign gets reprocessed with the new pipelines.

Ultimately, our archive will be smart enough to reprocess individual observations on the fly, as users request them through Leopard, thereby ensuring that all data are processed with the most recent pipelines as soon as anyone accesses the data. We will provide updates on this as our plans progress.