wiki:Archive

Archive

Servers

The archive is available from

  • ratir.astroscu.unam.mx
  • ratir.astrosen.unam.mx
  • mira.la.asu.edu

The examples below use ratir.astroscu.unam.mx, but you could equally use one of the other servers.

Access

The archive is accessed using the rsync program, which is installed by default on Mac OS X and Ubuntu. If it isn't installed on your computer, consult your computer support staff.

Contents

The archive contains the following files:

  • log files;
  • proposal identifier files;
  • image header files; and
  • image FITS files.

The image header files are the FITS headers with newlines inserted every 80 characters.

The image FITS files are compressed losslessly using bzip2 and can be uncompressed using the bunzip2 program, which is installed by default on Mac OS X and Ubuntu. If it isn't installed on your computer, consult your computer support staff.

Log Files

The log files list all science and calibration images take by the instrument in chronological order.

The log files are available as a plain text file with fixed-width column (for humans) and a CSV file (for spreadsheets and other programs).

To obtain the log files, use:

rsync rsync://ratir.astroscu.unam.mx/public/log.txt .

or

rsync rsync://ratir.astroscu.unam.mx/public/log.csv .

In both files, the columns are:

  1. The UTC date and time of the start of the exposure.
  2. The proposal identifier.
  3. The visit identifier.
  4. The name of the image FITS file.
  5. The camera (C0, C1, C2, or C3).
  6. The image type (object, bias, dark, flat).
  7. The exposure time, in seconds.
  8. The filter name.
  9. The J2000 RA at the start of the exposure, in degrees.
  10. The J2000 declination at the start of the exposure, in degrees.
  11. The hour angle at the start of the exposure, in degrees.
  12. The airmass at the start of the exposure.
  13. Whether the data is public ("true") or not ("false").

In both files, the rows are ordered by the date and time of the start of the exposure.

The log files contain only limited information on the images; for more detailed information, you can download the public image header files.

Proposal Identifier Files

The proposal identifier files list all science and calibration proposal identifiers.

The proposal identifier files as a plain text file with fixed-width column (for humans) and a CSV file (for spreadsheets and other programs).

To obtain the proposal identifier files, use:

rsync rsync://ratir.astroscu.unam.mx/public/proposalidentifiers.txt .

or

rsync rsync://ratir.astroscu.unam.mx/public/proposalidentifiers.csv .

In both files, the columns are:

  1. The proposal identifier.
  2. Whether the data have a proprietary period ("true") or not ("false").
  3. The name of the PI.
  4. The title of the proposal.

Public Files

Public files are:

  • All image FITS files from calibration proposals.
  • All image FITS files from science proposals after the propriety period of two years.
  • All image header files, even those from science proposals during the propriety period of two years.

Anyone can obtain public files using:

rsync -av rsync://ratir.astroscu.unam.mx/public/<proposalid> .

The files organized into directories as:

<proposalid>/<visitid>/<file>

in which <proposalid> is a proposal identifier and <visitid> is a visit identifier.

Calibration Files. Please Read

The proposalidentifiers file (see above) contains project identifiers for bias (2012A-0006), darks (2012A-0007) and flats (2012A-0008). The 2012 is correct, that only refers to the original identifier for calibration files. All calibration files are stored as visit = 0, which adds a subdirectory /0/ in the path for the rsync command. Bias files also have an identification letter "b" after the detector name (C0, C1, etc) before the .fits extension in the filename. For darks, the identification letter is "d", and for flats the identification letter is "f". Following this same logic, data files have an identification letter "o", for "object".

In the following examples, we show how to download calibration files in bulk

Example 1

To obtain all files corresponding to images taken for proposal 2012A-0006 (which is a calibration project), use:

rsync -av rsync://ratir.astroscu.unam.mx/public/2012A-0006 .

As you can see, this command will download ALL bias files ever taken with RATIR. The following example will help you to refine this selection.

Example 2

To obtain a more specific group of files, you can specify a visit and use wildcards to limit dates. For instance, to obtain all bias frames obtained on the night of august 12th, 2013, for the C1 detector only, you may write:

rsync -av rsync://ratir.astrosen.unam.mx/public/2012A-0006/0/20130813*C1b.* .

Notice that, as discussed above, the proposal identifier is 2012A-0006 (has nothing to do with the actual date of observation). Also notice the detector name, C1, and the "b", which is probably not necessary as 2012A-0006/0 will only contain bias images (but it could help the user to remember what a command does in an old script, for example).

Proprietary Files

Science proposal PIs can access all files obtained for their proposals (both proprietary files and public files) using:

rsync -av rsync://<proposalid>@ratir.astroscu.unam.mx/<proposalid> .

Here, <proposalid> is a proposal identifier. rsync will prompt for a password. This password will be distributed to the PI by Alan Watson <alan@…>. The PIs can then distribute the password as they wish.

The files organized into directories as:

<proposalid>/<visitid>/<file>

in which <proposalid> is a proposal identifier and <visitid> is a visit identifier.

Example 2

To obtain all files corresponding to images taken for proposal 2012A-1003, use:

rsync -av rsync://2012A-1003@ratir.astroscu.unam.mx/2012A-1003 .

Example 3

To obtain files corresponding to images taken on a specific date (March 03 UT in this particular example) for all completed visits of an imaginary proposal 2017A-1099:

rsync -rav --include='*/' --include='*/20170310*bz2' --exclude='*' rsync://2017A-1099@ratir.astrosen.unam.mx/2017A-1099 .

And, same case but downloading only data from visit # 4 (which, by the way, corresponds to "visitidentifier" in the phase2 definitions)

rsync -rav --include='*/' --include='4/20170310*bz2' --exclude='*' rsync://2017A-1099@ratir.astrosen.unam.mx/2017A-1099 .

Please notice the difference between adding a space and a period at the end of the command, which will indicate rsync to download the files. If you don't include the space and the period, it will only list the files. You can use this last option to test your command without having to make downloads until you are sure of the right set of files.

Last modified 5 months ago Last modified on Feb 7, 2017 9:31:26 PM