Reduction

This page describes the standard data reduction process for DECam images using kred. All scripts should be run from the top-level working directory containing the DECam_MEF/ folder.

Overview

The reduction pipeline processes multi-extension FITS (MEF) files through a sequence of steps to produce calibrated, background-matched tile mosaics. There are two main workflows:

Standard processing: Creates tile mosaics without background matching
Background matching: Improves background uniformity across overlapping images

Both workflows use the same initial preparation steps and share many scripts.

Note

Several programs have a -bsub flag that changes which data they operate on. The descriptions below assume -bsub is NOT used unless explicitly stated. Background-subtracted processing is covered in a separate section.

Standard Processing

To process data without background matching, run the following programs in order:

MefCheck.py

Run whenever files have been added to the data repository. Performs basic checks on files to ensure they can be read and have required header keywords. Also checks for duplicate files that could cause issues downstream.

MefCheck.py LMC_c42

MefSum.py

Summarizes the MEF files in a field. Creates a Summary/ directory and writes two astropy tables containing summary information. Files missing required keywords (e.g., MAGZERO) are placed in a separate summary file and excluded from further processing.

MefSum.py -np 8 LMC_c42    # Process single field with 8 processes
MefSum.py -all             # Process all fields

The routine also checks that filter/exposure time combinations appear in the instrument configuration file (config/DeMCELS_images.txt).

MefPrep.py

Rescales all images in a field, optionally subtracts background, and stores results in DECam_PREP/. For example, processing LMC_c45 creates data in DECam_PREP/LMC_c45/data/.

MefPrep.py -np 8 LMC_c42

By default, subtracts the median value of the mode across all CCDs from each exposure. The flux scaling uses the MAGZERO keyword from each MEF header as supplied by the NOIRLAB community pipeline, placing all images on a scale where 1 DN corresponds to magnitude 28. This is the standard approach and is sufficient in most cases.

If you need to verify that the MAGZERO values are mutually consistent across exposures, or to replace them with empirically-derived zero points, see Relative Photometry.

Important

If you plan to use the -use_all_data option in SetupTile (to include all possible images in a tile), you must run MefPrep on all data first. Otherwise, only images processed by MefPrep will be included.

SetupTile.py

Identifies CCD images that need to be processed for each tile. Creates tables in Summary/ identifying which CCDs contribute to each tile, and creates directories like DECam_PREP/LMC_c45/Tile01/ containing links to the appropriate data files.

SetupTile.py -all LMC_c42

This routine uses two configuration files from the config/ directory:

MC_tiles.txt: Defines tile centers and geometries. Modify this to create tiles for different regions.
DeMCELS_images.txt: Defines which filter/exposure combinations to include in final images. Multiple exposure times can be combined for the same filter.

The -use_all_data option examines all CCD images in DECam_PREP/ to find any that overlap the tile area, not just those from the field’s exposure sequence.

SwarpSetup.py

Creates directories and input files (.run and .default files) for running SWarp to create tile mosaics. Creates one tile image per filter and exposure time.

SwarpSetup.py -all LMC_c42

Output is written to DECam_SWARP/.

Swarp.py

Runs all SWarp commands created by SwarpSetup.

Swarp.py -all LMC_c42

SwarpEval.py

Creates evaluation figures for the swarped images. Scaling is designed to highlight flaws. Images are stored in DECam_SWARP/field/eval/.

SwarpEval.py -all DECam_SWARP LMC_c42

CleanStars.py

Creates continuum-subtracted images by generating emission-line-free images from N708 and r-band data, then subtracting these from the emission line images. Results are stored in DECam_SUB/field/tile/.

CleanStars.py -all LMC_c42

Background Matching

The standard processing does not match backgrounds between overlapping images. For improved background uniformity, run the following additional scripts after MefPrep (the other standard processing steps are not required first).

FindOverlaps.py

Uses header information (corner coordinates) to identify which images overlap. Only considers overlaps between images with the same filter and exposure time.

FindOverlaps.py -all LMC_c42

BackPrep.py

Creates a DECam_BACK/ directory structure for storing background estimation images. Projects all prepared images onto a common WCS with larger pixels to keep file sizes manageable. With the -run option, executes SWarp to create these images.

BackPrep.py -all -run LMC_c42

Warning

The files created by BackPrep are large. Processing all of LMC_c42 creates approximately 420 GB of images.

BackStats.py

Determines the background in overlap regions using images from BackPrep. Produces a single file per tile containing background estimates.

BackStats.py -all -np 8 -rm LMC_c42

The -rm option removes BackPrep files after processing to recover disk space.

BackCalc.py

Uses the differences calculated by BackStats to determine optimal offsets for each image to produce better background matching.

BackCalc.py -all LMC_c42

BackSub.py

Applies the offsets from BackCalc to create new images with corrected backgrounds. Output appears in DECam_PREP2/field/tile/.

BackSub.py -all -np 8 LMC_c42

Re-running SwarpSetup and Swarp

After background subtraction, re-run SwarpSetup and Swarp with the -bsub flag to create mosaics from the background-corrected images:

SwarpSetup.py -all -bsub LMC_c42
Swarp.py -all -bsub LMC_c42

This creates output in DECam_SWARP2/.

Finally, run CleanStars with the -bsub flag to create continuum-subtracted images from the background-matched data:

CleanStars.py -all -bsub LMC_c42

Output is written to DECam_SUB2/.

Complete Example

The following command sequence processes a field with full background matching:

# Initial preparation
MefSum.py -np 8 LMC_c42
MefPrep.py -np 8 LMC_c42

# Setup tiles
SetupTile.py -all LMC_c42
SwarpSetup.py -all LMC_c42

# Optional: create mosaics without background matching
# Swarp.py -all LMC_c42
# SwarpEval.py -all DECam_SWARP LMC_c42

# Background matching
FindOverlaps.py -all LMC_c42
BackPrep.py -all -run LMC_c42
BackStats.py -all -np 8 -rm LMC_c42
BackCalc.py -all LMC_c42
BackSub.py -all -np 8 LMC_c42

# Create background-matched mosaics
SwarpSetup.py -all -bsub LMC_c42
Swarp.py -all -bsub LMC_c42
SwarpEval.py -all DECam_SWARP2 LMC_c42

# Continuum subtraction
CleanStars.py -all -bsub LMC_c42
SwarpEval.py -all DECam_SUB2 LMC_c42

Tip

Save these commands in a script file and comment out completed sections as you progress through the reduction.