MefPrep

Prepare files for combining with Swarp

Space Telescope Science Institute

Command Line Usage

MefPrep.py [-h] [-all] [-finish] [-back_none] [-back_min] [-np N]
           [-zp COLUMN] FIELD [FIELD ...]

Rescales individual DECam CCD images to a common flux scale (1 DN = mag 28) and optionally subtracts an initial background estimate. Output files are written to DECam_PREP/{field}/data/.

Prerequisites: MEF files downloaded into the standard directory structure and MefSum.py run to create the Summary/{field}_mef.tab inventory tables.

Optional Arguments

-h: Print this help and exit.
-all: Process all fields found under DECam_MEF/ (interactive confirmation required).
-finish: Skip output files that already exist (do not reprocess).
-back_none: Do not subtract any background estimate.
-back_min: Subtract the minimum CCD background across all detectors in each exposure (instead of the default mode estimate).

-np N: Number of parallel worker processes (default: 1).
-zp COLUMN: Use empirical zero points from the named column in Summary/{field}_mef.tab (e.g. -zp ZP_smash_r) instead of the MAGZERO header keyword. The value must satisfy 20 < ZP < 35; if the column is absent or the value is invalid, the script falls back to the header MAGZERO with a warning. Generates ZP_USE and ZP_SRC keywords in every output header.

Output Header Keywords

ZP_USE – zero point actually applied for flux scaling
ZP_SRC – source of that zero point (column name or MAGZERO)
XFACTOR – multiplicative scale factor applied to pixel data
SUB_BACK – TRUE / FALSE depending on whether background was subtracted
GAIN, SATURAT, RDNOISE – rescaled detector properties

Examples

Standard single-field run with background subtraction (mode):

MefPrep.py -np 8 LMC_c42

Re-run using empirical SMASH r-band zero points:

MefPrep.py -np 8 -zp ZP_smash_r LMC_c42

Skip already-processed files, no background subtraction:

MefPrep.py -finish -back_none LMC_c42

Notes

Unlike MefSum.py, individual files are processed in separate threads so all requested cores are typically utilised. If too many files are open simultaneously, reduce -np.

Version History

230513 ksl: Coding begun
230621 ksl: Revised so that the default is to subtract background
260608 ksl: Add -zp COLUMN to use empirical zero points from Summary/_mef.tab instead of the header MAGZERO. ZP_USE and ZP_SRC written to output headers. Fixed pre-existing bug where computed back value was not passed to prep_one_det.

Attributes

`CCDDIR`
`MEFDIR`
`SUMDIR`

Functions

`get_no_jobs`(jobs)	Check how many jobs are running
`prep_one_det`([filename, ext, prepdir, back, redo, ...])	Prepare one file for combining with swarp. This version scales by MAGZERO
`prep_one_field`([field, back_type, redo, outdir, zp_col])	Prep a field with a single processor
`prep_one_mef`([field, root, back_type, redo, outdir, ...])	Split the mef files into their individual extenstions and prepare them for the downstream parts of the processing
`steer`(argv)	This is just a steering routine
`xprep_one_field`([field, back_type, redo, outdir, ...])	Process the mef files in a field usingwith using more than one core.

Module Contents

MefPrep.CCDDIR = 'DECam_CCD/'

MefPrep.MEFDIR = 'DECam_MEF/'

MefPrep.SUMDIR = 'Summary/'

MefPrep.get_no_jobs(jobs): Check how many jobs are running

MefPrep.prep_one_det(filename='DECam_MEF/LMC_c42/mef/c4d_211111_024404_ooi_N673_v1.fits.fz', ext='S2', prepdir='DECam_PREP/LMC_c42/T07/', back=0, redo=True, magzero=None, zp_col='')

Prepare one file for combining with swarp. This version scales by MAGZERO where 1DN will be a flux corresponding to mag 28 History:

230504 - Removed .fz from output file names. Note that this does not mean that the data is not compressed. Added extra keywords to primary header of output file that may be useful for swarp, including the rescaled saturation level

230611 - Addapted from prep_one_file. This version allows for several backgroud options, based on information assembled by MEFSum

:

MefPrep.prep_one_field(field='LMC_c42', back_type='none', redo=False, outdir='', zp_col='')

Prep a field with a single processor

This routine identifies the mefs to be processed when one is in single processor mode

MefPrep.prep_one_mef(field='LMC_c42', root='c4d_190109_061931_ooi_N662_v1', back_type='min', redo=False, outdir='', zp_col='')

Split the mef files into their individual extenstions and prepare them for the downstream parts of the processing

This version stores all of the actual output files in a single directory for each field. Setup_Tiles (will be modified for the new file structure

Note that background subtraction here, if is carriout out, subtractis the same value from all of the CCDs in a single image.

What is done is slightly confusing. MEFSum.py has estimated a different background in each of the CCDs in the exposure and stored this in the det file. Here we take all of those measurements for the mef image and either take the median of those values or the minimum.

MefPrep.steer(argv): This is just a steering routine

MefPrep.xprep_one_field(field='LMC_c42', back_type='none', redo=False, outdir='', nproc=4, zp_col='')

Process the mef files in a field usingwith using more than one core.

This routine creates a bundle of files to process at once. This is necessary because multiprocessing opens many files (note that in linux everything is a file, so this does not refer to fits files), and this can exceed the file limit. The sympton of this is a message that says too many files are open. If one sees this message, one may need to reduce n below.