GaiaCat

GaiaCat - GAIA Database Interaction Module.

Space Telescope Science Institute

Synopsis

Retrieve GAIA catalog data either from the archive or from a local file. Can determine the search center from a FITS image WCS or from explicit RA/Dec coordinates.

Command Line Usage

GaiaCat.py [-h] [-archive] [-redo] [-gfile FILENAME] [-rad DEGREES] [-out OUTROOT]
           input.fits or RA Dec

Arguments

input: Either a FITS file (to extract center from WCS), or RA in degrees. If RA is given, dec must also be provided.

RA, Dec RA and DEC of field center

or

whatever.fits a fits file with a WCS, note that size is not taken from WCS

Options

-h: Print this help message and exit
-archive: Retrieve from GAIA archive instead of local file. By default, tries local file first, then falls back to archive.
-redo: Force re-retrieval from the archive even if the output file already exists. Implies -archive.

-gfile FILENAME: Name of local GAIA catalog file. Searches locally first, then in $KRED/xdata/. Default: Gaia_MagClouds.fits
-rad DEGREES: Search radius in degrees. Default: 0.5
-out OUTROOT: Output filename root. Default: derived from RA/Dec or FITS filename

Description

This module provides functions to retrieve GAIA DR3 photometric and spectroscopic data. Catalog cone searches query MAST first (astroquery.mast.Catalogs) because it is synchronous and avoids the async-job storage bugs on the ESA server; ESA (astroquery.gaia) is retained as a fallback. The primary functions are:

get_gaia_from_archive(): Query MAST (primary) or ESA (fallback) with cone search
get_gaia_from_file(): Extract from a local pre-downloaded catalog
get_gaia_spectra_batch(): Bulk-retrieve Gaia XP spectra for a list of source IDs, pre-populating the per-star cache used by get_gaia_spec()

Notes

This module requires the astroquery package for archive access. All archive-dependent functions will provide clear error messages if astroquery is not available or if GAIA services are unreachable.

History:

240318 ksl Coding begun 240527 ksl Speed up the catalog matching 251105 ksl Split finding sources in an image from doing photometry 251130 ksl Cleaned up to focus on GAIA catalog interaction 251130 ksl Robust handling of astroquery import vs service availability 251211 ksl Handle gaiaxpy version compatibility (2.1.1 vs 2.1.2) 250116 ksl Added command-line steering with FITS/WCS support 260429 ksl Switch catalog queries to MAST as primary source; ESA async endpoint retained as fallback. Add get_gaia_spectra_batch() for bulk XP spectrum retrieval. 260502 ksl Add redo parameter to get_gaia_from_file; skip re-extraction when output file already exists.

Example Usage

Command line with FITS file:

$ GaiaCat.py myimage.fits -rad 0.3

Command line with RA/Dec:

$ GaiaCat.py 84.925 -66.274 -rad 0.5

Force archive retrieval (skip local file):

$ GaiaCat.py 84.925 -66.274 -archive -rad 0.3

Force re-retrieval from archive even if output file exists:

$ GaiaCat.py 84.925 -66.274 -redo -rad 0.3

Use a different local catalog file:

$ GaiaCat.py 84.925 -66.274 -gfile my_gaia_catalog.fits

Python usage:

>>> from GaiaCat import get_gaia_from_archive
>>> outfile = get_gaia_from_archive(ra=84.925, dec=-66.274, rad_deg=0.3)
>>> print(f"Catalog saved to: {outfile}")

Functions

`get_gaia`(ra, dec, size)	Retrieve data from a file if possible, but if it is not possible try
`get_gaia_from_archive`([ra, dec, rad_deg, outroot, ...])	Query the GAIA archive with cone search and automatic retry on errors.
`get_gaia_from_archive_old`([ra, dec, rad_deg, outroot, ...])	Query the GAIA archive with cone search (legacy version without retry).
`get_gaia_from_file`([ra, dec, size_deg, filename, ...])	Extract GAIA sources from a local catalog file within a sky region.
`get_gaia_spec`(gaiaID[, GAIA_CACHE_DIR, redo])	Retrieve a GAIA XP spectrum with robust version handling.
`get_gaia_spectra_batch`(source_ids[, GAIA_CACHE_DIR, redo])	Retrieve XP spectra for multiple Gaia sources in a single archive call.
`get_wcs_center`(fitsfile)	Extract the center RA/Dec from a FITS file WCS.
`load_Gaia`([probe_service, credentials_file])	Return the astroquery.gaia.Gaia class with service validation.
`old_get_gaia_spec`(gaiaID[, GAIA_CACHE_DIR, redo])	Load or download a GAIA XP spectrum and convert to physical units.
`random_rows`(tab, nrows[, seed])	Randomly select rows from an Astropy Table without replacement.
`simple_test`()	Run basic functionality tests for GaiaCat module.
`steer`(argv)	Execute command-line interface for GaiaCat module.
`unique_rows_within_tol`(tab[, tol])	Return unique rows based on approximate equality of position and size.

Module Contents

GaiaCat.get_gaia(ra, dec, size): Retrieve data from a file if possible, but if it is not possible try to get the data from the archive

GaiaCat.get_gaia_from_archive(ra=84.92500000000001, dec=-66.27416666666667, rad_deg=0.3, outroot='', nmax=-1, redo=False, max_retries=3, retry_delay=5)

Query the GAIA archive with cone search and automatic retry on errors.

This function queries the GAIA DR3 archive for photometric data within a specified cone, with robust handling of network errors through automatic retries.

Parameters

rafloat, optional: Right Ascension of the cone center in degrees (J2000). Default is 84.925.
decfloat, optional: Declination of the cone center in degrees (J2000). Default is -66.274.
rad_degfloat, optional: Cone search radius in degrees. Default is 0.3°.
outrootstr, optional: Root name for the output file. If empty, constructs from RA and Dec.
nmaxint, optional: Maximum number of rows to return. If -1 (default), returns all matching sources.
redobool, optional: If True, re-query even if the output file exists. Default is False.
max_retriesint, optional: Maximum number of retry attempts for failed queries. Default is 3.
retry_delayfloat, optional: Delay in seconds between retry attempts. Default is 5.

Returns

outfilestr: Path to the output file containing the catalog, or empty list if no sources were retrieved.

Raises

RuntimeError: If astroquery is not installed or GAIA services are unavailable.
IncompleteRead: If max retries are exceeded due to persistent network errors.

Notes

The function automatically renames columns to a simplified convention: - ra → RA - dec → Dec - source_id → Source_name - phot_g_mean_mag → G - phot_bp_mean_mag → B - phot_rp_mean_mag → R - teff_gspphot → teff - logg_gspphot → log_g - distance_gspphot → D - has_xp_continuous → xp_spec_exists (boolean flag)

Output is written to Gaia/Gaia.<outroot>.fits in FITS table format.

Examples

Basic cone search with default retry behavior:

>>> outfile = get_gaia_from_archive(
...     ra=180.0, dec=45.0,
...     rad_deg=0.5,
...     outroot='ngc1234'
... )
>>> print(f"Catalog written to: {outfile}")

Query with custom retry parameters:

>>> outfile = get_gaia_from_archive(
...     ra=10.0, dec=-5.0,
...     rad_deg=0.1,
...     max_retries=5,
...     retry_delay=10
... )

GaiaCat.get_gaia_from_archive_old(ra=84.92500000000001, dec=-66.27416666666667, rad_deg=0.3, outroot='', nmax=-1, redo=False)

Query the GAIA archive with cone search (legacy version without retry).

Deprecated since version 251211: Use get_gaia_from_archive() instead, which includes retry logic for improved reliability.

This is the original cone search function without automatic retry handling. It is retained for backward compatibility but is not recommended for new code.

Parameters

rafloat, optional: Right Ascension of the cone center in degrees (J2000). Default is 84.925.
decfloat, optional: Declination of the cone center in degrees (J2000). Default is -66.274.
rad_degfloat, optional: Cone search radius in degrees. Default is 0.3°.
outrootstr, optional: Root name for the output file. If empty, constructs from RA and Dec.
nmaxint, optional: Maximum number of rows to return. If -1 (default), returns all sources.
redobool, optional: If True, re-query even if the output file exists. Default is False.

Returns

outfilestr: Path to the output file, or empty list if no sources retrieved.

Raises

RuntimeError: If astroquery is not installed or GAIA services are unavailable.

Parameters

rafloat, optional: Right Ascension of the field center in degrees (J2000). Default is 84.925 (approximately LMC).
decfloat, optional: Declination of the field center in degrees (J2000). Default is -66.274 (approximately LMC).
size_degfloat, optional: Size of the extraction region in degrees. Default is 0.3°.
filenamestr, optional: Name of the local GAIA catalog file. Searches locally first, then in $KRED/xdata/. Default is ‘Gaia_MagClouds.fits’.
outrootstr, optional: Root name for the output file. If empty, constructs from RA and Dec.
redobool, optional: If True, re-extract and overwrite the output file even if it already exists. Default is False.

Returns

outfilestr: Path to the output FITS file containing the extracted sources.

Raises

IOError: If the input file cannot be located in either the current directory or $KRED/xdata/.

Notes

The function performs a rectangular (RA, Dec) selection rather than a cone search. The RA range is adjusted for declination to approximate equal angular sizes in both dimensions.

Output files are written to a Gaia/ subdirectory in FITS format.

Examples

Extract sources from a local catalog:

>>> outfile = get_gaia_from_file(
...     ra=150.0, dec=-30.0,
...     size_deg=0.5,
...     filename='gaia_dr3_subset.fits'
... )
>>> print(f"Extracted catalog: {outfile}")

GaiaCat.get_gaia_spec(gaiaID, GAIA_CACHE_DIR='./GaiaSpec', redo=False)

Retrieve a GAIA XP spectrum with robust version handling.

This is the recommended function for retrieving GAIA XP spectra. It provides improved error handling and compatibility with different gaiaxpy versions.

Parameters

gaiaIDint or str: GAIA DR3 source identifier.
GAIA_CACHE_DIRstr, optional: Directory path for caching spectrum files. Default is ‘./GaiaSpec’. Will be created if it doesn’t exist.
redobool, optional: If True, re-download the spectrum even if cached. Default is False.

Returns

wavenumpy.ndarray: Wavelength array in Angstroms.
fluxnumpy.ndarray: Flux array in erg/s/cm²/Å, or empty list if retrieval fails.

Notes

This function wraps old_get_gaia_spec() with additional error handling to gracefully handle failures in spectrum retrieval or conversion.

Examples

Retrieve a spectrum with error handling:

>>> result = get_gaia_spec(4658615927801509760)
>>> if isinstance(result, list):
...     print("Spectrum retrieval failed")
... else:
...     wave, flux = result
...     print(f"Retrieved spectrum with {len(wave)} points")

GaiaCat.get_gaia_spectra_batch(source_ids, GAIA_CACHE_DIR='./GaiaSpec', redo=False)

Retrieve XP spectra for multiple Gaia sources in a single archive call.

Uses gaiaxpy.calibrate() on the full list, then splits and caches results in the same per-star format used by get_gaia_spec(), so subsequent calls to get_gaia_spec() for the same IDs will be served from cache.

Parameters

source_idslist of int: Gaia DR3 source identifiers.
GAIA_CACHE_DIRstr, optional: Cache directory. Default is ‘./GaiaSpec’.
redobool, optional: If False (default), skip IDs already in cache.

Returns

n_okint: Number of spectra successfully retrieved and cached.
n_failint: Number of IDs for which retrieval failed.

GaiaCat.get_wcs_center(fitsfile)

Extract the center RA/Dec from a FITS file WCS.

Parameters

fitsfilestr: Path to FITS file with valid WCS in header

Returns

rafloat: Right Ascension of image center in degrees
decfloat: Declination of image center in degrees

Raises

ValueError: If WCS cannot be extracted from the FITS file

GaiaCat.load_Gaia(probe_service=True, credentials_file=None)

Return the astroquery.gaia.Gaia class with service validation.

This function provides a controlled import of the GAIA query interface, distinguishing between installation issues and service availability problems.

Parameters

probe_servicebool, optional: If True (default), perform a minimal network check to verify that external GAIA services are reachable. If False, only import the Gaia class without network validation.
credentials_filestr, optional: Path to file containing Gaia credentials (username on line 1, password on line 2). If None, defaults to ~/.gaia_credentials if it exists.

Returns

Gaiaclass: The Gaia class from astroquery.gaia, ready for use in queries.

Raises

RuntimeError: If astroquery is not installed, or if probe_service=True and the GAIA external services are unavailable or unreachable.

Notes

This function also configures astropy logging to reduce verbosity during GAIA operations by setting the log level to ERROR and disabling IERS auto-downloads.

If a credentials file exists, the function will automatically login to the Gaia archive.

Examples

Load GAIA with service validation:

>>> Gaia = load_Gaia(probe_service=True)
>>> # Now safe to use Gaia for queries

Load GAIA without immediate service check:

>>> Gaia = load_Gaia(probe_service=False)
>>> # Service errors will occur at first query attempt

GaiaCat.old_get_gaia_spec(gaiaID, GAIA_CACHE_DIR='./GaiaSpec', redo=False)

Load or download a GAIA XP spectrum and convert to physical units.

This function retrieves the XP continuous spectrum for a given GAIA source, caching it locally for future use. Spectra are converted from the archive format to erg/s/cm²/Å.

Parameters

gaiaIDint or str: GAIA DR3 source identifier.
GAIA_CACHE_DIRstr, optional: Directory path for caching spectrum files. Default is ‘./GaiaSpec’. Will be created if it doesn’t exist.
redobool, optional: If True, re-download the spectrum even if cached. Default is False.

Returns

wavenumpy.ndarray: Wavelength array in Angstroms.
fluxnumpy.ndarray: Flux array in erg/s/cm²/Å, or empty list [] if the spectrum cannot be retrieved.

Notes

This function was adapted from the lvmdrp package. It handles two different CSV formats produced by different versions of gaiaxpy (2.1.1 vs 2.1.2): - Newer versions use numpy representation strings - Older versions use comma-separated values

The function requires the requests and gaiaxpy packages to download new spectra, but can read cached spectra without these dependencies.

Warning

This is the legacy spectrum retrieval function. Consider using get_gaia_spec() for the current recommended approach.

Examples

Retrieve and plot a GAIA spectrum:

>>> import matplotlib.pyplot as plt
>>> wave, flux = old_get_gaia_spec(4658615927801509760)
>>> plt.plot(wave, flux)
>>> plt.xlabel('Wavelength (Å)')
>>> plt.ylabel('Flux (erg/s/cm²/Å)')
>>> plt.show()

GaiaCat.random_rows(tab, nrows, seed=None)

Randomly select rows from an Astropy Table without replacement.

Parameters

tabastropy.table.Table: Input table from which to select rows.
nrowsint: Number of rows to randomly select. Must be less than or equal to the length of the table.
seedint, optional: Random seed for reproducibility. If None, uses system entropy.

Returns

subtabastropy.table.Table: New table containing the randomly selected rows.

Warnings

If nrows exceeds the table length, a warning is printed and the entire table is returned unchanged.

Examples

Select 10 random rows from a catalog:

>>> from astropy.table import Table
>>> catalog = Table.read('gaia_stars.fits')
>>> random_subset = random_rows(catalog, 10, seed=42)
>>> len(random_subset)
10

GaiaCat.simple_test()

Run basic functionality tests for GaiaCat module.

This function performs a simple test of core functionality: 1. Retrieves a small catalog from the GAIA archive 2. Retrieves a spectrum for a known source

Returns

None

Notes

This test requires network access and will create files in the Gaia/ and GaiaSpec/ directories.

Examples

Run the test suite:

>>> simple_test()
Here we just check if a small amount of the SW works
Can we retrieve a catalog of stars
...
If there were no errors we have success

GaiaCat.steer(argv)

Execute command-line interface for GaiaCat module.

Parses command-line arguments and retrieves GAIA catalog data. By default, tries local file first, then falls back to archive.

Parameters

argvlist of str: Command-line arguments

Returns

str: Path to output file, or None if error

GaiaCat.unique_rows_within_tol(tab, tol=0.01)

Return unique rows based on approximate equality of position and size.

This function identifies and removes duplicate rows where RA, Dec, and Size are within a specified tolerance, keeping one representative row per group.

Parameters

tabastropy.table.Table: Table containing at minimum the columns ‘RA’, ‘Dec’, and ‘Size’, all in degrees.
tolfloat, optional: Matching tolerance in degrees. Default is 0.01° (~36 arcseconds).

Returns

unique_tabastropy.table.Table: New table containing one representative row per unique group.

Notes

The algorithm uses a greedy approach: rows are processed sequentially, and the first row in each group is kept as the representative.

Examples

Remove near-duplicate observations:

>>> catalog = Table.read('observations.fits')
>>> unique_catalog = unique_rows_within_tol(catalog, tol=0.001)
>>> print(f"Reduced from {len(catalog)} to {len(unique_catalog)} rows")

GaiaCat

Synopsis

Command Line Usage

Arguments

Options

Description

Notes

Example Usage

Functions

Module Contents

Parameters

Returns

Raises

Notes

Examples

Parameters

Returns

Raises

See Also

Parameters

Returns

Raises

Notes

Examples

Parameters

Returns

Notes

Examples

Parameters

Returns

Parameters

Returns

Raises

Parameters

Returns

Raises

Notes

Examples

Parameters

Returns

Notes

Examples

Parameters

Returns

Warnings

Examples

Returns

Notes

Examples

Parameters

Returns

Parameters

Returns

Notes

Examples