ImageFileCollection

class ccdproc.ImageFileCollection(location=None, keywords=None, find_fits_by_reading=False, filenames=None, glob_include=None, glob_exclude=None, ext=0)[source]

Bases: object

Representation of a collection of image files.

The class offers a table summarizing values of keywords in the FITS headers of the files in the collection and offers convenient methods for iterating over the files in the collection. The generator methods use simple filtering syntax and can automate storage of any FITS files modified in the loop using the generator.

Parameters:
locationstr or None, optional

Path to directory containing FITS files. Default is None.

keywordslist of str, ‘*’ or None, optional

Keywords that should be used as column headings in the summary table. If the value is or includes ‘*’ then all keywords that appear in any of the FITS headers of the files in the collection become table columns. Default value is ‘*’ unless info_file is specified. Default is None.

find_fits_by_reading: bool, optional

If True, read each file in location to check whether the file is a FITS file and include it in the collection based on that, rather than by file name. Compressed files, e.g. image.fits.gz, will NOT be properly detected. Will be ignored if `filenames` is not ``None``.

filenames: str, list of str, or None, optional

List of the names of FITS files which will be added to the collection. The filenames may either be in location or the name can be a relative or absolute path to the file. Default is None.

glob_include: str or None, optional

Unix-style filename pattern to select filenames to include in the file collection. Can be used in conjunction with glob_exclude to easily select subsets of files in the target directory. Default is None.

glob_exclude: str or None, optional

Unix-style filename pattern to select filenames to exclude from the file collection. Can be used in conjunction with glob_include to easily select subsets of files in the target directory. Default is None.

ext: str or int, optional

The extension from which the header and data will be read in all files.Default is 0.

Raises:
ValueError

Raised if keywords are set to a combination of ‘*’ and any other value.

Attributes Summary

ext

str or int, The extension from which the header and data will be read in all files.

files

list of str, Unfiltered list of FITS files in location.

glob_exclude

str or None, Unix-style filename pattern to select filenames to exclude in the file collection.

glob_include

str or None, Unix-style filename pattern to select filenames to include in the file collection.

keywords

list of str, Keywords currently in the summary table.

location

str, Path name to directory containing FITS files.

summary

Table of values of FITS keywords for files in the collection.

Methods Summary

ccds([ccd_kwargs])

Generator that yields each CCDData in the collection.

data([do_not_scale_image_data])

Generator that yields each image in the collection.

files_filtered(**kwd)

Determine files whose keywords have listed values.

filter(**kwd)

Create a new collection by filtering the current collection.

hdus([do_not_scale_image_data])

Generator that yields each HDU in the collection.

headers([do_not_scale_image_data])

Generator that yields each header in the collection.

refresh()

Refresh the collection by re-reading headers.

sort(keys)

Sort the list of files to determine the order of iteration.

values(keyword[, unique])

List of values for a keyword.

Attributes Documentation

ext

str or int, The extension from which the header and data will be read in all files.

files

list of str, Unfiltered list of FITS files in location.

glob_exclude

str or None, Unix-style filename pattern to select filenames to exclude in the file collection.

glob_include

str or None, Unix-style filename pattern to select filenames to include in the file collection.

keywords

list of str, Keywords currently in the summary table.

Setting the keywords causes the summary table to be regenerated unless the new keywords are a subset of the old.

Changed in version 1.3: Added deleter for keywords property.

location

str, Path name to directory containing FITS files.

summary

Table of values of FITS keywords for files in the collection.

Each keyword is a column heading. In addition, there is a column called file that contains the name of the FITS file. The directory is not included as part of that name.

The first column is always named file.

The order of the remaining columns depends on how the summary was constructed.

If a wildcard, * was used then the order is the order in which the keywords appear in the FITS files from which the summary is constructed.

If an explicit list of keywords was supplied in setting up the collection then the order of the columns is the order of the keywords.

Methods Documentation

ccds(ccd_kwargs=None, **kwd)[source]

Generator that yields each CCDData in the collection.

If any of the parameters save_with_name, save_location or overwrite evaluates to True the generator will write a copy of each FITS file it is iterating over. In other words, if save_with_name and/or save_location is a string with non-zero length, and/or overwrite is True, a copy of each FITS file will be made.

Parameters:
save_with_namestr, optional

string added to end of file name (before extension) if FITS file should be saved after iteration. Unless save_location is set, files will be saved to location of the source files self.location. Default is ''.

save_locationstr, optional

Directory in which to save FITS files; implies that FITS files will be saved. Note this provides an easy way to copy a directory of files–loop over the CCDData with save_location set. Default is ''.

overwritebool, optional

If True, overwrite input FITS files. Default is False.

clobberbool, optional

Alias for overwrite. Default is False.

do_not_scale_image_databool, optional

If True, prevents fits from scaling images. Default is True. Default is True.

return_fnamebool, optional

If True, return the tuple (header, file_name) instead of just header. The file name returned is the name of the file only, not the full path to the file. Default is False.

ccd_kwargsdict, optional

Dict with parameters for fits_ccddata_reader. For instance, the key 'unit' can be used to specify the unit of the data. If 'unit' is not given then 'adu' is used as the default unit. See fits_ccddata_reader for a complete list of parameters that can be passed through ccd_kwargs.

regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

Any additional keywords are used to filter the items returned; see files_filtered examples for details.

Returns:
astropy.nddata.CCDData

If return_fname is False, yield the next CCDData in the collection.

(astropy.nddata.CCDData, str)

If return_fname is True, yield a tuple of (CCDData, file name) for the next item in the collection.

data(do_not_scale_image_data=False, **kwd)[source]

Generator that yields each image in the collection.

If any of the parameters save_with_name, save_location or overwrite evaluates to True the generator will write a copy of each FITS file it is iterating over. In other words, if save_with_name and/or save_location is a string with non-zero length, and/or overwrite is True, a copy of each FITS file will be made.

Parameters:
save_with_namestr, optional

string added to end of file name (before extension) if FITS file should be saved after iteration. Unless save_location is set, files will be saved to location of the source files self.location. Default is ''.

save_locationstr, optional

Directory in which to save FITS files; implies that FITS files will be saved. Note this provides an easy way to copy a directory of files–loop over the image with save_location set. Default is ''.

overwritebool, optional

If True, overwrite input FITS files. Default is False.

clobberbool, optional

Alias for overwrite. Default is False.

do_not_scale_image_databool, optional

If True, prevents fits from scaling images. Default is False. Default is True.

return_fnamebool, optional

If True, return the tuple (header, file_name) instead of just header. The file name returned is the name of the file only, not the full path to the file. Default is False.

ccd_kwargsdict, optional

Dict with parameters for fits_ccddata_reader. For instance, the key 'unit' can be used to specify the unit of the data. If 'unit' is not given then 'adu' is used as the default unit. See fits_ccddata_reader for a complete list of parameters that can be passed through ccd_kwargs.

regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

Any additional keywords are used to filter the items returned; see files_filtered examples for details.

Returns:
numpy.ndarray

If return_fname is False, yield the next image in the collection.

(numpy.ndarray, str)

If return_fname is True, yield a tuple of (image, file name) for the next item in the collection.

files_filtered(**kwd)[source]

Determine files whose keywords have listed values.

Parameters:
include_pathbool, keyword-only

If the keyword include_path=True is set, the returned list contains not just the filename, but the full path to each file. Default is False.

regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

**kwd is dict of keywords and values the files must have. The value ‘*’ represents any value. A missing keyword is indicated by value ‘’.

Returns:
filenameslist

The files that satisfy the keyword-value restrictions specified by the **kwd.

Notes

Value comparison is case insensitive for strings, whether matching exactly or matching with regular expressions.

Examples

Some examples for filtering:

>>> keys = ['imagetyp','filter']
>>> collection = ImageFileCollection('test/data', keywords=keys)
>>> collection.files_filtered(imagetyp='LIGHT', filter='R')
>>> collection.files_filtered(imagetyp='*', filter='')

In case you want to filter with keyword names that cannot be used as keyword argument name, you have to unpack them using a dictionary. For example if a keyword name contains a space or a -:

>>> add_filters = {'exp-time': 20, 'ESO TPL ID': 1050}
>>> collection.files_filtered(imagetyp='LIGHT', **add_filters)
filter(**kwd)[source]

Create a new collection by filtering the current collection.

Parameters:
regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

**kwd is dict of keywords and values the files must have. The value ‘*’ represents any value. A missing keyword is indicated by value ‘’.

Returns:
ImageFileCollection

A new collection with the files matched by the arguments to filter.

hdus(do_not_scale_image_data=False, **kwd)[source]

Generator that yields each HDU in the collection.

If any of the parameters save_with_name, save_location or overwrite evaluates to True the generator will write a copy of each FITS file it is iterating over. In other words, if save_with_name and/or save_location is a string with non-zero length, and/or overwrite is True, a copy of each FITS file will be made.

Parameters:
save_with_namestr, optional

string added to end of file name (before extension) if FITS file should be saved after iteration. Unless save_location is set, files will be saved to location of the source files self.location. Default is ''.

save_locationstr, optional

Directory in which to save FITS files; implies that FITS files will be saved. Note this provides an easy way to copy a directory of files–loop over the HDU with save_location set. Default is ''.

overwritebool, optional

If True, overwrite input FITS files. Default is False.

clobberbool, optional

Alias for overwrite. Default is False.

do_not_scale_image_databool, optional

If True, prevents fits from scaling images. Default is False. Default is True.

return_fnamebool, optional

If True, return the tuple (header, file_name) instead of just header. The file name returned is the name of the file only, not the full path to the file. Default is False.

ccd_kwargsdict, optional

Dict with parameters for fits_ccddata_reader. For instance, the key 'unit' can be used to specify the unit of the data. If 'unit' is not given then 'adu' is used as the default unit. See fits_ccddata_reader for a complete list of parameters that can be passed through ccd_kwargs.

regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

Any additional keywords are used to filter the items returned; see files_filtered examples for details.

Returns:
astropy.io.fits.PrimaryHDU, ` astropy.io.fits.ImageHDU`

If return_fname is False, yield the next HDU in the collection.

(astropy.io.fits.PrimaryHDU, ` astropy.io.fits.ImageHDU`, str)

If return_fname is True, yield a tuple of (HDU, file name) for the next item in the collection.

headers(do_not_scale_image_data=True, **kwd)[source]

Generator that yields each header in the collection.

If any of the parameters save_with_name, save_location or overwrite evaluates to True the generator will write a copy of each FITS file it is iterating over. In other words, if save_with_name and/or save_location is a string with non-zero length, and/or overwrite is True, a copy of each FITS file will be made.

Parameters:
save_with_namestr, optional

string added to end of file name (before extension) if FITS file should be saved after iteration. Unless save_location is set, files will be saved to location of the source files self.location. Default is ''.

save_locationstr, optional

Directory in which to save FITS files; implies that FITS files will be saved. Note this provides an easy way to copy a directory of files–loop over the header with save_location set. Default is ''.

overwritebool, optional

If True, overwrite input FITS files. Default is False.

clobberbool, optional

Alias for overwrite. Default is False.

do_not_scale_image_databool, optional

If True, prevents fits from scaling images. Default is True. Default is True.

return_fnamebool, optional

If True, return the tuple (header, file_name) instead of just header. The file name returned is the name of the file only, not the full path to the file. Default is False.

ccd_kwargsdict, optional

Dict with parameters for fits_ccddata_reader. For instance, the key 'unit' can be used to specify the unit of the data. If 'unit' is not given then 'adu' is used as the default unit. See fits_ccddata_reader for a complete list of parameters that can be passed through ccd_kwargs.

regex_matchbool, keyword-only

If True, then string values in the **kwd dictionary are treated as regular expression patterns and matching is done by regular expression search. The search is always case insensitive.

**kwd

Any additional keywords are used to filter the items returned; see files_filtered examples for details.

Returns:
astropy.io.fits.Header

If return_fname is False, yield the next header in the collection.

(astropy.io.fits.Header, str)

If return_fname is True, yield a tuple of (header, file name) for the next item in the collection.

refresh()[source]

Refresh the collection by re-reading headers.

sort(keys)[source]

Sort the list of files to determine the order of iteration.

Sort the table of files according to one or more keys. This does not create a new object, instead is sorts in place.

Parameters:
keysstr, list of str

The key(s) to order the table by.

values(keyword, unique=False)[source]

List of values for a keyword.

Parameters:
keywordstr

Keyword (i.e. table column) for which values are desired.

uniquebool, optional

If True, return only the unique values for the keyword. Default is False.

Returns:
list

Values as a list.