NIR Calibrated Frame

Metadata

Data product name	DpdNirCalibratedFrame
Data product custodian	NIR
Name of the Schema file	euc-nir-CalibratedFrame.xsd
Last Edited for DPDD Version	1.1
Processing Element(s) creating/using the data product	Creators: The full NIR Pipeline Consumers: NIR_Resampling
Processing function using the data product	Indicate here the PF you are aware of using the product. Can be the custodian PF itself for internal product. Can also be Science Working Group for final products. The list does not need to be exhaustive. PF-NIR, PF-MER, PF-SIR
Proposed for inclusion in EAS/SAS	This product is proposed for inclusion in the SAS: yes This is one of the main PF-NIR products and represents a single NISP Raw Frame Product image fully calibrated.

Data Product Elements

Header	object of type sys:genericHeader
Data	object of type nir:nirCalibratedFrame
QualityFlags	object of type dqc:sqfPlaceHolder
Parameters	object of type ppr:genericKeyValueParameters

Detailed Description of the Data Product

The DpdNirCalibratedFrame is the OU-NIR Data Product produced by the execution of PF-NIR scientific pipeline on a NISP Raw Frame Product image. Pipeline tasks correct and flag the image for instrumental signatures and provides a fully astrometric and photometric calibrated image.

This Data Product is used by the Processing Function MER as input for its pipeline.

As most of the other Euclid Dpds, the DpdNirCalibratedFrame Data section is designed extending base classes with increasing specificity on the data product. Because of the DpdNirCalibratedFrame represents an image, the base class, on top of the data product (see image Fig. 16), is the generic class imgBaseFrame.

Fig. 16 : Graphical representation of first section of Data section: the imgBaseFrame.

This class collects all metadata relative to the image parameters itself.

• ImgType contains metadata describing the type of Image (i.e. photometric or spectroscopic, calibration).

• AxisNumber and AxisLengths describe the FITS file structure axis

• DataSize and DataLength refers to the size of file itself.

Fig. 17 : Graphical representation of second section of Data section: the img:singleExposure.

The higher class structure img:singleExposure (see fig Fig. 17) stores metadata needed for the description of an Euclid single exposure image.

• Instrument is the name of Euclid instrument used for the exposure.

• Filter hosts the name and some basic information of filter used.

• ImgSpatialFootprint contains the polygon coordinates of image sky coverage.

• Masks is the name of bit mask adopted.

• DetectorList contains a list of Detector, a structure dedicated to each detector metadata as:

  • DetectorId: the single detector coordinates on grid (i.e. 11,`12`, … ,`34`, 44) used as unique identifier

  • WCS: single detector WCS

  • ZeroPoint: the photometric Zeropoint value in AB mag

Fig. 18 : Graphical representation of third section of Data section: the nir:nirCalibratedFrame.

Metadata dedicated exclusively to the DpdNirCalibratedFrame are contained in the nir:nirCalibratedFrame structure (see fig Fig. 18).

• ObservationDateTime is the date and the time of the observation according the on-board clock, on UTC time and in julian date

• ObservationSequence is a set of metadata needed to identify the single observation

• ReadoutMode

• TargetPointing

Metadata in the nir:nirCalibratedFrame structure (see fig Fig. 18) contain:

• the Observation Sequence, that are numbers needed to identify the specific pointing, the date, the ReadoutMode adopted and the TargetPointing

• the name of the 4 files in the Storage metadata (see below)

• a section dedicated to the QualityParams, a set of statistical values and validation parameters useful to evaluate the data quality and identify anomalies

The DpdNirCalibratedFrame Data Product contains references to some files:

1. Data.DataStorage contains the Scientific Image: a multi-extension FITS (MEF) file, with FitsFormat id = “nir.calibratedScienceFrame”, containing three extensions per detector representing calibrated science image (SCI), RMS, and Data Quality flags (DQ). Hence the final MEF file has 49 layers: a primary header + 3 extensions for each of the 16 detectors.

2. Data.BackgroundStorage contains the Background model: a MEF file with the same structure as the Scientific Image, containing the estimated background.

3. Data.PsfStorage contains the PSF measured combining all the pipeline input images. The PSF is stored in a double format:

   1. Data.PsfStorage.PsfModelFile stores the PSF Model: a Point Spread Function (PSF) model as provided by PSFEx software (.psf).

   2. Data.PsfStorage.PsfFITSFile stores a PSF Image that is a single extension FITS file containing the PSF image associated to the science image. It can include wavelength and focal plane dependency.

4. Data.PsfSingleStorage is an optional metadata to store the PSF estimated from each single dither. It has the same structure of Data.PsfStorage.

Each file is provided together with a dedicated Quality Parameter section; therefore the Data.QualityParams is structured in 4 sections: ScienceQualityParams, BackgroundQualityParams, PsfQualityParams and PsfSingleQualityParams for science, background and PSF (global and single) respectively.

Scientific image FITS file

The main FITS file associated to this Data Product, linked on Data.DataStorage, contains the scientific image derived from the application of PF-NIR pipeline NIR Pipeline on the NISP Raw Frame Product data product.

The FITS file structure, therefore, is an evolution of the NISP Raw Frame Product one. It has a primary header with main informations on poitings and 3 layer for each detector, 49 layers in total. The layer name, in EXTNAME keyword, follow the convention DETxy.SCI, DETxy.RMS, and DETxy.DQ, for Science, RMS, and Data Quality layers respectively. xy are the detector coordinated on the NISP Detector System Assembly as illustrated on :numref:`NISP_detector_position_ids`image.

Layer name	Data Format	Units	Content
PRIMARY			Main headers
DETxy.SCI	float32	electron	Scientific data
DETxy.RMS	float32	electron	RMS data
DETxy.DQ	int32		Data Quality mask

Fig. 19 : Graphical representation of NIR calibrated frames 3-layer per detector structure

Note that background is not subtracted from science images, and pixel values are still in native units (electrons). Relative and absolute calibration factors for each detector, for conversion to microJy, are included in the metadata, as well as zero-points for conversion to AB magnitudes.

Reference pixels, the border of 4 not exposed pixels around the image, are removed from all detector, which size is therefore 2040x2040 pixels.

Primary layer

The PRIMARY header keywords report main observational pointing and instrumental information and other parameters relative to the NIR pipeline execution.

HDU PRIMARY: header
	Name	Description	Value
Keywords	FITS_DEF	FITS name definition	string = nir.calibratedScienceFrame
	FITS_VER	FITS version	string = 0.4
	TELESCOP		string = Euclid
	INSTRUME	or NISPsim for simulations	string = NISP
	VERSION	Data Release version	string
	DATE	UT date when this file was created	string
	ORIGIN	Euclid SGS origin	string
	SOFTVERS	Version of the processing function	string
	IMG_CAT	ImgType:Category DP category (SCIENCE, CALIB, …)	string
	IMG_T1	ImgType:FirstType DP type (OBJ, SRD, DARK, …)	string
	IMG_T2	ImgType:SecondType DP type (LAMP, SKY, …)	string
	OBSMODE	ImgType:ThirdType DP type (WIDE, DEEP, CALIBRATION)	string
	OBSTYPE	ImgType:Technique DP technique (IMAGE, SPECTROIMAGE)	string
	DATE-OBS	img:spaceObservationDateTime:OBT Date of observation (start), e.g. 2014-03-15T09:30:09.313	string
	UTC-OBS	img:spaceObservationDateTime:UTC UTC time correlated to the on-board time by the Mission Operation Centre, e.g. 2014-03-15T09:30:09.313Z	string
	MJD-OBS	img:spaceObservationDateTime:MJD MJD start time, e.g. 56731.39594113	double
	READMODE	ReadoutMode:ReadoutModeMethod Readout Mode Method: (Multiaccum, UpTheRamp, FowlerSamples)	string
	NR	ReadoutMode:MACCValues:NRead Number of reads	integer
	NG	ReadoutMode:MACCValues:NGroup Number of Group	integer
	ND	ReadoutMode:MACCValues:NDrop Number of drops	integer
	FRTIME	ReadoutMode:FrameTime Exposure time in seconds for single readout sample. Default value =1.41 sec	double
	EXPTIME	ExposureTime Effective integration time in seconds, e.g. 60.0005	double
	ELAPTIME	Elapsed time of the observation in seconds	double
	RA	TargetPointing:RA Center of field of view right ascension (deg)	double
	DEC	TargetPointing:DEC Center of field of view right declination (deg)	double
	PA	TargetPointing:Orientation Position angle (deg)	double
	EQUINOX	Standard FK5 (years), e.g. 2000.	double
	RADESYS	Coordinate reference frame, e.g. FK5	string
	OBS_ID	eso:observationSequence:ObservationId Running Sequence number of the Observed block	integer
	DITHOBS	eso:observationSequence:DitherObservation Number sequence of dithering i.e. 1-4 for WIDE survey	integer
	PTGID	eso:observationSequence:PointingID Integer number defining exposure inside observation	integer
	EXPNUM	eso:observationSequence:Exposure Internal integer counter for single images	integer
	TOTEXP	eso:observationSequence:TotalExposure Total number of Exposures in specific pointing	integer
	FILTER	Filter name, e.g. NIR_J	string
	FWA_POS	FWA position (J,H,Y,OPEN,CLOSED)	string
	GWA_POS	GWA position (RGS270,RGS000,RGS180,BGS000,OPEN)	string
	FWA_ANG	Commanded Filter Wheel Angle in the range 0,360	integer
	FWA_REF	Filter Wheel Angle Reference position to compute angle	string
	GWA_ANG	Commanded Grism Angle in the range 0,360	integer
	GWA_REF	Grism Wheel Angle Reference position to compute angle	string
	GWA_TILT	Commanded Grism Tilt Angle	integer
	PATCH_ID	Patch ID	integer
	S_OFFSET	Offset applied on board	integer
	LED_ID	Id of calibration led	string
	LED_INT	Intensity of calibration led	string
	LED_PWM	PWM of calibration led	string
	CU_STATE	State of calibration unit. True/False	string
	POS_X		double
	POS_Y		double
	POS_Z		double
	VEL_X		double
	VEL_Y		double
	VEL_Z		double
	PIXSCALE	Value of Pixel Scale in arcsec derived from MDB	double
	CALBLKID	ObservationSequence.CalblockId	string
	CALBLKVR	ObservationSequence.CalblockVariant	string
	BPIXCAL	Bad pixel calibration file	string
	SATCAL	Saturation limit file	string
	NLCAL	Non-linearity coefficients file	string
	PERSMAKS	Pers mask file	string
	THRESH	Applied threshold [e] for pers mask	double
	NTHRESH	Applied threshold on noise (n sigma)	double
	MDARKCAL	Master dark calibration file	string
	FLTCORR	True => flat field Corrected, False => no	string
FLTCAL	Flat calibration file	string
REFCAT	Gaia referencd catalog	string
ASTRIOFF	Initial astr offset (arcmin)	double
ASTRIORA	Initial astr offset (arcmin)	double
ASTRIODE	Initial astr offset (arcmin)	double
ASTRDEN	Flagged detections /arcmin**2	double
ASTRREF	Lowest number of ref sources/detector	integer
ASTRREFD	Detector with lowest number of ref sources	integer
ASTRNREF	Number of reference sources	integer
ASTRNM	Flagged matched ref source	integer
ASTRM2R	Matched/ref sources (pct)	double
ASTRNMF	Total matched ref sources	integer
ASTRD2R	Detected/ref sources (pct)	double
ASTRELL	Median ellipticity ref sources	double
WASTRELL	High median ellipticity. True/False	string
ERRASTR	Dither should be removed. True/False	string
PHRELOB	Correction factor for relative photometric calibration for observation	double
PHRELOBE	Uncertainty in PHRELOB	double
PHRELEX	Correction factor for relative photometric calibration	double
PHRELEXE	Uncertainty in PHRELEX	double
QPRELEX	quality parameter: relative photo offset among dithers	double
QPRELEXE	quality parameter: error in QPRELEX	double
CAL_VER	version of calibration factors	string
WAVE_CAL	wavelength used for calibration [microns]	double
WAVELEN	central wavelength of bandpass [microns]	double
WAVEHPBW	half-power bandwidth [microns]	double
PHOTFNU	inverse sensitivity [micro Jy s / e ]	double
PHOTFNUE	error in PHOTFNU [micro Jy s / e ]	double

Scientific layer

The DETxy.SCI layer contains the input image corrected from instrumental effects and converted in electrons. The astrometrical and photometrical correction factors are reported on detector header.

HDU DETn.SCI: header
	Name	Description	Value
Keywords	EXTNAME	e.g. DET11.SCI, meaning Detector number 0, Science data	string
	DET_ID	Detector:DetectorId e.g. Detector ID, e.g. 11	string
	BITPIX	array data type	integer
	NAXIS	e.g. number of array dimensions	integer
	NAXIS1	e.g. pixel numbers on first axis	integer
	NAXIS2	e.g. pixel numbers on second axis	integer
	BUNIT	physical units of the array values	string
	RPIX_PRC	Reference Pixel processing mode (0=no_px_correction,1=horiz_px_correction,2=vertical_px_correction)	integer
	GAIN	Detector:Gain e- to ADU conversion factor, e.g. 1.5	double
	EQUINOX	Standard FK5 (years), e.g. 2000.	double
	RADESYS	Coordinate reference frame, e.g. FK5	string
	CRVAL1	Right ascension at ref pixel	double
	CRVAL2	Declination at ref pixel	double
	CRPIX1	Reference pixel x coordinate	double
	CRPIX2	Reference pixel y coordinate	double
	CTYPE1	Coordinate 1 type	double
	CTYPE2	Coordinate 2 type	double
	CUNIT1	Axis 1 unit	string
	CUNIT2	Axis 2 unit	string
	CD1_1	Translation matrix element	double
	CD1_2	Translation matrix element	double
	CD2_1	Translation matrix element	double
	CD2_2	Translation matrix element	double
	PV1_0	Projection distortion parameter	double
	PV1_1	Projection distortion parameter	double
	PV1_2	Projection distortion parameter	double
	PV1_4	Projection distortion parameter	double
	PV1_5	Projection distortion parameter	double
	PV1_6	Projection distortion parameter	double
	PV1_7	Projection distortion parameter	double
	PV1_8	Projection distortion parameter	double
	PV1_9	Projection distortion parameter	double
	PV1_10	Projection distortion parameter	double
	PV2_0	Projection distortion parameter	double
	PV2_1	Projection distortion parameter	double
	PV2_2	Projection distortion parameter	double
	PV2_4	Projection distortion parameter	double
	PV2_5	Projection distortion parameter	double
	PV2_6	Projection distortion parameter	double
	PV2_7	Projection distortion parameter	double
	PV2_8	Projection distortion parameter	double
	PV2_9	Projection distortion parameter	double
	PV2_10	Projection distortion parameter	double
	ASTIRMS1	Astrom. dispersion RMS (intern., high S/N)	double
	ASTIRMS2	Astrom. dispersion RMS (intern., high S/N)	double
	ASTRRMS1	Astrom. dispersion RMS (ref., high S/N)	double
	ASTRRMS2	Astrom. dispersion RMS (ref., high S/N)	double
	ZPAB	zero point for conversion to AB magnitude	double
	ZPABE	error in ZPAB determination	double
	ZPVEGA	zero point for conversion to Vega magnitude	double
	ZPVEGAE	error in ZPVEGA determination	double
	PHRELDT	Correction factor for relative photometric calibration for detector	double
	PHRELDTE	Uncertainty in PHRELDT	double
	CRSNGALG	Algorithm used in finding cosmic rays	string
	CRMULALG	Algorithm used in finding cosmic rays	string
	DARKFILL	Dark fill value in e-	double
	NBADPIXT	Number of bad pixels for detector	integer
	NREJNL	Number of high noise pixels not corrected for non-linearity	integer
	NSATPIX	Number of pixels flagged for saturation	integer
	NDFILL	Number of high noise pixels with filled dark values	integer
	NCRPIXS	Number of pixels masked as CR pixels in the single frame rejection	integer
	NCRPIXM	Number of pixels masked as CR pixels in the multi frame rejection	integer

Standard Deviation layer

The pixel value stored into the DETxy.RMS data layer is derived from a variance layer intenally used by NIR Pipeline tasks. It is initially instantiated by the NIR Init task application according the formula 12 of Kubik (2016) paper.

The final variance layer value is the result of the calibration file contributions added during the application of NIR Pipeline tasks.

Data Quality layer

The DETxy.DQ layer is a bit maks containing flags associated to each pixel during the pipeline execution.

The first bit INVALID identifies pixels that are considered not to be used. It is activated by any pipeline task that identify a critical feature (i.e. bad pixels, cosmic hits and saturaded pixels).

The single bitmask flag description is available in dedicated wiki and shown in Fig. 20

Fig. 20 : NIR Data Quality Bit value, flag name and description.

Background FITS file

The Background FITS file contains the background value estimated from images by the NIR Background estimation task.

The FITS file structure mirrors the scientific file one with 49 layers: a primary header with main information and 3 layers (SCI, RMS and DQ) for each detector.

PSF model and image files

The NIR PSF estimation task estimates a PSF model from scientific images thanks a PSFEx run.

The software produces a .psf file containing model parameters and a .fits with an image of the PSF model. This last product is used by PF-MER.

Data Quality Parameters

The Data.QualityParams section provide a set of Quality Parameters derived from each of the files.

The Data.QualityParams.ScienceQualityParams contains a set of statistical values (min, max, mean, median std and masked pixel fraction) for the whole image and some Quality Parameters correlated to the astrometric and photometric calibration. For each detector are indeed provided the same set of statistics and the numbers of pixel flagged for each bit mask value.

Fig. 21 : NIR scientific calibrated frame DQC

The Data.QualityParams.BackgroundQualityParams contains the set of statistical values for each detector. Despite the background image pixel value is in [electron], the background detector statistical values are converted in [Myr/sr].

Data.QualityParams.PsfQualityParams and Data.QualityParams.PsfSingleQualityParams contain some quality information on PSF as the FWHM measured on the 2 axes, the number of sources used for PSF measurement and the size of radius encicling energy.

Fig. 22 : NIR PSF DQC