DQC Flag Rule Set¶
Metadata¶
Data product name |
DpdDqcFlagRuleSet |
Data product custodian |
DQC |
Name of the Schema file |
|
Last Edited for DPDD Version |
10.0 |
Processing Element(s) creating/using the data product |
Creators:
Consumers:
|
Processing function using the data product |
None. |
Proposed for inclusion in EAS/SAS |
Data Product Elements¶
Header |
object of type sys:genericHeader |
Data |
object of type dqc:flagRuleSet |
QualityFlags |
object of type dqc:sqfPlaceHolder |
Parameters |
object of type ppr:genericKeyValueParameters |
Detailed Description of the Data Product¶
FlagRuleSets are used as input to the DynamicFlags PE together with data products. A FlagRuleSet contains a collection of flagConditions, which test elements in the data products whether they fulfill a certain condition. If a condition returns ‘True’ the element(s) used in that comparison will be flagged.
FlagRuleSets are saved as xml-files and ingested into EAS. The schema of the xml-files is part of the DataModel under dpd/dqc and it is called DpdDqcFlagRuleSet. FlagRuleSets can be created using the tool ST_DM_FRuleCreatorGUI, which is part of ST_DataModelTools.
See Quality Flags for details.
Elements¶
Name Name of the flagRuleSet.
Selector A tag to be referenced when writing pipeline scripts, for selecting a (group of) FlagRuleSet(s) to be applied in a given run of the DynamicFlags PE. E.g. “include all FlageRuleSets with Selector=’dpdVisFrame’”.
Category Defines categories of flags. There is a separate table of these, indicating the allowed values of flags. New flagCategory entries are not supposed to be entered into the archive by output from the pipeline - they are meant to be manually entered/updated in the archive for “proliferation control” purposes. New categories may or may not be allowed to be entered by manual quality checks or quality flag reassessments (a policy question). This is not an enumeration type, in order to allow insertion of new categories without having to modify the data model.
Subcategory A more specific classification of a flag. A given flag subcategory can only occur together with a specific flag category, whereas a flag category can “contain” many flag subcategories. This table should also be “read only” seen from the pipeline, perhaps also for human flags - or at least with strict “proliferation control”.
InputDpNames The name of the Input data product (e.g. DpdVisAstrometricParametersDetailed, DpdMerBksCalibratedFrame, etc.)
Severity Severity of the dynamic flag, enumerated string.
Explanation Explanation of the flagRuleSet
AdditionalItemsToFlag If a flagCondition evaluates to true, all items mentioned in all of its conditions will be automatically flagged - even those that only occur in conditions that do fail. In addition, all of their ancestors in the data model hierarchy will be flagged - to enable the display of flags occurring deep down in the data product when looking at higher levels of the data product. If additional items need to be flagged (e.g. the statistics of a list where one or more items have been flagged), they should be listed here.
FlagConditions List of conditions to be evaluated. NOTE: This data structure may be replaced by a plain string containing a potentially complex condition to be parsed by the DQCT module. Until then, the following comment applies: Specifies simple comparisons like “Data.NumResults eq Data.NumExpectedResults” or “Data.NumResults gt 5”.
- LeftSide
Left side of a condition is always an item, never a literal value. An item always requires an XPath and optionally one or more Functions (which could be either
Statistics, Transforms, Mathematics or ReadFits), which are applied to the data at ‘XPath’.
- Operator
An operator to be applied on the left side, this can be unary or binary.
- RightSide
Operator might be unary - no right side. In case of a binary operator, the right hand side is either a Literal value or an Item (which requires an XPath and optionally
one or more Functions (which could be either Statistics, Transforms, Mathematics or ReadFits), which are applied to the data at ‘XPath’.
- AbortProcessing
“No” (default) means raise flag in case of condition evaluates to true, but continue evaluating subsequent conditions. “Yes” means raise flag in case of condition evaluates to true, and skip subsequent conditions. “Silent” means skip (do not evaluate) subsequent conditions without raising any flag. E.g. if numStars eq 0 succeeds, don’t go on to calculate the average of the (empty) list of star magnitudes.