Data Quality Common Tools (DQCT) ================================ Data Quality (DQ) assessment is the process of rejecting/marking data that is incomplete, inaccurate, non-conforming or redundant. DQ refers to the state of completeness, validity, consistency, timeliness and accuracy that makes data appropriate for a specific use in the Euclid data flow. This means checking the right syntax of metadata and columns, detecting missing values, optimizing relationships, and any other inconsistencies. The most valuable asset of Euclid are the data and, due to the huge data volume, quality control becomes a crucial aspect for all scientific data available at all various intermediate stages of the acquisition and processing workflows and pipelines, as it is foreseen during normal operations. Our team defined and developed Data Quality Common Tools (DQCT) to help assess the quality of the data. We mainly created three separate systems to flag data that is in some way incorrect. These 'Quality Flags' are either part of the Data Product they belong to, and are saved in the top-level section 'QualityFlags' of every Data Product. Or the Quality Flags are saved in a separate Data Product. Either way, all the Quality Flags can be retrieved from the Euclid Archive System to be used in a pipeline for example. All Quality Flags associated to a specific Data Product will be displayed by Euclid Metadata Explorer. See the `DQCT redmine page `__ or read about `Quality Flags `__ in more detail. Quality Flags ============= Quality Flags are used to flag any values or collection of values that are out of range, incorrect, non-existent or just suspicious in a given Data Product. There are three different types of Quality Flags defined by the DQCT team in Euclid: Static Flags ------------ Static Quality Flags are boolean elements stored inside the QualityFlags section at the top level of every Data Product. The Static Quality Flags are called "static" because they are defined within the Data Model and are always present inside the Data Products. Each Data Product Definition in the Data Model should have Static Quality Flags. Tools +++++ **SQFCreator**: Creates Static Quality Flag Limits Data Products (:ref:`DpdDqcStaticFlagLimits`) **fill_quality_flags()**: Calculates and fills in all Static Quality elements within a Data Product Dynamic Flags ------------- Dynamic Quality Flags are stored outside of the flagged Data Product. They are stored in their own Data Product. This allows for more flexibility to add flags without having to create a "new pipeline version" (no need to change the pipeline code or change the Data Model). This system also allows to compare and flag multiple Data Products, even Data Products of different types. Tools +++++ **FRuleCreator**: Creates the Flag Rule Sets Data Product (:ref:`DpdDqcFlagRuleSet`) **ST_DQT_DynamicFlags**: Creates Dynamic Quality Flags Data Product of the input Data Product(s) (:ref:`DpdDqcDynamicFlags`) Human Flags ----------- Human Quality Flags are stored outside of the flagged Data Product. They are stored in their own Data Product. This system is used to manually flag elements in a Data Product. Tools +++++ **HFCreator**: Creates Human Quality Flag Data Products (:ref:`DpdDqcHumanFlags`) QualityViewer ------------- The QualityViewer is a web browser tool included in the Euclid Metadata Explorer. It can be found with this `link `__. This tool will show all Quality Flags (Static, Dynamic and Human) that are associated with a given Data Product. Workflow for the Data Products associated with Data Quality Flags ----------------------------------------------------------------- .. figure:: Workflow_Quality_Flags.jpg :scale: 100 % :align: center :figclass: align-center Workflow of the three Quality Flag systems