Business Rules - Data Engineering

This document lists the fundamental business rules of the Data Engineering domain, each identified by a unique BR-NNN code. Every rule is technology-independent and describes an invariant constraint on the behavior of a data platform.

---

BR-001 - Pipeline run idempotency

Code: BR-001 Name: Every DataPipeline run must be idempotent Description: Re-running the same pipeline over the same partition interval (e.g., the same logical date) must produce the same output Dataset, with no duplicates and no cumulative side effects. Writes must use overwrite-partition, merge/upsert on a key, or insert-if-not-exists patterns - never a blind append. Rationale: Enable safe retries after transient failures and repeatable backfills without corrupting data. Priority: CRITICAL Actors: Data Engineer, Scheduler Violation: row duplication in the Dataset; the uniqueness QualityCheck fails and blocks publication. Traceability: glossary Idempotency, Idempotent Write, Backfill; Apache Airflow idempotency guidelines.

---

BR-002 - Quality validation before publication

Code: BR-002 Name: No Dataset reaches the published state without passing the blocking QualityChecks Description: Before promoting a dataset to published, the pipeline must run the defined quality checks (non-null on key columns, uniqueness of the logical primary key, freshness within SLA, value range/domain, row count within expected thresholds). Checks marked blocking halt promotion; those marked warning allow it but raise an alert. Rationale: Prevent corrupt or incomplete data from reaching downstream consumers. Priority: HIGH Actors: Data Steward, Data Engineer Violation: DataQualityFailed event; the Dataset stays in validated and does not transition to curated/published. Traceability: glossary Data Quality, Great Expectations; BR-001.

---

BR-003 - Backward schema compatibility

Code: BR-003 Name: A schema evolution cannot introduce breaking changes without versioning Description: Adding an optional column or promoting to a compatible type (int → long) is allowed in-place. Removing a column, renaming, an incompatible type change, or tightening a constraint are breaking changes: they require a new Dataset version (or an explicit SchemaEvolution with a coexistence period) so as not to break existing consumers. Rationale: Protect downstream jobs and queries from silent data contract breakage. Priority: HIGH Actors: Data Engineer, Data Consumer Violation: SchemaBreakingChange event; deployment of the new schema blocked by the Schema Registry in BACKWARD mode. Traceability: glossary Schema Evolution, Schema Registry, Apache Avro; Confluent Schema Registry compatibility modes.

---

BR-004 - Freshness SLA for every published dataset

Code: BR-004 Name: Every Dataset in the published state declares and honors a freshness SLA Description: Every dataset exposed to consumers must have an explicit SLA (e.g., "updated by 06:00 UTC with data up to T-1"). The system monitors the timestamp of the latest valid partition; if it exceeds the tolerance threshold, the SLA is violated and consumers must be notified. Rationale: Give consumers reliable expectations about data recency. Priority: HIGH Actors: Data Engineer, Data Consumer, Scheduler Violation: SLAViolated and DataFreshnessDegraded events; alert to the owning team. Traceability: glossary SLA, Data Freshness, Partitioning; BR-002.

---

BR-005 - End-to-end traceable lineage

Code: BR-005 Name: Every Dataset records its DataLineage upstream and downstream Description: For every produced dataset, the list of input sources (upstream) and applied transformations must be recorded so that the origin of each column is reconstructible. Removing or modifying a source must propagate a lineage update to derived datasets. Rationale: Enable impact analysis, debugging, compliance (GDPR data provenance), and trust in the data. Priority: HIGH Actors: Data Steward, Data Engineer Violation: incomplete lineage → impossible to assess the impact of a SchemaBreakingChange. Traceability: glossary Data Lineage, Data Catalog; OpenLineage spec; EU Regulation 2016/679 (GDPR) art. 30.

---

BR-006 - Minimization and classification of personal data

Code: BR-006 Name: Personal data must be classified, minimized, and masked in non-protected zones Description: Every column containing personal data (PII) must be labeled in the DataCatalog. In general-consumption zones (e.g., published datasets for self-service analytics) sensitive fields must be masked, pseudonymized, or excluded, unless the consumer has legitimate authorized access. Rationale: GDPR compliance (minimization, purpose limitation) and reduced data-breach risk. Priority: CRITICAL Actors: Data Steward, Data Consumer Violation: PII exposed in cleartext to unauthorized consumers → compliance incident. Traceability: glossary Data Governance, Data Catalog; EU Regulation 2016/679 art. 5, art. 25.

---

BR-007 - Zone separation (raw / curated / published)

Code: BR-007 Name: Data flows through zones with distinct responsibilities and does not skip stages Description: The architecture adopts zones of progressive maturity: raw (faithful ingest from the source, immutable), validated/curated (clean, conformed, typed), and published (modeled for consumption, documented). An analytics consumer never reads directly from the raw zone; every promotion between zones is explicit and traced. Rationale: Isolate failures, guarantee reproducibility (the raw zone enables re-processing), and decouple ingest from consumption. Priority: HIGH Actors: Data Engineer, Data Consumer Violation: direct consumption of the raw zone → fragile dependency on ungoverned raw formats. Traceability: glossary Data Lake, Data Lakehouse, Medallion Architecture; state-machine dataset-lifecycle.md.

---

BR-008 - Cost and partitioning under control

Code: BR-008 Name: Large tables must be partitioned and queries must use partition pruning Description: Datasets that exceed a volume threshold must be partitioned on a selective key (typically temporal). Read queries must filter on the partition column to enable pruning and avoid costly full scans. Columnar storage (Parquet/ORC) is mandatory for analytical datasets. Rationale: Control compute/storage costs and analytical query latency. Priority: MEDIUM Actors: Data Engineer, Data Consumer Violation: recurring full table scans → costs and query times over budget. Traceability: glossary Partitioning, Columnar Storage, Apache Parquet; PartitionCreated event.