Data Pipeline
A data pipeline is a sequence of processing steps that moves data from one or more sources to one or more destinations, applying transformations along the way. Pipelines can be batch or streaming, and are typically expressed as a directed acyclic graph of dependent tasks. Their core concerns are correctness, idempotency, observability, and recoverability after failure.
ETL (Extract, Transform, Load)
ETL is a pattern in which data is extracted from sources, transformed into the target shape on a dedicated processing engine, and then loaded into the destination store. Transformation happens before loading, so only conformed data lands in the warehouse. ETL was the dominant pattern when storage and compute were expensive and tightly coupled.
ELT (Extract, Load, Transform)
ELT extracts data from sources and loads it raw into the destination first, then transforms it in place using the warehouse's own compute. It exploits cheap cloud storage and elastic warehouse engines so raw data is preserved and transformations can be re-run. Tools like dbt popularised ELT by running SQL transformations directly inside the warehouse.
Data Warehouse
A data warehouse is a central analytical store optimised for fast aggregate queries over large, structured, historical data, usually using columnar storage. It serves business intelligence and reporting workloads rather than transactional ones. Examples include Snowflake, Google BigQuery, and Amazon Redshift.
Data Lake
A data lake stores raw data of any structure (structured, semi-structured, unstructured) at low cost on object storage, deferring schema definition until read time (schema-on-read). It is flexible and cheap but risks becoming a "data swamp" without governance, cataloguing, and quality controls. Common backings are Amazon S3, Azure Data Lake Storage, and Google Cloud Storage.
Data Lakehouse
A lakehouse combines the low-cost open storage of a data lake with warehouse-style management features such as ACID transactions, schema enforcement, and time travel, implemented via open table formats. It aims to serve both BI and machine-learning workloads from a single copy of data. Delta Lake, Apache Iceberg, and Apache Hudi are the dominant table formats.
Schema-on-Read
Schema-on-read defers interpreting the structure of data until it is queried, storing raw bytes and applying a schema at read time. It makes ingestion fast and flexible for evolving or unknown structures, at the cost of pushing parsing and validation to consumers. It contrasts with schema-on-write, where data must conform before it is stored.
Schema-on-Write
Schema-on-write enforces a defined structure at ingestion time, rejecting or coercing data that does not conform before it is persisted. It guarantees that stored data is clean and queryable but makes ingestion less flexible to source changes. Relational databases and traditional warehouses use schema-on-write.
Idempotency
In data engineering an operation is idempotent if running it multiple times produces the same result as running it once. Idempotency is essential for safe retries and backfills, because pipelines fail and tasks rerun. It is commonly achieved with deterministic partition overwrites, merge/upsert keys, or deduplication on a stable business key.
Backfill
A backfill reprocesses historical data through a pipeline, typically to populate a new column, fix a past bug, or rebuild a table after logic changes. Safe backfills require idempotent, partition-scoped tasks so a rerun does not duplicate or corrupt data. Backfills are often run partition by partition with bounded concurrency to control cost.
Partitioning
Partitioning divides a table physically into segments, commonly by a date or category column, so queries can prune irrelevant data and scan less. Good partitioning dramatically reduces query cost and enables idempotent partition-level overwrites. Over-partitioning (too many tiny partitions, the small-files problem) hurts performance and metadata overhead.
Idempotent Write
An idempotent write produces the same final table state regardless of how many times it runs, usually by overwriting a deterministic target (a partition) or merging on a key. It is the foundation of reliable batch pipelines, which must tolerate retries. Append-only writes are not idempotent unless paired with deduplication.
Slowly Changing Dimension (SCD)
A slowly changing dimension is a dimension table whose attribute values change infrequently over time, and the SCD types define how to record those changes. Type 1 overwrites the old value, Type 2 keeps full history by adding a new row with validity dates, and Type 3 keeps limited prior values in extra columns. SCD Type 2 is the standard way to preserve historical accuracy in a warehouse.
Star Schema
A star schema is a dimensional model with a central fact table referencing surrounding dimension tables via foreign keys, forming a star shape. It is denormalised for query performance and intuitive analytics. It is the classic Kimball design for warehouse reporting.
Fact Table
A fact table stores the measurable, quantitative events of a business process (such as sales or clicks), with foreign keys to dimensions and numeric measures. Facts are typically additive and stored at a defined grain (one row per event or per aggregation level). Choosing the grain is the first and most important step in dimensional modelling.
Dimension Table
A dimension table holds the descriptive, textual context for facts, such as customer, product, date, or location, that analysts filter and group by. Dimensions are denormalised and relatively wide, and may track history via slowly changing dimension techniques. They provide the "who, what, where, when" around each fact.
Data Modeling
Data modeling is the discipline of structuring data into tables and relationships that fit how it will be queried and maintained. Analytical modeling favours dimensional designs (star/snowflake) or wide denormalised tables; transactional modeling favours normalisation to reduce redundancy. The right model depends on the access pattern, not on a universal ideal.
Normalization
Normalization organises relational data to eliminate redundancy by decomposing tables according to normal forms, so each fact is stored once. It optimises write integrity and storage at the cost of more joins on read. Transactional (OLTP) systems normalise; analytical systems often deliberately denormalise for read speed.
Denormalization
Denormalization intentionally introduces redundancy, such as pre-joining or duplicating columns, to reduce joins and speed up reads. It trades storage and write complexity for query performance, which suits analytical and columnar systems. Wide tables and star schemas are common denormalised designs.
OLAP (Online Analytical Processing)
OLAP refers to workloads and systems optimised for complex aggregate queries over large historical datasets, such as reporting and dashboards. OLAP engines typically use columnar storage and are read-heavy with infrequent bulk loads. It contrasts with OLTP, which handles many small transactional reads and writes.
OLTP (Online Transaction Processing)
OLTP refers to systems optimised for high-volume, low-latency transactional operations, such as inserting an order or updating a balance. OLTP databases are usually row-oriented and heavily normalised for write integrity. Operational source systems are typically OLTP; their data is later moved to OLAP stores for analytics.
Columnar Storage
Columnar storage physically lays out a table's values by column rather than by row, so analytical queries read only the columns they need. This enables strong compression and fast scans for aggregations, which is why warehouses and formats like Parquet are columnar. It is poorly suited to row-at-a-time transactional workloads.
Apache Parquet
Parquet is an open, columnar, binary file format designed for efficient analytical storage and query, with built-in compression, encoding, and column statistics. Its column-level statistics and row-group structure let engines skip data (predicate pushdown). It is the de facto standard physical format for data lakes and lakehouses.
Apache Spark
Apache Spark is a distributed compute engine for large-scale batch and stream processing, exposing DataFrame, SQL, and RDD APIs across languages. It executes jobs as a DAG of stages over partitioned data, with in-memory caching and a cost-based optimiser (Catalyst). It is widely used for ETL/ELT, machine learning, and lakehouse transformations.
Apache Kafka
Apache Kafka is a distributed, append-only commit log used as a durable, high-throughput backbone for streaming data and event-driven systems. Producers write records to partitioned topics and consumers read them at their own offset, enabling replay and decoupling. It provides ordering within a partition and configurable retention and replication.
Apache Airflow
Apache Airflow is an open-source orchestrator that schedules and monitors workflows defined as Python DAGs of tasks with explicit dependencies. It handles retries, backfills, scheduling, and observability of batch pipelines. Each task should be idempotent so reruns and catchups are safe.
Orchestration
Orchestration coordinates the execution order, scheduling, retries, and dependencies of the tasks in a data pipeline. It answers what runs, in what order, when, and what happens on failure. Tools include Apache Airflow, Dagster, and Prefect; orchestration is distinct from the compute engine that does the work.
dbt (data build tool)
dbt is a transformation framework that lets analysts define warehouse transformations as version-controlled SQL models with dependencies, tests, and documentation. It compiles models into SQL, runs them in dependency order inside the warehouse (the T in ELT), and materialises them as tables or views. It brings software-engineering practices (testing, CI, lineage) to analytics.
Change Data Capture (CDC)
Change Data Capture detects and streams row-level changes (inserts, updates, deletes) from a source database, usually by reading its transaction log. It enables near-real-time replication and incremental ingestion without full table scans or heavy queries on the source. Debezium is a common open-source CDC platform built on Kafka Connect.
Stream Processing
Stream processing computes over unbounded, continuously arriving data, producing results incrementally with low latency. It must handle event-time semantics, out-of-order events, windowing, and exactly-once or at-least-once delivery. Engines include Apache Flink, Kafka Streams, and Spark Structured Streaming.
Batch Processing
Batch processing runs computations over a bounded, finite dataset, typically on a schedule (hourly, daily). It favours throughput and simplicity over latency and is the default for most analytical pipelines. The trade-off against streaming is freshness versus operational complexity.
Windowing
Windowing groups unbounded stream events into finite chunks for aggregation, by time (tumbling, sliding, session windows) or by count. Time windows must decide between event time and processing time and handle late-arriving data. Windowing is what makes aggregations over an infinite stream tractable.
Watermark
A watermark is a heuristic timestamp in stream processing that asserts no events earlier than it are expected, allowing windows to close and emit results. It trades completeness for latency: a later watermark waits longer for stragglers, an earlier one emits sooner but may drop late events. Watermarks are central to correct event-time processing in engines like Flink.
Exactly-Once Semantics
Exactly-once semantics guarantees each input record affects the result exactly once despite failures and retries, avoiding both loss and duplication. It is typically achieved with idempotent writes plus transactional offset commits, or with two-phase commit sinks. True end-to-end exactly-once requires cooperation between the source, the processor, and the sink.
Data Governance
Data governance is the framework of policies, roles, and processes that ensure data is accurate, secure, compliant, and well-defined across an organisation. It covers ownership, access control, classification, retention, privacy, and standards. Governance turns scattered data into a trusted, auditable asset.
Data Catalog
A data catalog is a searchable inventory of an organisation's datasets enriched with metadata such as schemas, owners, descriptions, lineage, and quality status. It helps users discover, understand, and trust data and underpins governance. Tools include DataHub, Amundsen, and OpenMetadata.
Data Lineage
Data lineage is the traceable record of where data originated and how it flowed and was transformed across systems to its current state. It supports impact analysis, debugging, compliance, and trust by showing upstream sources and downstream consumers. Lineage can be captured at the table or column level.
Data Quality
Data quality is the degree to which data is fit for its intended use, measured along dimensions such as accuracy, completeness, consistency, timeliness, validity, and uniqueness. It is enforced with automated tests, constraints, and monitoring that fail or quarantine bad data early. Frameworks include Great Expectations, dbt tests, and Soda.
Data Observability
Data observability is the practice of monitoring the health of data and pipelines across freshness, volume, schema, distribution, and lineage to detect issues before consumers do. It extends software observability concepts to the data domain. The goal is to catch broken or anomalous data automatically rather than via downstream complaints.
Data Contract
A data contract is an explicit, versioned agreement between a data producer and its consumers specifying schema, semantics, quality guarantees, and SLAs. It prevents silent breaking changes by making the interface a first-class, testable artefact. Contracts shift data quality left, to the point of production.
Medallion Architecture
The medallion architecture organises a lakehouse into progressive quality layers: bronze (raw ingested data), silver (cleaned, conformed, deduplicated), and gold (curated, business-level aggregates). Each layer adds structure and trust, isolating raw landing from consumption. It is a common pattern for incremental refinement in Delta Lake and similar platforms.
Apache Iceberg
Apache Iceberg is an open table format that adds database-like features to data-lake files, including ACID transactions, schema and partition evolution, snapshots, and time travel. It tracks table state through metadata and manifest files rather than directory listings, avoiding listing-based correctness problems. It lets multiple engines (Spark, Trino, Flink) safely read and write the same tables.
Delta Lake
Delta Lake is an open storage layer that brings ACID transactions, schema enforcement, and time travel to Parquet files via a transaction log. It enables reliable upserts (MERGE), deletes, and concurrent reads/writes on a data lake. It is the foundation of the lakehouse pattern on platforms such as Databricks.
Time Travel
Time travel is the ability to query a table as of a previous version or timestamp, enabled by table formats that retain immutable snapshots and a transaction log. It supports auditing, reproducing past results, debugging, and rolling back bad writes. Delta Lake, Iceberg, and Hudi all expose time-travel queries.
Idempotent Consumer
An idempotent consumer processes the same message more than once without changing the outcome, which is required because most messaging systems deliver at least once. It is implemented by deduplicating on a message key or by making the downstream effect itself idempotent. This pattern lets streaming systems achieve effectively-once results.
Data Mesh
Data mesh is a decentralised socio-technical approach that treats data as a product owned by domain teams, supported by a self-serve platform and federated governance. It shifts responsibility from a central data team to the domains that know the data best. Its four principles are domain ownership, data as a product, self-serve platform, and federated computational governance.
Reverse ETL
Reverse ETL moves transformed, modelled data from the warehouse back into operational tools such as CRMs, ad platforms, and support systems. It operationalises analytics by putting curated metrics where business users act on them. It is the inverse of traditional ingestion, with the warehouse as the source of truth.
Data Skew
Data skew is the uneven distribution of data across partitions or keys, causing some tasks or reducers to handle far more work than others and become stragglers. It is a leading cause of slow distributed jobs, especially on joins and aggregations over hot keys. Mitigations include salting keys, repartitioning, and broadcast joins for small tables.
Small Files Problem
The small files problem arises when a data lake accumulates many tiny files, inflating metadata, listing, and task-scheduling overhead and degrading read performance. It commonly results from high-frequency streaming writes or over-partitioning. It is addressed by compaction (combining files) and table formats that manage file sizes.
Materialized View
A materialized view stores the precomputed result of a query physically, so reads are fast at the cost of keeping it refreshed as underlying data changes. It trades storage and refresh complexity for query latency, useful for expensive recurring aggregations. Some engines refresh incrementally; others require full recomputation.
Data Freshness
Data freshness measures how recently data was updated relative to the events it represents, often expressed as the lag between source and destination. It is a core data-quality and SLA dimension that trades off against cost and pipeline complexity. Streaming improves freshness; batch schedules cap it at the interval length.
Apache Flink
Apache Flink is a distributed stream-processing engine built for stateful, event-time computations with exactly-once guarantees and low latency. It treats batch as a special case of streaming and uses checkpointing to recover state after failure. It is widely used for real-time analytics, CDC processing, and event-driven applications.