ETL
Extract, Transform, Load. A pipeline pattern that transforms data on a dedicated engine before loading it into the destination, so only conformed data lands in the target store.
ELT
Extract, Load, Transform. A pattern that loads raw data into the destination first and transforms it in place using the warehouse's compute; the model behind tools like dbt.
CDC
Change Data Capture. The technique of capturing row-level inserts, updates, and deletes from a source database (usually via its transaction log) to stream changes downstream incrementally.
OLAP
Online Analytical Processing. Read-heavy, aggregate-oriented analytical workloads over large historical data, typically served by columnar storage engines.
OLTP
Online Transaction Processing. High-volume, low-latency transactional workloads (small reads and writes), typically served by row-oriented operational databases.
SCD
Slowly Changing Dimension. A dimension whose attributes change over time; SCD types (1 overwrite, 2 historical rows, 3 limited prior columns) define how those changes are recorded.
DAG
Directed Acyclic Graph. The structure used to represent task dependencies in an orchestrator, where edges denote ordering and no cycles are allowed, so execution order is well-defined.
dbt
data build tool. A framework for defining warehouse transformations as version-controlled SQL models with tests, documentation, and dependency-aware execution (the T in ELT).
ACID
Atomicity, Consistency, Isolation, Durability. The transactional guarantees that open table formats like Delta Lake and Iceberg bring to data-lake files.
HDFS
Hadoop Distributed File System. The distributed, replicated file system of the Hadoop ecosystem, historically the storage layer for big-data processing before cloud object storage became dominant.
S3
Simple Storage Service. Amazon's object storage, a common backing store for data lakes and lakehouses; "S3-compatible" object stores expose the same API.
RDD
Resilient Distributed Dataset. Spark's low-level immutable, partitioned, fault-tolerant collection abstraction, beneath the higher-level DataFrame and SQL APIs.
Parquet
A columnar, compressed, binary file format with column statistics and row groups, the de facto physical storage format for analytical data lakes.
Avro
A compact, row-oriented binary serialization format with a self-describing schema, commonly used for streaming records and Kafka message payloads.
ORC
Optimized Row Columnar. A columnar file format from the Hive ecosystem with strong compression and indexing, an alternative to Parquet.
JSONL
JSON Lines. A text format with one JSON object per line, convenient for streaming, appending, and line-by-line processing of semi-structured records.
SLA
Service Level Agreement. A commitment to data consumers about delivery, typically covering freshness, availability, and quality; SLO (objective) and SLI (indicator) make it measurable.
SLO
Service Level Objective. A target value for a measurable property of a data service (for example, "table refreshed within 30 minutes of source"), against which an SLA is judged.
PII
Personally Identifiable Information. Data that can identify an individual; it triggers privacy obligations (such as GDPR) and requires classification, access control, and often masking.
GDPR
General Data Protection Regulation. The EU data-protection law governing processing of personal data, with requirements such as lawful basis, data minimisation, and the right to erasure.
MPP
Massively Parallel Processing. An architecture that distributes a query across many nodes that each process a shard of the data, the basis of cloud analytical warehouses.
ANSI SQL
The standardised SQL dialect; "ANSI-compliant" engines aim for portable query syntax, though most warehouses add proprietary extensions.
UPSERT
Insert-or-update. A write that inserts a row if its key is absent and updates it otherwise, expressed as MERGE in SQL and central to idempotent incremental loads.
MERGE
A SQL statement that conditionally inserts, updates, or deletes target rows based on a match with a source, used for upserts and SCD maintenance in warehouses and lakehouses.
TTL
Time To Live. A retention setting after which data (a cache entry, a streaming record, a table partition) is automatically expired or eligible for deletion.
DWH
Data Warehouse. A central analytical store optimised for aggregate queries over structured, historical data.
ODS
Operational Data Store. An integrated store of current operational data, often a staging area between source systems and the warehouse for near-real-time operational reporting.
EOS
Exactly-Once Semantics. A processing guarantee that each input affects the result exactly once despite failures, achieved with idempotent writes and transactional offset management.
Debezium
An open-source CDC platform built on Kafka Connect that streams database transaction-log changes into Kafka topics for downstream consumption.
Trino
A distributed SQL query engine (formerly PrestoSQL) for interactive analytics across heterogeneous sources, including object storage and open table formats, without moving the data.
ZooKeeper / KRaft
Coordination layers for Apache Kafka: ZooKeeper was the legacy metadata and consensus service, now superseded by KRaft (Kafka Raft), Kafka's built-in self-managed consensus mode.
Backpressure
A flow-control mechanism in streaming where a slow downstream consumer signals upstream producers to slow down, preventing unbounded buffering and memory exhaustion.
Throughput vs Latency
The core streaming/batch trade-off: throughput is volume processed per unit time (favoured by batch), latency is time from event to result (favoured by streaming).