Sources

Authoritative Sources

This knowledge pack is grounded in the following authoritative, primary references. Each was used to verify factual claims about formats, engines, semantics, and patterns. Upstream sources retain their own licenses; verify critical claims against the primary documentation before relying on them in production.

Processing engines and frameworks

  • Apache Spark Documentation - https://spark.apache.org/docs/latest/
  • Apache Spark Structured Streaming Programming Guide - https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
  • Apache Flink Documentation - https://nightlies.apache.org/flink/flink-docs-stable/
  • Apache Kafka Documentation - https://kafka.apache.org/documentation/
  • Apache Airflow Documentation - https://airflow.apache.org/docs/
  • Trino Documentation - https://trino.io/docs/current/

Table formats and storage

  • Apache Parquet - https://parquet.apache.org/docs/
  • Apache Iceberg - https://iceberg.apache.org/docs/latest/
  • Delta Lake Documentation - https://docs.delta.io/latest/index.html
  • Apache Hudi - https://hudi.apache.org/docs/overview/
  • Apache Avro Specification - https://avro.apache.org/docs/

Transformation, quality, and CDC

  • dbt Documentation - https://docs.getdbt.com/docs/introduction
  • Great Expectations Documentation - https://docs.greatexpectations.io/docs/
  • Debezium Documentation - https://debezium.io/documentation/reference/stable/index.html

Cloud warehouses and platforms

  • Google BigQuery Documentation - https://cloud.google.com/bigquery/docs
  • Snowflake Documentation - https://docs.snowflake.com/
  • Amazon Redshift Documentation - https://docs.aws.amazon.com/redshift/
  • Databricks Lakehouse Documentation - https://docs.databricks.com/

Modeling, architecture, and concepts

  • Kimball Group - Dimensional Modeling Techniques - https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/
  • Martin Kleppmann, "Designing Data-Intensive Applications" (O'Reilly) - https://dataintensive.net/
  • Data Mesh (Zhamak Dehghani / martinfowler.com) - https://martinfowler.com/articles/data-mesh-principles.html
  • The Streaming Systems book (Akidau et al.) - https://www.oreilly.com/library/view/streaming-systems/9781491983867/
  • DataHub Project - https://datahubproject.io/docs/

Governance and privacy

  • GDPR - Official Text (EUR-Lex Regulation 2016/679) - https://eur-lex.europa.eu/eli/reg/2016/679/oj