The problem: versioning data in production

In traditional data pipelines, once data is written to S3, HDFS, or local storage, there's no simple way to:

• Track who modified which data and when
• Rollback to a previous version if a data job introduces corrupted data
• Test transformations on an isolated copy of the data lake without touching production
• Perform controlled merge of separate data branches
• Recover accidentally deleted data (soft delete)

lakeFS solves this problem by sitting as a layer above S3 (or generic object storage), offering a Git-like API and CLI that transparently manages versioning.

Core concepts: branch, commit, merge

If you know Git, lakeFS will feel familiar. The key concepts are:

• Repository: logical container for your data (typically an S3 bucket).
• Branch: isolated view of the data lake. By default main exists, but you can create dev, feature/schema-migration, etc.
• Commit: snapshot of data with timestamp and metadata (author, message).
• Merge: integration of one branch into another with automatic conflict resolution.

Unlike Git (which tracks individual files), lakeFS tracks entire datasets and can operate at millisecond-level granularity on timestamps.

Installation and initial setup

The quickest way is via Docker. Create a docker-compose.yml file:

version: '3.8'
services:
  lakefs:
    image: treeverse/lakefs:latest
    ports:
      - "8000:8000"
    environment:
      LAKEFS_DATABASE_CONNECTION_STRING: postgresql://postgres:password@postgres:5432/lakefs
      LAKEFS_AUTH_ENCRYPTION_SECRET_KEY: secretkey1234567890abcdefgh
      LAKEFS_BLOCKSTORE_TYPE: s3
      AWS_ACCESS_KEY_ID: minioadmin
      AWS_SECRET_ACCESS_KEY: minioadmin
      AWS_ENDPOINT_URL: http://minio:9000
      AWS_S3_FORCE_PATH_STYLE: "true"
  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: lakefs
  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    command: server /data --console-address ":9001"

Start with docker-compose up. Access the UI at http://localhost:8000 and create a repository.

Basic operations with lakeFS CLI

Install the Python client:

pip install lakefs-sdk

Now you can operate on your data lake from Python. Example: create a branch and upload data:

import lakefs
from lakefs.client import Client

client = Client(
    host="http://localhost:8000",
    username="AKIAIOSFODNN7EXAMPLE",  # from UI setup
    password="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)

# Create a branch for testing
repo = client.repositories.get("my-lake")
repo.branches.create(name="feature/new-transform", source_branch="main")

# Upload data to the branch
lakefs_file = repo.object("feature/new-transform/data/processed.parquet")
with open("processed.parquet", "rb") as f:
    lakefs_file.upload(f)

# Commit
repo.commits.commit(
    branch="feature/new-transform",
    message="Add processed data from new transform"
)

Merge into the main branch once validated:

repo.branches.merge(
    source_branch="feature/new-transform",
    destination_branch="main",
    message="Merge new transform to production"
)

Integration with DuckDB and Polars

DuckDB can read directly from S3 (and therefore from lakeFS). Use DuckDB for on-the-fly OLAP queries on versioned data:

import duckdb
import polars as pl

# Query on a specific branch
connection = duckdb.connect()
result = connection.execute(
    """
    SELECT
        customer_id,
        SUM(amount) as total_spent
    FROM read_parquet('s3://my-lake/feature/new-transform/data/customers.parquet')
    GROUP BY customer_id
    ORDER BY total_spent DESC
    """
).fetchall()

# Or use Polars for lazy evaluation
df = pl.scan_parquet(
    "s3://my-lake/main/data/sales.parquet"
).filter(
    pl.col("date") > "2026-01-01"
).select([
    "customer_id",
    "amount",
    "date"
]).collect()

df.write_parquet("s3://my-lake/dev/aggregated.parquet")

The advantage is that you can develop and test on isolated branches, then merge into main only when quality is guaranteed.

Branching and merge strategy best practices

• main = production pure: The main branch contains certified data. No direct uploads; only merges from control branches.
• Staging branch: Create a staging branch for pre-production data. Final testing here before merge into main.
• Isolated feature branches: For each transformation/cleaning, create feature/<name>. Delete after merge.
• Merge policy: Define who can merge (senior data engineer, not junior developers).
• Metadata in commits: Always include schema version, source system, rowcount in commit message.

Rollback and disaster recovery

One of lakeFS's superpowers is instant rollback. If you discover that a job corrupted data in main, restore the previous snapshot in seconds:

from lakefs.client import Client

client = Client(host="http://localhost:8000", ...)
repo = client.repositories.get("my-lake")

# List recent commits
commits = repo.commits.list(branch="main", max_amount=10)
for commit in commits:
    print(f"{commit.id} - {commit.message} ({commit.creation_date})")

# Revert to the previous commit
rollback_commit_id = "abc123def456"
repo.refs.revert(
    branch="main",
    ref=rollback_commit_id,
    type="commit"
)
print(f"Rolled back to {rollback_commit_id}")

The operation doesn't require copying data; lakeFS simply repoints to the original files (copy-on-write).

Monitoring, audit, and compliance

lakeFS automatically logs every operation. Use the audit APIs for tracking:

# Download the complete audit log
audit_log = repo.audit_logs.list(max_amount=100)
for log_entry in audit_log:
    print(f"[{log_entry.creation_date}] {log_entry.user_id}: {log_entry.action}")

# Export to CSV for compliance
import pandas as pd
df = pd.DataFrame([
    {
        "timestamp": e.creation_date,
        "user": e.user_id,
        "action": e.action,
        "resource": e.resource_id
    }
    for e in audit_log
])
df.to_csv("audit_trail.csv", index=False)

Perfect for GDPR, SOX, or internal audits. Every data point is traceable back to the user who touched it.

Performance and trade-offs

Pros:

• Native versioning, zero overhead if you use S3/object storage
• Instant rollback (metadata only, doesn't recopy data)
• Branch isolation enables parallel testing
• Complete audit trail for compliance

Cons:

• Requires separate PostgreSQL database (management overhead)
• Merging large branches can be slow if many files
• Cost increment: lakeFS adds ~10-50ms latency per S3 operation
• For data lakes > 100TB, garbage collection requires planning

Comparison with alternatives

How does lakeFS compare to other solutions?

• Delta Lake: Focused on ACID transactions on Parquet in Spark. lakeFS is format-agnostic.
• Iceberg: Table format with snapshots, great for structured analytics. lakeFS aims at global data lake versioning.
• Git + dvc (Data Version Control): Lightweight solution for small data, but doesn't scale to terabytes.
• Native S3 Versioning: Free but without branch/merge semantics, rudimentary audit.

Decision: use lakeFS if you have a large team, strict compliance, and mature data lake; use Delta Lake if you use Spark and structured tables; use native S3 Versioning if budget-limited and data < 10TB.

Limits and final considerations

lakeFS is powerful but not a silver bullet. Remember that:

• It doesn't handle real-time streaming data; it's optimized for batch ETL
• Merge conflicts on the same files remain human responsibility (lakeFS doesn't auto-merge content)
• Garbage collection must be explicitly scheduled to save storage
• For > 10 billion objects, upgrade to lakeFS Enterprise is recommended

However, for most modern data engineering teams, lakeFS is a game-changer that eliminates entire classes of bugs ("Oops, I overwrote the data lake!").