A faulty load job overwrites the production folder of your data lake: how do you go back, and how could you have tried it first on a copy? lakeFS answers by applying Git's ideas to data: branches that are pointers, immutable commits, three-way merges. Before installing it, though, there is one thing to know that many tutorials do not mention: since version 1.87.0 the project's license is no longer Apache 2.0.
Prerequisites. Familiarity with Git (commit, branch, merge) and with object storage; the Python listings use the same environment as the previous articles.
The license, first of all
On 22 September 2026 the project announced that, starting from version 1.87.0, the Community edition moves from the Apache 2.0 license to the Business Source License 1.1; the versions published before remain Apache 2.0. The repository's LICENSE file, read on 2026-11-13, names Treeverse Labs Ltd. as the Licensor, "lakeFS 1.87.0" as the licensed work and Apache 2.0 as the change license, four years after the publication of each version. The BSL, the text itself says, is not a recognized open source license, even though the code stays public. In practice, the additional use grant allows production use, at any scale, on two conditions:
- the program must be the one distributed by the vendor, without modifications, forks or derivative works (configuring it and writing software that uses it through the documented public interfaces does not count as a modification);
- the use must be only for your own organization's internal needs, so you cannot offer it, host it or make its functionality available to third parties;
For a team that versions its own data inside its own company, these conditions are compatible with normal use. If instead you want to resell a service built on lakeFS, or maintain a fork of your own in production, you need a commercial agreement. The license text and the announcement are legal and commercial documents: read them directly and, if your case is borderline, write to the vendor as the announcement itself invites you to. The lakefs Python library is another matter and remains Apache-2.0, according to its metadata on PyPI.
There are three editions, according to the documentation page: Community (free under the BSL), Team (with access control, Iceberg REST Catalog, retention management, audit; the page lists 499 dollars a month for 5 users and 500 GB after the 30-day trial, price read on 2026-11-13) and Enterprise (by contract). Prices change: check them at the source on the day you decide.
The model: branches that copy nothing
The part that makes lakeFS useful is simple to state: a branch is a pointer to a commit plus a set of changes not yet committed, and creating one does not duplicate the data, as the documentation explains. The data stay in your object storage and lakeFS manages pointers and metadata. A commit is an immutable snapshot of the whole repository, with author, date, message and any key-value pairs you like.
The merge works as in Git: it finds the nearest common ancestor commit (the merge base) and compares, for each file, how it was at the base, in the source branch and in the destination one. The documentation gives the complete table of cases; the listing below reproduces it in a toy model of about thirty lines and verifies all ten rows. It is not lakeFS's code: it is there to see the rules, because the rules are what you need to understand.
"""Toy model: it is NOT lakeFS's code, it reproduces the rules described in its documentation."""
from types import MappingProxyType
class Commit:
def __init__(self, files, parents=()):
self.files = MappingProxyType(dict(files)) # frozen content: a commit never changes again
self.parents = parents
def ancestors(c):
seen, queue = [], [c]
while queue:
x = queue.pop(0)
if x not in seen:
seen.append(x)
queue.extend(x.parents)
return seen
def common_base(a, b):
sa = ancestors(a)
return next(x for x in ancestors(b) if x in sa) # the first ancestor of b that is also an ancestor of a
def merge(base, source, dest, strategy=None):
"""Three-way, file by file. None = file absent."""
result, conflicts = {}, []
for p in sorted(set(base) | set(source) | set(dest)):
a, b, c = base.get(p), source.get(p), dest.get(p)
if b == c: chosen = b # same (or deleted) on both sides
elif a == b: chosen = c # changed only in the destination
elif a == c: chosen = b # changed only in the source
elif strategy == "source-wins": chosen = b
elif strategy == "dest-wins": chosen = c
else:
conflicts.append(p); continue
if chosen is not None:
result[p] = chosen
return result, conflicts
# 1) the ten rows of the "How Merges Work" table (A, B, C = contents; X = absent)
ROWS = [("A","A","A","A"), ("A","B","B","B"), ("A","B","C","conflict"), ("A","A","B","B"),
("A","B","A","B"), ("A","X","X","X"), ("A","B","X","conflict"), ("A","X","B","conflict"),
("A","A","X","X"), ("A","X","A","X")]
val = lambda s: None if s == "X" else s
for base, src, dst, expected in ROWS:
res, conf = merge({"f": val(base)} if base != "X" else {}, {"f": val(src)} if src != "X" else {},
{"f": val(dst)} if dst != "X" else {})
outcome = "conflict" if conf else (res.get("f") or "X")
assert outcome == expected, (base, src, dst, outcome)
print(f"base {base} source {src} destination {dst} -> {outcome}")
# 2) a branch is a pointer: creating it copies nothing
main = Commit({"orders/2026-11-12.parquet": "h1"})
branches = {"main": main}
branches["etl"] = branches["main"]
print("branch created without copies:", branches["etl"] is branches["main"])
# 3) the work happens on the branch; main does not change until you merge
branches["etl"] = Commit({**branches["etl"].files, "orders/2026-11-13.parquet": "h2"}, (main,))
print("main sees the new file:", "orders/2026-11-13.parquet" in branches["main"].files)
res, conf = merge(common_base(branches["main"], branches["etl"]).files, branches["etl"].files, branches["main"].files)
print("merge:", res, "conflicts:", conf)
# 4) two branches that change the same file in different ways
a = Commit({**main.files, "orders/2026-11-12.parquet": "h-fix-A"}, (main,))
b = Commit({**main.files, "orders/2026-11-12.parquet": "h-fix-B"}, (main,))
base = common_base(a, b).files
print("without strategy:", merge(base, a.files, b.files))
print("source-wins: ", merge(base, a.files, b.files, "source-wins"))
# Output:
# base A source A destination A -> A
# base A source B destination B -> B
# base A source B destination C -> conflict
# base A source A destination B -> B
# base A source B destination A -> B
# base A source X destination X -> X
# base A source B destination X -> conflict
# base A source X destination B -> conflict
# base A source A destination X -> X
# base A source X destination A -> X
# branch created without copies: True
# main sees the new file: False
# merge: {'orders/2026-11-12.parquet': 'h1', 'orders/2026-11-13.parquet': 'h2'} conflicts: []
# without strategy: ({}, ['orders/2026-11-12.parquet'])
# source-wins: ({'orders/2026-11-12.parquet': 'h-fix-A'}, [])
The merge function is the heart: if source and destination have the same content, that content wins; if only one of the two differs from the base, the one that changed wins; if both changed in different ways, or one modified and the other deleted, there is a conflict. The final scene shows two branches that fix the same file in different ways: without a strategy the merge stops and the conflicting file does not enter the result, while with source-wins the source wins. The documentation provides source-wins and dest-wins and specifies that the strategy applies to all conflicting files, without being able to decide case by case.
Two practical consequences come from the fact that lakeFS is format-independent: the merge works on whole files, and it cannot merge two versions of the same Parquet file row by row. If two pipelines rewrite the same partition, you will get a conflict, not a fusion. The project rule that follows is simple: each pipeline writes to its own paths, for example one partition per day, and the merge almost always becomes an addition.
Where the data live and where the metadata do
lakeFS does not contain the data: it leaves them where they are, in the object storage, and keeps the metadata aside. The documentation describes two levels, both worth knowing because they are what you will have to administer:
- committed commits are stored as SSTable files in the same object storage, under the
_lakefs/prefix; - repositories, branches, tags and uncommitted changes live in a key-value store, for which there are drivers for PostgreSQL and DynamoDB, while an in-memory implementation is offered only for experimenting, without persistence.
The operational meaning is that the database is not a detail: it holds the state of repositories, branches, tags and changes in progress. Treat it as the main state of the service. The documentation describes a backup of the references with the lakefs-refs.py script (dump and restore commands), which I read but did not try. Do not use the in-memory store outside a trial. In this environment I did not run a lakeFS server; what follows is taken from the documentation.
The write, audit, publish flow
The most useful way to use lakeFS in a pipeline is a three-step path that the documentation calls write-audit-publish: you write to a branch, you check the data, and only if the check passes do you merge into main. The Python package to use is lakefs, the high-level SDK. The following block prints the real signatures of the methods you use in the flow, so as not to leave you to a tutorial that might be old.
import inspect
from importlib.metadata import version
import lakefs
def params(function):
"""Name and default value of each parameter, without self and without **kwargs."""
out = []
for p in inspect.signature(function).parameters.values():
if p.name == "self" or p.kind is p.VAR_KEYWORD:
continue
out.append(p.name if p.default is p.empty else f"{p.name}={p.default!r}")
return ", ".join(out)
print("lakefs", version("lakefs"), "| lakefs-sdk", version("lakefs-sdk"))
entries = [(lakefs.Repository, "__init__"), (lakefs.Repository, "branch"),
(lakefs.Branch, "create"), (lakefs.Branch, "commit"), (lakefs.Branch, "merge_into"),
(lakefs.Branch, "revert"), (lakefs.Branch, "transact"), (lakefs.Branch, "delete"),
(lakefs.WriteableObject, "upload")]
for cls, name in entries:
print(f"{cls.__name__}.{name}({params(getattr(cls, name))})")
# Output:
# lakefs 0.16.0 | lakefs-sdk 1.88.0
# Repository.__init__(repository_id, client=None)
# Repository.branch(branch_id)
# Branch.create(source_reference, exist_ok=False)
# Branch.commit(message, metadata=None)
# Branch.merge_into(destination_branch)
# Branch.revert(reference, parent_number=0, reference_id=None)
# Branch.transact(commit_message='', commit_metadata=None, delete_branch_on_error=True, tag=None)
# Branch.delete()
# WriteableObject.upload(data, mode='w', pre_sign=None, content_type=None, metadata=None)
The signatures say more than a description: to create a branch you pass source_reference, the commit requires the message and accepts a dictionary of metadata, merge_into takes the destination, and transact wraps several operations in a transaction on an ephemeral branch that is merged into the source only if everything succeeds. The following flow is a schema to adapt, not executed: it needs a lakeFS server and credentials. The SDK reads them from ~/.lakectl.yaml or from the LAKECTL_* variables.
# Schema to adapt, not executed: it requires a reachable lakeFS server and credentials.
# The signatures used here are the ones printed by the previous block.
import duckdb
import lakefs # reads endpoint and keys from ~/.lakectl.yaml or from the LAKECTL_* variables
repo = lakefs.Repository("analytics")
main = repo.branch("main")
work = repo.branch("etl-2026-11-13").create(source_reference="main") # zero copy: just a pointer
# 1) I write to the branch, never to main
with open("orders_2026-11-13.parquet", "rb") as f:
work.object("orders/2026-11-13/orders.parquet").upload(data=f.read(), mode="wb")
# 2) I check the data BEFORE publishing it: DuckDB reads the branch from lakeFS's S3 gateway
con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")
null_rows = con.execute(
"SELECT count(*) FROM read_parquet('s3://analytics/etl-2026-11-13/orders/2026-11-13/orders.parquet') "
"WHERE customer_id IS NULL").fetchone()[0] # the S3 settings must be configured as in the SQL block
# 3) I publish or discard
if null_rows == 0:
work.commit(message="Orders of 2026-11-13", metadata={"source": "erp", "check": "ok"})
work.merge_into(main)
else:
work.delete() # the branch and the uncommitted changes disappear
In the schema the check is a single query, but it can be anything: a schema, counts, tolerances. The important point is where it happens: on the branch, before the merge. If it fails, work.delete() removes the branch with its uncommitted changes and main was never touched. To have the check enforced by the server and not by your script, the documentation describes pre-commit and pre-merge hooks, defined in a YAML actions file: if a check fails, the merge into main is blocked. They are a governance feature, so check in your edition that it is included.
Reading a branch with DuckDB
To query the data of a branch with DuckDB, the simplest way is lakeFS's S3 gateway: the URI has the form s3://<repository>/<reference>/<path>, where the reference is a branch, a tag or a commit, according to the reference expressions described in the documentation. The DuckDB integration page shows the configuration and warns that, going through the gateway, lakeFS enters the data path and proxies the operations, so it is less efficient than the Iceberg REST catalog that it recommends as the main route. The Iceberg REST catalog appears among the features of the Team edition.
-- Schema to adapt, not executed: example values from the DuckDB page of the lakeFS documentation.
INSTALL httpfs;
LOAD httpfs;
SET s3_region = 'us-east-1'; -- with non-S3 storage the documentation still indicates us-east-1
SET s3_endpoint = 'lakefs.example.com'; -- host (and port) of the lakeFS server, not of the bucket
SET s3_access_key_id = '<lakefs-key>';
SET s3_secret_access_key = '<lakefs-secret>';
SET s3_url_style = 'path';
-- SET s3_use_ssl = false; -- only if the server listens in clear, for example locally
-- the URI schema is s3://<repository>/<reference>/<path>
SELECT count(*)
FROM parquet_scan('s3://analytics/main/orders/**/*.parquet', HIVE_PARTITIONING = 1);
The settings are those of any S3 connection (see the article DuckDB in production), with two differences: the endpoint is the lakeFS server and not the bucket, and the keys are those of the lakeFS user. To make an analysis reproducible, put the identifier of a commit in the path instead of the branch name: the branch moves, the commit does not. Any query of today will then give the same result in six months, as long as the commit is not removed by retention.
Going back: revert from the command line
If a faulty load is already in main, the documented command is lakectl branch revert, which records a new commit that undoes the effects of the indicated commit. It does not rewrite history: it adds to it, and the faulty commit remains viewable. The rollback guide presents it as an emergency measure to stop an error while you analyze the cause. The file shows the whole cycle with the commands of the CLI reference page; this too is a schema, not executed.
# Schema to adapt, not executed: the commands and options are those of the lakectl page of the documentation.
lakectl branch create lakefs://analytics/etl-2026-11-13 -s lakefs://analytics/main
lakectl commit lakefs://analytics/etl-2026-11-13 -m "Orders of 2026-11-13" --meta source=erp
lakectl merge lakefs://analytics/etl-2026-11-13 lakefs://analytics/main
# if a load brought wrong data into main: undo the effects of that commit
lakectl log lakefs://analytics/main --amount 5
lakectl branch revert lakefs://analytics/main <id-of-the-commit-to-revert>
The order of the commands is Git's: branch create with the -s option to indicate the source, commit with -m and --meta key=value pairs, merge from source to destination. The log serves to find the identifier of the commit to undo: use the --amount option to limit the rows. Writing the day of the data, the source and the outcome of the checks in the commit metadata saves you a trip through memory when you need to work out which commit to undo.
Really deleting: retention is not automatic
A point that people coming from Git underestimate: by default lakeFS keeps all objects forever, and that is what makes time travel possible. Deleting a file in a branch does not remove it from storage. The documentation describes a garbage collection job (a Spark program in the standard version, while Enterprise users can use a standalone program) that removes from storage the objects that were deleted or replaced and have expired under retention rules you define per branch, plus the uncommitted objects that have become unreachable.
This matters if you have deletion obligations, for example a request to erase personal data: deleting and committing is not enough. You need an adequate retention rule and the execution of the job, and you must verify that the object is not reachable from other branches or commits. It is a technical matter with legal consequences, which you will not find solved by a line of code: bring it to your data protection officer.
Where lakeFS is not enough
lakeFS versions objects, not tables. If your data are Iceberg or Delta tables, those formats already have their own snapshots and time travel, per table; lakeFS adds consistent versioning of several tables together and of non-tabular files, with a single commit. The comparison between the three formats is in the article Iceberg, Delta Lake and Hudi, processing files with DataFrames is in Polars and pandas, with the measurements, and a pipeline that uses them with Polars is in DuckDB and Polars: a lean pipeline. To version small sets of data together with the code there is DVC, which lakeFS has acquired and which, according to the announcement, remains under Apache 2.0.
On the cost of adoption, two items to estimate before starting: the server and its database are additional services to keep running and to upgrade, and every access through the gateway goes through there. For a small team with few datasets and a daily pipeline, a folder convention with dates and a check before publishing might be enough; lakeFS is justified when publication mistakes cost more than running it.
What you can do now
You can evaluate lakeFS with the license in hand, explain why a branch does not copy data and when a merge conflicts, set up the write-audit-publish flow with the Python SDK, read a branch or a commit from DuckDB, undo a load with revert and know that deleting a file does not free storage until garbage collection runs. To fit this piece into a complete architecture, the last article of the series is Modern Data Stack: a minimal architecture.
Frequently asked questions
Is lakeFS still open source?
Since version 1.87.0 the Community edition is distributed under the Business Source License 1.1, which is not a recognized open source license; the code is public and each version becomes Apache 2.0 four years after publication. The earlier versions remain Apache 2.0.
Can I use lakeFS in production at my company?
Yes, under the terms of the use grant: unmodified program and internal use only. You cannot offer it as a service to third parties nor use a modified version in production. For doubtful cases the vendor invites you to contact it.
Which database does lakeFS require?
The documentation describes a key-value store with drivers for PostgreSQL and DynamoDB; there is also an in-memory store, useful only for experimenting because it does not persist the data.
How do you handle a merge conflict?
The merge stops if the same file changed in different ways in the two branches, or was modified on one side and deleted on the other. You can rerun it with the source-wins or dest-wins strategy, which applies to all conflicts, or fix the branches beforehand.
Sources
- lakeFS: Community Edition is Moving to the Business Source License - move from Apache 2.0 to BSL from version 1.87.0, what is allowed, consulted on 2026-11-13
- lakeFS: LICENSE file (Business Source License 1.1) - Licensor, additional use grant, change date and change license, consulted on 2026-11-13
- lakeFS documentation: Internals - zero-copy branches, immutable commits, three-way merge table, strategies, metadata and key-value store, consulted on 2026-11-13
- lakeFS documentation: Editions - differences between Community, Team and Enterprise and the price of the Team plan, consulted on 2026-11-13
- lakeFS documentation: DuckDB - S3 gateway configuration and URI form, consulted on 2026-11-13
- lakeFS documentation: lakectl (CLI reference) - syntax of branch create, commit, merge, log and branch revert, consulted on 2026-11-13
- lakeFS documentation: Garbage Collection - default object retention and retention rules, consulted on 2026-11-13
- lakeFS documentation: Backup and Restore - the
lakefs-refs.pyscript to save and restore the references, consulted on 2026-11-13 - lakeFS documentation: Actions and Hooks - pre-commit and pre-merge hooks and action files, consulted on 2026-11-13
- PyPI: lakefs - version 0.16.0 of the high-level SDK and Apache-2.0 license, consulted on 2026-11-13
The toy model and the block with the SDK signatures were run with Python 3.14.4 and lakefs 0.16.0, twice with identical output. No lakeFS server was started: the schemas for the flow, the gateway and the CLI are to be adapted and were not executed, and they report syntax read in the official documentation.

Commenti
Caricamento commenti...
Sign in to comment