Introducing Data Versioning in LambdaDB

Version the knowledge and memory behind AI applications.

A knowledge-base update goes live, and a question your RAG assistant answered correctly yesterday now gets a different answer. The code and model have not changed, but the searchable data has. You need to inspect the earlier data and test a correction while current users continue reading a known version.

We're introducing data versioning in LambdaDB: branches for preparing changes, immutable tags for keeping validated states, and aliases for selecting what readers use. You can also create a branch from retained history with asOf, even if you did not save a checkpoint beforehand.

Prepare, validate, and choose what readers use

Prepare changes without changing what users read. Evaluate a fixed version, serve it through a stable read name, and keep the previous release available for rollback.

Every LambdaDB collection starts with main. References are scoped to that collection. Three reference types support these steps:

• Branches hold independent, writable document histories. Use one to prepare an update or test a correction.
• Tags pin committed snapshots. They preserve a named data version as the branch continues to change.
• Aliases are stable read names pointing to a branch or tag. Retarget an alias to change the version an application reads.

Creating a branch or tag does not require a full copy of the dataset or its search index. Versions share unchanged files, so teams can keep earlier searchable states without rebuilding a separate full index for every release. Shared files can also reuse cache entries across versions.

Use a candidate branch to prepare updates and a tag to identify the data version you evaluate. Point the serving alias at that tag once it passes evaluation, so readers use the same data you tested. The candidate branch can also become the production write target if the application routes future writes there.

If production writes stay on main, apply the changes there and create a tag from main. Evaluate that tagged version before pointing the serving alias at it. LambdaDB does not merge changes between branches.

To return readers to an earlier release, point the alias back to its tag. This changes the read target; writes still explicitly select a branch.

Start from an earlier state

Forking today's data helps you test the next change. It does not, by itself, recover yesterday's data after a problem appears.

With source.asOf, you can create a writable branch from the latest committed snapshot at or before a chosen time. For the unexpected RAG answer, choose a cutoff before the update, inspect the earlier knowledge, and test a correction independently.

You can also use a branch-based asOf source to create a historical tag when you need a fixed reference.

Historical branching requires the relevant branch history to remain available under the collection's snapshot retention policy. Snapshots pinned by existing tags remain preserved beyond the ordinary retention window while those tags exist.

For investigations, also record the model, prompts, and retrieval settings: keeping the data fixed does not make generated answers deterministic.

Where you can use data versioning

The following workflows can be built with these APIs. Each integration supplies its own ingestion and application logic.

RAG knowledge bases and code search

A support assistant can evaluate new product documentation or revised policies before making them available to users. Keep the previous version so that, when answers change, you can compare retrieval against the knowledge used before the update.

Code search adds another requirement: a coding assistant should retrieve the code version relevant to its task. An integration can map Git commits or releases to LambdaDB tags, keeping search results and source reads on the selected version.

After the initial import, the integration processes Git diffs, maps changed files to documents, and applies additions, updates, and deletions to a branch. Use tags to identify indexed revisions and check that each contains the intended source revision before making it available to readers. Earlier tags remain searchable without resubmitting the entire repository for every revision.

An assistant investigating a deployed release can then search that release's source. For a problem spanning a server and an SDK, it can select the relevant indexed version of each repository independently.

Character chat memory

Imagine a saved scene followed by two choices: trust a character or challenge them. Each path should develop its own memories, without leaking them into the other story.

To keep a saved scene available for future branching, create a checkpoint branch for its memory state and keep it unchanged. Create each new story path from that branch, without asOf, so resuming later does not depend on the original branch's historical retention window. A tag preserves the saved state for reads, but the current API cannot create a branch directly from a tag.

If you did not keep a checkpoint branch, asOf can recover an earlier state while the required branch history remains available.

The application routes memory reads and writes to the chosen story branch. It must also coordinate the transcript, selected responses, and character state; restoring retrieval data alone does not restore the whole conversation.

RL post-training for RAG and search

In reinforcement learning (RL) post-training, a model can learn to use search tools. The searchable corpus becomes part of its environment. If both the model and corpus change between runs, it becomes harder to identify what improved the results.

Select a fixed tag for a run while preparing corpus updates on another branch. You can then compare models against the same data, or data versions with the same model. Record the tag and snapshot identity alongside the model, evaluation questions, retrieval settings, and reward configuration. When source facts change, review the corresponding evaluation answers as well.

LambdaDB versions the searchable data; the training framework manages model checkpoints and experiment execution.

Get started

Anyone can sign up for LambdaDB Cloud and use data versioning, including on the Free plan.

Use the LambdaDB SDK for Python (0.9.0+), TypeScript (0.5.1+), or Go (0.4.0+).

Start with an initialized client and an existing knowledge collection containing a body text field. Pass the cutoff as Unix epoch milliseconds within the available history, and choose an unused branch name. Run one language example, or rename investigation between runs. For Python, call the function inside a with LambdaDB(...) as client: block using your client configuration.

Choose your language below. Each example creates a historical branch and searches it:

from lambdadb import BranchSource, LambdaDB, Ref

def search_past_state(client: LambdaDB, cutoff_ms: int):
    collection = client.collection("knowledge")
    collection.branches.create(
        "investigation",
        source=BranchSource.branch("main", as_of=cutoff_ms),
    )
    return collection.query(
        query={"queryString": {"query": "refund", "defaultField": "body"}},
        size=5,
        ref=Ref.branch("investigation"),
    )

Queries select the resulting branch through ref; asOf is a branch-creation selector here.

The Quickstart covers setup. The data versioning guide explains retention, read consistency, and API limits. Detailed integrations will follow in separate articles as runnable open-source demos become ready.