AI Memory with 95% Fewer Tokens, Without Graphs or LLM Bookkeeping

As conversations grow, sending the full history with every request increases token costs and can slow responses. Relevant facts become mixed with unrelated exchanges, and models may use them less reliably even before the context window is full. If the conversation continues long enough, its history eventually exceeds that window and no longer fits in a single request. Long-term memory must keep earlier information accessible while remaining reliable, responsive and affordable to build and use.

We built a LambdaDB memory pipeline that preserves source text and retrieves relevant passages for each answer. It builds no knowledge graph and uses no generative LLM calls to construct or maintain memories. LambdaDB data versioning also lets conversation branches and checkpoint restores reuse indexed memory without re-embedding shared history, while keeping later changes separate.

On all 500 questions in LongMemEval, the pipeline scored 96.0%, compared with 95.2% for full history under the same Jev evaluation protocol. Mean answer input fell from 105,183 to 4,886 tokens. Including initial memory preparation, retrieval, answer generation and LambdaDB usage, estimated execution cost for the 500 cases was $7.51–$9.42, compared with $105.53 for full history: about 91–93% lower.

Our open-source SillyMemory extension brings this design to SillyTavern, a platform for creating and chatting with AI characters. You can inspect the code and try the memory workflow in your own conversations.

Build Memory Without Graphs or LLM Bookkeeping

The design echoes the move from ETL (extract, transform, load) to ELT (extract, load, transform): preserve source data for future questions and uses. Affordable cloud storage and processing helped make that shift practical. Conversational memory needs the same economics: storing, indexing and searching source text must remain affordable as histories grow.

For AI memory, this flexibility matters even more because future natural-language questions are open-ended. We preserve original wording, source references and timestamps so interpretations made at ingestion do not decide which evidence survives. Chunking and embeddings make the text searchable; the answer model interprets retrieved passages in the context of each question. Edits, deletions and conversation branches remain ordinary data operations.

The memory pipeline follows three steps:

1. Index source text. Split conversation text into chunks with source references. LambdaDB native embeddings generate and store vectors directly from the text. No generative LLM extracts entities, infers relationships, resolves conflicts between facts, or rewrites memories into summaries.
2. Embed the query, search and rerank in one request. Send the current question as text. LambdaDB generates its query vector, combines lexical and semantic retrieval through Bayesian hybrid search, then applies Jev-based native reranking to return ranked source passages. The client receives these passages without generating vectors, coordinating separate inference calls or managing embedding and reranking provider keys.
3. Assemble the answer context. Select passages by relevance within a bounded memory budget, then pass them to the answer model with the question and instructions.

The application handles source integrity and prompt assembly. LambdaDB provides persistent storage and search, with independently scalable indexing and retrieval compute over durable object storage.

Full-History Quality with 95% Fewer Input Tokens

LongMemEval tests fact retrieval, reasoning across sessions, knowledge updates, temporal reasoning and abstention. We evaluated our pipeline on the cleaned LongMemEval-S dataset using gpt-6.1-sol with low reasoning.

All configurations use the same Jev 1.13.0 Noul judge, original task-specific rules and Yes probability ≥0.50 threshold.

Memory with LambdaDB used text-embedding-3-small embeddings, 800-character chunks, 200 hybrid-search candidates and Jev-based native reranking returning up to 50 chunks. Selected memory fit a 4,096-token budget and was presented chronologically alongside at least five recent messages kept outside the index, with adjacent user/assistant pairs kept together.

The full-history reference received the complete transcript. Built-in Vector Storage used Small embeddings and 400-character chunks; built-in Summarize targeted 200 words every ten messages. The separate 400-character summary method used an LLM to create and select summaries, reducing answer input while adding memory-processing work.

Configuration	Accuracy (measured cohort)	Mean answer input tokens	Additional memory tokens / case (mean)	Estimated cost, USD /500 cases
Plain 32K	32.6% (163/500)	28,056	None	$28.37 Answer only
Built-in Vector Storage, 32K	34.6% (173/500)	28,057	Embeddings: 104,209 input	$29.42 Answer: $28.38 Memory: $1.04
Built-in Summarize, 32K	34% (17/50)†	28,100	Summary LLM: 1,221,188 input + 14,329 output†	$1,321.27† Answer: $28.44 Memory: $1,292.83
400-character segment summaries	68% (34/50)†	1,555	Summary + selection LLM: 122,912 input + 1,449 output†	$131.94† Answer: $1.78 Memory: $130.16
Full history, 128K	95.2% (476/500)	105,183	None	$105.53 Answer only
Memory with LambdaDB	96.0% (480/500)	4,886	Embeddings: ≈103,482 input Jev reranking: ≈28,860–115,440 input (modeled)	$7.51–$9.42 Answer: $5.20 Memory: $2.31–$4.23

The pipeline achieved comparable full-history quality with 95.35% fewer answer input tokens and about 91–93% lower estimated execution costs than full history, including memory preparation, retrieval, answer generation, and LambdaDB.

Combining Facts Across Conversations

This example from the recorded 500-question run shows how Memory with LambdaDB supplied facts from two earlier sessions for the answer model to combine. The question was:

How much did I save on the Jimmy Choo heels?

Among the retrieved passages were two user messages from different sessions. Shortened source excerpts:

• May 23, 2023: “...Jimmy Choo heels ... at the outlet mall for $200...”
• May 29, 2023: “...originally retailed for $500.”

Both passages were included in the final prompt, among 27 selected chunks. The recorded answer was:

You saved $300 on your Jimmy Choo heels—you paid $200 at the outlet mall, compared with the original retail price of $500. That’s 60% off!

Memory with LambdaDB preserved both prices in the final prompt, and the answer model calculated their difference. The resulting $300 saving matches the dataset's reference answer. This request used 4,999 answer input tokens, including selected memory, recent dialogue and instructions. The example comes from the public LongMemEval dataset, question bb7c3b45.

The Cost of Remembering

Estimates cover initial memory preparation and one answer for each of 500 separate histories, using standard LLM rates and LambdaDB Pay as You Go rates without cache adjustments or free allowances. Reranking and storage costs are modeled, with two hours of storage; application hosting and benchmark judging are excluded. The benchmark report records usage and operation counts.

For Memory with LambdaDB, answer generation accounts for $5.20, while preparing and retrieving memory adds an estimated $2.31–$4.23. The memory cost includes $1.09 for initial embeddings, $0.58 for database writes, reads and query embeddings, $0.64–$2.55 for reranking, and less than $0.02 for storage during the run.

Scale Memory Without Scaling the Prompt

Searchable histories reached 10M tokens while mean answer input stayed around 13K. BEAM and BEAM-10M extend the evaluation to longer conversations. We followed past.dev’s published BEAM evaluation protocol, including its answer-generation hints, across 100 conversations and 2,000 questions.

This configuration uses text-embedding-3-small embeddings, 800-character chunks, 200 hybrid-search candidates and Jev-based native reranking, returning up to 100 chunks. Evidence receives an 8,000-token estimated source-text allocation; metadata and instructions add to the final input. gpt-5.6-luna, with reasoning disabled, handles answer generation and judging.

History bucket	Conversations / questions	Memory with LambdaDB /100	past.dev published /100	Mean answer input tokens (our pipeline)
100K	20 / 400	89.00	92.08	12,746
500K	35 / 700	85.99	89.63	13,126
1M	35 / 700	86.62	90.65	12,995
10M	10 / 200	80.64	85.03	13,641

Using the same answer and scoring protocol, our pipeline came within 3.1–4.4 points of past.dev’s published results across all four history sizes. This is a strong result for memory built from original-text indexing and retrieval, without a knowledge graph or generative memory construction.

The current pipeline makes a single retrieval pass per question, leaving more adaptive search strategies to explore. Multi-session reasoning is a clear improvement target: at 10M tokens, it scored 29.42/100, compared with 28.17 in the published reference. The ability breakdown provides concrete targets for further optimization.

Both evaluations have public configurations, recorded aggregates and reproduction commands in the benchmark directory. The LongMemEval and BEAM guides cover ingestion, retrieval, generation, judging and verification; neither standalone benchmark requires SillyTavern.

Branch the Conversation Without Rebuilding Memory

Branching a conversation should reuse shared history while keeping later choices separate. We map each story to a LambdaDB collection and each conversation path to a branch. Original chat files stay on the SillyTavern host; searchable passages and vectors live in LambdaDB.

• Branch: Create a LambdaDB branch from the parent path, then apply the changes needed to match the selected transcript.
• Frozen checkpoint: Preserve the complete original JSONL transcript, including recent messages, together with an unchanged LambdaDB branch of indexed older history.
• Resume: Copy the saved transcript into a new chat and create a writable branch from the checkpoint branch. Later changes affect only the resumed path.

LambdaDB data versioning shares unchanged data and index files across branches, so inherited memory needs no full copy or re-embedding. The checkpoint design describes the implementation.

Branch Indexed Memory in Milliseconds

For a 500-message restored chat, full memory preparation took about 2.4 minutes with built-in Vector Storage's local embeddings, compared with about 1 second with SillyMemory. The LambdaDB branch-create call itself took 64–77 ms across both transcript sizes. These measurements start after the original transcript is loaded.

Restored conversation	Built-in Vector Storage: full memory preparation	SillyMemory: full memory preparation	LambdaDB branch-create API round trip (included)
50 messages	12.68–12.88 s	0.45–0.55 s	67.8–76.9 ms
500 messages	142.17–142.47 s	1.00–1.01 s	64.4–68.8 ms

With ten times as many messages, branch creation stayed in the same latency range. SillyMemory's full preparation also compares inherited passages with the restored transcript, so it includes synchronization work that grows with the history.

Built-in Vector Storage re-embedded all 50 or 500 messages per restored path using its default local quantized Jina model. LambdaDB reused the existing memory with zero new document submissions on restore.

Try It in SillyTavern

SillyMemory brings LambdaDB's storage, search and data versioning into a working conversational application, with source synchronization and prompt assembly handled by the extension.

You can try this pipeline with SillyMemory, an open-source extension for SillyTavern. Start with the official SillyTavern installation guide for SillyTavern 1.19.0 and configure a chat model connection.

1. Prepare your LambdaDB project. Sign up for LambdaDB for free and create a project in your preferred region. Once the project is ready, open its API Keys tab and select Create key (or Create your first key). Name and create the key, then save it securely—the full value is shown only once. Keep the project's regional base URL, project name and API key ready for step 3. The LambdaDB quickstart walks through these steps.
2. Install. Open Extensions → Install extension and paste https://github.com/lambdadb/SillyMemory. Leave the branch field empty to install from main, then reload.
3. Connect. Open Extensions → SillyMemory, enter the regional endpoint (base URL), project name and API key from step 1, then select Connect and prepare.
4. Enable memory. Open a character chat. If built-in Vector Storage's chat vectorization or other extensions that rewrite chat history are enabled, turn them off. Expand Memory above the composer and select Turn memory on in this chat. Wait for synchronization to finish, then start chatting.

Once memory is ready, ask about details from earlier in the conversation and inspect the selected passages in Memory. View full memory opens the prompt inspector. Use SillyTavern's Create Branch and Save checkpoint controls to explore another conversation path and return to a saved point. For the frozen checkpoints described above, use Advanced → Frozen memory snapshots and its separate save/resume controls. Each chat has its own memory on/off setting, and the original transcript stays available.

The extension installation guide covers the supported setup. Older chat text, metadata and retrieval queries are processed by LambdaDB and its managed inference providers; your chat model's charges are separate.

To build memory into your own assistant, start with LambdaDB's native embeddings and native reranking guides. The SillyMemory source shows how these APIs fit together with source tracking, prompt assembly and conversation branching.

From Retrieval to Query-Time Reasoning

Next, we want to explore Extract, Load, Think for AI memory: preserve original conversations in searchable storage, then add model-guided reasoning at query time. A small, cost-efficient model could interpret questions, rewrite queries and decide when to search again.

The goal is better reasoning across sessions at a cost that makes memory practical as a default capability. We would trace evidence through to the final prompt and evaluate answer quality, token use, cost and latency together.