Fractera Memory Starter
The deterministic, multimodal, self-evolving memory engine for autonomous AI agents
An autonomous, self-hosted long-term memory engine and the cognitive core for AI agents. Built to work as the architect's personal command centre through Telegram and a unified REST API, it closes the gap between a volatile context window and real cognitive continuity.
The engine ingests raw, unstructured real-world input — text, images, voice notes, whole PDF documents, video, precise spatial-temporal coordinates and dates — and turns it into an indexed knowledge graph and structured relational stores, without unnecessary model calls and without per-request token costs.
- Zero per-request fees
- Zero vendor lock-in
- Full privacy on your server
The architect's operating system
Standard RAG pipelines and vector stores make agents lose critical context at every session reset, burn compute re-reading long logs, and never synthesise personal experience over time.
Fractera Memory works as a black box engine: in go multimodal input and runtime context parameters, out come structured objects, synthesised data, verified conclusions or actionable reports. One architecture unifies four storage layers under a deterministic multi-level router.
How a request travels
One entry point, one router, two very different costs behind it.
Zero tokens, no model, sub-10 ms
A model turn: hypothesis chains and reports
End-to-end schema adaptability
No manual migrations, no static schema design. The engine adapts its schema on the fly — adding columns, and generating fully typed relational SQL tables whenever new structured entities and relationships appear.
Cost-first architecture: the cost ladder
Every request is resolved with the minimum compute that can answer it. A query escalates only when the cheaper, deterministic tiers fail to produce a complete answer.
| Level | Retrieval mechanism | Cost and purpose | Opened by |
|---|---|---|---|
| Level 1 | Direct SQL / key-value query, no model | $0 / 0 tokens. Sub-10 ms latency. Exact factual properties. | Engine router |
| Level 2 | Single-pass model call without conversation history | Minimal. Direct execution and simple parsing. | Engine router |
| Level 3 | Knowledge graph traversal plus a context session | Low. Context retrieved without generating model tokens. | Engine router |
| Level 4 | Semantic vector store retrieval | Higher. Fuzzy semantic search across historical context. | Caller — depth: deep |
| Level 5 | Bounded recursive deep reasoning, up to 10 minutes | Maximum. Multi-hypothesis research and unstated facts. | Caller — depth: extreme |
«What is my passport number?» resolves instantly at level 1 for zero tokens. «Which of my contacts could have known this person?» escalates through levels 3–5 and comes back as a probabilistic reasoning chain.
Spatial-temporal context
Time and coordinates are first-class indexes here, not flat text tags.
An empty spatial-temporal scope means «location and time unknown» — never «everywhere and always».
«Which taxi service do I usually use here?» asked in Madrid returns Madrid knowledge, and never collides with or overwrites the same question answered in London.
A built-in spatial index over lat, lon and radius_m answers proximity queries: notes, expenses and records near this point.
Knowledge becomes an object, not a paragraph
Asked to summarise complex data — last month's spending, a project's state — memory does not hand back a wall of text. It builds the thing you asked for:
- Instantiates a structured entity: a typed table with the columns the answer needs.
- Compiles, sorts and formats a clean Markdown artifact with its own Object ID.
- Returns a short executive summary next to the artifact, so the answer reads well and the detail stays referenceable.
The memoization loop
Nothing expensive is paid for twice. Every high-cost chain is folded back down into the cheaper tiers.
- An expensive computation or research loop runs at level 4 or 5
- An artifact is created with its ID, alongside a concise conclusion
- The conclusion is indexed into the vector store, the knowledge graph and the tables
- Repeat questions are answered in 0.2 s at levels 1–3, for zero tokens
A self-evolving skill core with shadow A/B testing
When the engine detects repeated misses or a sub-optimal path, it writes a candidate skill and runs it as a challenger in the shadow — on real production traffic, while people keep being answered by the verified champion.
The model is forbidden from scoring its own work. Verdicts come from outside — explicit architect feedback and strict compute-cost ratios.
A challenger is promoted to champion only when it wins on external quality metrics with no regression in speed or cost.
Every modification is a commit. One click reverts the instructions to the baseline version through Git, with no data loss.
Four unified storage tiers
Four layers, one contract. The caller never learns which of them answered.
Tabular structures, typed facts, exact entity properties.
High-dimensional semantic embeddings for fuzzy similarity search.
Directional links between entities, people and events.
Local binary storage for raw attachments: PDF, images, audio, video.
Native multimodality
Not a preprocessor bolted on the side. The pipeline lives inside the engine.
Local speech-to-text transcription through a Whisper pipeline.
Scene captioning through vision, plus OCR text extraction.
Audio track extracted and transcribed, key frames processed by vision.
Native text parsing, OCR fallback for scans, structural summarisation.
Testing and verification in the built-in playground
The engine ships with an interactive bench. It is not a demo page: it is where an integration is proven before it is written.
- Execute direct API requests against the memory core with no front-end abstraction in the way.
- Inspect raw JSON payloads, execution timings and exact model token usage.
- Verify the request body before committing a line of client code.
/{lang}/settings?section=memory-test
How it compares
Two comparisons: one against the categories of memory tooling, one against a ready-made assistant of a different philosophy.
| Capability | Fractera Memory | Standard RAG frameworks | MemGPT / Letta | Mem0 / Zep |
|---|---|---|---|---|
| Storage architecture | Hybrid: graph + vector + relational + object store | Vector DB only | Relational / text files | Vector plus a basic graph |
| Zero-token reads | Yes — deterministic paths at levels 1–3 | No | No | Partial |
| Native multimodality | Built in: audio, video, PDF, images | Requires external parsers | Requires external parsers | Text focused |
| Spatial proximity indexing | Native lat/lon radius search | Text matching only | Function calling only | Basic metadata |
| Skill evolution | Champion / challenger A/B testing | None | Manual prompt edits | None |
| Self-hosted / open source | 100% on-premise, single node | Varies | Yes | Freemium / cloud |
| Capability | Fractera Memory | IVA Agent (smixs/iva-agent) |
|---|---|---|
| System classification | An autonomous memory engine behind an API, for any front-end | An end-to-end Telegram assistant tied to an Obsidian vault |
| Architecture | A decoupled microservice; the Telegram bot is an optional client | A monolith: Telegram, userbot and vault manager in one codebase |
| Cost optimisation | A five-tier deterministic router; instant zero-token reads | Every operation leans on model passes, BM25 and vector lookups |
| Data processing | Dynamic SQL tables, structured artifacts, a knowledge graph | Markdown cards written to a folder for Obsidian to sync |
| Integrations | Many front-ends at once over one REST API | Bound to one Telegram account and an Obsidian setup |
API quickstart
One REST API, one key. Every example below runs against a live instance as it stands.
curl -X POST https://memory.your-domain.com/v1/remember \
-H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
-d '{
"who": "roman",
"text": "Office lease note",
"media": [{ "kind": "audio", "url": "https://.../note.oga" }],
"scope": [{ "at": "2026-09-11", "lat": 40.4168, "lon": -3.7038, "radius_m": 500 }]
}'curl -X POST https://memory.your-domain.com/v1/recall \
-H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
-d '{
"who": "roman",
"text": "What notes or files did I save within 500 meters of here?",
"scope": [{ "lat": 40.4168, "lon": -3.7038, "radius_m": 500 }]
}'curl -X POST https://memory.your-domain.com/v1/recall \
-H "Content-Type: application/json" -H "x-memory-key: YOUR_MEMORY_KEY" \
-d '{
"who": "roman",
"text": "Summarize all my taxi expenses from last month into a table",
"depth": "deep",
"want_chain": true
}'Installation
There is exactly one thing to know about installing this.
One run of the Fractera installer robot on your own server brings up every microservice of the platform, memory included — nginx, certificates and the access key are arranged for you. There is nothing to assemble by hand.
Design principles
All data, graphs and media stay strictly on your machine. No telemetry, no hidden cloud dependency.
What a person stated is logged as fact (said); what the engine inferred is flagged as hypothesis (guess) and stored only with its evidence (basis).
Connect the official Telegram starter, or attach your own web chat, mobile app and automation pipelines over HTTP. The bundled console is a microservice of its own, and it is optional.
Questions and answers
Short answers to what people ask before they integrate.
Does every request cost tokens?
No. The engine answers levels 1 to 3 without a model at all: a direct lookup, a graph traversal, a conclusion already folded back into the stores. A model turn is spent only when the cheap deterministic paths return nothing, and the answer reports depth_used so you can see what you paid for.
Can it answer questions about a place by coordinates, not by a word?
Yes. A scope entry carries lat, lon and an optional radius_m, and the coordinates are spatially indexed. You can ask what you know within 500 metres of a point, and knowledge recorded in Madrid never merges with knowledge recorded in London.
What can I send besides text?
Voice notes, images, video, PDF and HTML. The pipeline lives inside the engine: audio is transcribed, images are captioned and read by OCR, video has its track transcribed and its key frames captioned, PDFs are parsed with an OCR fallback. The original binary stays in the built-in object store and is referenced from answers by id.
What schema do I have to design first?
None. You send a sentence. The engine adds columns as new kinds of fact appear and generates typed relational tables when a kind grows into an entity. There are no migrations to write.
What happens after an expensive research run?
It folds the result back. The artifact goes to the object store, its summary into text, into the vector store and into the knowledge graph, and the relation tables are updated. The same question is then answered from the cheap levels, in fractions of a second.
How does it improve itself without breaking what works?
It writes a second version of the skill and runs it as a challenger in the shadow, on real traffic, while people keep being answered by the champion. Promotion needs an external verdict and no regression in cost: the engine is never allowed to grade its own work.
What can I connect to it?
Any HTTP client: a Telegram bot, a web chat, a mobile app, a scheduled job. The engine also ships with its own console, already connected, and that console is optional: nothing in the API path depends on it.
Where does my data live?
On your server, in your database, in your object store, behind a key you can revoke in one click. There is no metered API in the middle and no telemetry leaving the machine.
The Fractera project on GitHub
Fractera Memory is one microservice of the Fractera platform, the engineering infrastructure for autonomous agents. The whole project, this engine included, is open source.
The Fractera project on GitHubSee how it is built
Read the full design in the passport — the document written before the code and kept in step with it ever since.