OpenClaw Memory System: File-First Agent Memory and Search

Published on 5/27/2026

OpenClaw Memory System

This document describes OpenClaw's memory implementation as it exists in code. It focuses on durable Markdown memory, plugin registration, indexing, recall, safe reads, compaction flushes, dreaming, and optional backend/plugin branches.

The short version:

  • OpenClaw memory is file-first. Durable memory is ordinary Markdown in the agent workspace.
  • MEMORY.md is canonical long-term memory.
  • memory/YYYY-MM-DD.md and memory/YYYY-MM-DD-<slug>.md are working daily or topical memory.
  • DREAMS.md and memory/dreaming/** are review artifacts, not the durable fact database.
  • The default memory-slot plugin is memory-core.
  • memory-core registers memory_search, memory_get, prompt guidance, pre-compaction flush behavior, runtime access, public artifacts, CLI commands, and optional dreaming.
  • The builtin backend indexes files into SQLite tables for files, chunks, FTS, vector rows, and embedding cache.
  • QMD can replace the builtin search backend, with builtin fallback.
  • Companion plugins such as memory-wiki can add corpora without replacing the active memory slot.
  • Active memory is a separate optional proactive recall plugin. It does not replace memory_search.

There is no hidden durable model state. The index is derived state and can be rebuilt. If memory looks wrong, reindex before assuming the source files are wrong.

Table of Contents

Core Concepts

OpenClaw has two different memory layers that should not be confused:

LayerDurable?Purpose
Markdown filesYesHuman-readable source of truth for long-term and working memory.
Search indexNoSQLite/QMD derived state used for recall. Rebuildable.

Main memory files:

File or directoryPurpose
MEMORY.mdCanonical long-term memory: durable preferences, facts, standing decisions, and behavior guidance.
memory/YYYY-MM-DD.mdDaily working notes and session summaries.
memory/YYYY-MM-DD-<slug>.mdTopic/session-specific daily notes.
DREAMS.mdHuman-readable dreaming diary and review output.
memory/dreaming/**Separate dreaming phase reports.
memory/.dreams/**Machine-facing short-term recall and promotion state.

Important boundaries:

  • MEMORY.md is the durable long-term database, even though it is stored as a plain Markdown file.
  • DREAMS.md is review output. It should not become a source for promotion.
  • memory_search searches indexed chunks; it does not read arbitrary files.
  • memory_get reads bounded excerpts from allowed memory paths or configured extra Markdown paths.
  • Companion corpora such as wiki are supplements, not replacements for the active memory backend.

Layer Map

LayerMain filesResponsibility
User docsdocs/concepts/memory.md, docs/concepts/memory-search.md, docs/reference/memory-config.mdUser-facing model, providers, backends, search tuning, dreaming, and CLI.
Plugin entryextensions/memory-core/index.tsRegisters memory capability, lazy tools, CLI, command, embedding providers, and dreaming hooks.
Tool entry pointsextensions/memory-core/src/tools.tsImplements memory_search and memory_get.
Shared tool helpersextensions/memory-core/src/tools.shared.ts, tools.citations.tsManager lookup, unavailable results, corpus supplements, citations, and injected character limits.
Prompt sectionextensions/memory-core/src/prompt-section.tsAdds "Memory Recall" guidance to the system prompt when tools exist.
Flush planextensions/memory-core/src/flush-plan.tsBuilds the pre-compaction memory flush turn.
Runtime bridgeextensions/memory-core/src/runtime-provider.tsExposes memory manager access to the host runtime.
Backend selectorextensions/memory-core/src/memory/search-manager.tsChooses builtin vs QMD, caches managers, and applies fallback behavior.
Builtin managerextensions/memory-core/src/memory/manager.tsPublic search, sync, status, read, probe, and close behavior.
Sync layerextensions/memory-core/src/memory/manager-sync-ops.tsWatchers, interval sync, session sync, vector readiness, safe reindex, dirty state.
Watch settleextensions/memory-core/src/memory/watch-settle.tsDebounces file watcher events until paths stop changing.
Status stateextensions/memory-core/src/memory/manager-status-state.ts, manager-source-state.tsAggregates source/file/chunk counts and resolves provider/search-mode status.
Embedding layerextensions/memory-core/src/memory/manager-embedding-ops.tsChunk embedding, cache use, batch fallback, vector/FTS writes.
Search helpersextensions/memory-core/src/memory/manager-search.tssqlite-vec KNN, fallback cosine scan, FTS, LIKE fallback, lexical boosts.
Search preflightextensions/memory-core/src/memory/manager-search-preflight.tsTrims empty queries and avoids provider initialization when no indexed content exists.
Rankingextensions/memory-core/src/memory/hybrid.ts, mmr.ts, temporal-decay.tsBM25/vector merge, temporal decay, diversity rerank.
Host SDKpackages/memory-host-sdk/src/host/*File discovery, safe reads, schema, config resolution, session export, multimodal helpers.
Host registrysrc/plugins/memory-state.ts, memory-runtime.ts, registry.tsActive memory capability registry and slot enforcement.
Agent promptsrc/agents/bootstrap-files.ts, src/agents/system-prompt.tsLoads bootstrap files and appends memory prompt sections.
Dreamingextensions/memory-core/src/dreaming*.ts, short-term-promotion.tsManaged cron, phase sweeps, recall tracking, scoring, and promotion.
Promotion budgetextensions/memory-core/src/memory-budget.tsKeeps auto-promoted MEMORY.md sections inside the bootstrap budget.
CLI/statusextensions/memory-core/src/cli.ts, cli.runtime.ts, src/gateway/server-methods/doctor.memory-core-runtime.tsHuman and doctor surfaces for status, reindex, search, promote, REM harness, and repairs.
Public artifactsextensions/memory-core/src/public-artifacts.ts, src/plugin-sdk/memory-host-core.tsExposes memory files and event logs to companion systems.

Data Flow

Startup and Registration

plugin loader
  -> selected memory-slot plugin
  -> memory-core/register()
  -> register memory capability
  -> register lazy memory_search and memory_get
  -> register CLI and dreaming command
  -> prompt builder can add Memory Recall guidance
model calls memory_search
  -> tools.ts parses query/maxResults/minScore/corpus
  -> get active memory manager
  -> builtin or QMD search
  -> filter session transcript visibility
  -> decorate citations if enabled
  -> optionally search corpus supplements such as wiki
  -> merge and balance results
  -> optionally record short-term recall signals
  -> JSON result

Exact Read

model calls memory_get
  -> tools.ts parses path/from/lines/corpus
  -> wiki supplement read, or active backend read
  -> readMemoryFile safety checks
  -> bounded excerpt with continuation metadata

Builtin Indexing

workspace memory files
  -> listMemoryFiles()
  -> buildFileEntry()
  -> chunkMarkdown() or buildMultimodalChunkForIndexing()
  -> embedding provider, if available
  -> files/chunks/chunks_fts/chunks_vec/embedding_cache

Compaction Flush

conversation approaches compaction
  -> resolveMemoryFlushPlan()
  -> embedded memory flush agent turn
  -> write tool restricted to memory/YYYY-MM-DD.md
  -> append durable notes or silent reply token

Dreaming Promotion

memory_search recall results
  -> short-term recall store
  -> managed dreaming cron/heartbeat trigger
  -> light/REM/deep phase sweep
  -> rank promotion candidates
  -> append qualified entries to MEMORY.md
  -> write review output to DREAMS.md or memory/dreaming

Durable File Layout

The workspace is resolved through config helpers such as resolveAgentWorkspaceDir(). Public docs describe the default as:

~/.openclaw/workspace

Code should not hardcode that path.

Typical layout:

<workspace>/
  MEMORY.md
  DREAMS.md
  memory/
    2026-05-25.md
    2026-05-25-project-slug.md
    dreaming/
      deep/
        2026-05-25.md
    .dreams/
      short-term-recall.json
      phase-signals.json
      short-term-promotion.lock

MEMORY.md

MEMORY.md is the canonical long-term memory file. Uppercase is intentional. Lowercase memory.md is treated as legacy or auxiliary in several root-memory paths.

Relevant files:

  • src/memory/root-memory-files.ts
  • packages/memory-host-sdk/src/host/openclaw-runtime-memory.ts
  • packages/memory-host-sdk/src/host/internal.ts

memory/*.md

Daily and topical notes are indexed for recall but are not always injected into the prompt. The pre-compaction memory flush writes the date-only form memory/YYYY-MM-DD.md and explicitly avoids timestamped variants.

DREAMS.md

DREAMS.md is a review diary. Dreaming output can also go under memory/dreaming/<phase>/YYYY-MM-DD.md. The code has filters that prevent dreaming narrative and promotion metadata from promoting themselves back into MEMORY.md; otherwise memory may recursively include its own bookkeeping output.

Plugin Capability Registry

The active memory slot is managed through src/plugins/memory-state.ts, src/plugins/memory-runtime.ts, and src/plugins/registry.ts.

memory-core registers one primary memory capability:

Capability fieldRegistered value
promptBuilderbuildPromptSection
flushPlanResolverbuildMemoryFlushPlan
runtimememoryRuntime
publicArtifacts.listArtifactslistMemoryCorePublicArtifacts

It also registers:

  • builtin embedding providers through registerBuiltInMemoryEmbeddingProviders
  • dreaming hooks through registerShortTermPromotionDreaming
  • lazy memory_search
  • lazy memory_get
  • /dreaming command
  • openclaw memory CLI group

memory-state.ts stores:

Registry itemMeaning
primary memory capabilityThe selected memory plugin's prompt, flush, runtime, and artifacts.
corpus supplementsExtra corpora searched by memory_search corpus=wiki or corpus=all.
prompt supplementsExtra prompt guidance appended after the primary memory section.

registerMemoryCapability() has a subtle merge rule: if an existing primary capability exists and a new registration only contributes publicArtifacts, the existing runtime/prompt/flush fields are preserved. This lets companion plugins publish artifacts without overriding the active memory runtime.

registry.ts enforces that only memory-kind plugins can register primary memory features. Dual-kind plugins must actually be selected for the memory slot before their primary capability is accepted.

Prompt Bootstrap and Recall Guidance

Memory reaches the model through two routes:

RouteSourceBehavior
Bootstrap contextsrc/agents/bootstrap-files.ts, src/agents/system-prompt.tsSelected workspace files such as MEMORY.md can be loaded directly into prompt context.
Tool recallmemory_search, memory_getThe model searches and reads indexed memory during a turn.

When MEMORY.md is loaded as project context, system-prompt.ts labels it as:

MEMORY.md: durable user preferences and behavior guidance.

extensions/memory-core/src/prompt-section.ts adds guidance telling the agent to search memory before answering questions about prior work, decisions, dates, people, preferences, or todos, and to use memory_get when exact lines matter.

Memory Tools

Schema is registered in extensions/memory-core/index.ts and implemented in extensions/memory-core/src/tools.ts.

Parameters:

ParameterMeaning
queryRequired search text.
maxResultsOptional result cap. Defaults come from memory search config.
minScoreOptional score floor.
corpusOptional memory, sessions, wiki, or all.

Corpus behavior:

CorpusSearch scope
unset/defaultActive backend sources. Usually memory files; can include sessions when configured.
memoryDurable memory file chunks only.
sessionsIndexed session transcript chunks only, with visibility filtering.
wikiRegistered wiki/corpus supplements only.
allMemory plus supplements, with balancing because scores are not comparable.

Returned JSON can include:

  • results
  • provider
  • model
  • fallback
  • citations
  • mode
  • debug

If memory is unavailable, the tool returns a disabled/unavailable JSON shape instead of exposing raw runtime failures directly to the model.

memory_get

Parameters:

ParameterMeaning
pathRequired memory path or supplement lookup.
fromOptional 1-indexed start line.
linesOptional line count.
corpusOptional memory, wiki, or all.

For builtin backend reads, memory_get calls readAgentMemoryFile(), which uses readMemoryFile() in the host SDK. Non-builtin backends can implement their own manager.readFile().

Builtin Index Manager

The builtin backend is implemented by MemoryIndexManager in:

extensions/memory-core/src/memory/manager.ts

It extends:

MemoryManagerEmbeddingOps
  -> MemoryManagerSyncOps

Responsibilities:

Class/fileResponsibility
manager.tsSearch, sync orchestration, status, read, probes, lifecycle close, manager cache.
manager-sync-ops.tsDatabase open/schema, watchers, dirty flags, vector table readiness, safe reindex, session sync.
manager-embedding-ops.tsChunk embedding, embedding cache, batch fallback, FTS/vector/chunk writes.

The manager cache key includes agent id, workspace path, resolved settings, and purpose (default, status, or cli). Status and CLI managers are transient so health checks do not accidentally shut down the live manager.

File Discovery and Chunking

File discovery lives in:

packages/memory-host-sdk/src/host/internal.ts

listMemoryFiles() collects:

  • canonical MEMORY.md
  • files under memory/
  • configured extraPaths

It skips:

  • symlinks
  • root-memory repair artifacts
  • legacy lowercase root memory in workspace paths
  • non-Markdown files unless multimodal indexing is enabled and the file matches allowed modality settings

buildFileEntry() records:

FieldMeaning
pathWorkspace-relative normalized path.
absPathAbsolute file path.
mtimeMs, sizeChange detection signals.
hashContent or structured multimodal hash.
kindmarkdown or multimodal.
dataHash, mimeType, modalityMultimodal metadata when applicable.

chunkMarkdown() splits content using token-budget estimates and overlap. Defaults are documented around:

SettingDefault
chunk sizeabout 400 tokens
overlapabout 80 tokens

Long lines are split. There is a second pass for CJK-heavy text so token estimates remain reliable.

Session transcript chunks use remapChunkLines() because session JSONL is flattened into text before chunking; search results should still point back to original JSONL lines.

SQLite Schema

ensureMemoryIndexSchema() in packages/memory-host-sdk/src/host/memory-schema.ts creates:

TablePurpose
metaKey/value metadata for index state.
filesOne row per indexed source file.
chunksChunk text, line spans, source, model, JSON embedding fallback, hash, updated time.
embedding_cacheProvider/model/provider key/hash to embedding cache, when cache is enabled.
chunks_ftsFTS5 virtual table when hybrid/FTS is enabled.
chunks_vecsqlite-vec virtual table created lazily when vector search is available.

Important columns:

ColumnTablesMeaning
sourcefiles, chunks, chunks_ftsmemory or sessions.
modelchunks, chunks_ftsEmbedding model or fts-only.
embeddingchunksJSON fallback vector storage.

Indexes:

  • idx_chunks_path
  • idx_chunks_source
  • idx_embedding_cache_updated_at

meta stores memory_index_meta_v1, including provider/model/provider key, configured sources, scope hash, chunking, FTS tokenizer, and vector dimensions. Provider, chunking, tokenizer, scope, or source changes can trigger a full reindex.

Sync and Reindexing

Sync can run because of:

  • initial search bootstrap when no indexed content exists
  • sync.onSearch
  • sync.onSessionStart
  • file watcher events
  • interval sync
  • targeted session transcript updates
  • CLI reindex
  • provider/model/chunking/scope changes

manager-sync-ops.ts uses chokidar and ignores noisy directories such as:

  • .git
  • node_modules
  • .pnpm-store
  • .venv
  • venv
  • .tox
  • __pycache__

Full reindex is normally safe/atomic:

current index
  -> temporary sqlite database
  -> seed embedding cache
  -> rebuild files/chunks/FTS/vector rows
  -> write meta
  -> atomic replacement

Test-only unsafe reindex can be enabled through environment flags, but normal runtime uses the safe path. Read-only database errors have recovery paths that reopen/rebuild state before retrying.

Watcher Settling and Dirty State

File watcher changes are not applied immediately. manager-sync-ops.ts records paths into a settle queue through recordMemoryWatchEventPath(), then settleMemoryWatchEventPaths() rechecks size and mtime before sync proceeds.

Why this exists:

  • editors often write Markdown through temporary files or multi-step saves
  • session JSONL files can still be growing when a watcher event fires
  • indexing a half-written file creates invalid chunks that can appear as memory corruption later

Settle behavior:

CaseBehavior
Empty pathIgnored.
Directory eventStored as a null snapshot.
File event with statsStores size and mtimeMs.
Missing baseline but file appearsRechecks after 100 ms.
Snapshot changes on recheckPath remains queued and sync waits.
Queue drainsSync can proceed.

Dirty state is then consumed by normal sync triggers: on-search sync, interval sync, watcher sync, session-start sync, targeted session updates, and CLI reindex. Status-only managers start dirty only when memory files exist but indexed metadata is missing, while normal managers are willing to sync more eagerly.

manager-source-state.ts loads existing file hashes by source so unchanged files can be skipped. That source separation is important because memory and sessions share physical tables but have different discovery and visibility rules.

Embeddings

Embedding provider creation lives under:

extensions/memory-core/src/memory/embeddings.ts
extensions/memory-core/src/memory/provider-adapters.ts

Supported provider IDs in public docs include:

ProviderID
Bedrockbedrock
DeepInfradeepinfra
Geminigemini
GitHub Copilotgithub-copilot
Local GGUFlocal
Mistralmistral
Ollamaollama
OpenAIopenai
Voyagevoyage

Auto-detection order:

  1. local, only when memorySearch.local.modelPath is configured and exists.
  2. github-copilot
  3. openai
  4. gemini
  5. voyage
  6. mistral
  7. deepinfra
  8. bedrock

ollama is supported but must be configured explicitly.

Important behavior:

  • No provider means FTS-only mode, not a total memory failure.
  • Provider key data participates in embedding cache identity.
  • Batch embedding is supported when provider runtime exposes batchEmbed.
  • Batch failures fall back to inline embedding and can disable batch after repeated failures.
  • Retry policy handles retryable embedding errors.
  • Local/self-hosted providers get longer inline timeouts.
  • Ollama defaults to lower non-batch concurrency.
  • Multimodal chunks require a provider with structured input support.

Embedding cache table identity:

provider + model + provider_key + chunk_hash

Vector search helper:

extensions/memory-core/src/memory/manager-search.ts

If sqlite-vec is available, OpenClaw uses native KNN:

embedding MATCH ? AND k = ?

It still computes cosine distance in the select path so downstream scores remain cosine-like.

If sqlite-vec is unavailable, it falls back to scanning JSON embeddings from chunks in bounded batches and computing cosine similarity in process. The bounded batch size prevents large indexes from blocking the event loop for long periods.

Keyword search uses chunks_fts when FTS is available.

buildFtsQuery() tokenizes by letters, numbers, and _, quotes each token, and joins them with AND.

Scoring:

PathScoring
FTS MATCHBM25 mapped with bm25RankToScore().
MATCH failureFalls back to LIKE-based substring conditions.
FTS-only boosted rankingAdds path match, token overlap, density, and useful text length boosts.

When the tokenizer is trigram, short CJK tokens are handled with substring fallback because trigram FTS is unreliable for very short tokens.

Hybrid Ranking

Hybrid merge logic lives in:

extensions/memory-core/src/memory/hybrid.ts

Search flow:

if no provider:
  FTS keyword search only
else:
  keyword search if FTS is available
  embed query
  vector search
  if hybrid enabled:
    merge vector and keyword results
  else:
    vector results only

Default public config:

SettingDefault
query.hybrid.enabledtrue
query.hybrid.vectorWeight0.7
query.hybrid.textWeight0.3
query.hybrid.candidateMultiplier4
query.hybrid.mmr.enabledfalse
query.hybrid.mmr.lambda0.7
query.hybrid.temporalDecay.enabledfalse
query.hybrid.temporalDecay.halfLifeDays30

Combined score:

vectorWeight * vectorScore + textWeight * textScore

Optional reranking:

  • temporal decay in temporal-decay.ts
  • MMR diversity in mmr.ts

If strict hybrid scoring drops everything but keyword results exist, the manager relaxes the score floor for keyword-backed matches. This keeps exact lexical recall alive when vector weighting gets too fancy for its own good.

Session memory is experimental and opt-in.

Relevant files:

  • packages/memory-host-sdk/src/host/session-files.ts
  • extensions/memory-core/src/memory/manager-session-sync-state.ts
  • extensions/memory-core/src/memory/manager-targeted-sync.ts
  • extensions/memory-core/src/session-search-visibility.ts

Config surfaces:

SettingMeaning
experimental.sessionMemoryEnables transcript indexing.
sourcesAdd "sessions" to include session source.
sync.sessions.deltaBytesByte threshold for incremental session reindex.
sync.sessions.deltaMessagesMessage threshold for incremental session reindex.

Session transcript export:

  • converts JSONL transcripts into sanitized plain text
  • strips internal runtime context and metadata
  • skips compaction checkpoints and non-usage-counted archive artifacts
  • preserves usage-counted reset/deleted archives
  • wraps very long messages
  • maps flattened content lines back to original JSONL lines
  • classifies dreaming narrative and cron-run transcripts

Search visibility filtering is applied after retrieval. The index may contain session chunks, but filterMemorySearchHitsBySessionVisibility() decides what the current requester may see.

Safe Reads

Safe read implementation:

packages/memory-host-sdk/src/host/read-file.ts

readMemoryFile() allows:

Path classRule
workspace memory pathsMust satisfy isMemoryPath(): MEMORY.md, DREAMS.md, or memory/**.
configured extra fileMust be the exact configured Markdown file.
configured extra directoryTarget must be inside the directory, with symlink containment checks.

Other rules:

  • path is required
  • result must be Markdown
  • workspace reads are resolved through root safety helpers
  • missing files return empty text instead of crashing
  • output is bounded by line and character limits

memory_get is therefore not an arbitrary filesystem read with a memory sticker on the laptop.

QMD Backend

QMD is selected with:

memory.backend = "qmd"

Relevant files:

  • packages/memory-host-sdk/src/host/backend-config.ts
  • extensions/memory-core/src/memory/search-manager.ts
  • extensions/memory-core/src/memory/qmd-manager.ts
  • packages/memory-host-sdk/src/host/qmd-*

QMD is a sidecar search engine. OpenClaw manages collections and invokes QMD subprocesses for updates, embeddings, and search.

Selector behavior in search-manager.ts:

  1. Resolve backend config.
  2. Check workspace availability.
  3. Check QMD binary availability.
  4. Create or reuse a cached QMD manager.
  5. Wrap full QMD managers in FallbackMemoryManager.
  6. Use transient QMD managers for CLI/status.
  7. Use BorrowedMemoryManager for status against cached live managers so a health probe does not close active search.
  8. Fall back to builtin if QMD is unavailable.

Failure behavior:

FailureBehavior
QMD binary missingUse builtin backend.
QMD workspace unavailableUse builtin backend.
QMD open failsRecord a 60 second cooldown to avoid retry storms.
QMD search fails laterClose primary, evict cache, switch wrapper to builtin fallback.

QMD adds local-first BM25/vector search, query expansion, reranking, extra configured paths, and optional session transcript collections.

Companion and Alternative Memory Plugins

Only one plugin owns the active memory slot at a time, but companion plugins can add corpora, prompt supplements, artifacts, or separate tools.

PluginRole
memory-coreDefault file-backed memory slot plugin.
memory-lancedbAlternative memory slot plugin backed by LanceDB; supports embeddings, auto-recall, auto-capture, and memory_recall style behavior.
memory-wikiCompanion plugin that compiles durable memory into a wiki vault with claims, provenance, pages, dashboards, and wiki tools.
active-memoryOptional proactive recall plugin using a blocking sub-agent before prompt build.
Honcho memoryPlugin install path documented in docs/concepts/memory-honcho.md; AI-native cross-session memory, not the builtin file index.

Memory Wiki

memory-wiki does not replace the memory slot. It registers supplement corpora so memory_search corpus=wiki or corpus=all can search wiki output alongside active memory results.

Active Memory

extensions/active-memory/ registers a /active-memory command and a before_prompt_build hook. It checks session toggles, target agent, interactive session eligibility, chat type, and allow/deny rules. Then it starts an embedded agent with lightweight bootstrap context and memory-only tools.

Defaults:

Active memory slotTools
memory-corememory_search plus memory_get
memory-lancedbmemory_recall

The summary is injected as untrusted hidden <active_memory_plugin> context. It is proactive recall, not a new durable store.

Pre-Compaction Memory Flush

Relevant files:

  • extensions/memory-core/src/flush-plan.ts
  • src/auto-reply/reply/agent-runner-memory.ts
  • src/agents/pi-embedded-runner/run/attempt.tool-run-context.ts

memory-core registers buildMemoryFlushPlan() as the memory flush plan resolver.

Defaults:

SettingDefault
enabledenabled unless agents.defaults.compaction.memoryFlush.enabled === false
soft threshold4000 tokens
force transcript threshold2 MiB
reserve tokens floorDEFAULT_PI_COMPACTION_RESERVE_TOKENS_FLOOR
target filememory/YYYY-MM-DD.md
date timezoneresolved user timezone

Safety hints are enforced even when prompts are customized:

  • store durable memories only in memory/YYYY-MM-DD.md
  • append only
  • do not overwrite bootstrap/reference files such as MEMORY.md, DREAMS.md, SOUL.md, TOOLS.md, and AGENTS.md
  • do not create timestamped variants
  • reply with the silent token when nothing durable should be stored

The model can be overridden with:

agents.defaults.compaction.memoryFlush.model

When the flush runs, the embedded agent receives memoryFlushWritePath; the write wrapper restricts writes to that one target and append behavior.

Dreaming and Promotion

Dreaming is opt-in and disabled by default.

Relevant files:

  • extensions/memory-core/src/dreaming.ts
  • extensions/memory-core/src/dreaming-phases.ts
  • extensions/memory-core/src/dreaming-narrative.ts
  • extensions/memory-core/src/short-term-promotion.ts
  • extensions/memory-core/src/dreaming-markdown.ts
  • docs/concepts/dreaming.md

Dreaming has two related systems:

SystemPurpose
Short-term recall storeRecords which daily-note snippets are repeatedly recalled.
Managed dreaming sweepRuns phase review, ranks candidates, promotes durable snippets, writes diary/report output.

Short-Term Recall Store

Files:

memory/.dreams/short-term-recall.json
memory/.dreams/phase-signals.json
memory/.dreams/short-term-promotion.lock

Recall entries track:

  • path and line span
  • source
  • snippet
  • recall count
  • daily count
  • grounded count
  • total and max score
  • first and last recall timestamps
  • query hashes
  • recall days
  • concept tags
  • optional claim hash
  • promotion timestamp

memory_search records recall signals when dreaming is enabled. Only memory source hits from short-term memory paths participate. Dreaming artifacts are filtered to avoid self-promotion loops.

Promotion Scoring

Default gates:

GateDefault
minimum score0.75
minimum signal count3
minimum unique queries/context diversity2
recency half-life14 days

Default weights:

ComponentWeight
frequency0.24
relevance0.30
diversity0.15
recency0.15
consolidation0.10
conceptual richness0.06

Score:

weighted components + phase signal boost

Phase signal boosts:

  • light phase max boost: 0.06
  • REM phase max boost: 0.09
  • phase signal half-life: 14 days

Promotion applies only after candidates are rehydrated from source files. If the line range moved, the code tries to relocate the snippet. If it lands inside a dreaming fence or looks contaminated by dreaming narrative, promotion is refused.

Deep promotion appends a "Promoted From Short-Term Memory" section to MEMORY.md with source and score metadata. If the file would exceed its budget, older auto-promotion sections can be compacted out before writing.

MEMORY.md Promotion Budget

extensions/memory-core/src/memory-budget.ts keeps automatic promotion from silently pushing MEMORY.md past the bootstrap injection cap.

Default budget:

DEFAULT_MEMORY_FILE_MAX_CHARS = 10000

The compactor only drops sections matching ## Promoted From Short-Term Memory (DATE).

Guarantees:

  • user-authored memory content is preserved
  • non-matching headings are preserved
  • auto-promotion sections are dropped oldest first
  • the freshest new section is still written even if the final file cannot be made perfect
  • a small writer-overhead reserve is subtracted so the final on-disk file stays within the caller's intended budget

This is intentionally narrow. It trims the machine-owned growth ring, not the user's hand-written long-term memory. Tiny scalpel, not chainsaw.

Managed Cron

registerShortTermPromotionDreaming() reconciles a managed cron job:

  • creates or updates the unified dreaming job when enabled
  • removes duplicate managed jobs
  • migrates legacy light/REM jobs
  • retries cron resolution after startup if the cron service is not ready yet
  • handles heartbeat or cron triggers that include the managed dreaming token

The sweep can write phase reports and generate dream diary narrative through a subagent. Cron-triggered narratives may run detached so maintenance does not block the whole house. Sensible. Very adult.

Public Artifacts

memory-core exposes public artifacts through:

extensions/memory-core/src/public-artifacts.ts
src/plugin-sdk/memory-host-core.ts

listMemoryHostPublicArtifacts() walks resolved memory workspaces and emits:

KindPath
memory-rootMEMORY.md
daily-noteMarkdown under memory/
dream-reportMarkdown under memory/dreaming/
event-logMemory host event log JSON

listActiveMemoryPublicArtifacts() clones and sorts artifacts by workspace, relative path, kind, content type, agent ids, and absolute path. Companion systems such as memory-wiki depend on this boundary, so artifact shape changes should be treated as API changes, not as small internal cleanups.

Memory Host Event Journal

Memory host events are stored as JSONL under:

memory/.dreams/events.jsonl

Implementation:

  • src/memory-host-sdk/events.ts
  • extensions/memory-core/src/short-term-promotion.ts
  • extensions/memory-core/src/dreaming-markdown.ts

Event types:

EventWritten byMeaning
memory.recall.recordedShort-term recall trackingA memory_search result was recorded for promotion scoring.
memory.promotion.appliedShort-term promotion apply pathOne or more candidates were processed into MEMORY.md.
memory.dream.completedDreaming markdown writerA light/REM/deep phase wrote inline and/or report output.

appendMemoryHostEvent() writes through a regular-file append helper with symlink-parent rejection. readMemoryHostEvents() tolerates missing logs and skips malformed lines. The event log is also exposed as a public artifact of kind event-log, so companion systems can consume it without poking private runtime state.

The event journal is diagnostic and integration-facing. It is not searched as durable memory and should not become a promotion source.

CLI, Status, and Doctor Surfaces

The memory CLI is registered by extensions/memory-core/src/cli.ts and executed by cli.runtime.ts.

Commands:

CommandPurpose
openclaw memory statusShow backend, provider, source, dirty, vector, and dreaming status.
openclaw memory status --deepProbe embedding provider and vector readiness.
openclaw memory status --indexReindex if dirty while collecting deep status.
openclaw memory status --fixRepair stale recall locks and normalize/repair dreaming artifacts.
openclaw memory index --forceForce a full reindex.
openclaw memory search "query"Search from the CLI, optionally JSON-formatted.
openclaw memory promoteRank short-term recall candidates; --apply appends qualified entries.
openclaw memory promote-explain <selector>Explain one candidate's score and threshold pass/fail state.
openclaw memory rem-harnessPreview REM reflections, candidate truths, and deep promotion output without writing.
openclaw memory rem-backfillWrite grounded historical REM summaries into DREAMS.md; optionally stage short-term candidates.

Status aggregation uses:

  • manager.status()
  • collectMemoryStatusAggregate()
  • resolveStatusProviderInfo()
  • vector readiness from manager status and optional probes
  • dreaming audit and repair helpers

Deep status can show whether the embedding provider is ready, whether semantic vectors are available, whether the sqlite-vec store loaded, per-source file and chunk counts, dreaming configuration, and repairable artifact issues.

Doctor integration in src/gateway/server-methods/doctor.memory-core-runtime.ts reuses bundled memory-core runtime helpers rather than reimplementing repairs. That means CLI and doctor repairs should stay behaviorally aligned; if one reports a healthy memory pipeline and the other reports errors, the issue may be outside the shared repair logic.

Configuration Map

Most builtin search settings live under:

agents.defaults.memorySearch

Agent-specific overrides can live under:

agents.list[].memorySearch

Backend selection:

memory.backend
memory.qmd

Dreaming:

plugins.entries.memory-core.config.dreaming

Active memory:

plugins.entries.active-memory.config

Important config areas:

AreaExamples
provider selectionprovider, model, fallback, enabled
provider auth/endpointsremote.baseUrl, remote.apiKey, remote.headers
provider-specific inputinputType, queryInputType, documentInputType, output dimensionality
query rankingquery.minScore, query.maxResults, query.hybrid.*
chunkingchunking.tokens, chunking.overlap
extra filesextraPaths
multimodalmultimodal.enabled, modalities, maxFileBytes
embedding cachecache.enabled, cache.maxEntries
batch indexingremote.batch.*, remote.nonBatchConcurrency
session memoryexperimental.sessionMemory, sources, sync.sessions.*
SQLitestore.path, store.vector.*, store.fts.tokenizer
QMDmemory.qmd.command, searchMode, paths, sessions, update behavior

The full key list is in:

docs/reference/memory-config.md

Failure Modes

AreaFailure modeExpected behavior
No memory configMemory tools are not registered or manager is null.
Memory manager unavailableTool returns disabled/unavailable JSON.
First search after restartBuiltin manager forces synchronous bootstrap sync if index is empty.
Empty search querySearch preflight refuses provider initialization and returns no hits.
Watch event during active writeSettle queue waits until size/mtime stops changing before sync.
No embedding providerBuiltin search degrades to FTS-only mode.
FTS unavailableHybrid loses keyword path; vector-only can still work if provider exists.
sqlite-vec unavailableVector search falls back to bounded in-process cosine scan.
Embedding batch failureBatch falls back to inline embeddings and may disable batch after repeated failures.
Provider/model/chunking/scope changesFull reindex is triggered.
Read-only databaseManager runs readonly recovery and can reopen/rebuild.
QMD binary/workspace unavailableBackend falls back to builtin.
QMD open failureShort cooldown avoids retry storms.
QMD runtime failureWrapper closes QMD, evicts cache, and switches to builtin fallback.
Session visibility mismatchSession hits are filtered after retrieval.
Unsafe read pathmemory_get returns an error instead of reading arbitrary files.
Missing memory fileSafe read returns empty text for the allowed path.
Dreaming contaminated snippetCandidate is skipped or refused during rehydration.
Stale short-term lockAudit/repair can remove it when safe.
Auto-promotion grows MEMORY.md too largeOldest machine-owned promotion sections are dropped within budget; user-authored content is preserved.
Malformed event journal lineEvent reader skips the bad line and continues.

Useful investigation commands:

openclaw memory status --deep --agent main
openclaw memory status --fix --agent main
openclaw memory index --force --agent main
openclaw memory search "query"
openclaw memory promote
openclaw memory promote --apply
openclaw memory promote-explain "query"
openclaw memory rem-harness --json
openclaw memory rem-backfill --path ./memory --stage-short-term

Maintenance Checklist

When changing this system:

  • If durable memory layout changes, update file discovery, safe reads, public artifacts, docs, and companion plugin assumptions.
  • If MEMORY.md canonical handling changes, audit root memory bootstrap, listMemoryFiles(), and isMemoryPath().
  • If memory_search result shape changes, update tool docs, active memory, wiki supplements, citation decoration, and recall tracking.
  • If memory_get safety changes, audit symlink and containment tests.
  • If chunking changes, update reindex meta logic and tests that assert line spans.
  • If watcher behavior changes, update settle-queue logic and dirty-state tests so half-written files are not indexed.
  • If schema changes, update ensureMemoryIndexSchema(), status aggregation, migration/compat paths, and reindex behavior.
  • If provider identity changes, update embedding cache key logic and force reindex rules.
  • If hybrid weights or scoring change, update hybrid.ts, manager.ts, and search quality tests.
  • If session memory changes, update transcript export, visibility filtering, targeted sync, and source filtering.
  • If QMD integration changes, check fallback behavior, status/CLI transient managers, cooldown logic, and active-memory search mode overrides.
  • If dreaming promotion changes, update recall store normalization, scoring docs, contamination filters, audit/repair, and MEMORY.md budget behavior.
  • If promotion section headings change, update memory-budget.ts so compaction still trims only machine-owned sections.
  • If CLI/status output changes, update cli.runtime.ts, doctor surfaces, and public CLI docs.
  • If memory host event types change, update event readers, public artifacts, and companion consumers.
  • If public artifact kinds change, treat it as a plugin-facing API change.

Useful tests:

  • extensions/memory-core/src/memory/manager*.test.ts
  • extensions/memory-core/src/tools*.test.ts
  • extensions/memory-core/src/memory-events.test.ts
  • extensions/memory-core/src/memory-budget.test.ts
  • extensions/memory-core/src/memory/watch-settle.test.ts
  • extensions/memory-core/src/memory/manager-status-state.test.ts
  • extensions/memory-core/src/memory/manager-source-state.test.ts
  • extensions/memory-core/src/memory/manager-search-preflight.test.ts
  • extensions/memory-core/src/dreaming*.test.ts
  • extensions/memory-core/src/short-term-promotion*.test.ts
  • src/plugins/memory-state.test.ts
  • src/plugins/memory-runtime.test.ts
  • src/plugins/registry.dual-kind-memory-gate.test.ts
  • src/agents/system-prompt.memory.test.ts
  • src/agents/bootstrap-files.test.ts
  • src/auto-reply/reply/agent-runner-memory.test.ts
  • src/plugin-sdk/memory-host-*.test.ts
  • src/commands/doctor-memory-search.test.ts
  • src/commands/status.scan-memory.test.ts

Implementation References

FilePurpose
docs/concepts/memory.mdPublic memory overview.
docs/concepts/memory-search.mdPublic search behavior and tuning.
docs/reference/memory-config.mdFull config reference.
extensions/memory-core/index.tsDefault memory plugin registration.
extensions/memory-core/src/tools.tsmemory_search and memory_get.
extensions/memory-core/src/tools.shared.tsTool runtime helpers and corpus supplements.
extensions/memory-core/src/tools.citations.tsCitation decoration and injected-character clamping.
extensions/memory-core/src/prompt-section.tsMemory Recall prompt section.
extensions/memory-core/src/flush-plan.tsPre-compaction memory flush plan.
extensions/memory-core/src/runtime-provider.tsRuntime manager bridge.
extensions/memory-core/src/public-artifacts.tsMemory-core artifact provider.
src/memory-host-sdk/events.tsMemory event journal read/write helpers.
extensions/memory-core/src/memory/search-manager.tsBuiltin/QMD selection and fallback.
extensions/memory-core/src/memory/manager.tsBuiltin manager public behavior.
extensions/memory-core/src/memory/manager-sync-ops.tsSync, watcher, reindex, vector readiness, sessions.
extensions/memory-core/src/memory/watch-settle.tsFile watcher settle queue.
extensions/memory-core/src/memory/manager-status-state.tsStatus provider/search-mode and source aggregate helpers.
extensions/memory-core/src/memory/manager-source-state.tsExisting source file hash loading.
extensions/memory-core/src/memory/manager-embedding-ops.tsChunk embedding and index writes.
extensions/memory-core/src/memory/manager-search.tsVector and keyword query helpers.
extensions/memory-core/src/memory/manager-search-preflight.tsEmpty-query and empty-index preflight behavior.
extensions/memory-core/src/memory/hybrid.tsBM25/vector merge.
extensions/memory-core/src/memory/mmr.tsDiversity rerank.
extensions/memory-core/src/memory/temporal-decay.tsRecency scoring.
extensions/memory-core/src/memory/qmd-manager.tsQMD backend integration.
packages/memory-host-sdk/src/host/internal.tsFile discovery, chunking, hashing, multimodal chunks.
packages/memory-host-sdk/src/host/memory-schema.tsBuiltin SQLite schema.
packages/memory-host-sdk/src/host/read-file.tsSafe memory reads.
packages/memory-host-sdk/src/host/session-files.tsSession transcript export.
packages/memory-host-sdk/src/host/backend-config.tsBuiltin/QMD backend resolution.
src/plugins/memory-state.tsMemory capability and supplement registry.
src/plugins/memory-runtime.tsLazy runtime loading for the selected memory plugin.
src/plugins/registry.tsMemory plugin slot enforcement.
src/agents/bootstrap-files.tsBootstrap file loading.
src/agents/system-prompt.tsPrompt assembly and memory context labels.
src/memory/root-memory-files.tsCanonical root memory path logic.
extensions/memory-core/src/dreaming.tsManaged dreaming cron and trigger handling.
extensions/memory-core/src/dreaming-phases.tsLight/REM/deep sweep work.
extensions/memory-core/src/dreaming-narrative.tsDream diary generation.
extensions/memory-core/src/dreaming-markdown.tsDream report writes.
extensions/memory-core/src/short-term-promotion.tsRecall tracking, ranking, promotion, audit, repair.
extensions/memory-core/src/memory-budget.tsBounded MEMORY.md auto-promotion compaction.
extensions/memory-core/src/cli.tsMemory CLI command registration.
extensions/memory-core/src/cli.runtime.tsMemory CLI command implementation.
src/gateway/server-methods/doctor.memory-core-runtime.tsDoctor runtime bridge for memory-core repair helpers.
src/plugin-sdk/memory-host-core.tsPublic artifact listing.
extensions/memory-wiki/Wiki companion plugin.
extensions/memory-lancedb/LanceDB memory slot plugin.
extensions/active-memory/Proactive memory recall plugin.

One-Screen Architecture

Runtime startup
  -> plugin loader selects memory slot
  -> memory-core registers capability, tools, CLI, and dreaming hooks
  -> prompt builder adds Memory Recall guidance

User asks about prior context
  -> model calls memory_search
  -> tools.ts resolves active manager
  -> builtin SQLite or QMD returns ranked chunks
  -> session hits are visibility-filtered
  -> wiki supplements may be merged
  -> dreaming may record recall signals

Memory files change
  -> watcher/interval/session update marks dirty
  -> sync scans files/transcripts
  -> chunks are embedded or FTS-indexed
  -> stale rows are pruned

Context approaches compaction
  -> flush plan starts embedded memory turn
  -> durable notes append to memory/YYYY-MM-DD.md

Dreaming runs
  -> phase sweep records signals and review output
  -> candidates are ranked and rehydrated
  -> qualified snippets append to MEMORY.md

Practical Gotchas

  • MEMORY.md uppercase is canonical.
  • The search index is derived; rebuild it before assuming the source files are wrong.
  • memory_get is intentionally narrow and safe.
  • Session memory is a separate sessions source and is filtered after search.
  • DREAMS.md is a review surface, not the durable fact store.
  • QMD is a backend sidecar; builtin search is the fallback.
  • Wiki is a companion corpus, not the active memory backend.
  • Active memory is proactive recall, not a replacement for durable Markdown.
  • Multimodal indexing applies to configured extraPaths, not the default memory roots.
Back to all posts