Memory architecture

What

Castellan memory is the coordination substrate plus durable episode artifacts — it is not a vector store, not a conversation archive, and not RAG. There is no embedding index in the default path; “recall” means querying structured episode/checkpoint/observation records, and “remember” means persisting a typed observation, not chunking text for similarity search.

Why

Hot field state (blackboard + pheromones) must stay fast and ephemeral; warm episodes must remain replayable for evolve. Lexicon: Conventions.

If you’re deciding whether to enable SQLite, wiring castellan_recall/castellan_remember from an MCP client, or figuring out where a run’s state actually lives on disk, this page is the map.

Anatomy: hot / warm / cold layers

flowchart TB
    subgraph hot["Hot"]
        BB[Blackboard]
        FIELD[PheromoneField]
        SCHED[Scheduler]
    end

    subgraph warm["Warm"]
        EP[episodes]
        TR[traces]
        TA[topology archive]
    end

    RUN[castellan run] --> SCHED
    SCHED --> FIELD
    RUN --> BB
    RUN --> EP
    RUN --> TR
    EVOLVE[castellan evolve] --> EP
    EVOLVE --> TA
    MCP[castellan mcp] --> warm
    REPLAY[castellan replay] --> EP

    classDef runtime stroke:#0969da,stroke-width:2px
    classDef substrate stroke:#5b6b8c,stroke-width:2px
    classDef evolve stroke:#d97706,stroke-width:2px

    class RUN,REPLAY runtime
    class BB,FIELD,SCHED,MCP substrate
    class EVOLVE,EP,TR,TA evolve

On-disk layout

How operators use it

1. Default path — nothing to configure. Every run writes episode JSON automatically:

castellan run --goal "reach target" --plugin gridworld
castellan replay --episode .castellan/episodes/<id>.json
castellan memory status

2. Read live substrate state from an editor/MCP client while a run is active:

3. Switch to the SQLite backend when you want queryable aggregates instead of scanning JSON files. Add to castellan.toml:

[memory]
backend = "sqlite"
path = ".castellan/castellan.db"
mirror_json = true
auto_import = true

mirror_json (default true) keeps writing .castellan/episodes/*.json alongside SQL so tooling that reads plain JSON keeps working. auto_import (default true) imports any existing JSON corpus the first time SQLite opens.

4. Query durable memory from MCP tools once SQLite is enabled:

5. Confirm which backend is active and where the DB lives:

castellan memory status

6. Mid-episode durable checkpoints: persist scheduler state during a run so castellan run --resume survives process loss — not just episode-end snapshots.

[memory]
checkpoint_interval_ticks = 10   # 0 = episode-end only (default, backward-compatible)

Checkpoints land in .castellan/checkpoints/ and SQLite field_history when backend = "sqlite". The engine fires on_checkpoint every N ticks; CLI wires this to MemoryStore::save_checkpoint_json. Integration tests: durable_checkpoint.rs, durable_resume.rs.

Recipes

When you’re prototyping a plugin, stay on the JSON default — episodes are human-readable, diffable, and git-friendly for fixture corpora (see tests/fixtures/evolve-episodes/).

When your episode corpus grows past a few hundred runs and castellan evolve gets slow, switch to backend = "sqlite" so evolve reads SQL aggregates instead of re-parsing every JSON file.

When you need memory to survive across machines (e.g. a shared team archive), configure [memory.sync] with a Turso/libSQL URL and set sync_on_episode_end = true — tokens are referenced by env var name (turso_token_env), never stored inline in castellan.toml. On each castellan run / MCP session open, Castellan pulls newer remote episodes into the local SQLite file; after each episode ends it pushes that episode’s rows when sync_on_episode_end = true.

Check sync health:

castellan doctor --json | jq '.stages[] | select(.name=="memory_sync")'

The memory_sync stage reports reachability, last pull/push timestamps, and the last error (if any). State is stored in .castellan/sync_state.json (no tokens).

When you want a clean corpus for a benchmark run, cp a known-good fixture set into .castellan/episodes/ before castellan evolve rather than hand-editing the live archive.

What never goes in the database

Failure paths / troubleshooting

Engineering detail

Full SQL schema, the MemoryStore trait, and sync policy internals: MEMORY_ARCHITECTURE.md (maintainer spec, not a book page).

See also

• Coordination model — the hot layer this page’s “warm” tier is a snapshot of
• Topology evolution — the primary reader of the warm layer
• castellan.toml reference — full [memory] key list
• MCP tools — castellan_recall, castellan_remember, castellan_query_field_history