Topology evolution

What

Evolution that changes coordination geometry — wake thresholds, topology nodes, per-zone decay — not prompt strings. castellan evolve reads real episode logs, proposes structural mutations, gates them through drift + immune checks, and keeps only genomes that measurably improve plugin-grounded fitness.

Why

Outcome: improve harness topology from episode logs under governance gates — fitness from geometry you can replay, not from rewriting prompts. Round-trip seeding: Evolve round-trip. Lexicon: Conventions.

If you’re trying to prove the evolve loop works, debug why a generation got rejected, or wire evolve into CI, this page walks the pipeline end to end.

What this proves

1. Episode logs from real castellan run / swarm-demo runs ground fitness via plugin_grounded_fitness() — not a synthetic benchmark.
2. castellan evolve --episodes 10 runs accept/reject over generations, gated by drift, convergence, and immune gates.
3. Drift gate: structurally incompatible candidates (e.g. edges referencing missing nodes) are rejected on the production path; drift_rejected and drift_violations appear in CLI JSON output when the gate fires.
4. Convergence gate (potential-game): candidates with zero decay, unreachable wake thresholds, or disconnected stigmergy topology are rejected outright, and any candidate whose potential Φ drops below the incumbent’s is rejected even when fitness improves (convergence:potential_decline); convergence_rejected and convergence_violations appear in CLI JSON.
5. Gen-0 vs gen-N fitness delta and full genome lineage are observable directly in CLI output — nothing is hidden behind an opaque score.
6. MAP-Elites (H5): behavior descriptors illuminate a grid of viable geometries (castellan genome map); multi-objective FitnessVector keeps success + circulation per cell.
7. Clonal immune memory (H8): failure antigens clone detectors into .castellan/immune_memory.json so similar failures quarantine faster than static enums alone.

Anatomy: the evolve pipeline

flowchart LR
    EP[episode corpus] --> PROP[proposer]
    PROP --> MUT[mutation candidate]
    MUT --> DRIFT{drift gate}
    DRIFT -->|fail| REJ1[rejected: drift_violations]
    DRIFT -->|pass| CONV{convergence gate}
    CONV -->|fail| REJ1B[rejected: convergence_violations]
    CONV -->|pass| IMM{immune gate}
    IMM -->|auto_reject| REJ2[rejected: quarantine candidate]
    IMM -->|pass| FIT[plugin_grounded_fitness]
    FIT --> ARCH[topology archive]

    classDef evolve stroke:#d97706,stroke-width:2px
    classDef gate stroke:#1a7f37,stroke-width:2px

    class EP,PROP,MUT,FIT,ARCH evolve
    class DRIFT,IMM gate

How operators use it

1. Build the CLI and run the scripted proof end to end:

cargo build --release -p castellan-cli
bash docs/demo/evolve-proof.sh

2. Or step through it manually — generate real episodes, then evolve from them:

castellan swarm-demo
castellan run --goal "reach target" --plugin gridworld   # repeat 3x for a small corpus
castellan evolve --episodes 10 --workspace .castellan/episodes
castellan genome list
castellan genome lineage <genome_id>

3. Do a read-only analysis pass before committing to a mutation:

castellan evolve --plan

--plan emits an evolution plan JSON (incumbent, proposed coordination, pressure summary, drift preview, fitness estimate) with zero writes — safe to run against a production archive.

4. Two-phase gated apply when you want a human or CI check between propose and apply:

castellan evolve --dry-run                      # writes .castellan/proposals/<uuid>.json, no archive writes
castellan evolve --apply-proposal <uuid>        # re-checks drift gate, then applies

5. Live pane topology apply — after field-shaped mutations, apply the seed→target delta to an in-process multiplexer pane tree (typed birth/prune, not chat routing):

castellan evolve --episodes 3 --workspace .castellan/episodes --apply-live

JSON includes live_apply (born, pruned, pane_count). When gates reject the incumbent update, --apply-live still trials the last field-shaped candidate onto live panes and writes .castellan/live/<goal>.topology_apply.jsonl (MCP attach remains instrument-only).

Mid-run drain: while a live-instrumented castellan run is active, the engine drains that JSONL on scheduler ticks (latest-wins), updates topology + wake threshold, emits topology_apply_drained / CastellanEvent::TopologyApplyDrained, and on --mux reconciles panes via RAH apply — without a separate --apply-live pass.

6. Wire the same two-phase flow through MCP (castellan_propose_mutation / castellan_resolve_mutation) when driving evolve from an editor agent instead of the CLI.

Sample output shape

{
  "generations": 10,
  "accepted": 4,
  "rejected": 6,
  "drift_rejected": 1,
  "drift_violations": [
    { "check_id": "topology_edge_refs", "message": "edge agent-1→agent-2 references missing node(s)" }
  ],
  "gen0_fitness": 0.69,
  "final_fitness": 0.74,
  "fitness_delta": 0.05,
  "field_mutations": ["birth_cold_zone", "tune_wake"],
  "instrument_informed": true,
  "seed_source": "episodes",
  "pareto_front": []
}

Recipes

When you want a fast, LLM-free CI smoke test of the evolve loop, run the fixture-backed proof rather than a live corpus:

cargo run -q -p evolve_proof

This uses tests/fixtures/evolve-episodes/ (4 gridworld episodes) — no network, no LLM, deterministic.

When a generation rejects everything, check drift_violations first — a structural issue (missing node reference, invalid zone) will reject before fitness is even computed, so a 100% rejection rate often means one bad mutation shape, not a fitness problem.

When you want continuous improvement without babysitting every generation, use --seed-archive + --write-back on production runs (see Evolve round-trip) so successful runs feed fitness back automatically.

When reviewing a mutation before it goes live, prefer --plan → --dry-run → --apply-proposal over a bare castellan evolve, so you get an inspection point between proposal and archive write.

Honest limits

Failure paths / troubleshooting

Fast-slow co-evolution

Castellan splits fast per-episode RLM remediation from slow topology evolution — two timescales on the same episode corpus, not a monolithic agent chat loop.

[coevolution]
fast_remediation = true
slow_evolve_every = 4
slow_evolve_generations = 2

Demo:

bash docs/demo/fast-slow-coevolution.sh

State is tracked in .castellan/flywheel_state.json; slow evolve emits a slow_evolve event in episode JSON when it runs.

Related

• Evolve round-trip — seeding runs from the archive and writing fitness back
• Organism model — genomes, circulation vitals, and the immune gate in depth
• Stigmergy ablation
• castellan evolve reference