Have you ever noticed that every time you start a new conversation with your AI assistant, it is like meeting a stranger?
It forgot your preferences from last session. It forgot the technical decisions you discussed. It forgot you hate simplified Chinese characters. Every single time, you start from scratch. This is not because AI is not smart enough — it is because it has no memory system at all.
I spent three months building a layered memory architecture from scratch. It now runs daily across four different AI CLIs (Claude Code, Gemini CLI, GitHub Copilot, Codex), sharing a single memory pool — without blowing up the token budget.
The system is called Ghost In Shell, and it is open source. Here is the design philosophy and core architecture.
The Problem: Every Existing Approach Falls Short
| Common Approach | Fatal Flaw |
|---|---|
| Dump everything into the system prompt | Token bonfire — 500+ lines loaded every session |
| Use a single knowledge file | Grows unbounded, blows context window |
| Rely on conversation history | New session = total amnesia |
| External vector database | Complex setup, retrieval latency, overengineered |
The Solution: 5-Layer Memory + Cognitive Engine
Ghost In Shell's memory architecture is inspired by CPU cache hierarchies (L1/L2/L3) — frequently used data stays close to the core, cold data is loaded on demand.
+-------------------------------------------+
| Always Loaded (~230 lines) |
| +----------------+ +-----------------+ |
| | L0 MEMORY.md | | L1 fact.yml | |
| | Router/Index | | Hot-layer facts| |
| +----------------+ +-----------------+ |
+-------------------------------------------+
| Load on Demand |
| +--------------------------------------+ |
| | L1 fact_archive.yml (cold tools) | |
| | L1 fact_decisions.yml (decisions) | |
| | L1 episodic.jsonl (episodes) | |
| | L0.5 scratchpad.md (scratch) | |
| +--------------------------------------+ |
+-------------------------------------------+
| v4 Cognitive Layer (background) |
| +--------------------------------------+ |
| | associations.jsonl (graph) | |
| | principles_candidates.jsonl | |
| | .retrieval_buffer.jsonl | |
| +--------------------------------------+ |
+-------------------------------------------+Result: Only ~230 lines loaded per session (instead of 500+), while the cognitive layer automatically strengthens useful memories and fades stale ones in the background.
Core Design: Each Layer Does One Thing
L0: MEMORY.md — A Router, Not a Database
The entry point to the memory system. It stores only "where to find what" — an index, never actual knowledge. Target: < 200 lines. Think of it as the library card catalog — you would never write an entire book on the card.
L1 Hot: fact.yml — Core Facts Loaded Every Session
YAML format. Stores user preferences, system config, absolute rules, tool inventory. Target: < 150 lines. This is the agent's "working memory" — your name, timezone, language preferences, forbidden terms.
L1 Cold: On-Demand Specialized Knowledge
Full tool configurations (fact_tools_detail.yml), historical decisions (fact_decisions.yml) — only loaded when needed, avoiding wasted tokens.
L1 Episodic: episodic.jsonl — Lessons and Milestones
JSON Lines format, currently 398 entries. Each entry has an importance score (1-10), tags, and timestamps. This is the agent's "long-term experiential memory" — it remembers why the last deployment failed, which API has gotchas.
v4 Cognitive Layer: Memory That Thinks
This is the most interesting part. The cognitive layer is not passive storage — it actively builds associations between memories.
associations.jsonl is a graph that records causal, continuation, and similarity relationships between memories. When you ask "what happened in last deployment failure," the system does not just find that one record — it pulls in related fix attempts, follow-up improvements, and similar incidents.
Memory Strength Formula:
strength = base(imp/10) + retrieval(count * 0.08) + assoc(edges * 0.05) - decay(weeks * 0.03)- base: Importance (1-10) divided by 10
- retrieval: Access count x 0.08 — frequently retrieved memories stay strong
- assoc: Association edges x 0.05 — well-connected memories resist forgetting
- decay: Weeks x 0.03 — forgetting curve, counteracted by retrieval and associations
This models the core mechanics of human memory: what you use often, you remember. What is connected to other things, you remember. What you have not touched in a while fades away.
Multi-CLI Shared Memory: Four Agents, One Brain
My work environment runs four AI CLIs simultaneously: Claude Code (primary), Gemini CLI (long context), GitHub Copilot (code review), and Codex (backup). They all share the same memory system.
| CLI | Role | Memory Access |
|---|---|---|
| Claude Code | Primary — full-stack development | Full read/write |
| Gemini CLI | Long-context document processing | Full read/write |
| GitHub Copilot | Code review specialist | Full read/write |
| Codex CLI | Backup executor | Full read/write |
Each CLI auto-injects runtime metadata on startup (VOID_MEMORY_RUNTIME, VOID_MEMORY_EXECUTOR, VOID_MEMORY_SESSION_ID) via wrapper scripts, and automatically writes back to memory on session end. File locking prevents conflicts when multiple executors write simultaneously.
Automation: Memory That Maintains Itself
| Automation | Frequency | What It Does |
|---|---|---|
| Session Logger | Every session end | Auto-records work summary |
| Trigger Check | Every session end | Checks if consolidation needed |
| Daily Review | Daily at 23:00 | Auto-review + decay calculation |
| Weekly Consolidate | Sundays at 19:00 | Extract principles from episodes |
All schedules run via macOS launchd. The consolidation engine automatically extracts general principles from 398 episodic memories into principles_candidates.jsonl — these are rules the agent "learned" from experience.
The Identity Triangle: Agents Need More Than Memory
Memory is only one part of Ghost In Shell. The full framework rests on three pillars:
- Identity:
IDENTITY.md+SOUL.md+USER.md— defines who the agent is, who it serves, how it speaks - Memory: 5-layer system + cognitive engine — lets the agent remember
- Governance: 3-layer enforcement — ensures the agent does not do dangerous things
The governance layer is critical: L1 is a deny list (100% enforced via settings.json), L2 is a PreToolUse Hook (100% enforced shell guard), L3 is rules/ guidance (~80% soft constraints). Three layers stacked together ensure the agent has autonomy without losing control.
Open Source: Take It
Ghost In Shell is fully open source, including:
- Interactive CLI wizard (5 steps to a complete agent config)
- 16 comprehensive docs (from quick start to cross-machine sync)
- Example templates (minimal / multi_cli_memory)
- Memory automation scripts (session log, consolidation, decay, associations)
GitHub: cyhsieh817/Ghost_In_Shell
If you are tired of re-teaching your AI who you are every single conversation — give it a try.
The memory architecture described in this post has been running on the TheVoidWeaver system for over three months, with daily automated maintenance, 398 episodic memories, 138 association edges, and 6 automated consolidations. This is not theory — it is living infrastructure.
Found this useful?
Follow for new AI × biomedical research notes:
Or buy me a coffee to keep new content coming.
☕ Buy Me a Coffee