Pi Stateful Memory Extension

This project adds long-term statefulness to Pi via an extension. It maintains a persistent persona (stateful-memory/SOUL.md), user profile (stateful-memory/USER.md), per-session memory logs, and topic-based knowledge addenda (stateful-memory/persona_topics/).

Documentation

• Project State (current behavior, config defaults, validation): docs/PROJECT_STATE.md
• Plan (four-pillar roadmap): docs/PLAN.md

This repo mirrors the global install layout: agent/ maps to ~/.pi/agent, and stateful-memory/ maps to ~/.pi/stateful-memory.

What’s Included

• agent/extensions/stateful-memory/: extension implementation (memory store, retrieval, topic routing).
• .pi/extensions/: empty in the global-dev layout (global extension is symlinked into ~/.pi/agent/extensions).
• stateful-memory/: persona, profile, topic addenda, and session memory logs.
  • stateful-memory/memory/sessions/*.md: per-session summaries.
  • stateful-memory/persona_topics/, stateful-memory/PERSONALITY_MATRIX.md: topic addenda + routing table.
• .pi/stateful-memory.json: configuration overrides.

Architecture Diagram (High-Level)

User Prompt
   │
   ▼
Pi Agent Session
   │
   ├─ before_agent_start
   │    ├─ System Prompt = Base + Memory Context + Topic Addenda
   │    └─ Uses stateful-memory/SOUL.md + stateful-memory/USER.md
   │       + stateful-memory/memory/sessions + stateful-memory/persona_topics
   │
   ├─ Tools
   │    ├─ remember / remember_session / recall
   │    └─ list_topics / load_topic
   │
   └─ Session lifecycle hooks
        ├─ session_shutdown / session_before_switch / session_before_fork
        └─ Memory summary → stateful-memory/memory/sessions/*.md

Design Philosophy

Memory vs. Topics: Two Kinds of Knowledge

The extension deliberately separates two different kinds of persistent knowledge:

• Memory (what happened) — a growing archive of past sessions, facts, and events. Accessed on demand via recall when context enrichment from past experience is needed. Not injected wholesale into every prompt — the three-tier hierarchy makes targeted retrieval cheap enough to be practical.
• Topic addenda (how to approach a domain) — approach knowledge, opinions, and framing for specific subject areas. Loaded automatically based on what the current prompt is about. Not facts — these are the "vibes" layer that shapes how to think and engage, rather than what to recall.

Both systems feed into the recall and summarisation LLM personas, so memories written during domain-specific conversations are shaped by the relevant topic framing being active at the time.

Recall Without Vector Embeddings

The memory retriever uses a small LLM as a retriever agent rather than vector similarity search. This is made practical by the three-tier navigability hierarchy built into the session archive:

1. Slug filenames — file names are keyword slugs derived from the session summary content. The retriever can narrow candidates from a large index by scanning filenames alone, at minimal token cost.
2. Session summaries — paragraph-length narratives for shortlisted files. Resolves most recall queries without going deeper.
3. Raw JSONL sessions — every memory file includes a Session: path to the original JSONL transcript on disk. Available for fine-grained extraction when the summary isn't specific enough.

The retriever pays only for what it needs. Most queries resolve at tier 2. Tier 3 is the escape hatch for detailed historical reconstruction.

Running With Pi

From this directory:

pi

Pi auto-discovers the extension globally via ~/.pi/agent/extensions/stateful-memory. To load explicitly from the repo during tests:

pi -e agent/extensions/stateful-memory/extension.js

If you ever want to go back to local-only development (no global extension), re-add a project-local shim:

mkdir -p .pi/extensions/stateful-memory
printf 'export { default } from "../../../agent/extensions/stateful-memory/extension.js";\n' \
  > .pi/extensions/stateful-memory/index.js

Then remove/disable the global symlink at ~/.pi/agent/extensions/stateful-memory to avoid double-loading conflicts.

Configuration

Edit .pi/stateful-memory.json to override defaults in this repo. For global installs, use ~/.pi/agent/stateful-memory.json. If you want a single global memory store even while working in this repo, point .pi/stateful-memory.json at the global paths (same as the global config). See docs/PROJECT_STATE.md for current defaults and environment overrides.

Global Install + Dev Symlinks

Recommended layout for developing the extension while running it globally:

• Extension code: symlink ~/.pi/agent/extensions/stateful-memory → agent/extensions/stateful-memory (repo)
• Persona + topic files: symlink from ~/.pi/stateful-memory/ to repo files
• USER.md: keep global (not symlinked)
• Memory logs: keep global at ~/.pi/stateful-memory/memory/sessions
• .pi/extensions/ should stay empty to avoid double-loading the extension

Example setup:

mkdir -p ~/.pi/agent/extensions ~/.pi/stateful-memory/memory/sessions
ln -sfn /media/psf/Repos/memory/agent/extensions/stateful-memory \
  ~/.pi/agent/extensions/stateful-memory

ln -sfn /media/psf/Repos/memory/stateful-memory/SOUL.md \
  ~/.pi/stateful-memory/SOUL.md
ln -sfn /media/psf/Repos/memory/stateful-memory/STYLE.md \
  ~/.pi/stateful-memory/STYLE.md
ln -sfn /media/psf/Repos/memory/stateful-memory/REGISTER.md \
  ~/.pi/stateful-memory/REGISTER.md
ln -sfn /media/psf/Repos/memory/stateful-memory/PERSONALITY_MATRIX.md \
  ~/.pi/stateful-memory/PERSONALITY_MATRIX.md
ln -sfn /media/psf/Repos/memory/stateful-memory/persona_topics \
  ~/.pi/stateful-memory/persona_topics

# USER.md stays global (no symlink)
cp /media/psf/Repos/memory/stateful-memory/USER.md \
  ~/.pi/stateful-memory/USER.md

Tools

• remember: store explicit facts.
• remember_session: force a session summary.
• recall: memory retrieval.
• list_topics: list available topic addenda.
• load_topic: load a topic addendum by id.

Tests

npm test
npm run smoke:memory
npm run smoke:topics

See docs/PROJECT_STATE.md for validation steps and smoke test expectations.