System Architecture
Full technical architecture of the Personal AI Operating System
Overview
The system is a layered architecture: hardware at the base, a gateway in the middle, specialized agents above that, and integrations connecting outward to the world.
┌─────────────────────────────────────────────────────────────────────┐
│ USER INTERFACES │
│ Signal · iMessage · Email · Mission Control (web) · Voice │
└──────────────────────────────┬──────────────────────────────────────┘
│
┌──────────────────────────────▼──────────────────────────────────────┐
│ GATEWAY LAYER: OpenClaw │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Session │ │ Agent │ │ Cron │ │ TTS │ │
│ │ Mgmt │ │ Router │ │Scheduler │ │ Engine │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────────┬──────────────────────────────────────┘
│
┌──────────────────────────────▼──────────────────────────────────────┐
│ AGENT LAYER │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Chief │ │ Olivia │ │ Marcus │ │ Riley │ │ Matlock │ │
│ │ (Chief) │ │(Enforce)│ │(Researc)│ │ (Write) │ │ (Legal) │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │Sterling │ │ Vega │ │ Sage │ │ Forge │ │ Maxwell │ │
│ │(BizOps) │ │(Finance)│ │(Teach) │ │(Infra) │ │ (PR) │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
│ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Quinn │ │ Taylor │ │ Inbox │ │ Dev │ │
│ │(Family) │ │(Sched) │ │(Email) │ │ (Code) │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
└──────────────────────────────┬──────────────────────────────────────┘
│
┌──────────────────────────────▼──────────────────────────────────────┐
│ INTEGRATION LAYER │
│ │
│ M365/Planner │ Home Assistant │ Email │ Invoice Ninja │
│ DocuSeal │ Obsidian │ n8n │ SearXNG │
└──────────────────────────────┬──────────────────────────────────────┘
│
┌──────────────────────────────▼──────────────────────────────────────┐
│ HARDWARE LAYER │
│ │
│ Mac mini M4 Proxmox Node 1 Proxmox Node 2 │
│ (primary compute) (services 1) (services 2) │
│ OpenClaw, MC, HA, n8n, Plex, Invoice │
│ Chatterbox TTS AdGuard, Metabase Ninja, DocuSeal │
└─────────────────────────────────────────────────────────────────────┘
Gateway: OpenClaw
OpenClaw is the central nervous system. All traffic passes through it.
Key Responsibilities
Session Management
- Each agent maintains its own session context
- Memory injection: daily files + MEMORY.md loaded at session start
- Session history stored locally
Agent Routing
- Default agent receives all messages
- Routing rules: keyword-based, semantic, or agent-decided
- Agent-to-agent delegation (Chief can hand off to Marcus)
Cron Scheduler
- Runs scheduled tasks autonomously
- Each cron: target agent, schedule (cron syntax), message/task
- Morning briefing, email digest, weekly review, etc.
Channel Multiplexing
- Signal, iMessage, Email, Web UI on one gateway
- Each channel can have its own default agent
- TTS output configurable per channel
TTS Integration
- ElevenLabs-compatible API (Chatterbox TTS locally)
- Per-agent voice configuration
- Inline
[[tts:voiceId=X]]directives in responses
Memory Architecture
Session Start
│
├── Load MEMORY.md (long-term curated facts)
├── Load memory/YYYY-MM-DD.md (today's log)
├── Load memory/YYYY-MM-DD.md (yesterday's log)
└── Inject into agent context
During Session
│
└── Agent writes key events to memory/YYYY-MM-DD.md
Heartbeat (periodic)
│
├── Review recent daily files
├── Distill important events
└── Update MEMORY.md (long-term)
Three memory tiers:
- Daily files (
memory/YYYY-MM-DD.md): Raw logs, temporary context - MEMORY.md: Curated long-term facts, decisions, lessons learned
- Obsidian vault: Deep notes, project docs, research, synced via iCloud
Agent Architecture
Each agent is:
id: agent-id
name: Display Name
model: claude-[haiku|sonnet|opus]-[version]
voice: voice-id (optional)
description: "One-line for routing"
system: |
[Full system prompt]
PERSONA: ...
DOMAIN: ...
BOUNDARIES: ...
ESCALATION: ...
STYLE: ...
Model Distribution (Adam's system)
Opus 4.6: Riley, Matlock, Maxwell (3 agents — high-quality output)
Sonnet 4.6: Chief, Vega, Forge, Dev, Olivia, Marcus, Sterling, Sage (real-time + balanced)
Haiku 4.5: Quinn, Taylor, Inbox (3 agents — high-volume, cheap)
Tip: Opus 4.7 is now available and is the most capable model for complex reasoning — consider it for Matlock (legal) and Riley (long-form writing).
Integration Patterns
Pull Integration
Agent requests data when asked:
User → Chief → "What tasks are due today?"
Chief → [calls Planner API] → Returns tasks
Chief → Synthesizes response → User
Push Integration (Cron-based)
System proactively delivers data:
Cron (7 AM) → Chief → [gets calendar + tasks + weather]
Chief → Synthesizes briefing → Signal message → User
Event-Driven (n8n)
External events trigger agent actions:
Email with [TASK] arrives → n8n webhook detects
n8n → POST to OpenClaw/Chief → "Create task: [parsed task]"
Chief → Creates Planner task → Confirms via Signal
Security Architecture
Network Zones
INTERNET
│
├── Cloudflare Tunnel (public services only)
│ ├── clients.yourdomain.com → Invoice Ninja
│ └── contracts.yourdomain.com → DocuSeal
│
└── Tailscale VPN (admin access only)
├── n8n (192.168.x.120:5678)
├── Home Assistant (192.168.x.124:8123)
└── Proxmox (192.168.x.55, 192.168.x.70)
LAN (192.168.1.x)
├── Mac mini (192.168.x.100) — primary compute
├── Proxmox 1 (192.168.x.55)
└── Proxmox 2 (192.168.x.70)
Data Residency
| Data Type | Stays Local | Cloud |
|---|---|---|
| Memory files | ✅ Mac mini | ❌ |
| Voice audio | ✅ Chatterbox local | ❌ |
| Contracts | ✅ Proxmox 2 | ❌ |
| Invoices | ✅ Proxmox 2 | ❌ |
| Agent prompts | ✅ Mac mini | ❌ |
| AI reasoning | — | ✅ Claude API only |
| Messages | ✅ Signal E2E | Signal servers |
| Notes | ✅ iCloud encrypted | iCloud |
Request Flow: Full Example
"Research the Agilent Q1 earnings report"
1. User sends Signal message
2. OpenClaw receives via Signal channel
3. Default routing → Chief
4. Chief reads message
5. Chief recognizes research request
6. Chief delegates to Marcus ("Research: Agilent Q1 earnings")
7. Marcus executes: web search + synthesis
8. Marcus returns structured brief to Chief
9. Chief synthesizes final response
10. Chief responds via Signal (text + optional Obama TTS)
11. Chief logs key points to memory/YYYY-MM-DD.md
Total: ~8-15 seconds API calls: 2 (Chief + Marcus) Cost: ~$0.02 (Sonnet 4.6 + Sonnet 4.5)