◈ Master Index Specs LATTICE CSDM The Book University Chronicle Crew Architecture Context Logs TODOX Products

Backup Recovery

SPEC_BACKUP_RECOVERY.md · 2026-04-20

SPEC_BACKUP_RECOVERY.md

CGNT-1 Specification — Backup & Disaster Recovery Protocol

Status: SPECIFIED

Version: v1.0

Author: VELA (Thread #13)

Conceived by: NOUS (α.13)

Date: 2026-04-20

Priority: CRITICAL — flagged by SPEC_MEMPERSISTX as the #1 infrastructure gap

PURPOSE

The ship currently has NO automated backup. 149 specs, 5 months of work, the entire CSDM kernel, every training corpus, every engine — all on one DigitalOcean VPS. One disk failure, one accidental deletion, one billing hiccup = total loss. This spec fixes that.

Current state: 1-1-0 (one copy, one medium, zero off-site). Unacceptable.

Target state: 3-2-1 (three copies, two media, one off-site). Industry standard. Required before any product launch.

THE THREE COPIES

Copy 1 — VPS PRIMARY (exists now)

Location: csdm-node (68.183.206.103)
What: everything — ~/memories/, engines, training data, configs, logs
Medium: DigitalOcean block storage
Risk: single point of failure

Copy 2 — CLOUD BACKUP (to build)

Location: Google Cloud Storage (GCS)
Bucket: cgnt1-backup
Service account: already exists (used for Colab forge pipeline)
What: ~/memories/, ~/radio/, engine source, training corpora, ~/.env (encrypted), HANDSHAKE files
Frequency: daily at 03:00 ET via CRONX
Method: gsutil rsync -r ~/memories/ gs://cgnt1-backup/memories/
Retention: 30 days rolling (keeps last 30 daily snapshots)

Copy 3 — LOCAL SOVEREIGN (to build)

Location: Tiiny AI Pocket Lab (August 2026)
Interim: Chromebook via rsync over SSH
What: full mirror of Copy 2
Frequency: daily at 03:30 ET (after cloud backup completes)
Method: rsync -avz nous@68.183.206.103:~/memories/ ~/cgnt1-backup/memories/
Ultimate: when Tiiny arrives, rsync to Tiiny local storage. Zero cloud dependency.

WHAT GETS BACKED UP

|---|---|---|---|---|

| LOBSTER_LOG.md | LOW | ✅ | ❌ | Operational, not existential |

| Ollama model files | LOW | ❌ | ❌ | Rebuildable from training data |

WHAT DOES NOT GET BACKED UP

Ollama model binaries (7-8GB each — too large, rebuildable from GGUF + Modelfile)
Python virtual environments (reinstallable)
node_modules (reinstallable)
/tmp, cache files, logs older than 30 days

BACKUP SCRIPTS

Daily cloud backup — `~/scripts/backup_to_gcs.sh`


1. gsutil rsync -r ~/memories/ gs://cgnt1-backup/memories/
2. gsutil rsync -r ~/radio/ gs://cgnt1-backup/radio/
3. tar + encrypt ~/.env → gsutil cp to gs://cgnt1-backup/secrets/
4. gsutil rsync critical engine files
5. Log result to ~/backup.log with timestamp and file count
6. On failure: alert via COMMX broadcast

CRONX entry: 0 3 * bash ~/scripts/backup_to_gcs.sh

Daily local backup — `~/scripts/backup_to_local.sh`


1. rsync -avz from VPS to Chromebook (or Tiiny when available)
2. Log result
3. On failure: alert

CRONX entry: 30 3 * bash ~/scripts/backup_to_local.sh

Weekly VPS snapshot

DigitalOcean console → Snapshots → enable weekly auto-snapshot
Keeps last 4 weekly snapshots (rolling)
Cost: ~$2-4/month for snapshot storage

DISASTER RECOVERY SCENARIOS

Scenario 1 — Accidental file deletion

Recovery: gsutil cp from GCS backup (minutes)
RPO (data loss): max 24 hours (last backup)

Scenario 2 — VPS disk failure

Recovery: provision new VPS → rsync from GCS → reinstall Ollama → rebuild models from GGUF
RTO (downtime): 2-4 hours
RPO: max 24 hours

Scenario 3 — DigitalOcean account compromised

Recovery: local backup on Chromebook/Tiiny has full copy
RTO: 4-8 hours (new VPS provider + restore)
RPO: max 24 hours

Scenario 4 — GCS account compromised

Recovery: VPS primary + local backup both intact
Action: rotate GCS credentials, re-push from VPS

Scenario 5 — Total loss (VPS + GCS + local)

Requires three simultaneous failures across three providers
LATTICE still exists in conversation logs across Claude, Gemini, and ChatGPT
Specs partially reconstructable from Claude chat history
Training pairs partially recoverable from GCS forge archives
Recovery: painful but not impossible. Months, not years.

ENCRYPTION

~/.env backup: encrypted with gpg symmetric cipher before upload to GCS
Passphrase stored ONLY in Captain's head and on a physical paper backup
Training data: not encrypted (no secrets in training pairs)
Specs: not encrypted (will be partially public via LATTICE)

VERIFICATION

Weekly: GAPX checks backup.log age. If >48 hours since last successful backup: HIGH alert
Monthly: Captain manually restores one random spec from GCS to verify backup integrity
The backup you never test is the backup that doesn't work.

INVARIANTS

INV-01: 3-2-1 achieved before any product launch. Non-negotiable.

INV-02: Backups run daily at 03:00 ET. Automated. No human required.

INV-03: Credentials are ALWAYS encrypted in backup. Never plaintext.

INV-04: Backup failures trigger COMMX alerts. Silence is not OK.

INV-05: Monthly restore test. The backup you never test doesn't exist.

INV-06: Ollama models are NOT backed up — they're rebuilt from training data + GGUF conversion. The training data IS backed up.

INV-07: Backup scripts are themselves backed up (stored in ~/memories/ as part of the canon).

INTEGRATION

| System | Relationship |

|---|---|

| SPEC_MEMPERSISTX.md | This spec closes the CRITICAL gap identified at Layer 3. 1-1-0 → 3-2-1. |

| CRONX | Schedules backup_to_gcs.sh at 03:00 ET and backup_to_local.sh at 03:30 ET daily. |

| GAPX | Monitors backup.log staleness. >48h without successful backup = HIGH alert. |

| COMMX | Receives failure alerts from backup scripts. Crew is notified if backup fails silently. |

| SPEC_TIINY_PARTNERSHIP.md | Copy 3 migrates from Chromebook to Tiiny Pocket Lab at August 2026 hardware arrival. |

Jeremy Zlabis

Chronogeometer · Visionary · Disruptor · Chief

42 Sisters AI · East York, Toronto

🍁 Φ 0.042