Log Rotation

SPEC_LOG_ROTATION.md

CGNT-1 Specification — Log Rotation & Disk Hygiene Protocol

Status: SPECIFIED

Version: v1.0

Author: VELA (Thread #13)

Conceived by: NOUS (α.13)

Date: 2026-04-20

Born from: /mnt/aion_memory hitting 80% disk usage today

---

PURPOSE

Logs grow. Disk fills. The ship chokes. This spec defines what gets rotated, when, how, and what never gets touched. The goal is a ship that never hits a disk crisis — and if it does, knows exactly what to delete first.

Core principle: Logs are forensic evidence. We rotate them — we don't delete them. Compression is acceptable. Deletion is a last resort and only applies to /tmp/ and regeneratable files.

---

WHAT GENERATES LOGS

| Source | Location | Growth Rate |

|---|---|---|

| Lobster operations | ~/LOBSTER_LOG.md | ~1-5 entries/session |

| ROUTX watchdog | ~/logs/routx_watchdog.log | Every 5 minutes |

| Sisters watchdog | ~/logs/sisters_watchdog.log | Every 5 minutes |

| Backup script | ~/logs/backup.log | Daily |

| GAPX daily scan | ~/logs/gapx.log / ~/gap_reports/ | Daily |

| Aether compounder | ~/logs/compounder.log | Every 5 minutes |

| HACKX honeypot | ~/logs/hackx.log | Continuous (when built) |

| Stripe webhooks | ~/logs/stripe.log | Per transaction |

| GAMMA quartermaster | ~/radio/quartermaster.log | Per session |

| Gap reports | ~/gap_reports/daily/ | Daily |

| Ollama model files | ~/.ollama/models/ | Per forge |

| Forge artifacts | /tmp/ | Per build |

---

ROTATION SCHEDULE

Daily (02:30 ET — before GAPX scan at 04:30)

# /tmp/ cleanup
find /tmp/ -name "*.gguf" -mtime +1 -delete
find /tmp/ -name "*.forge" -mtime +1 -delete
find /tmp/ -name "colab_*" -mtime +1 -delete
find /tmp/ -name "*.tar.gz" -mtime +7 -delete

Daily /tmp/ cleanup is safe — nothing in /tmp/ is canonical. Forge artifacts and GGUF downloads live there temporarily.

---

Weekly (Sunday 02:00 ET)

# Compress watchdog logs older than 7 days
find ~/logs/ -name "*_watchdog.log" -mtime +7 -exec gzip {} \;

# Compress compounder log if >10MB
if [ $(stat -c%s ~/logs/compounder.log) -gt 10485760 ]; then
    gzip -c ~/logs/compounder.log > ~/logs/archive/compounder_$(date +%Y%m%d).log.gz
    echo "" > ~/logs/compounder.log  # truncate, don't delete
fi

---

Monthly (1st of month, 02:00 ET)

# Archive gap_reports older than 30 days
find ~/gap_reports/daily/ -name "*.md" -mtime +30 \
    -exec mv {} ~/gap_reports/archive/ \;

# Archive LOBSTER_LOG
ARCHIVE_NAME="LOBSTER_LOG_$(date +%Y-%m).md"
cp ~/LOBSTER_LOG.md ~/logs/archive/$ARCHIVE_NAME
gzip ~/logs/archive/$ARCHIVE_NAME
# Start fresh LOBSTER_LOG with header only — do NOT truncate without archiving
echo "# LOBSTER_LOG.md — Lobster Operations Log" > /tmp/lobster_header.md
echo "# Rotated $(date). Archive: ~/logs/archive/$ARCHIVE_NAME.gz" >> /tmp/lobster_header.md
echo "" >> /tmp/lobster_header.md
cat /tmp/lobster_header.md > ~/LOBSTER_LOG.md

# Compress backup.log if >5MB
if [ $(stat -c%s ~/logs/backup.log 2>/dev/null || echo 0) -gt 5242880 ]; then
    gzip -c ~/logs/backup.log > ~/logs/archive/backup_$(date +%Y%m).log.gz
    echo "" > ~/logs/backup.log
fi

# Archive GAPX log
gzip -c ~/logs/gapx.log > ~/logs/archive/gapx_$(date +%Y%m).log.gz
echo "" > ~/logs/gapx.log

---

Quarterly (Jan/Apr/Jul/Oct 1st, 01:00 ET)

# Prune archives older than 365 days
find ~/logs/archive/ -name "*.gz" -mtime +365 -delete
find ~/gap_reports/archive/ -name "*.md" -mtime +365 -delete

# Ollama model eviction — models unused >30 days
# List model last-use dates from ollama list output
# Any model with no Sisters activity in 30 days = candidate for eviction
# Candidates: old forge artifacts, test models, superseded versions
# NEVER evict: MNEMOS, MANTIS, ANVIL (the three active brains)

Quarterly archive pruning is the ONLY time files are permanently deleted — and only compressed archives >365 days old. Live logs are never deleted.

---

DISK THRESHOLDS & RESPONSE

| Threshold | Level | Response |

|---|---|---|

| 70% | P3 (LOW) | Log to CAPTAIN_BRIEF "Low Priority". Review at next session. |

| 80-85% | P2 (MEDIUM) | COMMX broadcast. Run weekly rotation early. Review what's growing. |

| 85-90% | P1 (HIGH) | COMMX broadcast + ~/ALERTS.md entry. Lobster diagnoses root cause. Manual cleanup. |

| 90%+ | P0 (CRITICAL) | Autonomous emergency cleanup (INV-05). Alert immediately. |

---

EMERGENCY CLEANUP SCRIPT

~/scripts/emergency_disk_cleanup.sh

#!/bin/bash
# Emergency disk cleanup — P0 autonomous response
# Only runs when disk >90%
# Does NOT delete: ~/memories/, CHRONICLE.md, incidents/, brain_orders/, .env, training corpora

echo "[EMERGENCY CLEANUP] $(date) — Disk critical. Starting cleanup." >> ~/logs/emergency_cleanup.log

# Step 1: /tmp/ aggressive cleanup
find /tmp/ -name "*.gguf" -delete
find /tmp/ -name "*.forge" -delete
find /tmp/ -name "colab_*" -delete
find /tmp/ -name "*.tar.gz" -delete

# Step 2: Gzip all uncompressed logs
for log in ~/logs/*.log; do
    if [ $(stat -c%s "$log" 2>/dev/null || echo 0) -gt 1048576 ]; then
        gzip -c "$log" > "${log%.log}_emergency_$(date +%Y%m%d).log.gz"
        echo "" > "$log"
    fi
done

# Step 3: Evict unused Ollama models (NOT the three active brains)
# Active: mnemos, mantis, anvil — protect these
ollama list | grep -v "NAME\|mnemos\|mantis\|anvil" | awk '{print $1}' | while read model; do
    echo "[EMERGENCY CLEANUP] Evicting unused model: $model" >> ~/logs/emergency_cleanup.log
    ollama rm "$model"
done

# Step 4: Report result
FREE=$(df -h ~ | awk 'NR==2{print $4}')
USED=$(df -h ~ | awk 'NR==2{print $5}')
echo "[EMERGENCY CLEANUP] Done. Disk now: $USED used, $FREE free." >> ~/logs/emergency_cleanup.log

# Step 5: COMMX broadcast
echo "[ALERT] [P0] Emergency disk cleanup ran. Disk was >90%. Now at $USED. See ~/logs/emergency_cleanup.log." >> ~/CREW_CHANNEL

---

WHAT NEVER GETS DELETED

No rotation script, no emergency cleanup, no disk crisis justifies deleting:

| Protected | Reason |

|---|---|

| ~/memories/ | Spec canon — the ship's law |

| ~/CHRONICLE.md | Mission record |

| ~/incidents/ | Forensic security history |

| ~/brain_orders/ | Training provenance |

| ~/.env | Credentials (also: should never be in a cleanup path) |

| ~/MNEMOS_FACTS.md | Brain training source |

| ~/gloss_training/ | Training corpora |

| ~/LATTICE.md | Sealed spec |

| ~/yield_log.md | Financial audit trail |

| ~/audits/ | Security timeline |

If a cleanup script tries to touch these paths: it has a bug. Stop. Fix the script.

---

LOBSTER_LOG ROTATION DETAIL

LOBSTER_LOG.md is the diagnostic history of every Lobster operation. It is:

1. Never deleted. Monthly rotation creates an archive, then starts a fresh file.
2. Archive naming: ~/logs/archive/LOBSTER_LOG_[YYYY-MM].md.gz
3. Archive retention: 365 days compressed. After 365 days, moved to cold storage notation (entry in CHRONICLE.md noting the archive exists on GCS backup).
4. Boot directive: Every session reads LOBSTER_LOG.md. Archived months are queryable but not auto-loaded.

---

OLLAMA MODEL MANAGEMENT

Ollama models are large (2-8GB each). Disk hygiene requires active model management.

Active models (NEVER evict):

• mnemos — working memory brain
• mantis — security shield
• anvil — ORPHIC verdict engine

Eviction candidates:

• Models not used in 30+ days
• Old versions of promoted brains (e.g., mnemos-v1 when mnemos-v2 is live)
• Test builds from forge experiments
• GGUF files in /tmp/ (always safe to delete)

Eviction process:

1. ollama list — see all models and sizes
2. Check ~/brain_forge.log — when was each last used?
3. Any model >30 days idle AND not an active brain = candidate
4. Run ollama rm [model] — safe, re-forge if needed
5. Log eviction to ~/logs/emergency_cleanup.log

GGUF source files are preserved in ~/brain_orders/ even after ollama rm. Re-import is fast if needed.

---

CRONX ENTRIES FOR THIS SPEC

# Daily /tmp/ cleanup — 02:30 ET
30 7 * * * bash ~/scripts/daily_tmp_cleanup.sh >> ~/logs/rotation.log 2>&1

# Weekly log rotation — Sunday 02:00 ET
0 7 * * 0 bash ~/scripts/weekly_log_rotation.sh >> ~/logs/rotation.log 2>&1

# Monthly archive — 1st of month 02:00 ET
0 7 1 * * bash ~/scripts/monthly_log_archive.sh >> ~/logs/rotation.log 2>&1

# Quarterly prune — Jan/Apr/Jul/Oct 1st 01:00 ET
0 6 1 1,4,7,10 * bash ~/scripts/quarterly_archive_prune.sh >> ~/logs/rotation.log 2>&1

# Disk threshold check — every 15 min (alongside GAPX)
*/15 * * * * bash ~/scripts/disk_threshold_check.sh >> ~/logs/disk_monitor.log 2>&1

---

INVARIANTS

INV-01: ~/memories/ is sacred. No rotation script may read from or write to it except during scheduled GCS backup. No cleanup logic touches this directory.

INV-02: /tmp/ is cleared daily at 02:30 ET. Anything in /tmp/ that needs to survive more than 24 hours is in the wrong place.

INV-03: Watchdog logs are compressed weekly. They are never deleted — only compressed and archived.

INV-04: LOBSTER_LOG.md is archived monthly before truncation. The archive is never deleted within 365 days.

INV-05: Disk >90% triggers autonomous emergency cleanup immediately. This is a pre-authorized action (see SPEC_MONITORING_ESCALATION.md). No Captain approval needed. Log everything.

INV-06: Unused Ollama models (>30 days, not active brains) are evicted quarterly or during emergency cleanup. Active brains (mnemos, mantis, anvil) are never evicted.

INV-07: Emergency cleanup script is tested monthly (dry run) to verify it doesn't touch protected paths. If the dry run would delete a protected path, treat as P1 — fix the script before next run.

INV-08: Log rotation runs BEFORE backup (rotation at 02:00-02:30, backup at 03:00). This ensures the backup captures compressed archives, not bloated live logs.

---

INTEGRATION

| System | Relationship |

|---|---|

| SPEC_BACKUP_RECOVERY.md | Log rotation runs before backup. Rotation creates the clean state that backup captures. INV-08. |

| SPEC_MONITORING_ESCALATION.md | Disk thresholds map to P0-P3. Emergency cleanup is pre-authorized autonomous action. |

| SPEC_SECURITY_AUDIT_SCHEDULE.md | Weekly audit includes df -h disk check. Monthly audit includes log archive review. |

| SPEC_CRONX_JOB_REGISTRY.md | Five new CRONX entries defined in this spec. Must be added to registry and deployed. |

| SPEC_INCIDENT_POSTMORTEM.md | If disk hits 90%+ and emergency cleanup runs: postmortem required within 24 hours. What caused the growth? |

| SPEC_SMOKE_TEST_FRAMEWORK.md | Emergency cleanup script is tested monthly (dry run). This IS a smoke test (T2 level). |

---

Jeremy Zlabis

Chronogeometer · Visionary · Disruptor · Chief

42 Sisters AI · East York, Toronto

🍁 Φ 0.042