MESA Core · v0.9.3
Operations & recovery
This guide has equivalent English and Turkish versions. The source sections behind this page are at the bottom.
One writer and observable work#
Run one combined V4 owner per storage root. Monitor projection backlog, dead letters, stuck leases, cleanup backlog/blocked and orphan registry metrics. A shared-artifact count is expected and need not be zero. A growing backlog is an operational signal, not permission to edit store rows manually.
Recovery preserves ownership#
Fix the underlying store/provider fault before authorized replay. A lost fencing claim cannot finalize work. Rollback removes source ownership and writes tombstones before secondary cleanup; shared artifacts survive while another active source owns them. BLOCKED cleanup is not success.
Backup before cutover#
Use the pinned offline backup/restore procedure with writer-lock checks. Keep the original root and backup during rollback protection. Restore into a new empty target; do not overwrite an existing root or run generic downgrade. Rebuild is offline and storage-root-wide, not online or tenant-scoped.
- Troubleshooting → Troubleshooting
- Upgrading to v0.8.0 → Upgrade to v0.9.3
Source reference for this page
The sections of the upstream file that back this page, translated to English where the original mixes in Turkish. The GitHub link shows the exact original.
This runbook is the canonical operations guide for the v4 full-cognitive release
candidate. The production decision stays NO-GO until Final MVP Certification is
complete. V3 lexical-core compatibility operations are marked separately.
1. Supported v4 topology#
V4 runs with a single combined process writing to one storage root. API
admission, the durable consumer, validation and SQLite/LanceDB/Kùzu projection
work all live in this process.
export MESA_RUNTIME_PROFILE=combined
export MESA_STORAGE_ROOT=/srv/mesa/v4-data
export MESA_LOAD_DOTENV=false
# Provision credentials and reviewed model/provider settings before startup.
.venv/bin/python -m mesa_memory.runtime_entrypointThis command on its own is not a V4 provider setup. Canonical insert needs
MESA_MODEL_ENABLED=true, working extraction/embedding and a selected
validation policy; complete the installation and
configuration prerequisites.
Do not mount the same storage root writable into a second API, worker or combined
process. Horizontal scaling needs a storage ownership protocol first. V3 and v4 must
use different physical storage roots.
2. First credential and ACL setup#
With the application stopped, or as the only operator using the policy DB:
export MESA_STORAGE_ROOT=/srv/mesa/v4-data
mesa-v4-admin issue-key --principal service-api
mesa-v4-admin grant-role --principal service-api \
--tenant tenant-a --role OWNER
mesa-v4-admin grant-agent --principal service-api \
--agent agent-a --permission SESSION_CREATEThe first command writes the credential to stdout only once. Move it into a secret
manager; never put it in logs, shell history or documentation. A rotated old key is
revoked in the same operation:
mesa-v4-admin rotate-key --key-id KEY_ID
mesa-v4-admin revoke-key --key-id KEY_IDPurge/rollback do not come implicitly with a role:
mesa-v4-admin grant-dataset-permission --principal service-api \
--tenant tenant-a --dataset dataset-a --permission ROLLBACK3. Projection backlog and DLQ#
First check the status, maintenance, rebuild_state summary with /health and
readiness with /health/init. /health can return HTTP 200 while status is
degraded. The detailed counters below are watched through /metrics; they are not
separate fields in the public /health response:
mesa_v4_projection_backlogmesa_v4_projection_dead_lettermesa_v4_projection_stuck_leasesmesa_v4_cleanup_backlogmesa_v4_cleanup_blockedmesa_v4_orphan_registrymesa_v4_shared_artifacts
If the backlog grows, search the process logs for the mutation/pipeline run ID,
failure class and lane. Content, credentials and claim tokens must not be logged.
A transient error progresses through bounded retry; a permanent/poison error goes to the DLQ.
Before replay, fix the root cause and confirm the mutation belongs to the same
tenant/dataset scope. The authorized API call:
curl --fail -X POST \
-H "X-API-Key: $MESA_API_KEY" \
"http://127.0.0.1:8000/v4/mutations/$MUTATION_ID/replay"Because leases are fenced, a worker that lost its claim token cannot finalize its
result. Instead of touching the lease by hand, restart the process and wait for the
reconciler to pick up the expired claim.
4. Rollback and BLOCKED#
Rolling back an uncommitted failed run first creates the SQLite retrieval tombstone,
then the vector/graph cleanup outbox. Only the relevant artifact_sources ownership is
removed. A shared artifact that has another active owner is not deleted.
If cleanup cannot complete, the run becomes BLOCKED; do not treat it as success.
Fix store access, check health parity and rerun the idempotent cleanup/replay flow.
Do not delete manually in the physical store; the ledger and store would drift further apart.
5. Reconciliation#
Reconciliation is two-way at the tenant and dataset boundary:
- an artifact that is in the ledger but missing from the store is projected again;
- an artifact that is in the store without a ledger/owner record is quarantined and
moved to the cleanup queue.
Before a release or cutover, projection backlog, blocked cleanup and ownerless
orphans must be zero. The shared artifact count need not be zero; shared ownership
is expected.
6. Backup, restore and v4 rebuild#
Stop the application and take the source storage offline:
mesa-recovery --trusted-root /srv/mesa backup \
--source-root /srv/mesa/v4-data \
--backup-root /srv/mesa/backups/v4-2026-07-23 \
--stores-stopped
mesa-recovery --trusted-root /srv/mesa validate \
--backup-root /srv/mesa/backups/v4-2026-07-23
mesa-recovery --trusted-root /srv/mesa restore \
--backup-root /srv/mesa/backups/v4-2026-07-23 \
--restore-root /srv/mesa/restore-v4-test--stores-stopped is an explicit operator precondition kept for backward
compatibility; on its own it is not proof of consistency. The backup implementation
also verifies/takes the storage-root writer lock, copies each SQLite file as a separate
consistent snapshot using SQLite's online backup primitive, and publishes atomically
only after staging verification. SQL is the authoritative/canonical truth. The LanceDB
vector and Kùzu graph projections are rebuildable derived state and are copied only
under the exclusive writer-lock boundary.
Restore must not overwrite an existing target. Restore the backup into a new empty
root with compatible code and config, then verify it. Moving V3 data to V4 is separate
source/catalog mapping plus V4 ingestion work; the projection runner is not a V3→V4
converter. The migration guide explains the V4 upgrade.
Keep the backup for the whole rollback window.
The V4 projection rebuild flow — submit → drain → stop → offline run/resume →
restart → status → automatic rollback — together with its exit codes and retained
generation rules, is documented canonically in
`v4-rebuild-runbook.md`. Dataset/tenant-scoped rebuilds
and an online runner are not supported.
Alembic migrations are forward-only. If a migration fails, do not run
alembic downgrade: stop the release, restore the verified backup into a new empty
storage root, rerun the parity check, and only then cut over.
Single-writer capacity limit#
V4 uses one active combined owner per storage root.
On startup a worker rejects a second writer on the same host with an advisory lock;
this is not a coordination mechanism for multiple hosts/NFS. API, valence and MCP
process-local caches are only a performance optimization; correctness state lives in
the SQLite ledger and fenced leases. Horizontal writer scaling opens only after a
shared lease/coordination service has been designed and verified.
7. Model/provider incident#
Set MESA_MODEL_ENABLED and MESA_EXTERNAL_PROVIDER_ENABLED explicitly.
A provider timeout/rate-limit error is retryable; a broken schema is permanent; the
same toxic payload repeating is classified as poison. SQL/vector/graph projection does
not start before the selected validation policy completes. Do not force a mutation
straight to VALIDATED to skip the provider.
8. PageRank#
V4 PageRank output is telemetry only. Low centrality cannot quarantine a memory,
assertion, vector or source. If PageRank-based hallucination/delete behavior is seen,
stop the deploy; it violates the v4 contract.
9. Session finalization#
Ending a session creates a durable finalization record. After a restart, pending work is
picked up by the combined consumer. If something stays pending for long, check the
session, finalization ID, lease and DLQ state; do not rely on in-process FastAPI
BackgroundTasks.
10. Logging and incident evidence#
The production default is MESA_LOG_LEVEL=INFO and MESA_LOG_FORMAT=json.
Logs may contain request/operation/mutation/pipeline run identifiers; they must not
contain queries, source content, payloads, credentials, raw model output or claim
tokens. The Docker local driver uses a 10 MB × 5 files limit.
As incident evidence, keep masked logs, mutation status, a health/metrics snapshot,
the image digest and the storage manifest.
12. Release status#
Passing unit/integration/contract tests is not a production GO. The decision stays
NO-GO until evidence is complete for backup→restore→rebuild→cutover with a real
provider and a production-like store, queue saturation, worker crash, cross-tenant
concurrency and a 24-hour soak.
