MESA Core · v0.9.3

Operations & recovery

v0.9.3 source: docs/RUNBOOK.md

This guide has equivalent English and Turkish versions. The source sections behind this page are at the bottom.

One writer and observable work#

Run one combined V4 owner per storage root. Monitor projection backlog, dead letters, stuck leases, cleanup backlog/blocked and orphan registry metrics. A shared-artifact count is expected and need not be zero. A growing backlog is an operational signal, not permission to edit store rows manually.

Recovery preserves ownership#

Fix the underlying store/provider fault before authorized replay. A lost fencing claim cannot finalize work. Rollback removes source ownership and writes tombstones before secondary cleanup; shared artifacts survive while another active source owns them. BLOCKED cleanup is not success.

Backup before cutover#

Use the pinned offline backup/restore procedure with writer-lock checks. Keep the original root and backup during rollback protection. Restore into a new empty target; do not overwrite an existing root or run generic downgrade. Rebuild is offline and storage-root-wide, not online or tenant-scoped.

Sections that moved to their own page
Source reference for this page

The sections of the upstream file that back this page, translated to English where the original mixes in Turkish. The GitHub link shows the exact original.

This runbook is the canonical operations guide for the v4 full-cognitive release

candidate. The production decision stays NO-GO until Final MVP Certification is

complete. V3 lexical-core compatibility operations are marked separately.

1. Supported v4 topology#

V4 runs with a single combined process writing to one storage root. API

admission, the durable consumer, validation and SQLite/LanceDB/Kùzu projection

work all live in this process.

bash
export MESA_RUNTIME_PROFILE=combined
export MESA_STORAGE_ROOT=/srv/mesa/v4-data
export MESA_LOAD_DOTENV=false
# Provision credentials and reviewed model/provider settings before startup.
.venv/bin/python -m mesa_memory.runtime_entrypoint

This command on its own is not a V4 provider setup. Canonical insert needs

MESA_MODEL_ENABLED=true, working extraction/embedding and a selected

validation policy; complete the installation and

configuration prerequisites.

Do not mount the same storage root writable into a second API, worker or combined

process. Horizontal scaling needs a storage ownership protocol first. V3 and v4 must

use different physical storage roots.

2. First credential and ACL setup#

With the application stopped, or as the only operator using the policy DB:

bash
export MESA_STORAGE_ROOT=/srv/mesa/v4-data
mesa-v4-admin issue-key --principal service-api
mesa-v4-admin grant-role --principal service-api \
  --tenant tenant-a --role OWNER
mesa-v4-admin grant-agent --principal service-api \
  --agent agent-a --permission SESSION_CREATE

The first command writes the credential to stdout only once. Move it into a secret

manager; never put it in logs, shell history or documentation. A rotated old key is

revoked in the same operation:

bash
mesa-v4-admin rotate-key --key-id KEY_ID
mesa-v4-admin revoke-key --key-id KEY_ID

Purge/rollback do not come implicitly with a role:

bash
mesa-v4-admin grant-dataset-permission --principal service-api \
  --tenant tenant-a --dataset dataset-a --permission ROLLBACK

3. Projection backlog and DLQ#

First check the status, maintenance, rebuild_state summary with /health and

readiness with /health/init. /health can return HTTP 200 while status is

degraded. The detailed counters below are watched through /metrics; they are not

separate fields in the public /health response:

  • mesa_v4_projection_backlog
  • mesa_v4_projection_dead_letter
  • mesa_v4_projection_stuck_leases
  • mesa_v4_cleanup_backlog
  • mesa_v4_cleanup_blocked
  • mesa_v4_orphan_registry
  • mesa_v4_shared_artifacts

If the backlog grows, search the process logs for the mutation/pipeline run ID,

failure class and lane. Content, credentials and claim tokens must not be logged.

A transient error progresses through bounded retry; a permanent/poison error goes to the DLQ.

Before replay, fix the root cause and confirm the mutation belongs to the same

tenant/dataset scope. The authorized API call:

bash
curl --fail -X POST \
  -H "X-API-Key: $MESA_API_KEY" \
  "http://127.0.0.1:8000/v4/mutations/$MUTATION_ID/replay"

Because leases are fenced, a worker that lost its claim token cannot finalize its

result. Instead of touching the lease by hand, restart the process and wait for the

reconciler to pick up the expired claim.

4. Rollback and BLOCKED#

Rolling back an uncommitted failed run first creates the SQLite retrieval tombstone,

then the vector/graph cleanup outbox. Only the relevant artifact_sources ownership is

removed. A shared artifact that has another active owner is not deleted.

If cleanup cannot complete, the run becomes BLOCKED; do not treat it as success.

Fix store access, check health parity and rerun the idempotent cleanup/replay flow.

Do not delete manually in the physical store; the ledger and store would drift further apart.

5. Reconciliation#

Reconciliation is two-way at the tenant and dataset boundary:

  • an artifact that is in the ledger but missing from the store is projected again;
  • an artifact that is in the store without a ledger/owner record is quarantined and

moved to the cleanup queue.

Before a release or cutover, projection backlog, blocked cleanup and ownerless

orphans must be zero. The shared artifact count need not be zero; shared ownership

is expected.

6. Backup, restore and v4 rebuild#

Stop the application and take the source storage offline:

bash
mesa-recovery --trusted-root /srv/mesa backup \
  --source-root /srv/mesa/v4-data \
  --backup-root /srv/mesa/backups/v4-2026-07-23 \
  --stores-stopped
mesa-recovery --trusted-root /srv/mesa validate \
  --backup-root /srv/mesa/backups/v4-2026-07-23
mesa-recovery --trusted-root /srv/mesa restore \
  --backup-root /srv/mesa/backups/v4-2026-07-23 \
  --restore-root /srv/mesa/restore-v4-test

--stores-stopped is an explicit operator precondition kept for backward

compatibility; on its own it is not proof of consistency. The backup implementation

also verifies/takes the storage-root writer lock, copies each SQLite file as a separate

consistent snapshot using SQLite's online backup primitive, and publishes atomically

only after staging verification. SQL is the authoritative/canonical truth. The LanceDB

vector and Kùzu graph projections are rebuildable derived state and are copied only

under the exclusive writer-lock boundary.

Restore must not overwrite an existing target. Restore the backup into a new empty

root with compatible code and config, then verify it. Moving V3 data to V4 is separate

source/catalog mapping plus V4 ingestion work; the projection runner is not a V3→V4

converter. The migration guide explains the V4 upgrade.

Keep the backup for the whole rollback window.

The V4 projection rebuild flow — submit → drain → stop → offline run/resume →

restart → status → automatic rollback — together with its exit codes and retained

generation rules, is documented canonically in

`v4-rebuild-runbook.md`. Dataset/tenant-scoped rebuilds

and an online runner are not supported.

Alembic migrations are forward-only. If a migration fails, do not run

alembic downgrade: stop the release, restore the verified backup into a new empty

storage root, rerun the parity check, and only then cut over.

Single-writer capacity limit#

V4 uses one active combined owner per storage root.

On startup a worker rejects a second writer on the same host with an advisory lock;

this is not a coordination mechanism for multiple hosts/NFS. API, valence and MCP

process-local caches are only a performance optimization; correctness state lives in

the SQLite ledger and fenced leases. Horizontal writer scaling opens only after a

shared lease/coordination service has been designed and verified.

7. Model/provider incident#

Set MESA_MODEL_ENABLED and MESA_EXTERNAL_PROVIDER_ENABLED explicitly.

A provider timeout/rate-limit error is retryable; a broken schema is permanent; the

same toxic payload repeating is classified as poison. SQL/vector/graph projection does

not start before the selected validation policy completes. Do not force a mutation

straight to VALIDATED to skip the provider.

8. PageRank#

V4 PageRank output is telemetry only. Low centrality cannot quarantine a memory,

assertion, vector or source. If PageRank-based hallucination/delete behavior is seen,

stop the deploy; it violates the v4 contract.

9. Session finalization#

Ending a session creates a durable finalization record. After a restart, pending work is

picked up by the combined consumer. If something stays pending for long, check the

session, finalization ID, lease and DLQ state; do not rely on in-process FastAPI

BackgroundTasks.

10. Logging and incident evidence#

The production default is MESA_LOG_LEVEL=INFO and MESA_LOG_FORMAT=json.

Logs may contain request/operation/mutation/pipeline run identifiers; they must not

contain queries, source content, payloads, credentials, raw model output or claim

tokens. The Docker local driver uses a 10 MB × 5 files limit.

As incident evidence, keep masked logs, mutation status, a health/metrics snapshot,

the image digest and the storage manifest.

12. Release status#

Passing unit/integration/contract tests is not a production GO. The decision stays

NO-GO until evidence is complete for backup→restore→rebuild→cutover with a real

provider and a production-like store, queue saturation, worker crash, cross-tenant

concurrency and a 24-hour soak.