MESA Core · v0.9.3
Core mimarisi
Bu rehberin Türkçe ve İngilizce sürümleri aynı konuyu açıklar. Bu sayfayla ilgili kaynak bölümleri en altta yer alır.
Asıl kayıtlar SQLite’ta tutulur#
Kaynak kimliği, değiştirilemeyen belge sürümleri, bilgi kayıtları, değişikliklerin durumu ve çıktıların sahipliği SQLite’ta tutulur. LanceDB vektör araması, Kùzu ise ilişkiler üzerinde arama için kullanılır. Bu iki sistem, bir bilginin hâlâ geçerli olduğuna veya kimlerin erişebileceğine kendi başına karar vermez.
Veri alımı ve kurtarma nasıl ilerler?#
Bir istek kabul edildiğinde, yapılacak iş önce kalıcı olarak kaydedilir. Ardından bilgi çıkarımı ve seçilen doğrulamalar yapılır; kayıtlar SQL → vektör → çizge sırasıyla yazılır. Üç depo tek bir ACID işlemini paylaşmaz. Bu nedenle işin kime ait olduğu, hangi adımların tamamlandığı ve yeniden denendiğinde ne yapılacağı ayrıca izlenir. Yazma ancak COMMITTED durumuna ulaştığında tamamlanmış sayılır.
Ayrıntıları nerede bulabilirsiniz?#
Mimari sayfası veri akışını, depoları, aramayı ve kaynak izini ayrı ayrı açıklar. Uygulama ayrıntıları için veri alımı, arama, güvenlik ve işletim rehberlerine bakın.
Eşzamanlı arama ve kapsam önbelleği#
MESA_SEARCH_SCOPE_CACHE=true, kapsama özel geçici FTS indekslerini ve satır/metin özelliklerini dört kapsamlık LRU önbelleğinde yeniden kullanır. SQLite commit değişiklikleri önbelleği geçersiz kılar; her istekte yetkilendirme yeniden denetlenir. Sorgu embedding’i arama kilidinden önce hesaplanır; yeniden sıralama bağlantı bırakıldıktan sonra çalışır. scrypt doğrulama, anahtar oluşturma ve döndürme işlemleri event loop dışında yürütülür; yetki sonuçları önbelleğe alınmaz. Çizge kapalıysa başlangıç düğümlerinin hazırlığı atlanır.
Şemalar
Çağrılar yalnızca aşağı yönlüdür: API/MCP → işçiler → hafıza → depolama (ADR 0013). Kararı SQLite verir; LanceDB ve Kùzu ondan sonra sırayla yazılan görünümlerdir (ADR 0009).
Üç depo tek kök altındaki gömülü dosyalardır; birlikte yedeklenir ve geri yüklenir. Modeller kapalı combined çalışma belgeleri kabul eder ve BM25 araması sunar. Hukuk kuralları model istemez; açık uçlu SPO çıkarımı ayrıca açılır.
Tüm şemalar ve gerekçeleri: Mimari →
- Canonical entity and Graph V2 → Çizge dolaşımı
- Ecosystem and ownership → Ekosistem sınırları
- Performance and limitations → Performans ve sınırlar
Kaynak belgedeki ilgili bölümler
Bu bölüm, kaynak belgenin bu sayfayla ilgili kısımlarını değiştirmeden gösterir. Kaynak içindeki bir bölüme giden bağlantıyı açtığınızda metin otomatik olarak görünür.
Package version: 0.9.3, without production certification.
This document owns the current V4 design; production qualification remains
release candidate / NO-GO under the release policy.
The root architecture entry point links here. The obsolete
V3 whitepaper is available in Git history; see the
ADR index for decision applicability.
Module ownership#
| Module | Responsibility |
|---|---|
mesa_storage | Canonical SQLite, catalog repositories, ownership, retrieval and projection providers |
mesa_memory | Runtime configuration, extraction/validation, embedding, context, security and rebuild orchestration |
mesa_workers | Durable ingestion, projection and maintenance consumers |
mesa_api | HTTP schemas, authorized V3/V4 routes and control API |
mesa_client | Sync/async HTTP clients and optional integrations |
mesa_mcp | MCP service adapters, gateway, bindings and client lifecycle |
mesa_evals | Local repository diagnostics and deterministic retrieval contract tooling (not packaged in wheel) |
Higher layers may depend on lower layers: API/MCP → workers → memory → storage.
Runtime composition roots are explicit exceptions; the source guard is
`check_layer_imports.py`. MemoryDAO
remains the compatibility façade; catalog transactions have a focused repository
boundary. See ADRs 0012 and
0013.
Runtime boundary#
V4 has one supported storage-writing topology: one combined process owns
SQLite, LanceDB and Kùzu. API admission and background consumers run in that
process. A second writer must never mount the same storage root.
V3 remains available as the lexical-core compatibility surface and keeps its
existing API/worker topology. V3 and v4 must not write the same physical
storage directories concurrently.
Canonical v4 fact extraction is owned by FactExtractionService.
Security and catalog hierarchy#
The v4 security boundary is a tenant, not an agent:
Principal
└── Tenant
└── Workspace
└── Dataset
└── Document
└── immutable DocumentRevision
└── SourceChunkRoles are OWNER, WRITER and READER at tenant, workspace or dataset
scope. Roles inherit downward. Purge and rollback require explicit dataset
permissions. Agent IDs represent compute/persona context and do not grant
tenant access.
An authenticated session is generated by the server and binds one principal,
tenant, workspace, agent and immutable set of datasets. Insert, search,
context, status, replay and rollback derive their scope from this record.
API credentials use key_id.secret. Only the salted scrypt digest is stored;
plaintext is returned once by mesa-v4-admin issue-key. Multiple keys may
belong to one or more principals and can be revoked or rotated independently.
Canonical ingestion#
POST /v4/memory/insert durably admits exact source content and creates:
- one
MemoryCandidate; - one
memory_mutationsledger row; - one
pipeline_runsrow plus append-only events; - ordered SQL, vector and graph outbox lanes.
The candidate explicitly carries tenant, workspace, dataset, document,
revision, chunk, source reference, evidence span, agent/session, pipeline,
extraction and embedding versions. Authorization fields are never read from
free-form metadata.
An authenticated dataset writer may additionally supply the typed
legal_identity object (required law_id and article_id; optional
statute_code and version/paragraph/clause identifiers). MESA validates it against the bounded
legal alias ontology, binds it to the immutable mutation provenance and stores
it on canonical assertions. Reserved _mesa_* keys in caller metadata are
always discarded, so free-form metadata cannot opt into legal ranking. The
producer remains responsible for verifying that the declared identity belongs
to the source; MESA_Data must not populate it from a bare article heading.
Validation is an explicit policy gate configured via MESA_TIER3_MODE (0 = deterministic-only, 1 = single LLM validator, 2 = dual-LLM consensus). A rejected mutation may keep its raw
audit source, but it cannot create an active SQL, vector, entity, edge or
assertion artifact. In Mode 0, deterministic admission checks gate progression with zero LLM validator dependencies.
Pipeline and failure model#
The legal success path is:
QUEUED → RUNNING → EXTRACTED → VALIDATED → PROJECTING → COMMITTEDFailure/recovery states are RETRY_PENDING, DLQ, ROLLING_BACK,
ROLLED_BACK and BLOCKED. State changes use compare-and-swap and append an
event. Claims use a random fencing token, bounded lease, heartbeat and bounded
retry. A worker that loses its token cannot finalize the claim.
V4 retries restart extraction from RETRY_PENDING, skip model calls after
validation, and atomically mark the dispatch and any not-yet-validated mutation
DEAD_LETTER and the raw log failed after five unsuccessful attempts,
preserving final states.
Failed dispatch and projection attempts back off for 60, 120, 240 and 480 seconds,
capped at 900 seconds, before becoming eligible for another claim.
At startup, the combined runtime holds the storage writer lock and returns
abandoned processing raw logs to DEFERRED, clearing their claims.
Operator replay also resets an extraction dead letter's dispatch attempts and
raw-log claim, making it eligible to run immediately.
These are pipeline states; client-facing mutation states are listed in
concepts and ingestion. COMMITTED is the success
state; receiving an admission response alone does not establish completion.
Projection prerequisites are enforced in the database:
SQL → vector → graphThe projector only claims a later lane when its predecessor is complete.
Every lane is idempotent by mutation and physical artifact identity.
Legal citation retrieval selects trusted typed identities through the scoped
compound assertion index before lane ranking. Tenant, dataset, agent,
jurisdiction, validity and committed-state predicates are applied in that SQL
selection. Typed identity refines candidate recall and lane order; it does not
bypass deterministic weighted RRF or replace canonical evidence text.
Artifact ownership and rollback#
artifact_registry represents physical SQL, vector and graph artifacts.
artifact_sources allows several mutations/source chunks to own the same
entity or vector.
Rollback removes only the requesting run's source ownership. A shared artifact
remains active while another source owns it. The last-owner path writes the
SQLite retrieval tombstone first and schedules idempotent vector/graph cleanup.
An unfinished cleanup leaves the run BLOCKED, never falsely committed.
A new document revision does not erase history. Prior assertions become
SUPERSEDED. A legal purge retracts the requested source and removes only
artifacts that have no remaining active owner.
Reconciliation and health#
Reconciliation is bidirectional:
- ledger artifact missing from a store: requeue the missing lane;
- physical artifact without registry/ownership: quarantine it in the registry
and enqueue cleanup.
Authenticated /health returns the aggregate fields
status, maintenance and rebuild_state. /health/init reports readiness
and fails with HTTP 503 when required stores, workers or backlog checks fail.
Projection backlog, dead letter, stuck leases, cleanup backlog, blocked cleanup,
ownerless registry rows and shared-owner counts are aggregate gauges at /metrics;
they are not individual fields in the public /health response.
Retrieval#
All SQL, vector and graph lanes receive the authorized dataset filter before
ranking. V4 combines rank positions using real reciprocal rank fusion. Jurisdiction and validity constrain eligibility before ranking; assertion
confidence affects assertion-lane ordering. The bounded post-RRF legal factor
uses query statute/article identity. See retrieval. Candidate assertion identity is the deterministic final tie-break.
Responses include document, revision, chunk, evidence, assertion, pipeline and
embedding provenance. PageRank is not a retrieval eligibility signal.
Bounded adaptive context retrieval#
The raw V4 search endpoint remains the deterministic single-query primitive.
The context endpoint and mesa_get_context default to retrieval_mode=single.
Callers may explicitly select adaptive, which runs the original query first
and invokes one semantic reformulation call only when the result is empty or
none of its top three candidates has support from at least two retrieval lanes.
The planner receives only the normalized user query and may return at most two
validated query strings. It cannot change tenant, agent, dataset,
jurisdiction, temporal or graph scope. New statute/article citations are
rejected. Every expansion reuses MemoryDAO.search_v4_memory; query rankings
are fused with deterministic RRF weights Q0=1.0, Q1=0.5 and Q2=0.5, with a
Q0-preserving tie break. Inner lane scores and provenance remain unchanged,
and outer query_origins plus query_fusion_score remain separately auditable.
An adaptive request performs at most one planner call and three total
retrievals. Planner or optional expansion failures preserve the original
retrieval. The deduplicated final candidate list enters the existing
ContextBuilder once, so the 2048-token default and atomic proof packing remain
the sole model-visible budget authority. This feature performs no answer
generation, recursive planning, decomposition or reranking.
Migration and release#
V3 and V4 use separate storage roots. Moving V3 content into V4 requires an
explicit source/catalog mapping and ingestion through the V4 contract; there
is no general automatic V3-to-V4 converter documented here. The offline
projection runner rebuilds derived stores from an existing V4 canonical ledger,
not a new ledger from raw V3 data.
For V4 upgrades use migration,
backup/restore and the
projection rebuild procedure. Keep the prior root and
matching code/config for rollback.
Production remains NO-GO until migration/restore, v4 Compose, crash,
saturation, cross-tenant concurrency, benchmark and 24-hour soak gates all
pass with production-like storage and the selected model provider.
Canonical V4 insertion requires the combined processing consumer. With models
off, it stores records with a SKIPPED VECTOR lane and finds them through BM25.
Enabling embedding later backfills skipped lanes without hiding committed records.
