MESA Core · v0.9.3

Ingestion & memory lifecycle

v0.9.3 source: docs/concepts-and-ingestion.md

This guide has equivalent English and Turkish versions. The source sections behind this page are at the bottom.

The source hierarchy#

A tenant contains workspaces and datasets. A dataset owns documents, immutable revisions and exact source chunks. A server-created session binds the principal, tenant, workspace, agent and an immutable authorized dataset set. Caller metadata never grants scope.

Admission is not completion#

POST /v4/memory/insert requires session and source identities, title, source_ref and exact content. A 202 returns mutation, candidate, pipeline and raw-log IDs. Poll mutation state; wait_until_committed can also return a failed terminal state. Rejection must not create active retrieval artifacts.

Revision boundaries and retries#

For a multi-chunk revision, use distinct IDs/ordinals and finalize_revision=false until the last insert. The final insert freezes membership; activation waits for every required child. Retry with the same body idempotency key and exact payload. Conflicting retries are rejected. Correction creates a new revision; purge and rollback remove source ownership through explicit permissions.

Whole documents and purge#

POST /v4/documents handles exact-text chunking, duplicate detection and revision IDs on the server. It returns per-chunk mutation IDs; admission remains distinct from COMMITTED. Models-off operation stores text and supports BM25; optional vector and graph lanes can be SKIPPED and later backfilled. Document purge removes owned source/extracted text while retaining audit identifiers and hashes; shared text stays while another document owns it.

Article-aware chunking and retries#

MESA_CHUNKING=auto|plain defaults to auto. At least three Turkish article headings at line starts trigger article-boundary splitting; long pieces remain bounded to 512 tokens and preserve exact source text. plain keeps the previous plan for new revisions. Existing ACTIVE revisions do not change. A retry preserves the plan matching the admitted text, even after restart or a setting change. If the admitted prefix matches two different plans, admission fails closed with HTTP 409 without appending chunks.

Queue capacity and document deadlines#

Document admission waits briefly for room for the current chunk on queue_over_capacity, retaining the complete chunk plan within that request. Persistent pressure still returns 503 queue_over_capacity; retry the exact payload to resume stored ordinals. add_document(deadline=600) uses the remaining document deadline for the HTTP request. Admission and full readiness remain separate; large uploads can take minutes.

Legal rules default on for newly admitted chunks; model-based SPO extraction defaults off, even with models enabled. Legal items retain original chunk offsets and remain separate from SPO facts. Existing identities are immutable on resume.

Diagrams

UML · Sequence diagramWrite path: accept now, process in the background, poll for the result
«actor»Your app«boundary»API«entity»SQLite ledger«control»Worker«external»Model«entity»LanceDB · Kùzu insert(source chunk) legal items + mutation + run mutation_id 202 · mutation_id 202 = accepted, not processed.Processing continues below. claim run (lease) source chunk SPO only if enabled 0..N facts validate (mode 0–2) accept / reject alt[accepted] facts · SQL_APPLIED VECTOR_APPLIED / SKIPPED GRAPH_APPLIED / SKIPPED state = COMMITTED [rejected] state = REJECTED loop[every 2 s until a terminal state] GET /v4/mutations/{id} read state state final state

Admission snapshots legal items separately from SPO facts. Legal rules default on; model-based SPO extraction defaults off. Each enabled store write follows the previous one; retries reuse idempotent IDs. Mutation polling: 2 s; revision activation: 1 s.

UML · State machineOne mutation from QUEUED to a terminal state
QUEUEDRUNNINGEXTRACTEDVALIDATEDPROJECTINGCOMMITTEDDLQRETRY_PENDINGREJECTEDBLOCKEDROLLED_BACK worker claimsfacts or fallbackchecks passSQL writtenlanes complete transient error (any step) [validation rejects] retry retries exhausted replay rollback[cleanup done] rollback [cleanup unfinished] REJECTED keeps the audit source but creates no search data.

COMMITTED means required lanes completed; disabled vector/graph lanes may be SKIPPED. Retries are bounded; exhausted work parks in the dead-letter queue (DLQ) until someone replays it. A failure stays visible instead of being hidden by compensation.

All diagrams with their design rationale: Architecture →

Source reference for this page

The sections of the upstream file that back this page, unchanged. Source section links open the appendix automatically.

Identity and scope#

ConceptImplemented meaning
PrincipalServer-authenticated identity owning keys and ACL grants
TenantV4 authorization boundary
WorkspaceTenant-owned grouping of datasets
DatasetSource collection inside a workspace; session-bound access
AgentCompute/persona context; not a tenant credential
SessionServer-generated principal/scope binding with immutable dataset set
DocumentDataset-owned source identity with a current revision head
RevisionImmutable source version with a frozen chunk manifest
Source chunkExact source content, ordinal, source reference and content hash
AssertionCanonical subject/predicate/entity-or-literal fact with source evidence
MutationDurable admitted work and its state, separate from pipeline aggregate
ProjectionDerived SQL/vector/graph artifact work with fenced claim and receipt
EvidenceLinked assertion/chunk/source span; not independently verified truth

Catalog public IDs resolve inside the tenant. Physical storage keys are opaque;

passing a physical key as an external ID does not grant access. Agent and dataset

authorization are resolved from the session, never caller metadata.

Admission and completion#

POST /v4/memory/insert requires session, dataset, document, revision, chunk,

title, source_ref and exact content. See API for bounds.

Admission returns 202 with candidate, mutation, pipeline-run and raw-log IDs.

This is durable admission, not visibility in every store.

mermaid
stateDiagram-v2
    [*] --> RECEIVED
    RECEIVED --> EXTRACTED
    EXTRACTED --> VALIDATED
    EXTRACTED --> REJECTED
    VALIDATED --> SQL_APPLIED
    SQL_APPLIED --> VECTOR_APPLIED
    VECTOR_APPLIED --> GRAPH_APPLIED
    GRAPH_APPLIED --> COMMITTED

The pipeline aggregate uses `QUEUED → RUNNING → EXTRACTED → VALIDATED →

PROJECTING → COMMITTED`. Technical recovery states include retry, dead-letter,

rollback and blocked cleanup. The SDK terminal set is COMMITTED, REJECTED,

DEAD_LETTER, ROLLED_BACK, BLOCKED; check status, not just method name.

Internal outbox/pipeline records may use DLQ; inspect each object's state.

Canonical FactExtractionService preserves zero-to-many extracted facts.

Validation mode 0 checks deterministically, mode 1 uses validator A, and mode 2

requires distinct A/B provider-model pairs. The runtime-selected policy is

snapshotted into admitted work; reserved _mesa_* metadata cannot downgrade it.

Rejected work may retain audit source but creates no active retrieval artifact.

Projection order is SQL → vector → graph. Separate stores do not share an ACID

transaction: fenced leases, receipts, ownership and reconciliation provide

idempotent convergence. A lost lease cannot finalize another owner's work.

Backpressure yields explicit admission errors; do not retry without bounds.

Revision finalization and retries#

For a multi-chunk revision set finalize_revision=false on all but the final

insert, with distinct chunk IDs and ordinals. The last insert freezes the

manifest. No new chunks may then enter that revision; the revision activates

only after every required child mutation succeeds. One document has one active

head. Create a new revision to change source content.

Use an insert idempotency_key (body field, <=128 characters) with the exact

same payload on retries. Conflicting payloads are rejected. Rebuild uses the

separate Idempotency-Key HTTP header. The SDK avoids unsafe automatic POST

retry after ambiguous failures; application recovery must inspect durable IDs.

declared_content_hash is the caller's whole-revision hash, manifest_hash

identifies frozen chunk membership, and a chunk content_hash identifies only

that chunk. Direct insertion does not establish a whole-revision hash.

Correction, purge and rollback#

Supersession retains historical assertions. A time-qualified query can include

eligible superseded evidence, which requires explicit producer validity dates (valid_from / valid_to); without them, superseded evidence is not returned. An unqualified query uses active evidence.

Rollback removes source ownership; shared artifacts remain while another active

source owns them. Tombstones precede secondary cleanup; failure stays BLOCKED.

Purge requires dataset OWNER plus explicit PURGE; replay/rollback require

the originating scope and explicit ROLLBACK. Semantic rejection cannot be

reopened through generic replay (409 NON_REPLAYABLE). Non-head historical

rollback is a conflict. Do not edit store rows or leases manually.

Producer contract#

Core consumes exact content and identity; MESA_Data owns source

acquisition, chunking and verified source metadata. HTTP insertion supports a

typed legal_identity; the current SDK insert signature does not expose it.

Use the HTTP contract for that field. Bare “MADDE 1” text is insufficient to

prove law identity. Producer metadata accuracy remains an external responsibility.

Canonical V4 insertion requires the combined processing consumer. A combined

runtime without models stores documents and finds them with BM25; vector search

starts when embedding is enabled (degraded_sources shows vector until then);

503 canonical_processing_unavailable happens only on a non-combined profile.

Document ingestion briefly waits for queue capacity for the current chunk,

keeping its chunk plan within the request. Persistent pressure still returns

503 queue_over_capacity; retry the exact payload to resume stored ordinals.

SDK add_document(deadline=600) uses the remaining document budget for this

HTTP request, allowing a large upload to wait for capacity.