MESA Core · v0.9.3
Ingestion & memory lifecycle
This guide has equivalent English and Turkish versions. The source sections behind this page are at the bottom.
The source hierarchy#
A tenant contains workspaces and datasets. A dataset owns documents, immutable revisions and exact source chunks. A server-created session binds the principal, tenant, workspace, agent and an immutable authorized dataset set. Caller metadata never grants scope.
Admission is not completion#
POST /v4/memory/insert requires session and source identities, title, source_ref and exact content. A 202 returns mutation, candidate, pipeline and raw-log IDs. Poll mutation state; wait_until_committed can also return a failed terminal state. Rejection must not create active retrieval artifacts.
Revision boundaries and retries#
For a multi-chunk revision, use distinct IDs/ordinals and finalize_revision=false until the last insert. The final insert freezes membership; activation waits for every required child. Retry with the same body idempotency key and exact payload. Conflicting retries are rejected. Correction creates a new revision; purge and rollback remove source ownership through explicit permissions.
Whole documents and purge#
POST /v4/documents handles exact-text chunking, duplicate detection and revision IDs on the server. It returns per-chunk mutation IDs; admission remains distinct from COMMITTED. Models-off operation stores text and supports BM25; optional vector and graph lanes can be SKIPPED and later backfilled. Document purge removes owned source/extracted text while retaining audit identifiers and hashes; shared text stays while another document owns it.
Article-aware chunking and retries#
MESA_CHUNKING=auto|plain defaults to auto. At least three Turkish article headings at line starts trigger article-boundary splitting; long pieces remain bounded to 512 tokens and preserve exact source text. plain keeps the previous plan for new revisions. Existing ACTIVE revisions do not change. A retry preserves the plan matching the admitted text, even after restart or a setting change. If the admitted prefix matches two different plans, admission fails closed with HTTP 409 without appending chunks.
Queue capacity and document deadlines#
Document admission waits briefly for room for the current chunk on queue_over_capacity, retaining the complete chunk plan within that request. Persistent pressure still returns 503 queue_over_capacity; retry the exact payload to resume stored ordinals. add_document(deadline=600) uses the remaining document deadline for the HTTP request. Admission and full readiness remain separate; large uploads can take minutes.
Legal rules default on for newly admitted chunks; model-based SPO extraction defaults off, even with models enabled. Legal items retain original chunk offsets and remain separate from SPO facts. Existing identities are immutable on resume.
Diagrams
Admission snapshots legal items separately from SPO facts. Legal rules default on; model-based SPO extraction defaults off. Each enabled store write follows the previous one; retries reuse idempotent IDs. Mutation polling: 2 s; revision activation: 1 s.
COMMITTED means required lanes completed; disabled vector/graph lanes may be SKIPPED. Retries are bounded; exhausted work parks in the dead-letter queue (DLQ) until someone replays it. A failure stays visible instead of being hidden by compensation.
All diagrams with their design rationale: Architecture →
Source reference for this page
The sections of the upstream file that back this page, unchanged. Source section links open the appendix automatically.
Identity and scope#
| Concept | Implemented meaning |
|---|---|
| Principal | Server-authenticated identity owning keys and ACL grants |
| Tenant | V4 authorization boundary |
| Workspace | Tenant-owned grouping of datasets |
| Dataset | Source collection inside a workspace; session-bound access |
| Agent | Compute/persona context; not a tenant credential |
| Session | Server-generated principal/scope binding with immutable dataset set |
| Document | Dataset-owned source identity with a current revision head |
| Revision | Immutable source version with a frozen chunk manifest |
| Source chunk | Exact source content, ordinal, source reference and content hash |
| Assertion | Canonical subject/predicate/entity-or-literal fact with source evidence |
| Mutation | Durable admitted work and its state, separate from pipeline aggregate |
| Projection | Derived SQL/vector/graph artifact work with fenced claim and receipt |
| Evidence | Linked assertion/chunk/source span; not independently verified truth |
Catalog public IDs resolve inside the tenant. Physical storage keys are opaque;
passing a physical key as an external ID does not grant access. Agent and dataset
authorization are resolved from the session, never caller metadata.
Admission and completion#
POST /v4/memory/insert requires session, dataset, document, revision, chunk,
title, source_ref and exact content. See API for bounds.
Admission returns 202 with candidate, mutation, pipeline-run and raw-log IDs.
This is durable admission, not visibility in every store.
stateDiagram-v2
[*] --> RECEIVED
RECEIVED --> EXTRACTED
EXTRACTED --> VALIDATED
EXTRACTED --> REJECTED
VALIDATED --> SQL_APPLIED
SQL_APPLIED --> VECTOR_APPLIED
VECTOR_APPLIED --> GRAPH_APPLIED
GRAPH_APPLIED --> COMMITTEDThe pipeline aggregate uses `QUEUED → RUNNING → EXTRACTED → VALIDATED →
PROJECTING → COMMITTED`. Technical recovery states include retry, dead-letter,
rollback and blocked cleanup. The SDK terminal set is COMMITTED, REJECTED,
DEAD_LETTER, ROLLED_BACK, BLOCKED; check status, not just method name.
Internal outbox/pipeline records may use DLQ; inspect each object's state.
Canonical FactExtractionService preserves zero-to-many extracted facts.
Validation mode 0 checks deterministically, mode 1 uses validator A, and mode 2
requires distinct A/B provider-model pairs. The runtime-selected policy is
snapshotted into admitted work; reserved _mesa_* metadata cannot downgrade it.
Rejected work may retain audit source but creates no active retrieval artifact.
Projection order is SQL → vector → graph. Separate stores do not share an ACID
transaction: fenced leases, receipts, ownership and reconciliation provide
idempotent convergence. A lost lease cannot finalize another owner's work.
Backpressure yields explicit admission errors; do not retry without bounds.
Revision finalization and retries#
For a multi-chunk revision set finalize_revision=false on all but the final
insert, with distinct chunk IDs and ordinals. The last insert freezes the
manifest. No new chunks may then enter that revision; the revision activates
only after every required child mutation succeeds. One document has one active
head. Create a new revision to change source content.
Use an insert idempotency_key (body field, <=128 characters) with the exact
same payload on retries. Conflicting payloads are rejected. Rebuild uses the
separate Idempotency-Key HTTP header. The SDK avoids unsafe automatic POST
retry after ambiguous failures; application recovery must inspect durable IDs.
declared_content_hash is the caller's whole-revision hash, manifest_hash
identifies frozen chunk membership, and a chunk content_hash identifies only
that chunk. Direct insertion does not establish a whole-revision hash.
Correction, purge and rollback#
Supersession retains historical assertions. A time-qualified query can include
eligible superseded evidence, which requires explicit producer validity dates (valid_from / valid_to); without them, superseded evidence is not returned. An unqualified query uses active evidence.
Rollback removes source ownership; shared artifacts remain while another active
source owns them. Tombstones precede secondary cleanup; failure stays BLOCKED.
Purge requires dataset OWNER plus explicit PURGE; replay/rollback require
the originating scope and explicit ROLLBACK. Semantic rejection cannot be
reopened through generic replay (409 NON_REPLAYABLE). Non-head historical
rollback is a conflict. Do not edit store rows or leases manually.
Producer contract#
Core consumes exact content and identity; MESA_Data owns source
acquisition, chunking and verified source metadata. HTTP insertion supports a
typed legal_identity; the current SDK insert signature does not expose it.
Use the HTTP contract for that field. Bare “MADDE 1” text is insufficient to
prove law identity. Producer metadata accuracy remains an external responsibility.
Canonical V4 insertion requires the combined processing consumer. A combined
runtime without models stores documents and finds them with BM25; vector search
starts when embedding is enabled (degraded_sources shows vector until then);
503 canonical_processing_unavailable happens only on a non-combined profile.
Document ingestion briefly waits for queue capacity for the current chunk,
keeping its chunk plan within the request. Persistent pressure still returns
503 queue_over_capacity; retry the exact payload to resume stored ordinals.
SDK add_document(deadline=600) uses the remaining document budget for this
HTTP request, allowing a large upload to wait for capacity.
