Pipeline

The pipeline separates topic selection, prose generation, verification, constrained repair, and the only write gate. Agent stages emit JSON into staging/; only scripts/accept.py may append to the public corpus.

flowchart LR
  M["topic-miner"] --> R{"category routing"}
  R -->|standard| C["crafter"]
  R -->|MME category| MC["mme-crafter"]
  C --> V["verifier"]
  MC --> V
  V -->|accept| A["accept.py"]
  V -->|repair| P["repairer (one pass)"]
  P --> RV["verifier (re-verify)"]
  RV -->|accept| A
  V -->|reject or abstain| S["stop candidate"]
  RV -->|anything except accept| S
  A --> D["data JSONL + meta/verification.jsonl"]

Routing and stages

topic-miner produces a batch of exactly five topics. The category then selects its crafting prompt:

• visual-arts, music, film-tv, and sports-games use mme-crafter.
• The other eight categories use the standard crafter.

MME changes the crafting route, not the verification standard. Every draft goes to an independent verifier that attempts to disprove it, then checks the exact claim and unique answer against two live, unrelated authoritative sources.

The verifier can return four statuses:

Status	Consequence
accept	The draft and matching verification report may proceed to accept.py.
repair	One repairer pass may alter only allowed fields from q, w, and x; the result is independently re-verified.
reject	The candidate stops. A repairer may not rescue its premise or answer.
abstain	The candidate stops because the evidence is insufficient for a reliable judgment.

Only an accept report can cross the write gate. accept.py checks run metadata, topic identity, destination bucket, card rules, live source evidence, and duplication before appending the card to data/<cat>/<diff>.jsonl and evidence to meta/verification.jsonl.

Stable failure codes

Code	Meaning
R_SCHEMA	Malformed or incomplete data, invalid run metadata, mismatched topic or bucket, invalid identity, or a non-accept verification report at the write gate.
R_CHOICES	The four choices are not distinct or a normalized pair has Levenshtein distance 2 or less.
R_ANSWER_LEAK	The normalized answer occurs in the normalized question stem.
R_EXPLAINER_LENGTH	The explainer is empty or exceeds 20 words.
R_UNSTABLE	The question or explainer uses rejected time-sensitive language.
R_UNPROVABLE	There are fewer than two usable sources or a source URL is missing.
R_SRC_LAUNDER	The two evidence sources resolve to the same normalized domain.
R_SRC_DEAD	An evidence URL fails its live HTTP check.
R_DUP_EXACT	The normalized-question fingerprint duplicates an accepted card.
R_DUP_NEAR	A same-category question reaches Jaccard similarity 0.85 or higher against an accepted question.

Failure codes are stable machine tokens used in validator output, rejection records, and tests.

Locks and stopping rules

All stateful writes acquire locks in one mandatory order:

1. global meta/.state.lock;
2. per-bucket data/<cat>/<diff>.jsonl.lock.

Both locks use POSIX fcntl.flock. Acquiring them in the opposite order risks deadlock and violates the write contract.

At most four independent buckets run concurrently. A bucket trips its circuit breaker after 20 consecutive failed or abstained candidates; work stops rather than repeatedly lowering the evidence bar.