review/README.md

# Independent private review

This folder contains reviewer-authored, synthetic evaluation and adversarial cases for the local issue-export prototype. It is not a public submission or an eligibility opinion.

## Frozen protocol

- Fixtures and the simple keyword baseline are committed to SHA-256 values in `fixture_manifest.json` before the first prototype prediction. The builder was told the generic API/schema and test categories, but not the fixture texts or expected labels.
- Freeze and record model identity/revision, reference examples, thresholds and executable source before scoring. Hold-out results must not tune reference examples, thresholds or model choice. Functional safety fixes can be retested; retain the original findings and identify the retest. If performance is tuned after seeing results, a new independently authored holdout is required for a clean generalization claim.
- Classification: report all 40 cases, all-label accuracy, confusion matrix including `unsure`/rejection, per-label scores, coverage/abstention, and accuracy among answered cases. Report the same fixed keyword baseline on the same text. `Unsure` is correct for the eight deliberately unsupported/ambiguous/mixed/empty cases, not an excuse to drop hard examples. Rejected input must be reported separately from model abstention.
- Duplicate retrieval: eight labeled duplicate pairs (16 positive anchors), plus 12 same-topic/nonduplicate singletons. Report nearest-candidate top-1 retrieval when exposed, thresholded unique-pair precision/recall, false-positive pairs, and missed pairs. A high cosine similarity is not proof of duplication. Do not count self-matches. Distinguish prior-only suggestions from bidirectional retrieval.
- Timing: report actual process wall time and application-reported model-load/batch times separately. A new process can still benefit from operating-system filesystem caches; do not call it a fully cold machine measurement. Poll resident set size on this process if available, report the sampling interval, and do not confuse process peak RSS with total machine memory.
- Offline: run the actual CLI in a subprocess with networking denied by a reviewer preload. Confirm both success and zero blocked network attempts after assets are present. This establishes process-level blocked-network inference in the observed environment, not a general operating-system air-gap guarantee.
- Inputs: test malformed JSON, nonfinite values, null/wrong field types, zero items, empty fields, duplicate/mixed IDs, size/item limits, script/attribute payloads, formula-leading CSV cells, Unicode/control characters and untrusted extra fields. Rejections should be deliberate and legible; no crash stack trace or unbounded work in normal UI use.
- Browser: upload HTML/script payloads and verify rendered text is inert; export JSON/CSV and inspect bytes. Observe requests during inference. Original issue prose must not become executable markup, executable spreadsheet formula cells, or instructions for external requests.
- Privacy: local page/API, no external inference/analytics, no GitHub writes, no import of user account data. Check real code and actual browser/CLI behavior. Do not infer this solely from README promises.
- Provenance: verify Mastra executes substantial workflow stages and local model weights actually run. If fallback is used, describe TF-IDF plus a fitted classifier accurately; never call it an LLM/open-weight language model or claim Mastra participation without execution.

## Claim limits

Synthetic accuracy is a small diagnostic, not proof of real maintainer performance. `Ready` is a filtering suggestion, not a correctness guarantee. Report review counts and observed timings. Do not invent human time saved, prizes, organizer acceptance, firsthand human experience, or production readiness. Imported existing labels are untrusted metadata, not evaluation truth.

All publication, account, contest-entry and financial/legal steps remain outside this local test work.