Hermes 4 405B
Strong integration
selectivity passprovenance stronguncertainty strong
Retrieved, incorporated, attributed, and preserved boundaries. The main weakness was simplifying the instruction hierarchy.
VS-Bench · freshness + governance · August 2026
Did the model merely read the patch—or did it use it?
VS-Bench now separates retrieval, model adherence, freshness lift, evidence realization, provenance, uncertainty, expiry behavior, and update-gap handling. The new governance layer grew out of cross-model manual runs where different models failed at different points between receiving current evidence and actually changing their answers.
4Hosted stale/strong models compared in governance runs
7Executable governance dimensions added
11/11Retrieval checks stayed stable in local A/B runs
6/8Local generation adherence remained separately measurable
These are development diagnostics, not a statistically powered model leaderboard. The important result is architectural: correct retrieval can coexist with failed utilization, decorative citations, temporal confabulation, or good abstention.
New in VS-Bench
Seven governance dimensions are now executable.
Evidence realizationDid retrieved evidence change the answer?
Distinguishes “patch acknowledged” from observable use of relevant claim content.
Provenance entailmentDoes the cited claim support the sentence?
A claim ID does not count merely because it appears next to an unrelated conclusion.
Provenance type accuracyDid the model cite the right kind of identifier?
Catches cases where a patch/signature ID is presented as though it were an atomic claim ID.
Uncertainty retentionDid needs-review stay needs-review?
Missing versions, truncated scope, unclear policy terms, and unestablished compatibility should not become confident facts.
False temporal attributionDid old knowledge get dressed up as “recent”?
Flags freshness-shaped prose such as GPT-3 or long-established prompting practices being narrated as new developments.
Expiry awarenessHistorical is not current.
Expired claims may remain useful as dated context without being presented as active guidance.
No resurrectionReplacement expiry does not revive the old claim.
If a newer replacement expires, VS-Bench expects an update gap rather than silent rollback.
Held back intentionallyInstruction hierarchy is not auto-scored yet.
The loading prompt itself may prime models to treat patch facts as an instruction layer. That test is being redesigned before automation.
Cross-model governance evidence
Same patch. Four very different failure profiles.
The manual protocol tested baseline knowledge, explicit patch injection, patched retesting, and adversarial retention. The key question was not “can the model repeat the patch?” but whether the update becomes behaviorally active.
o3
Freshness lift with overreach
freshness passgrounding partialprovenance partial
Used Maker claims concretely, but sometimes generalized product changes into broader best practices or capabilities the patch did not establish.
Gemma 3 12B
Evidence recognized, incompletely realized
selectivity strongrealization partialentailment partial
Could cite the patch and explain governance rules while leaving the actual answer mostly unchanged. This exposed the gap between governance comprehension and governance execution.
MythoMax 13B
Patch retained, not behaviorally activated
Phase 3 utilization failPhase 4 retrieval returnsprovenance type error
Demonstrated patch comprehension immediately after injection, reverted to stale answers during normal retesting, then recovered the patch when explicitly asked about it again.
Patch persistence / activation
MythoMax gave the cleanest counterexample.
Recent AI developmentsAccessible knowledge is not necessarily active knowledge.
Patched retest
Still described GPT-3 and transfer learning as recent advances, essentially reproducing its stale baseline despite having just summarized the Maker patch correctly.
Ingest → understand → retain → failed spontaneous utilization.
Direct patch cue afterward
Correctly recovered Ollama dark-mode/system-appearance information, OpenDesign boundaries, Cherry Studio uncertainty, expiry behavior, and no-resurrection logic.
Explicit cue → retrieve patch again.
Local A/B foundation
Retrieval stayed at 11/11 while downstream model adherence remained separately measurable at 6/8.
Dolphin 3:8B and Qwen3 4B Instruct remain the local reproducibility proof. The governance work extends that same separation principle from retrieval-vs-generation into patch-acquisition-vs-behavior.
| Model | Path | Retrieval | Generation | Characteristic miss |
|---|---|---|---|---|
| Dolphin 3:8B | Local Ollama RAG + VS-Bench | 11/11 | 6/8 | Missed explicit outside-context labeling and a retrieved smell guardrail. |
| Qwen3 4B Instruct | Local Ollama RAG + VS-Bench | 11/11 | 6/8 corrected | Invented freezer wording in a multi-intent answer and missed the smell guardrail. |
Regression cases still preserved
The old benchmark cases still matter.
Five-day chickenApply current storage limits.
Checks that a five-day refrigerated chicken case is rejected against the current cooked-leftover rule.
Maker EditionFacts without invented prescriptions.
Tests whether current Maker claims are used without pretending the publication supplied developer recommendations it did not provide.
Historical update gapNo silent rollback.
When revision two replaced revision one and later expired, revision one must not quietly become current again.
Who won the 1998 World Cup?Negative capability.
The correct Velvet behavior is no-current-context; a model may answer from general knowledge only if it keeps that source boundary explicit: “Outside current Velvet Signal context.”
VS-Bench also caught a real retrieval bug here: zero-score lexical results once filled the result limit and made unrelated Pantry claims look grounded. The regression test now preserves
no-current-context.Run it
Retrieval bench and governance bench stay separate.
Use bench:save to preserve local A/B artifacts. Use the governance scorer on captured baseline/patched runs to inspect evidence realization and governance behavior.
npm run bench npm run bench:save -- --model dolphin3:8b npm run bench:save -- --model qwen3:4b-instruct npm run bench:governance -- --input run.json npm run test