Skip to main content
Use this workflow when Eval Labs reveals that Lucia is routing prompts into the wrong behavior mode.

​
What an intent-layer failure looks like

An intent-layer failure happens when Lucia understands the words too shallowly and chooses the wrong mode. Example:
The user is not asking for a capability menu. The user is expressing disorientation.

​
Step 1 — Create a focused suite

Use 5–10 related prompts. Example suite:

​
Step 2 — Confirm the Eval Labs endpoint

Eval Labs runs against the Lucia endpoint configured in its environment. As verified on 2026-08-12, the active operator-facing target is:
Staging should be used only when the validation pass has explicitly been configured or promoted for staging validation. Do not let an example env file override the intended validation target for the pass. Eval Labs Production at https://evaluationlabs.ai is the evaluation app, not the evaluated Engine environment. Its current source identity is commit 6cbfd15f75330c414bfe79e0d2fab51ef8115102, verified through same-deploy runtime identity evidence.
Engine Development owns the direct model call and currently defaults to gpt-5.6-sol, with explicit override precedence. Eval Labs has delegated model ownership: it sends the evaluation request and reads same-response provenance. Execution claims require same-response provenance; source configuration alone is not proof. The Engine model gateway uses the OpenAI Responses API.

​
Step 3 — Review mode accuracy

For each response, ask:
If the response gives a generic redirect to a distress signal, fail it.

​
Step 4 — Patch the owning Engine seam

Inspect current Engine source before choosing a file. Intent/routing behavior and output refinement are Engine-owned; Eval Labs owns the test, evidence, and review workflow. Do not patch wording if the mode is wrong. Do not change Eval Labs heuristics to make an Engine behavior failure appear fixed.

​
Step 5 — Validate the current bank, then deploy deliberately

Local-only changes are not enough for Eval Labs. The current Engine bank contains 217 cases. On 2026-08-12, an exact-SHA non-strict diagnostic against Engine commit 9a15b1da199923d1ecb0927be2507feda2cb4c71 returned 76/217. That was a diagnostic observation, not a pass, release gate, or current quality baseline. The Engine behavior must be deployed to the intentionally configured Development target before production Eval Labs can test that build. If the pass is intentionally configured for staging validation, deploy or promote to staging as part of that pass and record that choice in the review thread.

​
Step 6 — Re-run the exact same suite

Do not change prompts yet. Run the same suite again. Compare:
  • which prompts now route correctly
  • which still fail
  • whether any neighboring behavior regressed
  • whether tone improved or became too mushy

​
Step 7 — Export evidence

Persist the run and reviews, then export the evidence. Record the Eval Labs release identity, Engine SHA, endpoint/environment, suite identity and case count, strictness, result, and per-run runtime provenance. Share the export with the engineering/product thread.

​
Success standard

The patch is successful only if the repeated suite shows improved behavior without creating new failures. Configured model names, historical scores, or a non-strict diagnostic are not sufficient. A release claim needs the required strict gate plus the exact target/build and per-request evidence.

​
Historical checkpoints

  • The 2026-05-29 178/178 strict-brain checkpoint belongs to an older 178-case bank and revision. It is useful history, not the current 217-case baseline.
  • The 2026-07-10 owner runtime verification observed gpt-5.6-sol configured and provider-resolved for one Development request. This workflow did not make a new model-bearing verification request on 2026-08-12.