Skip to main content
Eval Labs is the dedicated platform for testing Lucia’s responses, capturing review judgments, and hardening behavior over time.

​
Product Role

Eval Labs is a first-class product surface. EvaluationLabs.ai is Lucia’s proprietary evaluation platform for shaping her human intent layer, emotional awareness, psychological understanding, natural language interpretation, warmth, empathy, judgment, and operational intelligence. It supports:
Verified 2026-08-12, re-read 2026-09-10: Eval Labs Production at https://evaluationlabs.ai serves commit 6cbfd15f75330c414bfe79e0d2fab51ef8115102, version 0.1.0, with same-deploy runtime identity evidence. There has been no Eval Labs release since 2026-07-29 (HLLC-41 audit read of the production deploy). The evaluated Engine target on 2026-09-10 is Development at e2166b8.

​
Why It Matters

Lucia’s intelligence cannot be trusted by vibes. It needs a system that can answer:

​
Review Philosophy

Eval Labs should not reward polished nonsense. It should reward:

​
Current Relationship to Lucia

Eval Labs currently tests operator-facing Lucia through Engine Development, Guest verification through Guest Agent Production, and controlled Fieldwork behavior through Development-only relays. Implemented product surfaces include:
The validation focus recorded on 2026-08-12 included Focus Ops behavior, semantic conversational intent assist, scoped entity routing, Calendar/booking-spine awareness, Workspace OS context awareness, Signal Stream active_context, prior recommendation and offer context, saved DAW workflow truth-state, the Signal → Action → Save → Reminder loop, route correctness, gpt-5.6-sol runtime evidence, and Lucia JSON gateway behavior through the OpenAI Responses API. Since 2026-09-04 the Engine composes Focus on Anthropic’s Fable behind a provider-aware gateway, with the intent-assist and refinement stages retired from the served Focus route (ALGO-39, ALGO-40); Eval Labs records whatever provenance the owning route returns and has not been changed for that shift. Eval Labs records evidence returned by the owning product routes. It does not expose a separate direct OpenAI model gateway.

​
Runtime Verification and Run Provenance

Eval Labs production targets Lucia Engine Development at:
The owner-only Lucia Runtime Verification panel shows direct evidence from the same Engine response that produced Lucia’s answer:
New Eval runs persist runtimeProvenance on the case that produced the response and mirror the latest evidence in session metadata. The record includes the target endpoint/environment, Engine identity, request ID, configured/resolved models, execution-path booleans, task evidence, and runtime_verified_at. Historical runs are not backfilled. When a run predates capture or otherwise has no record, the owner view says: Runtime provenance was not captured for this run. Missing evidence is not converted into an inferred endpoint or model. One live owner verification on 2026-07-10 observed:
This is a dated single-request receipt, not current proof for every run. The uninterrupted Staging-target configuration began at 2026-05-04T16:21:35Z; routing was corrected and live-verified against Development on 2026-07-10. That proves the routing drift and correction, not that every historical run individually used Staging. Guest Agent model evidence is a separate ownership boundary. Its latest accepted proof in this review is the 2026-07-18 same-deploy/provider receipt for source fbf77b662511cf004ee1f46787e05773a8a78d71, which recorded configured and provider-resolved gpt-5.6-sol for one production request. That receipt does not establish later executions, and this August documentation review made no fresh Guest Agent model call. Guest-facing Lucia adds a second required validation surface:
This is a first-class Guest-Facing Lucia Eval Track, separate from operator-facing Lucia evals. Purpose:
The track must cover identity orientation, orientation paths for already booked / joining someone / planning / exploring, booked-guest claim parsing, claim fragments across turns, weak vs strong claims, ambiguous/no-match cases, magic-link eligibility, email sent only to the booking email on file, no private data leakage, token consume/replay/expiry, verified session state, guest-to-operator signal creation, Admin Signal Stream visibility, Focus Ops no drift to unrelated synthetic fixtures (Synthetic Seed Guest A and Synthetic Seed Guest B), and warm hospitality tone. Those labels are generic test-fixture identifiers, not people. Payment truth adds a separate required financial-attention validation lane:
This is future Eval Labs coverage. The dated Synthetic Payment Guest fixture is Development runtime proof, not current truth or broad Eval Labs certification; the label is explicitly synthetic and does not identify a person.

​
Current Engine diagnostic posture

The current Engine bank contains 217 cases and targets:
The canonical Focus Ops route is:
On 2026-08-12 an exact-SHA non-strict diagnostic against Engine commit 9a15b1da… returned 76/217; on 2026-09-03 two runs on Development read 174/217 and 192/217 (ALGO-36). None was a pass, strict gate, baseline, or release proof; the standing behaviour gate is the replay harness described on Validation Battery. The earlier 178/178 strict-brain result was recorded on 2026-05-29 against an older 178-case bank and revision. The May 2026 60-run / 3,000-prompt readiness gate is historical platform-lifecycle evidence, not a current authenticated count.
Neither the 2026-08-12 verification nor the 2026-09-10 pass made a fresh Engine, Guest Agent, or Fieldwork model call, queried authenticated Eval Labs counts, or executed a Fieldwork cohort.

​
Persistence topology

Ordinary evaluation persists suites, runs/sessions, items/cases, human reviews, review positions, and optional Behavioral Observatory labels. Custom-suite definitions, review-resume state, and Guest verification history also use browser-local convenience storage. Platform Runtime Registry and Fieldwork evaluation use separate claims, seals, cohort records, signed receipts, review attestations, and recovery lineage. Those records are not generic Eval sessions and do not gain release authority merely by existing.

​
Critical Review Question

The central question is:
If not, the response fails even if technically correct. For guest-facing Lucia, the parallel question is:
If not, the response fails even if it sounds friendly.

​
See Also