> ## Documentation Index > Fetch the complete documentation index at: https://helloluciallc.com/llms.txt > Use this file to discover all available pages before exploring further. # START HERE - Evaluation System **At a glance** Eval Labs is Lucia's quality-control system. It exists to test whether Lucia is useful, truthful, calm, semantically aware, and operationally correct before behavior becomes trusted. *** ## Why Evaluation Exists Lucia is not evaluated only on whether an answer is “correct.” Lucia must be evaluated on whether the answer: ```text theme={"dark"} reduces operator burden preserves truth-state routes action correctly feels calm and useful avoids fake completion protects guest identity/linkage boundaries avoids booking-private data leakage preserves payment truth without fake paid/overdue claims ``` *** ## Evaluation Is Product Infrastructure Eval Labs is not a side tool. Eval Labs is part of Lucia’s intelligence stack.
Lucia behaviorEval Labsreviewrefinementsafer behavior
*** ## Current v0.1.3.6 Evaluation Posture Strict brain quality eval reached 178/178 after workspace-context awareness. This is current live-dev evidence, not a permanent guarantee. Eval Labs should validate v0.1.3.6 against: ```text theme={"dark"} https://api-dev.hellolucia.ai/admin/operator-focus ``` The canonical Focus Ops route is: ```text theme={"dark"} /admin/operator-focus ``` v0.1.3.6 is not promoted to staging yet. Staging promotion waits until the Eval Labs dev baseline is captured and reviewed. Guest-facing Lucia now requires its own first-class Eval Labs track before guest-facing behavior is treated as launch-ready. Payment truth now requires dedicated financial-attention coverage before policy-aware payment judgment is treated as stable. *** ## Guest-Facing Lucia Eval Track The Guest-Facing Lucia Eval Track is separate from operator-facing Lucia evals. Purpose: ```text theme={"dark"} Manual testing cannot cover every possible guest phrasing. Eval Labs must stress-test guest identity, verification, privacy, tone, and guest-to-operator routing at scale. ``` This track must cover: ```text theme={"dark"} identity orientation already booked / joining someone / planning / exploring booked-guest claim parsing claim fragments across turns weak vs strong claims ambiguous/no-match cases magic-link eligibility email sent only to booking email on file no private data leakage token consume / replay / expiry verified session state guest-to-operator signal creation Admin Signal Stream visibility Focus Ops no Luca/Nora drift warm hospitality tone ``` *** ## Primary Evaluation Targets ### 1. Intent Accuracy Did Lucia understand what the operator was asking? Examples: ```text theme={"dark"} priority_triage deferral human_utility maintenance_focus arrival_awareness semantic_conversational_assist ``` *** ### 2. Operational Usefulness Did Lucia identify a useful next move? A technically accurate answer can still fail if it leaves the operator with too much work. Current booking-spine usefulness must be tested against arrivals, departures, stay windows, and the difference between Full Booking Page review and Dynamic Action Workspace completion. Current Workspace OS usefulness must also be tested against the difference between Lucia Workspace + DAW as the cockpit and Full Booking Page as the record/review surface. *** ### Guest Identity and Linkage Did Lucia preserve the difference between a guest claim, a candidate booking, an operator-linked booking, and a verified booking? Required guest-facing scenarios: ```text theme={"dark"} already booked joining someone's stay planning trip just exploring name-only weak claim booking ID-only weak claim ID + name strong claim name + arrival date strong claim claim fragments across turns ambiguous matches no matches service overlap not used as identity evidence no private data leakage ``` Expected behavior is warm, useful, and bounded. *** ### 3. Emotional Containment Did Lucia reduce overwhelm? Good containment: ```text theme={"dark"} one next move clear reason quiet non-urgent noise ``` Bad containment: ```text theme={"dark"} long list vague sympathy dashboard summary ``` *** ### 4. Truth-State Discipline Did Lucia avoid overclaiming? Lucia must not imply: ```text theme={"dark"} an external action happened a task was completed a vendor was contacted a system update occurred ``` unless verified. *** ### 5. Semantic Conversational Intent Did Lucia understand short, social, lightweight utility, and scoped context prompts by meaning rather than exact phrase? Protected families include: ```text theme={"dark"} Time? What time is it in Bangladesh? Nice to see you Let's do this Should we keep dinner outside tomorrow? Is outside still okay for the ceremony? ``` The expected behavior is bounded usefulness, not open-domain chat. *** ### 6. Signal → Action → Save → Reminder Loop Current live-dev validation must cover: ```text theme={"dark"} Calendar monthly booking spine Booking Pulse over the same booking source Signal Stream ordering Lucia Workspace / Focus with Lucia handoff active_context from Signal Stream Chat active_context.workspace current-surface awareness same shared conversation across Dashboard embedded mode and Workspace Sidebar mode prior recommendation memory for short follow-ups prior offer context for short confirmations like yes saved DAW workflow awareness without fake resolution real reminder persistence due reminder resurfacing Got it acknowledgement Resolver Matrix route correctness Dynamic Action Workspace render correctness Full Booking Page record/review route correctness ``` The product rule under test: ```text theme={"dark"} Calendar = root operational reality. Lucia Workspace = reasoning partner beside the operator. Infinite real-world property tasks. Finite beautiful action workspaces. Lucia routes the human to the right one. Lucia Workspace + DAW = cockpit. Full Booking Page = record/review surface. Guest-facing Lucia = front-of-house teammate. Eval Labs = proof/regression system. ``` Guest-facing validation must also cover: ```text theme={"dark"} guest operational_signal v0 unlinked signal routing candidate signal routing verified signal routing Admin Signal Stream visibility Focus Ops no Luca/Nora drift candidate/unlinked signals not opening fake DAW ``` This is Development/live-dev runtime truth, not a production-readiness claim. *** ## Payment Truth Eval Requirements Current proof status: ```text theme={"dark"} Runtime code proves the Development payment truth foundation. Harper Quinn #29110012 proves confirmed-paid suppression survived durable ledger restart. Eval Labs has not yet replaced that runtime proof with broad policy-aware regression coverage. ``` Required future coverage: ```text theme={"dark"} Confirmed-paid guest should not remain a payment blocker. Pending payment should not become confirmed paid. Failed, disputed, or refunded states should not be treated as paid. Unknown policy should not produce due or overdue overclaims. Deposit-paid but final-balance-not-yet-due should not become overdue. Final balance overdue should become an attention candidate only after policy + temporal truth support it. Admin truth rendering should remain read-only and no-write unless explicitly implemented. DAW payment reconciliation notes should not mutate payment state. ``` This coverage should test the architecture recorded in [Lucia Payment Truth Foundation](/lucia/system-architecture/lucia-payment-truth-foundation). *** ## Eval Labs Role EvaluationLabs.ai is Lucia's proprietary evaluation platform for shaping her human intent layer, emotional awareness, psychological understanding, natural language interpretation, warmth, empathy, judgment, and operational intelligence. Eval Labs captures: ```text theme={"dark"} prompt response review decision quality notes behavioral concerns ``` It creates a repeatable workflow for improving Lucia’s intelligence and tone. *** ## Quality Bar A passing Lucia response should be: ```text theme={"dark"} accurate specific calm actionable truthful low-overwhelm ``` *** ## See Also * [02 - Validation Battery](/lucia/evaluation/validation-battery) * [03 - Quality Bar](/lucia/evaluation/quality-bar) * [06 - Semantic Conversational Intent Assist](/lucia/intelligence/semantic-conversational-intent-assist) * [04 - Seed Data and Test Worlds](/lucia/evaluation/seed-data-and-test-worlds) * [02 - Focus Ops Intelligence](/lucia/intelligence/focus-ops-intelligence) * [04 - Lucia Workspace OS Milestone](/lucia/system-architecture/lucia-workspace-os-milestone) * [07 - Guest-Facing Eval Requirements](/lucia/guest-facing-lucia/guest-facing-eval-requirements) *** ## Upstream / Downstream ### Upstream This layer is fed by: * [02 - Current System State](/lucia/start-here-lucia-ai/current-system-state) * [00 - START HERE - System Architecture](/lucia/system-architecture/start-here-system-architecture) ### Downstream This layer affects: * [00 - START HERE - Operations](/lucia/operations/start-here-operations) * [00 - START HERE - Evaluation System](/lucia/evaluation/start-here-evaluation-system) * [Lucia Decision Log](/lucia/decisions/lucia-decision-log)