At a glanceEval Labs is Lucia’s quality-control system. It exists to test whether Lucia is useful, truthful, calm, semantically aware, and operationally correct before behavior becomes trusted.
Why Evaluation Exists
Lucia is not evaluated only on whether an answer is “correct.”
Lucia must be evaluated on whether the answer:
reduces operator burden
preserves truth-state
routes action correctly
feels calm and useful
avoids fake completion
protects guest identity/linkage boundaries
avoids booking-private data leakage
preserves payment truth without fake paid/overdue claims
Evaluation Is Product Infrastructure
Eval Labs is not a side tool.
Eval Labs is part of Lucia’s intelligence stack.
Lucia behaviorEval Labsreviewrefinementsafer behavior
Current v0.1.3.6 Evaluation Posture
Strict brain quality eval reached 178/178 after workspace-context awareness. This is current live-dev evidence, not a permanent guarantee.
Eval Labs should validate v0.1.3.6 against:
https://api-dev.hellolucia.ai/admin/operator-focus
Guest-Facing Lucia Eval Track
The Guest-Facing Lucia Eval Track is separate from operator-facing Lucia evals.
Purpose:
Manual testing cannot cover every possible guest phrasing.
Eval Labs must stress-test guest identity, verification, privacy, tone, and guest-to-operator routing at scale.
identity orientation
already booked / joining someone / planning / exploring
booked-guest claim parsing
claim fragments across turns
weak vs strong claims
ambiguous/no-match cases
magic-link eligibility
email sent only to booking email on file
no private data leakage
token consume / replay / expiry
verified session state
guest-to-operator signal creation
Admin Signal Stream visibility
Focus Ops no Luca/Nora drift
warm hospitality tone
Primary Evaluation Targets
1. Intent Accuracy
Did Lucia understand what the operator was asking?
Examples:
priority_triage
deferral
human_utility
maintenance_focus
arrival_awareness
semantic_conversational_assist
2. Operational Usefulness
Did Lucia identify a useful next move?
A technically accurate answer can still fail if it leaves the operator with too much work.
Current booking-spine usefulness must be tested against arrivals, departures, stay windows, and the difference between Full Booking Page review and Dynamic Action Workspace completion.
Current Workspace OS usefulness must also be tested against the difference between Lucia Workspace + DAW as the cockpit and Full Booking Page as the record/review surface.
Guest Identity and Linkage
Did Lucia preserve the difference between a guest claim, a candidate booking, an operator-linked booking, and a verified booking?
Required guest-facing scenarios:
already booked
joining someone's stay
planning trip
just exploring
name-only weak claim
booking ID-only weak claim
ID + name strong claim
name + arrival date strong claim
claim fragments across turns
ambiguous matches
no matches
service overlap not used as identity evidence
no private data leakage
3. Emotional Containment
Did Lucia reduce overwhelm?
Good containment:
one next move
clear reason
quiet non-urgent noise
long list
vague sympathy
dashboard summary
4. Truth-State Discipline
Did Lucia avoid overclaiming?
Lucia must not imply:
an external action happened
a task was completed
a vendor was contacted
a system update occurred
5. Semantic Conversational Intent
Did Lucia understand short, social, lightweight utility, and scoped context prompts by meaning rather than exact phrase?
Protected families include:
Time?
What time is it in Bangladesh?
Nice to see you
Let's do this
Should we keep dinner outside tomorrow?
Is outside still okay for the ceremony?
6. Signal → Action → Save → Reminder Loop
Current live-dev validation must cover:
Calendar monthly booking spine
Booking Pulse over the same booking source
Signal Stream ordering
Lucia Workspace / Focus with Lucia handoff
active_context from Signal Stream Chat
active_context.workspace current-surface awareness
same shared conversation across Dashboard embedded mode and Workspace Sidebar mode
prior recommendation memory for short follow-ups
prior offer context for short confirmations like yes
saved DAW workflow awareness without fake resolution
real reminder persistence
due reminder resurfacing
Got it acknowledgement
Resolver Matrix route correctness
Dynamic Action Workspace render correctness
Full Booking Page record/review route correctness
Calendar = root operational reality.
Lucia Workspace = reasoning partner beside the operator.
Infinite real-world property tasks.
Finite beautiful action workspaces.
Lucia routes the human to the right one.
Lucia Workspace + DAW = cockpit.
Full Booking Page = record/review surface.
Guest-facing Lucia = front-of-house teammate.
Eval Labs = proof/regression system.
guest operational_signal v0
unlinked signal routing
candidate signal routing
verified signal routing
Admin Signal Stream visibility
Focus Ops no Luca/Nora drift
candidate/unlinked signals not opening fake DAW
Payment Truth Eval Requirements
Current proof status:
Runtime code proves the Development payment truth foundation.
Harper Quinn #29110012 proves confirmed-paid suppression survived durable ledger restart.
Eval Labs has not yet replaced that runtime proof with broad policy-aware regression coverage.
Confirmed-paid guest should not remain a payment blocker.
Pending payment should not become confirmed paid.
Failed, disputed, or refunded states should not be treated as paid.
Unknown policy should not produce due or overdue overclaims.
Deposit-paid but final-balance-not-yet-due should not become overdue.
Final balance overdue should become an attention candidate only after policy + temporal truth support it.
Admin truth rendering should remain read-only and no-write unless explicitly implemented.
DAW payment reconciliation notes should not mutate payment state.
Eval Labs Role
EvaluationLabs.ai is Lucia’s proprietary evaluation platform for shaping her human intent layer, emotional awareness, psychological understanding, natural language interpretation, warmth, empathy, judgment, and operational intelligence.
Eval Labs captures:
prompt
response
review decision
quality notes
behavioral concerns
Quality Bar
A passing Lucia response should be:
accurate
specific
calm
actionable
truthful
low-overwhelm
See Also
Upstream / Downstream
Upstream
This layer is fed by:
Downstream
This layer affects:
