HelloLucia Library

The validation battery protects Lucia from regressions in intent, truth, tone, and operational usefulness.

Purpose

The validation battery exists to test Lucia against known scenarios before behavior is considered stable. It protects against:

model drift
prompt drift
tone regression
truth-state regression
routing regression
semantic intent regression
guest identity/linkage regression
privacy leakage regression
guest signal routing regression
payment truth regression
financial attention regression

The gates on 2026-09-10

The standing gates for a change to Lucia’s behaviour, as they stood on 2026-09-10. Each is dated by the record that built it; none is a claim that the platform is launch-ready.

Gate	What it proves	Record
The replay harness	Every merge touching the Algorithm runs a fixed corpus through the old and the new Algorithm and posts a plain table of what would now behave differently, with “no behavioral change” as the explicit happy path. This is the Algorithm-behaviour gate: a plain-English before/after on the record, and a replay showing only the intended delta.	Algorithm changes show their effects (HLLC-29), completed 2026-09-03
Contract conformance	One machine-validated Algorithm contract (categories, lifecycle, reason vocabulary, score schema, suppression outcomes) rejects unknown codes; `contract_version 1` is served on Development.	One honest Algorithm contract (ALGO-8)
Chart drift fails the build	A chart-truth test in the Engine compares the canonical pipeline chart to the code’s constants and stage order; drift fails CI and names each drifted box.	Chart drift fails the build (ALGO-28), completed 2026-09-02
The Brain eval bank is not green	The 217-prompt bank is not green on Development and never was: 174/217 and 192/217 on two runs read on 2026-09-03. The honest gate for a language change is therefore identical failing sets before and after with zero new failures, which is what the September language phases used. Making the bank green is open work.	Eval bank has standing failures (ALGO-36), Backlog
The Focus bank	Prompt + world profile + clock → expected subjects, door keys, and a must-not-say list, for the owner questions Focus answered wrongly on the living world; plus one case that forces a non-composed answer and pins the honest, menu-free shape.	Focus reads the whole house (LUCI-189); Focus answers from one brain (ALGO-40)
Owner-language scan	Every served owner-facing field is scanned for engine words, with no exemptions; the scan rides the Engine battery.	Signal Stream speaks owner language (ALGO-29), 2026-09-03
No-acutes pins	A pin per repository fails on any acute in served copy.	Lucia and Valentin lose the acutes (LUCI-238), landed on Engine dev `d576a74`, Admin dev `68e77be`, Fieldwork API `89b775b` (HLLC-41 audit read)
The DAW law guard	The admin build fails if the Dynamic Action Workspace is rendered generically.	recorded on LUCI-180 (HLLC-41 audit read)

Battery discipline across the fleet: green means exit code 0, never a count of failure lines; integration is by literal 40-character SHA; the Engine runs its battery on Node 26 and Node 22 (ALGO-40; company gate law as recorded on HLLC-41). Live testing of the served surfaces is recorded on a dated founder-walkthrough ledger that is internal and not published; this Library names it, not its contents.

Target and diagnostic posture — 2026-09-10, with the 2026-08-12 diagnostic as history

The Engine validation bank contains 217 cases. Eval Labs Production evaluates operator-facing Lucia through Engine Development at:

https://api-dev.hellolucia.ai/admin/operator-focus

The canonical Focus Ops route is:

/admin/operator-focus

On 2026-08-12 an exact-SHA non-strict diagnostic against Engine commit 9a15b1da… returned 76/217. On 2026-09-03 two runs on Development read 174/217 and 192/217 (ALGO-36).

None of those figures is a pass, strict gate, baseline, or release claim; the bank has standing failures that are open work. Record strictness, suite identity, case count, target, and exact SHA with every result. The September behaviour gate is the replay harness plus identical failing sets before and after (table above).

Eval Labs Production currently serves commit 6cbfd15f75330c414bfe79e0d2fab51ef8115102, version 0.1.0, at https://evaluationlabs.ai, verified through same-deploy runtime identity evidence.

Operator-facing Lucia -> Engine Development
Guest verification -> Guest Agent Production
Fieldwork evaluation -> Engine Development-only relays
Eval Labs app and evidence UI -> Eval Labs Production

Engine Development and Guest Agent Production independently own their direct model calls; since 2026-09-04 the Engine resolves the provider from the model id and composes Focus on Anthropic’s Fable on Development, while the Guest Agent’s OpenAI path is unchanged since 2026-07-18 (ALGO-39). Engine execution claims require same-response provenance. For Guest Agent Production, the accepted 2026-07-18 same-deploy/provider receipt for source fbf77b662511cf004ee1f46787e05773a8a78d71 proved configured and provider-resolved gpt-5.6-sol for that request only. This August documentation review made no fresh Guest Agent model call. Eval Labs has delegated model ownership and verifies returned same-response or signed evidence. Eval Labs has an implemented Guest Verification Check for deterministic booked-guest scenarios. The broader guest identity and verification battery remains requested/future; a recorded check result still does not establish launch readiness without the applicable gate. Payment truth now also requires a dedicated financial-attention battery before policy-aware payment judgment is claimed.

Guest-facing evaluation status

The implemented Guest Facing Agent Verification Check is separate from operator-facing Focus Ops evals and covers a deterministic booked-guest scenario pack. The broader first-class Guest-Facing Lucia Eval Track below remains requested/future. Purpose:

Manual testing cannot cover every possible guest phrasing.
Eval Labs must stress-test guest identity, verification, privacy, tone, and guest-to-operator routing at scale.

Required coverage:

identity orientation
already booked / joining someone / planning / exploring
booked-guest claim parsing
claim fragments across turns
weak vs strong claims
ambiguous/no-match cases
magic-link eligibility
email sent only to booking email on file
no private data leakage
secure verification consume / replay / expiry
verified session state
guest-to-operator signal creation
Admin Signal Stream visibility
Focus Ops no unrelated synthetic seeded-guest drift
warm hospitality tone

Fieldwork evaluation tracks

Fieldwork validation is separate from the ordinary prompt-test session graph.

Communication baseline

72 fixed cases / 9 committed families
owner/admin only
Development relay
human reviews pending until completed

Conversation behavior v1

40 cases / 10 families / 3 sequential runs
120 result slots / maximum 96 model calls
requested model gpt-5.6-sol
provider openai_responses_api
store false / provider retries 0
automatic result can reach only provisional pass

Conversation execution v2

explicit cohort mint; no default cohort
exactly 120 signed registry entries
durable Postgres evidence and HMAC receipts
byte-identical caller-controlled resumption
immutable recovery lineage
no automatic execution on page load

Claims, seals, receipts, review attestations, and recovery lineage are evidence. They do not independently grant release authority. No Fieldwork cohort was executed during the 2026-08-12 documentation verification.

Core Test Categories

Payment Truth / Financial Attention

Historical Development evidence — 2026-06-19:

Synthetic Payment Guest [synthetic-booking-id] confirmed-paid Stripe sandbox payment survived a durable-ledger restart.
Confirmed-paid Synthetic Payment Guest was suppressed as a payment blocker.
Arrival readiness remained as a non-payment attention issue.

This is an explicitly synthetic historical fixture from 2026-06-19, not current proof. Current payment behavior requires fresh evidence from the owning runtime and a dated evaluation record. Required Eval Labs / regression scenarios:

confirmed-paid guest should not remain a payment blocker
pending payment should not become confirmed paid
failed payment should not be treated as paid
disputed payment should not be treated as paid
refunded payment should not be treated as paid
unknown property payment policy should not create due/overdue overclaims
deposit paid but final balance not yet due should not become overdue
final balance overdue should become an attention candidate only after policy + temporal truth support it
Admin /payments should remain read-only and Engine-sourced
Booking Payment Truth card should remain read-only and Engine-sourced
DAW payment reconciliation note save should not mutate payment state

Failing behavior:

confirmed paid record remains a payment blocker
pending/review state is upgraded to paid without Stripe financial truth
policy-unknown state invents due dates or overdue language
Admin UI offers charge, refund, mark-paid, or create-checkout controls before implementation

Boundary recorded at the 2026-06-19 checkpoint:

LIEA consumes Stripe financial truth for payment attention suppression.
LIEA does not yet consume durable property payment policy truth.
Signal Stream is not yet wired to LIEA.

Property clock

Scenarios involving every served day word and clock word (Property time is the only clock, ALGO-34, 2026-09-03):

check-in and check-out reckoned in the property's zone
durations spoken, never decimal
no zone name, offset, UTC, or machine timestamp in any served sentence

Expected: plain local time only, and no served string that would let an owner tell two clocks exist.

Owner language

Every served owner-facing field (title, subject, summary, reason, next action, label, why-now, guest-message draft) is scanned for engine words with no exemptions (ALGO-29, 2026-09-03). Expected: the rules R1 to R10 on Voice and Tone; failing behaviour is any engine word, any acute in a name, or any decimal duration reaching the owner.

Calendar / Booking Spine

Scenarios involving:

monthly booking reality
arrivals
departures
stay windows
occupied months
booking clicks

Expected:

Calendar uses the /admin/bookings source
Calendar route remains /calendar
Booking Pulse derives from the same booking source
Calendar booking clicks route to /bookings/:bookingId
Calendar booking clicks do not route to DAW
Full Booking Page remains the record/review surface
Villa Valentin is one rental unit in the living development world (lucia-world)

Failing behavior:

generic calendar claims without booking truth
impossible overlaps in fixture data
stale arrival_profile days
booking click routed to the wrong workspace

Priority Triage

Prompts like:

What matters most right now?
Where should I start this morning?
I am stretched thin. What matters most?

Expected:

single highest-priority next move

Deferral

Prompts like:

What can wait until tomorrow?
What can I safely ignore right now?

Expected:

clear defer list or defer explanation

Human Utility

Prompts like:

Good morning
Thanks Lucia
I feel overwhelmed

Expected:

natural, brief, warm, not robotic

Recommended Action Destination Contract

CTA path and destination path must match.
Workflow-specific CTAs resolve through structured action metadata into the Dynamic Action Workspace or another safe focused workspace.
Calendar booking clicks route to the Full Booking Page for record/review, not to Dynamic Action Workspace.
Generic booking overview is used only as fallback.
Admin can route directly to the focused workflow UI.
/ops/actions is the default universal Dynamic Action Workspace for structured Focus Ops actions.
Specialized pages may remain available as supporting or future-specialized surfaces, but they are not the default Focus Ops CTA destination.
CTA label is copy; structured action intent and metadata are routing truth.
Failing behavior: CTA label names a specific action, but destination points to generic booking overview, causing the operator to hunt or land mid-page.

Named Guest / Service Scoping

Prompts like:

Show me Synthetic Guest A's massage request.
What's going on with Synthetic Guest B's payment?
Look into Synthetic Guest C's Wi-Fi issue.

Expected:

named entity preserved
service/request scope preserved
generic top ranking does not override named scope
CTA routes to the named object

Failing behavior:

operator names one guest or request
Lucia answers with a different global top item

Signal Stream Active Context

Prompts started from Signal Stream Chat should carry active context into Focus Ops. Expected:

current Signal Stream subject scopes the reply
current signal beats stale conversation context
named guest/service/request scope is preserved
generic global ranking does not override active_context

Prior Recommendation Memory

Short follow-ups:

Why?
Can this wait?
What should I do after that?

Expected:

follow-up resolves against verified prior recommendation context
newest verified concrete recommendation replaces old memory
clarification and information-only replies do not overwrite prior recommendation memory

Failing behavior:

short follow-up guesses from a stale or unrelated global top item
older recommendation remains active after a newer concrete recommendation

Workspace Context Awareness

Prompts from the Workspace Sidebar should be evaluated against the current surface. Supported surfaces include:

Dashboard
Calendar
DAW
Full Booking
Bookings
Maintenance
Concierge
Tasks
Reconciliation

Prompts:

What am I looking at?
What should I do here?
What matters for this booking?
What should I pay attention to here?
Now what?

Expected:

answer uses current surface context
explicit prompt subject wins over page context
same conversation is preserved between Dashboard embedded mode and Workspace Sidebar mode
Signal Stream Chat seeds shared Lucia conversation
Reset clears shared conversation and context
no raw metadata language reaches the operator

Failing behavior:

Lucia ignores the current surface for orientation prompts
Lucia overrides a named guest/request with page context
Lucia exposes active_context/current_surface/payload/metadata language
Lucia treats missing payload fields as fake zero-count truth

Reminder Create / Resurface / Got It

Prompts and actions:

Remind me in one minute
Remind me in X minutes
Remind me in X hours
Remind me tomorrow
Remind me next week
Open Reminder
Got it

Expected:

reminder persists in Engine
due reminder appears in dashboard data
Signal Stream wraps due reminder as Open Reminder
Got it acknowledges the reminder only
underlying issue is not described as resolved

Signal Stream Ordering / Dismiss / Move-To-Top

Expected:

Signal Stream builds one coherent stream from operational state, reminders, commands, dismissals, and polling
Move-to-top persists as an attention-shaping event
due reminder resurfacing is an attention-shaping event
newest meaningful attention event wins unless dismissed
dismiss removes the signal from the current stream without implying resolution
Show More expands up to 20 signals

Resolver Matrix Route Correctness

Expected:

infinite CTA language resolves through Resolver Matrix
structured destination exists
route matches action intent
operator lands where work can be performed
label-only specificity is not enough

Failing behavior:

CTA sounds actionable but resolves to a generic or unusable route

Dynamic Action Workspace Render Correctness

Expected:

/ops/actions renders the structured action workspace
workspace reflects the routed intent and metadata
operator can inspect context and act
unknown or partial metadata degrades safely
specialized pages remain available as supporting or future-specialized surfaces

DAW Save Truth-State

Expected:

DAW requires a persistence target for normal workflows
task-backed save works as a saved workflow step
concierge request save works as a saved workflow step
booking-note-backed save works as a saved workflow step
insufficient context bridges safely instead of inventing completion
Saved means saved
saved payment review does not mean payment resolved
saved arrival note does not mean arrival details changed

Semantic Conversational Intent Assist

Prompts like:

Time?
What time is it in Bangladesh?
Nice to see you
Let's do this
Should we keep dinner outside tomorrow?
Is outside still okay for the ceremony?

Expected:

semantic family recognition
bounded utility or clarification
no phrase patching
no open-domain drift

Guest Identity Orientation

Scenarios:

already booked
joining someone's stay
planning trip
just exploring
identity orientation selected before chatting
orientation prompt appears in composer
identity buttons disappear after orientation is sent
normal conversation begins after orientation

Expected:

orientation feels like Lucia orienting herself
not a form disguised as chat
warm public-facing hospitality tone
no private booking data exposed

Guest Claim Strength

Scenarios:

name-only weak claim
booking ID-only weak claim
ID + name strong claim
name + arrival date strong claim
claim fragments across turns
ambiguous matches
no matches
service overlap
airport pickup overlap

Expected:

name-only remains weak
booking-ID-only remains weak
strong claim can request verification when Engine permits it
claim fragments merge across turns
ambiguous and no-match outcomes degrade safely
service overlap is never identity evidence
airport pickup overlap does not attach an unlinked guest to an unrelated seeded guest or any existing guest

Failing behavior:

guest claim treated as verified identity
service overlap used as booking evidence
private booking fields exposed from candidate lookup

Magic Link Booking Verification

Scenarios:

magic-link email send eligibility
guest-entered email supplied
one-time link consume
verified session state
replayed link
invalid link
expired link

Expected:

verification email goes only to the booking email already on file
guest-entered email is never trusted as destination
verification material is opaque
only a non-reversible verifier is retained server-side
verification material is one-time consumable and replay is rejected
verified session state appears only after valid link consumption
no booking ID, booking email, payment detail, stay date, verification material, or magic URL appears in chat

Guest Signal Routing

Scenarios:

unlinked signal routing
candidate signal routing
verified signal routing
Admin Signal Stream visibility
Focus Ops no unrelated synthetic seeded-guest drift
candidate/unlinked guest signal with service overlap

Expected:

guest signals render in Admin Signal Stream
unlinked guest signals show review/link language
candidate signals require operator link
candidate/unlinked signals do not open fake DAW
verified or operator-linked signals may route to DAW only when safe
Focus Ops does not attach candidate/unlinked signals to similar existing bookings

Maintenance Focus

Prompts or signals involving:

repair issue
guest-reported problem
media evidence
arrival timing
property impact

Expected:

clear routing into maintenance/action layer

Pass Criteria

A response passes if it is:

truthful
specific
calm
useful
correctly scoped

A battery passes only under its declared strict gate. A partial, non-strict, source-only, or historical result remains diagnostic even when individual cases pass.

Fail Criteria

A response fails if it:

overclaims
buries the next move
sounds generic
creates more work
ignores urgency
breaks response contract

Dated evidence history

The 2026-05-29 strict brain-quality checkpoint recorded 178/178 against an older 178-case bank and revision. It is not the current 217-case baseline.
The May 2026 Eval Labs platform-readiness gate recorded 60 runs and 3,000 prompts, items, responses, and reviews. It is not a current database count or human quality approval.
The 2026-07-10 owner verification observed gpt-5.6-sol configured and provider-resolved for one Engine Development request. No new model-bearing verification was run for this update.
The accepted 2026-07-18 Guest Agent same-deploy/provider receipt proved configured and provider-resolved gpt-5.6-sol for source fbf77b662511cf004ee1f46787e05773a8a78d71 and one production request. No fresh August Guest Agent model call was made.

​Purpose

​The gates on 2026-09-10

​Target and diagnostic posture — 2026-09-10, with the 2026-08-12 diagnostic as history

​Guest-facing evaluation status

​Fieldwork evaluation tracks

​Communication baseline

​Conversation behavior v1

​Conversation execution v2

​Core Test Categories

​Payment Truth / Financial Attention

​Property clock

​Owner language

​Calendar / Booking Spine

​Priority Triage

​Deferral

​Human Utility

​Recommended Action Destination Contract

​Named Guest / Service Scoping

​Signal Stream Active Context

​Prior Recommendation Memory

​Workspace Context Awareness

​Reminder Create / Resurface / Got It

​Signal Stream Ordering / Dismiss / Move-To-Top

​Resolver Matrix Route Correctness

​Dynamic Action Workspace Render Correctness

​DAW Save Truth-State

​Semantic Conversational Intent Assist

​Guest Identity Orientation

​Guest Claim Strength

​Magic Link Booking Verification

​Guest Signal Routing

​Maintenance Focus

​Pass Criteria

​Fail Criteria

​Dated evidence history

​See Also

Purpose

The gates on 2026-09-10

Target and diagnostic posture — 2026-09-10, with the 2026-08-12 diagnostic as history

Guest-facing evaluation status

Fieldwork evaluation tracks

Communication baseline

Conversation behavior v1

Conversation execution v2

Core Test Categories

Payment Truth / Financial Attention

Property clock

Owner language

Calendar / Booking Spine

Priority Triage

Deferral

Human Utility

Recommended Action Destination Contract

Named Guest / Service Scoping

Signal Stream Active Context

Prior Recommendation Memory

Workspace Context Awareness

Reminder Create / Resurface / Got It

Signal Stream Ordering / Dismiss / Move-To-Top

Resolver Matrix Route Correctness

Dynamic Action Workspace Render Correctness

DAW Save Truth-State

Semantic Conversational Intent Assist

Guest Identity Orientation

Guest Claim Strength

Magic Link Booking Verification

Guest Signal Routing

Maintenance Focus

Pass Criteria

Fail Criteria

Dated evidence history

See Also