Skip to main content
Eval Labs may borrow ideas from OpenAI-style eval frameworks, but it remains the Lucia-native evaluation product and preserves Lucia-specific runtime evidence.

​
Position

Eval Labs should not be replaced by a generic LLM eval framework. Lucia’s most important qualities require human judgment and product-specific review.

​
Current Lucia runtime boundary

Eval Labs production evaluates Lucia Engine Development through:
The Engine’s model gateway is the OpenAI Responses API. Eval Labs does not infer which model ran from the requested model name. It reads runtime and model evidence from the same stable ok / data / meta Engine response that contains the evaluated result, including the configured model, provider-returned resolved model, whether a model was invoked, whether no-attempt deterministic execution applied, and whether fallback was used independently of invocation. New runs persist that evidence as runtimeProvenance. Historical runs without captured evidence explicitly report that runtime provenance was not captured; they are not rewritten with current defaults or later verification results. Staging remains a promotion and validation environment. It is not the current Eval Labs production target.

​
Owner-only runtime verification

The owner-only Lucia Runtime Verification dashboard shows:
  • resolved endpoint and classified target environment
  • Engine environment and commit SHA
  • configured model and provider-resolved model
  • model-invoked, deterministic-path, and fallback states
  • model-task evidence and verification timestamp
On 2026-07-10, live verification returned Verified with configured and resolved model gpt-5.6-sol, model_invoked: true, deterministic_path: false, and fallback_used: false. This proves the observed Development request. It does not convert a configured-model claim into evidence for unrelated runs.

​
What external eval frameworks are good for

External eval frameworks can help with:
  • structured datasets
  • automated graders
  • model comparisons
  • JSONL exports
  • benchmark-style checks
  • repeatable scoring pipelines

​
What they do not solve for Lucia

They do not automatically answer:
  • Did Lucia reduce overwhelm?
  • Did Lucia choose the right emotional posture?
  • Did Lucia avoid overclaiming?
  • Did Lucia preserve trust?
  • Did Lucia sound like Lucia?
  • Did Lucia reduce operator scanning burden?

​
Future direction

Eval Labs may eventually export OpenAI-compatible eval datasets. Potential mapping:

​
Principle

Eval Labs is the source of truth. OpenAI eval concepts can become adapters. Do not invert that relationship.