Skip to main content
Eval Labs may borrow ideas from OpenAI-style eval frameworks, but it remains the Lucia-native evaluation product and preserves Lucia-specific runtime evidence.

​
Position

Eval Labs should not be replaced by a generic LLM eval framework. Lucia’s most important qualities require human judgment and product-specific review.

​
Deployed Lucia runtime contract

Evidence boundary — 2026-08-12: https://evaluationlabs.ai served Eval Labs Production commit 6cbfd15f75330c414bfe79e0d2fab51ef8115102. Inspection of that deployed source/config supports the contract below. This documentation check did not authenticate to Eval Labs, make a Lucia request, or observe provider execution, so it is not fresh runtime or persistence proof.
Eval Labs production source/config targets Lucia Engine Development through:
The Engine source owns an OpenAI Responses API model gateway. Eval Labs is designed not to infer which model ran from the requested model name. Its source reads runtime and model evidence from the same stable ok / data / meta Engine response that contains the evaluated result, including the configured model, provider-returned resolved model, whether a model was invoked, whether no-attempt deterministic execution applied, and whether fallback was used independently of invocation. The durable-run write contract stores that evidence as runtimeProvenance for new runs. Historical runs without captured evidence explicitly report that runtime provenance was not captured; they are not rewritten with current defaults or later verification results. Successful live persistence was not re-proven on 2026-08-12. Staging remains a promotion and validation environment. It is not the current Eval Labs production target.

​
Owner-only runtime verification

The deployed source defines an owner-only Lucia Runtime Verification dashboard that can show:
  • resolved endpoint and classified target environment
  • Engine environment and commit SHA
  • configured model and provider-resolved model
  • model-invoked, deterministic-path, and fallback states
  • model-task evidence and verification timestamp
On 2026-07-10, a separately recorded owner verification request returned Verified with configured and resolved model gpt-5.6-sol, model_invoked: true, deterministic_path: false, and fallback_used: false. This proves the observed Development request. It does not convert a configured-model claim into evidence for unrelated runs.

​
What external eval frameworks are good for

External eval frameworks can help with:
  • structured datasets
  • automated graders
  • model comparisons
  • JSONL exports
  • benchmark-style checks
  • repeatable scoring pipelines

​
What they do not solve for Lucia

They do not automatically answer:
  • Did Lucia reduce overwhelm?
  • Did Lucia choose the right emotional posture?
  • Did Lucia avoid overclaiming?
  • Did Lucia preserve trust?
  • Did Lucia sound like Lucia?
  • Did Lucia reduce operator scanning burden?

​
Future direction

Eval Labs may eventually export OpenAI-compatible eval datasets. Potential mapping:

​
Principle

Eval Labs owns evaluation orchestration, review, and evaluation evidence. OpenAI eval concepts can become adapters. Lucia Engine and each direct model-calling runtime remain authoritative for their own runtime and model-execution truth. Eval Labs may capture and assess that evidence, but it does not become its owner. Do not invert that relationship.