Quality Evaluation

The Quality Evaluation Layer measures whether Layerr is doing a good job. It benchmarks providers, calibrates strategies, validates code outputs, and produces quality scores that feed back into the adaptive learning loop.

Evaluation Subsystems

Subsystem	Purpose	Key File
Calibration Engine	Recalibrates providers and strategies based on outcomes	`evaluation/calibration/engine.ts`
Quality Scoring Engine	Scores provider reliability and strategy effectiveness	`evaluation/quality/engine.ts`
Coding Evaluation	Validates code outputs for correctness and completeness	`evaluation/coding/engine.ts`
Outcome Analysis	Aggregates execution outcomes into metrics	`evaluation/outcomes/engine.ts`
Benchmarks	Systematic A/B testing of providers and strategies	`evaluation/benchmarks/engine.ts`

Calibration

The calibration engine (evaluation/calibration/engine.ts) periodically runs:

function calibrateProviders(): CalibrationReport {
  // 1. Compare recent provider performance vs. historical baseline
  // 2. Detect drift (providers whose scores have changed significantly)
  // 3. Build recommendations for provider rotation
}

function calibrateStrategies(): StrategyCalibration {
  // 1. Measure each strategy's cost/quality/speed outcomes
  // 2. Detect strategies that are no longer optimal
  // 3. Recommend weight adjustments
}

Calibration Outputs

Output	Description
`driftedProviders`	Providers whose quality has deviated from baseline
`sparseWorkloads`	Workload types with insufficient data for reliable scoring
`riskScores`	Risk assessment per provider group
`buildRecommendations()`	Actionable recommendations for workspace admins

Quality Scoring Engine

The quality engine (evaluation/quality/engine.ts) computes:

Function	Purpose
`scoreProviderReliability()`	Composite reliability score from historical traces
`scoreStrategyEffectiveness()`	Measures how well a strategy achieves its stated goal
`computeTraceConfidence()`	Confidence that a trace’s quality score is accurate
`buildAggregateExplanations()`	Human-readable quality explanations

Quality Grades

Grade	Score	Meaning
A+	97-100	Exceptional
A	93-96	Excellent
A-	90-92	Very good
B+	87-89	Good
B	83-86	Above average
B-	80-82	Average
C+	77-79	Below average
C	73-76	Needs improvement
D	0-72	Poor

Functions: qualityGrade(), confidenceGrade()

Coding Evaluation

The coding validator (evaluation/coding/engine.ts and evaluation/coding/validators.ts) checks code outputs for:

Check	Validator	What It Tests
Code presence	`checkCodePresence()`	Response actually contains code
Code completeness	`checkCodeCompleteness()`	Code is complete, not truncated
Language consistency	`checkLanguageConsistency()`	Code matches the requested language
JSON validity	`checkJsonValidity()`	JSON outputs parse correctly
Schema compliance	`checkStructuredOutputSchema()`	Output matches expected schema
Brace balance	`checkBraceBalance()`	Brackets/parentheses are balanced
Response length	`checkResponseLength()`	Response is reasonably sized
Patch format	`checkPatchFormat()`	Diff/patch format is valid

Outcome Analysis

The outcomes engine (evaluation/outcomes/engine.ts) produces metrics like:

Metric	Description
Success rate	% of requests that succeeded
Average latency	Mean time to first token and completion
Average cost	Mean cost per request
Quality distribution	Histogram of quality grades
Fallback rate	% of requests that needed fallback
Provider distribution	Which providers are being used most

Frontend Components

Component	File	Purpose
`QualityScoringInspector`	`src/features/evaluation/quality/`	Deep quality score analysis
`CalibrationOutcomeView`	`src/features/evaluation/`	View calibration results
`ExecutionQualitySummary`	`src/features/evaluation/`	Summarised quality dashboard

File Reference

File	What It Does
`evaluation/calibration/engine.ts`	Recalibration engine for providers and strategies
`evaluation/quality/engine.ts`	Provider reliability and strategy effectiveness scoring
`evaluation/coding/engine.ts`	Code output validation and grading
`evaluation/coding/validators.ts`	Individual validator functions for code quality
`evaluation/outcomes/engine.ts`	Execution outcome aggregation and metrics
`evaluation/benchmarks/engine.ts`	Systematic provider/strategy benchmarking

Integration

Execution Engine → provides traces for evaluation
Replay → stores evaluation results alongside traces
Adaptive Learning → receives evaluation insights for weight updates
Explainability → includes quality grades in explanations
Strategy Engine → receives calibration recommendations