Provider Scoring & Selection

The Scoring Layer evaluates every candidate provider and model against the current workload, strategy, and workspace constraints. It produces a ranked candidate list that the Execution Engine uses to route requests.

Scoring Dimensions

Dimension	Weight Source	Key Function
Quality Score	Model capability + historical accuracy	scoreProviderReliability() in evaluation/quality/engine.ts
Speed Score	Historical latency p50/p99	scoreOneCandidate() in scoring/runtimeScorer.ts
Cost Score	Token price × estimated output size	scoreCandidate() in scoring/scorer.ts
Capability Match	Does model support the required features?	resolveCapabilities() in routing/enrichment.ts
Health Score	Current provider uptime / recent errors	Provider health aggregator
Workspace Fit	User preference + historical satisfaction	workspaceFit in capabilityModel.ts

The Scoring Pipeline

Strategy + Workload + Provider Pool
    |
    v
[1] Filter Ineligible    --> Remove providers that don't support required features
    |
    v
[2] Score Candidates     --> Score each remaining provider across dimensions
    |
    v
[3] Apply Strategy Weights -> Weight scores by strategy priority (cost/speed/quality)
    |
    v
[4] Build Fallback Chain  --> Ordered list: primary, backup1, backup2...
    |
    v
[5] Score Trace          --> Record scores for explainability and learning

Runtime Scoring

The runtime scorer (scoring/runtimeScorer.ts) does fast, per-request scoring:

function scoreOneCandidate(
  candidate: ProviderCandidate,
  workload: WorkloadProfile,
  strategy: Strategy,
  telemetry: ProviderTelemetry
): ScoreResult {
  const quality = scoreQuality(candidate, workload);
  const speed = scoreSpeed(candidate, telemetry);
  const cost = scoreCost(candidate, workload, strategy);
  const health = scoreHealth(candidate, telemetry);
  const fit = scoreWorkspaceFit(candidate, workspace);

  return weightedAggregate(quality, speed, cost, health, fit, strategy.weights);
}

Quality Scoring Engine

The quality engine (evaluation/quality/engine.ts) provides deep quality analysis:

Function	Purpose
scoreProviderReliability()	Aggregates historical success rate, error rate, retry rate
scoreStrategyEffectiveness()	Measures how well a strategy performs for a provider
computeTraceConfidence()	Confidence score for a single execution trace
buildAggregateExplanations()	Builds human-readable explanations for quality scores

Fallback Chain Construction

The fallback builder (orchestration/fallback.ts) creates the ordered chain:

scoreDeltas()     --> Compare candidate scores
scoredOrder()     --> Sort by composite score
scoreBuiltFallbackChain() --> Build chain with diversity rules

Diversity rules ensure the fallback chain isn’t just “the same provider three times”:

• Provider diversity: Primary and backup must be different providers where possible
• Model diversity: Different model families for resilience
• Geographic diversity: Different regions for availability
• Tier diversity: Mix of premium and standard models

File Reference

File	What It Does
scoring/scorer.ts	Core scoring logic. Aggregates all dimension scores
scoring/runtimeScorer.ts	Fast per-request runtime scoring
scoring/profiles.ts	Scoring profiles per strategy and workload type
evaluation/quality/engine.ts	Deep quality analysis and historical aggregation
orchestration/fallback.ts	Fallback chain construction and delta analysis
providers/capabilities.ts	Provider capability definitions and matching
src/features/providers/capabilities/capabilityModel.ts	Frontend capability matrix data model

Integration

1. Strategy Engine → provides weights
2. Workload Analysis → provides estimated token counts and complexity
3. Provider Registry → provides available providers and metadata
4. Provider Health → provides real-time health telemetry
5. Execution Engine → receives the ordered candidate list
6. Explainability → receives score breakdowns for transparency