Methodology

From evidence to score

1. Raw evidence — a benchmark result, internal test, or measured metric (e.g. latency, price) for one model on one dimension.
2. Normalization — raw values are scaled to 0–100 relative to the other models with evidence on that dimension. Metrics where lower is better (cost, latency) are inverted first.
3. Confidence adjustment — confidenceFactor = 0.70 + 0.30 × min(1, evidenceCount / targetEvidence). A single low-sample result is never discarded, but it never carries full weight either.
4. Weighted profile score — each ranking profile applies its own explicit, versioned weights across dimensions. Dimensions with no evidence for a model are excluded entirely and the remaining weights are renormalized — a missing dimension is never silently treated as zero.
5. Freshness & reliability adjustment — finalRankScore = overall × 0.90 + freshnessBonus × 0.05 + reliabilityScore × 0.05. Freshness decays as evidence ages; reliability reflects how much confidence backs the model’s scored dimensions overall.

Current formula version: 1.0.0.

Ranking profiles

A ranking profile is a named, versioned set of dimension weights — never scattered constants inside the application.

Evidence levels

Every piece of evidence is tagged with how it was produced, so provenance is never hidden:

L1

Source

L2

Benchmark Result

L3

Expert Review

L4

Internal Test

L5

Reproducible Run

What’s kept separate

• Benchmark results and expert evaluations are never merged into one score.
• Community ratings are a separate signal and are not folded into official rankings.
• Sample/demo data is explicitly marked wherever it appears in the product.