How we score AI provider reliability
The layered trust signal model behind every routing decision on Onlist: objective fulfillment, automated checks, and real-usage reviews.
A unified AI API has one job that flat REST APIs do not: pick a provider on every single call. If the cheapest provider is also the most reliable, the choice is easy. In practice they are almost never the same provider on the same day. Routing is where the gateway earns its keep, and routing is only as good as the signals it ranks providers by.
This post documents the layered trust signal model Onlist uses today. It is the framework behind every routing decision, every red flag shown on a provider's page, and every weight applied to the next request your application makes.
The four signals
We rank providers by four kinds of evidence. They are listed in order of how heavily routing weighs them, and how hard each one is to fake.
1. Objective fulfillment: the backbone
The primary signal is whether a provider successfully completed the calls we sent them. This is fully objective: every call settled through Onlist is logged with its outcome (success, upstream error, timeout, refused). A provider's recent fulfillment rate is computed from the last N hours of real traffic.
When a provider's fulfillment rate dips below threshold, that provider enters a cooling state. Cooling providers stay routable but receive lower routing weight for a configurable window. The buyer-facing listing shows a cooling badge so callers can decide for themselves whether to pin to a specific competing provider.
This signal is impossible for a provider to game without genuinely improving. A provider cannot fake having served thousands of successful calls.
2. Automated content checks: the red flag detector
We periodically run blind detection probes against Claude-family providers to check whether the upstream is genuinely the model it claims. The probes are a usage-pattern detector, not an authenticity certifier. They reliably flag certain failure modes (e.g. silently routing a request for Claude to a smaller open model) but cannot certify that a passing provider is in fact running the claimed model.
Because the probes have known false negatives, a passing check is not displayed as a positive trust signal. A failing check is a negative red flag that contributes to a provider's risk score.
We chose this asymmetry deliberately. Showing a green "verified" badge on a provider we cannot fully verify would be worse than showing nothing.
3. Real-usage reviews: the tiebreaker
Buyers who actually paid for calls on a provider can leave a rating and review. The rating is anchored to settled usage; there is no path to leave a review without a billed history. This eliminates the most common rating-fraud vector but does not eliminate small-account self-rating from a provider's own team.
Rating is treated as a tiebreaker among providers with similar fulfillment numbers. It does not override objective fulfillment metrics. A 5-star provider with degraded fulfillment is routed less than a 4-star provider in good standing.
The rating exists for buyer visibility more than for routing weight. The marketplace is more honest if buyers can see what other paying buyers think.
4. External reputation: currently absent
A complete trust model would weigh provider history outside the platform: incident reports, public statements, behavior on competing marketplaces. Onlist does not consume this signal yet. We treat its absence as a known gap, not a design choice.
Why this is layered, not a single score
The temptation is to roll all four signals into one number and rank providers by it. The reason we don't:
The signals have very different operational properties. Fulfillment is real-time and continuous. Detection probes are sparse and asymmetric. Reviews are slow and biased. External reputation is qualitative. Smoothing them into a single weighted average hides the differences from the buyer.
What we do instead: route by fulfillment first, surface cooling state and red flags second, expose ratings third. A buyer who pins to a specific provider sees all four pieces of evidence separately and decides for themselves.
What changes when
The four-signal model is the framework, not a fixed product. Specific routing weights, cooling thresholds, and badge presentation evolve as we accumulate more real data and as failure modes are exposed.
When a routing rule changes, that change is documented in Provider routing. When the detection probe coverage expands, that is logged in our changelog.
If you want the short version: Onlist trusts what the platform itself observed (calls, settlements, probes) more than what providers claim about themselves, and lets buyers see all of it.