model fatıgue

How we work

Every number has an owner and a date

Each figure on this site says whose it is (Anthropic's, Artificial Analysis's, a named person's, or ours) and links to where it came from, with the time we read it. When we tested something ourselves, the figure says so and gives the number of runs, the setting each run used and the date.

Every video page has a numbers table with all of this, and the same data as JSON and CSV.

Whose numbers they are

Our launch-day videos mostly report other people's measurements. When that is all we have, the page says so near the top, and every number names who measured it. A number of ours only appears once we have run the test, and the page marks it as ours.

Numbers that can move

Some figures change after a video is made, for example when Artificial Analysis re-runs a model on a fixed build. Those carry a ⟳ mark. When we read one again and it has moved, the page shows the value from the video next to the new one, both dated. We don't overwrite what the video said.

What we run on

Our own tests run on subscription plans, through Claude Code and Codex, at each tool's default settings unless the page says otherwise. Where we give a dollar cost for our runs, it is the notional cost at API list prices, not money we were billed.

Colours

Yellow marks the number the sentence is about. It never means better or cheaper. Grey marks the other data beside it, and it never means worse. BB, the lilac bar, is not data unless he is labelled as a bar in a chart.

When we're wrong

We list every change after publication on the corrections page, newest first, and say what kind of change it was.

Articles

Articles are written pieces with no video. Each one answers a single question about which model to use, at what effort or at what cost, and each has to add something its sources don't already say: arithmetic across sources, a dated comparison, a follow-up on a claim, or a measurement of ours.

A model writes them. The byline on each article names the model that wrote it, read from the run that produced it. Every figure goes through the same numbers table as the video pages, and the site's build refuses a number that has no source behind it.

Before an article is published, a second model that hadn't seen it being written checks every claim against the saved sources and nothing else. Its verdict for each claim is kept with the article, and an article isn't published while any claim stands as wrong or unsupported. Corrections after publication go on the corrections page, the same as for videos.