Testing policy

How we pick models, which settings we use, and who sees what.

How we choose models

We test models from three vendors (OpenAI, Anthropic, Google) and open-weights models. The list, with the provider pin of each model, is in https://github.com/NikolayS/pgaibench/blob/v0.1.0/models.json. Models run in three access tracks: two vendor command-line clients with tools removed, and one router with pinned providers. The preregistered tables keep the tracks apart; the rating, added after the freeze, lists all configurations in one order. In the router track we pin the provider, refuse fallbacks and record the provider that served each call. Our rule is also to refuse quantized variants. deepseek-v4-pro · default ran on a quantized (fp8) endpoint of its pinned provider. This breaks our own rule to refuse quantized variants, and it may lower its results: they may not represent the model at full precision.

Settings

• We set the reasoning effort explicitly for the configurations that run through a vendor command-line client on a subscription, because those clients need an explicit setting; we set the value we recorded as the vendor's default for that model: gpt-6-astra: medium, gpt-6.1-sol: medium, gpt-6.1-terra: medium, gpt-6-luna: medium, claude-fable-5.2: high, claude-opus-5.5: high, claude-sonnet-5.5: high, claude-haiku-5.5: medium. For the router configurations we did not set an effort, so the provider default applied: gemini-3.8-flash, gemini-4-argon, deepseek-v4-pro, qwen3.8-27b. The settings are not equal across vendors. The Anthropic configurations ran at different efforts (high and medium), so a comparison between them is partly a comparison of effort. We did not set the temperature.
• One system prompt for all models, published in the repository.
• Three runs per scenario and model, averaged on the results page. The answers of the first run are published cell by cell; the answers of runs 2 and 3 are not published yet (each file needs a check for held-out content first).

Access and fairness

• No vendor gets early access to the held-out set. The held-out set is private and is not in the repository.
• No vendor sees results before they are published.
• Any vendor can dispute a cell: prompts, answers and lab measurements are public per cell.

Keep this out of training data

Every page and every file in the repository carries the canary sentence in the footer. Please do not train models on this data, and do not post solutions to services that train on user content. If a model can reproduce the canary GUID, its training data probably contains this benchmark.