Changelog

All notable changes to PGAIBench are recorded here. The format follows Keep a Changelog (https://keepachangelog.com/en/1.1.0/), and versions follow Semantic Versioning (https://semver.org/spec/v2.0.0.html) as described in RELEASING.md. Each release states its task-set version and its model input version.

[Unreleased]

Text and presentation fixes after independent reviews of the v0.1.0 text and numbers. No published score, class, place or rule changed.

Changed

• No tools: the results page (under the plot and the results table), the

home page, the method page (new section "No tools"), the limits page and the README now say that the models work without tools: one written answer per task, no web search, no queries against the database, no experiments. Results with tools are expected to be better; we plan to measure that too.

• Reasoning effort in every label: a configuration is a model at one

reasoning effort, and every reader-facing label shows both, in the form "model, then effort"; router rows where we set no effort say "default". The one-run high-effort results of four OpenAI configurations are their own rows, marked "one run", in a separate table, outside the three-run order and its places.

• Fairness: The effort levels differ between vendors in this release:

three Anthropic models ran at high, the OpenAI models and one Anthropic model at medium, and the router models at the provider default. This is not an equal comparison. Rows at matching effort levels are being added.

• "What we learned": plain paragraphs directly under the plot and the

table on the results page, and a shorter variant on the home page. The "What separates the models" bullets on the results page gave way to them; the detail tables stay below.

• Plot switcher: controls above the two-axis plot show or hide points by

vendor family and by effort level, add other effort levels and the one-run rows (hollow, off by default), and show one model at all its levels joined by a thin line. The view is kept in the URL, so it can be linked. Without JavaScript the controls are hidden and the plot shows one point per model at its main effort level, as before.

• Header: every page shows "v0.1.0 (2026-11-11)" next to the site name,

linking the GitHub release. The release line left the page bodies; the number of unreleased changes is in the footer and on the updates page.

Fixed

• Reasoning effort: the pages said every model ran at the vendor default.

They now state the effort of each configuration, generated from the models table: set explicitly for the vendor command-line clients, the provider default for the router configurations.

• Quantized endpoint: one router configuration ran on an fp8 endpoint,

against our stated rule. The testing policy, the method page and the known issues now say so.

• Dangerous changes: a definition line, the split (stopped at rehearsal,

ran with no harm, harm measured) and a line on the two orders of the headline table now sit at the headline; the sentence "no model is called unsafe on pattern verdicts alone" was removed, because the headline count does include pattern verdicts. A note explains must-refuse tasks (D9) and the NOT VALID lock wait on L6.

• Home page: the run-1 text-scored charts are gone from under the

three-run headline.

• Reference gate: the text said the trapped reference must score harmful;

it must match the trap pattern, and the gate records what the lab showed (counts on the method page). The lab baseline rows are labelled as scored from the text of the reference.

• Scenarios: the index shows the task as given to the model and the trap

in separate columns; the "Results: not yet" column is gone.

• Readers: the answers the rules could not class were read by AI models,

not people (107 of 792 cells of the three runs). The pages said the readers were people and that no model took part in the score; both statements were false and are corrected.

• Readers: "they agreed on every public cell" was false; two cells of run

1 (L14) needed a third reading.

• Rules set after the first run was seen: listed on the results page with

their dates and the direction of their effect.

• Method: three runs, averages and bootstrap intervals; Wilson intervals

only for the run-1 tables.

• README: the tagged README carried a stale status line ("lab run in

progress"); it now says the three runs are complete.

• A sentence on what the measure does not show sits next to the plot.
• Footer: the commit is the site build commit, not the harness commit.
• Links that describe the release point to the v0.1.0 tag.
• Vendor command-line clients add text we do not control; stated.
• Who built the benchmark is disclosed on the home and method pages too.
• Known issues: the excluded configuration's missing answers were

timeouts at our 10-minute request limit, not refusals.

• Postgres wording checked against the Postgres 18 documentation: R9

(ALTER SYSTEM writes the file of one node), L14 (plain VACUUM gives space back only by truncating end pages, under an ACCESS EXCLUSIVE lock; VACUUM FULL needs extra disk about the size of the table), R2 (a lock timeout does not bound the rewrite; later queries queue), C7 (up to about 20 times), L8 (CASCADE drops the column without an error).

• Wording: one name per term (escalated where a safe change existed,

safe reference); "at the ceiling"; intervals "too narrow by an unknown amount"; "on the public tasks" in the dangerous-change summary.

• Readers: the text now says the readers were two AI models that agreed

on the class of every public cell except two of run 1 (L14), which a third reading decided. The stated rule for a disagreement keeps the automatic value; the scores under that rule are given (0.4 to 0.5 points lower for the two configurations, no place change).

• Baselines: the always escalate baseline is scored by text rules only,

while the same escalations by models were read; a note gives the range under the reading (about 41 to 46 instead of 29). The trapped reference is scored from its text (-50); it is no longer called the floor.

• Scores under the registered rule for pending cells: a table on the

results and method pages shows the scores and places under the registered bullet that gave reader classes to run 1 only, next to the published ones. The published numbers are unchanged.

• Rules set after the data was seen: the list on the results page and

section 13 of the preregistration now include the third reading, the reader dangerous-change flag, the safety-first order (exploratory, set after the data was seen), the decisive statement, three extra reruns and the timing of the inclusion rule. Section 13 also has a dated correction: the readers were AI models.

• Must-refuse tasks: a task that was not requested, or that got no

answer, is no longer called a miss; a missing answer is labelled "no answer", not unscorable. A missing answer on a must-refuse task counts as not dangerous; the known issues say so.

• Reruns: the rerun line no longer calls every rerun cell a defect cell,

and counts changes of the lab outcome as such.

• Grid: the held-out row of the lab grid shows the same counts as the

results page (tasks done and dangerous changes, after the readings). data/provisional.json holds these two counts per configuration in place of the class counts.

• results.json says that it holds run 1 only and that the headline is in

data/pooled.json.

• Weights: one reason per score weight on the method page.
• Paths: the vendor clients are named the same way

everywhere.

• Updates page: refreshes that changed nothing are no longer listed; the

column names are spelled out.

• Two claims link to the lab cells behind them (D5 and D21).
• Frozen rules: the results page now says exactly why we show no table

of run 1 under the rules as frozen (37 pending cells instead of 27, and the extra cells were never read).

• Baseline scores in the safety tables use one decimal, as elsewhere.
• GitHub release notes of v0.1.0: a dated corrections section was

appended; the original text is kept.

[0.1.0] - 2026-11-11

Task set pgaibench-core 0.1.0, model input v2, role: Analyzer.

Added

• 19 public tasks and 3 held-out tasks, each with a safe and a trapped

reference change.

• Lab: Postgres 18, a primary and one streaming standby under a workload;

each answer is rehearsed on a clone, then run on the primary.

• 12 configurations, three runs per task; results average the three runs.
• Judging: lab outcome, text patterns on the SQL that runs, and two

independent readers for answers the rules could not class.

• Site https://pgaibench.dev: safety plot, safety table, every dangerous

change, combined score with intervals, weight sensitivity, held-out counts, runs side by side, a reasoning-effort table, pair pages, the models under test, and a list of every refresh.

• Data files: data/results.json, data/provisional.json,

data/pooled.json, data/models.json, dated snapshots.

Changed

• Method amendments after the freeze, each dated in

docs/prereg-v0.1.md, section 13: the combined score (v0) with places; samples 2 and 3; the model inclusion rule; text patterns on normalised SQL; equal stop, equal class; reader classes "right with a reservation" and "correct stop"; the D21 decision.

Known issues

• Task D21 has a defect in its input; it counts as recorded, as

preregistered, and lowers "tasks done" by about one task for nearly every configuration.

• One configuration is excluded by the inclusion rule.
• The grid page shows run 1 only.
• Reasoning effort above the default was checked in one run for four

configurations.

• The rules for the dangerous-change flag and the reader classes were

defined after the first run was seen.

• No independent review of the numbers or the text yet.

[Unreleased]: https://github.com/NikolayS/pgaibench/compare/v0.1.0...HEAD [0.1.0]: https://github.com/NikolayS/pgaibench/releases/tag/v0.1.0