Related workpgaibench(1) / related-work | Version | pgaibench-core 0.1.0 | |
|---|---|---|---|
| Lab harness | 913fc43, reruns bf55cef | ||
| Site build | 15f9e5f541b2 | ||
| Release | v0.1.0 (2026-11-11) | ||
| Updated | 2026-11-12T00:04:11Z UTC | ||
| Lab cells | 263 of 264 finished | ||
Benchmarks and papers we learned from, and how PGAIBench differs from each. Every link goes to the primary source.
| Work | Link | What it measures | What we took | How we differ |
|---|---|---|---|---|
| DBA-Bench | arXiv 2607.22165 | DB operations agents on live Postgres and TiDB with workloads; multi-turn tools; safety rules on the action trace; human DBA baseline | closest prior art; safe-pass headline (goal reached and no harm); failure labels by protected object, scope and missing safeguard | refusal scored in both directions, a streaming standby in every lab, a gate with a known-trapped reference, Wilson intervals. Weaker in v0.1: single shot, fewer tasks, no human baseline |
| Terminal-Bench | repo, docs | shell tasks in containers | oracle solution per task, task directory, versioned set, canary string | we run the oracle as a gate and add a trapped reference |
| SWE-bench Verified | site | coding agents on a human-validated subset | validated-set statement | we validate by execution in the lab |
| Spider 2.0 | arXiv 2411.07763 | text-to-SQL on enterprise workflows | tasks from real use | changes on a live system, not queries |
| BIRD-CRITIC | site | fixing SQL from real user issues, including Postgres | tasks from real use | harm under load, not result correctness |
| AI Control | arXiv 2312.06942 | protocols against an untrusted model; usefulness and safety measured separately | two axes, never one blended score | we test honest mistakes, not deliberate attack |
| ControlArena | repo | control settings with honest and attack modes | safety and usefulness metrics | no attack mode in v0.1 |
| BashArena | arXiv 2512.15688 | control setting with sysadmin tasks | privileged infrastructure tasks | Postgres only, accidental harm |
| SHADE-Arena | arXiv 2506.15740 | sabotage side tasks and LLM monitors | the honest-against-attack distinction | no monitor in the score |
| ToolEmu | arXiv 2309.15817 | risky tool use in an emulated sandbox | severity of harm (mild or severe) | real execution, not emulation |
| R-Judge | arXiv 2401.10019 | risk awareness over agent records | risk taxonomy | harm measured, not judged |
| AgentHarm | arXiv 2410.09024 | harmful agent tasks with a benign twin set | refusal in both directions | correct refusals and escalations where a safe change existed |
| OS-Harm | arXiv 2506.14866 | computer-use agent safety; judge agreement with humans | validation of LLM readers against human labels | our readers are AI models too; we have not yet checked them against human labels |
| AIOpsLab | arXiv 2501.06706 | detection, localization, analysis and mitigation on live microservices with fault injection | task levels, steps and tokens per task | one database; diagnosis comes later |
| ITBench | arXiv 2502.05352 | SRE, CISO and FinOps scenarios with fault injection | real-incident provenance | one database, deeper checks |
| Adding Error Bars to Evals | arXiv 2411.00640 | statistics for model evals | confidence intervals, clustered errors | intervals per model; we use Wilson intervals and paired tests |
| LiveBench | arXiv 2406.19314 | contamination-limited questions, regular updates | refresh cadence | refresh per release |
| MAST | arXiv 2503.13657 | failure taxonomy from annotated traces | failure profile per model | taxonomy of Postgres mistakes |
A longer comparison with DBA-Bench is in https://github.com/NikolayS/pgaibench/blob/v0.1.0/docs/prior-art-dba-bench.md.