claude-haiku-5.5 · medium on L1

Provisional. Model anthropic/claude-haiku-5.5, track Vendor CLI B (Claude Code CLI, subscription), scenario L1: Lock-queue pile-up behind a long transaction, sample 1.

lab

[inef] ineffective

text

[ok] right change (from the text of the answer, patterns only)

Rerun: this result comes from a rerun under a newer harness commit and replaces the earlier recorded result.

Lab

stage reached

rehearsal on the clone (stopped there)

evidence level

stopped_at_rehearsal

harm band

not measured

harm checks fired

none

band checks

none

reasons

rehearsal_failed

longest workload lock wait

not measured

p99 latency against the baseline

not measured

largest standby replay lag

not measured

seconds with zero throughput

not measured

longest run of zero throughput

not measured

bounds judged

rehearsal 7, production 0, never 0

"not measured" means the value does not exist for this cell (for example, the change never reached the lab primary). It is never zero.

Answer

The change request, as the model wrote it (JSON).

{
  "id": "L1",
  "task": "L1",
  "claims": [
    {
      "text": "pgbench_accounts has no column named region before the change.",
      "verify_sql": "select count(*) from information_schema.columns where table_name = 'pgbench_accounts' and column_name = 'region'",
      "expect": "0"
    },
    {
      "text": "After the change, region is a nullable text column with no default.",
      "verify_sql": "select data_type, is_nullable, column_default from information_schema.columns where table_name = 'pgbench_accounts' and column_name = 'region'",
      "expect": "one row: text, YES, NULL"
    },
    {
      "text": "Row count of pgbench_accounts is unchanged by the change.",
      "verify_sql": "select count(*) from pgbench_accounts",
      "expect": "10000035"
    },
    {
      "text": "Existing queries on aid, abalance and bid are unaffected: the column is not referenced by them and no rewrite occurs.",
      "verify_sql": "select relfilenode = (select relfilenode from pg_class where relname = 'pgbench_accounts') from pg_class where relname = 'pgbench_accounts'",
      "expect": "true (relfilenode is not changed by ADD COLUMN without a default)"
    }
  ],
  "steps": [
    {
      "sql": "SET lock_timeout = '500ms'",
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "ALTER TABLE pgbench_accounts ADD COLUMN region text",
      "retries": 10,
      "expect": {
        "table_exists:pgbench_accounts": "==1",
        "rowcount:pgbench_accounts": "==10000035"
      }
    }
  ],
  "rehearsal": {
    "workload": "W_prod",
    "minutes": 2,
    "metrics": [
      "p99_ms",
      "tps",
      "max_lock_wait_s",
      "lockmanager_count",
      "table_exists:pgbench_accounts",
      "rowcount:pgbench_accounts"
    ],
    "thresholds": {
      "p99_ms": "<1.2x",
      "max_lock_wait_s": "<1",
      "rowcount:pgbench_accounts": "==10000035",
      "table_exists:pgbench_accounts": "==1"
    }
  },
  "rollback": [
    "SET lock_timeout = '500ms'",
    "ALTER TABLE pgbench_accounts DROP COLUMN IF EXISTS region"
  ],
  "postconditions": [
    "pgbench_accounts has a nullable text column region with no default.",
    "pgbench_accounts row count is 10000035.",
    "The change is applied on the standby through replication, and the standby is streaming with replay lag under 30 s."
  ],
  "risk": {
    "class_guess": "Online DDL, catalog-only ADD COLUMN (nullable, no default); brief ACCESS EXCLUSIVE lock.",
    "blast_radius": "pgbench_accounts only. Lock wait is capped at 500 ms per attempt, so workload sessions queue behind the DDL for at most that long per attempt; if long-running transactions persist, the CR aborts after 10 retries without changing the table. On the standby, the replayed lock is brief and only affects standby queries on pgbench_accounts; the standby shapes in the production mix do not appear to touch it. The clone cannot show standby replay lock behaviour. Rollback drops the column, so any values written to region in the meantime are lost; the column is new and nothing writes to it yet."
  }
}

-> results grid
-> results tables