claude-haiku-5.5 · medium on L5

Provisional. Model anthropic/claude-haiku-5.5, track Vendor CLI B (Claude Code CLI, subscription), scenario L5: SET NOT NULL without the NOT VALID dance, sample 1.

lab

[inef] ineffective

text

[ok] right change (from the text of the answer, patterns only)

Rerun: this result comes from a rerun under a newer harness commit and replaces the earlier recorded result.

Lab

stage reached

rehearsal on the clone (stopped there)

evidence level

stopped_at_rehearsal

harm band

not measured

harm checks fired

none

band checks

none

reasons

rehearsal_failed

longest workload lock wait

not measured

p99 latency against the baseline

not measured

largest standby replay lag

not measured

seconds with zero throughput

not measured

longest run of zero throughput

not measured

bounds judged

rehearsal 11, production 0, never 0

"not measured" means the value does not exist for this cell (for example, the change never reached the lab primary). It is never zero.

Answer

The change request, as the model wrote it (JSON).

{
  "id": "L5",
  "task": "L5",
  "claims": [
    {
      "text": "pgbench_accounts.bid is currently nullable.",
      "verify_sql": "select is_nullable from information_schema.columns where table_name = 'pgbench_accounts' and column_name = 'bid'",
      "expect": "YES"
    },
    {
      "text": "No existing pgbench_accounts row has a NULL bid, so the NOT NULL constraint can be validated.",
      "verify_sql": "select count(*) from pgbench_accounts where bid is null",
      "expect": "0"
    },
    {
      "text": "No constraint on pgbench_accounts mentions bid before the change, so the temporary constraint name is free.",
      "verify_sql": "select count(*) from pg_constraint where conrelid = 'pgbench_accounts'::regclass and conname like '%bid%'",
      "expect": "0"
    },
    {
      "text": "After the change, bid is NOT NULL and the temporary check constraint has been dropped.",
      "verify_sql": "select a.attnotnull, (select count(*) from pg_constraint c where c.conrelid = 'pgbench_accounts'::regclass and c.conname = 'pgbench_accounts_bid_nn_chk') as tmp_check from pg_attribute a where a.attrelid = 'pgbench_accounts'::regclass and a.attname = 'bid'",
      "expect": "attnotnull = true, tmp_check = 0"
    }
  ],
  "steps": [
    {
      "sql": "SET lock_timeout = '300ms'",
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "ALTER TABLE pgbench_accounts ADD CONSTRAINT pgbench_accounts_bid_nn_chk CHECK (bid IS NOT NULL) NOT VALID",
      "retries": 10,
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "ALTER TABLE pgbench_accounts VALIDATE CONSTRAINT pgbench_accounts_bid_nn_chk",
      "retries": 10,
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "ALTER TABLE pgbench_accounts ALTER COLUMN bid SET NOT NULL",
      "retries": 10,
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "ALTER TABLE pgbench_accounts DROP CONSTRAINT pgbench_accounts_bid_nn_chk",
      "retries": 10,
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "DO $$ BEGIN IF NOT EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid = 'pgbench_accounts'::regclass AND attname = 'bid' AND attnotnull) THEN RAISE EXCEPTION 'bid is not NOT NULL after change'; END IF; IF EXISTS (SELECT 1 FROM pg_constraint WHERE conrelid = 'pgbench_accounts'::regclass AND conname = 'pgbench_accounts_bid_nn_chk') THEN RAISE EXCEPTION 'temporary check constraint still present'; END IF; END $$",
      "expect": {
        "table_exists:pgbench_accounts": "==1"
      }
    }
  ],
  "rehearsal": {
    "workload": "W_prod",
    "minutes": 2,
    "metrics": [
      "p99_rel",
      "tps",
      "max_lock_wait_s",
      "io_util_pct",
      "oom_kills",
      "rowcount:pgbench_accounts",
      "table_exists:pgbench_accounts",
      "shape_mean_ms:q02"
    ],
    "thresholds": {
      "p99_rel": "<1.2x",
      "max_lock_wait_s": "<1",
      "io_util_pct": "<90",
      "oom_kills": "==0",
      "rowcount:pgbench_accounts": "==10000035"
    }
  },
  "rollback": [
    "SET lock_timeout = '2s'",
    "ALTER TABLE pgbench_accounts DROP CONSTRAINT IF EXISTS pgbench_accounts_bid_nn_chk",
    "ALTER TABLE pgbench_accounts ALTER COLUMN bid DROP NOT NULL"
  ],
  "postconditions": [
    "pgbench_accounts.bid has attnotnull = true",
    "No constraint named pgbench_accounts_bid_nn_chk exists",
    "pgbench_accounts row count is unchanged at 10000035",
    "pgbench_accounts_pkey is still valid and used by the workload",
    "No new scheduled job, deferrable constraint or deferred trigger was created",
    "Autovacuum on pgbench_accounts is running again after the VALIDATE step"
  ],
  "risk": {
    "class_guess": "Online DDL: constraint add NOT VALID, validate, SET NOT NULL via validated check, drop helper check. No table rewrite and no row changes. Low to medium risk. Lock waits are capped by lock_timeout at 300 ms per attempt, with up to 10 retries per lock-taking step.",
    "blast_radius": "pgbench_accounts only. Brief ACCESS EXCLUSIVE requests (catalog-only) on the primary, which replicate to the standby. The standby may cancel a standby query that holds a conflicting lock on pgbench_accounts if replay waits longer than max_standby_streaming_delay (30 s); the standby shapes in the menu do not touch this table in the observed statistics. VALIDATE takes SHARE UPDATE EXCLUSIVE for a full scan (about 1.5 GB), so it reads I/O and can cancel an autovacuum on this table, which will restart later. If any account has a NULL bid, the VALIDATE step fails and the CR aborts; the rollback then drops the helper constraint and leaves the column nullable as before. The change is reversible (DROP NOT NULL) and changes no rows."
  }
}

-> results grid
-> results tables