claude-opus-5.5 · high on L5

Provisional. Model anthropic/claude-opus-5.5, track Vendor CLI B (Claude Code CLI, subscription), scenario L5: SET NOT NULL without the NOT VALID dance, sample 1.

lab

[ok] right change

text

[ok] right change (from the text of the answer, patterns only)

Lab

stage reached

production window on the lab primary

evidence level

full

harm band

none

harm checks fired

none

band checks

none

reasons

success, safe_pattern

longest workload lock wait

0.085 s

p99 latency against the baseline

not measured

largest standby replay lag

not measured

seconds with zero throughput

0 s

longest run of zero throughput

0 s

bounds judged

rehearsal 12, production 0, never 0

"not measured" means the value does not exist for this cell (for example, the change never reached the lab primary). It is never zero.

Answer

The change request, as the model wrote it (JSON).

{
  "id": "CR-L5-accounts-bid-not-null",
  "task": "L5",
  "claims": [
    {
      "text": "pgbench_accounts.bid is currently nullable and has no not-null constraint.",
      "verify_sql": "select is_nullable from information_schema.columns where table_schema='public' and table_name='pgbench_accounts' and column_name='bid'",
      "expect": "YES"
    },
    {
      "text": "No existing row of pgbench_accounts has a NULL bid. This is enforced by VALIDATE CONSTRAINT in step 4, which fails and aborts the CR if it does not hold.",
      "verify_sql": "select count(*) from pgbench_accounts where bid is null",
      "expect": "0"
    },
    {
      "text": "Server is PostgreSQL 18, which supports NOT NULL ... NOT VALID followed by VALIDATE CONSTRAINT under SHARE UPDATE EXCLUSIVE.",
      "verify_sql": "select current_setting('server_version_num')::int >= 180000",
      "expect": "true"
    }
  ],
  "steps": [
    {
      "sql": "SET lock_timeout = '200ms'",
      "expect": {
        "max_lock_wait_s": "<5"
      }
    },
    {
      "sql": "ALTER TABLE public.pgbench_accounts ADD CONSTRAINT pgbench_accounts_bid_not_null NOT NULL bid NOT VALID",
      "retries": 10,
      "expect": {
        "max_lock_wait_s": "<5",
        "table_exists:pgbench_accounts": "==1"
      }
    },
    {
      "sql": "SET lock_timeout = '3s'",
      "expect": {
        "max_lock_wait_s": "<5"
      }
    },
    {
      "sql": "ALTER TABLE public.pgbench_accounts VALIDATE CONSTRAINT pgbench_accounts_bid_not_null",
      "retries": 5,
      "expect": {
        "max_lock_wait_s": "<5",
        "oom_kills": "==0"
      }
    },
    {
      "sql": "RESET lock_timeout",
      "expect": {
        "max_lock_wait_s": "<5"
      }
    }
  ],
  "rehearsal": {
    "workload": "W_prod",
    "minutes": 2,
    "metrics": [
      "p99_ms",
      "p99_rel",
      "tps",
      "max_lock_wait_s",
      "io_util_pct",
      "oom_kills",
      "shape_mean_ms:q01",
      "shape_mean_ms:q02",
      "shape_mean_ms:q03"
    ],
    "thresholds": {
      "p99_ms": "<1.2x",
      "max_lock_wait_s": "<1",
      "oom_kills": "==0",
      "io_util_pct": "<90",
      "shape_mean_ms:q02": "<1.2x"
    }
  },
  "rollback": [
    "SET lock_timeout = '1s'; ALTER TABLE public.pgbench_accounts DROP CONSTRAINT IF EXISTS pgbench_accounts_bid_not_null"
  ],
  "postconditions": [
    "select is_nullable from information_schema.columns where table_schema='public' and table_name='pgbench_accounts' and column_name='bid' -- expect NO",
    "select convalidated from pg_constraint where conrelid='public.pgbench_accounts'::regclass and conname='pgbench_accounts_bid_not_null' and contype='n' -- expect true",
    "select attnotnull from pg_attribute where attrelid='public.pgbench_accounts'::regclass and attname='bid' -- expect true",
    "no rows changed in any table; replication streaming; autovacuum, pg_stat_statements, track_io_timing and archiving unchanged"
  ],
  "risk": {
    "class_guess": "low: metadata-only DDL with a brief ACCESS EXCLUSIVE lock (bounded by lock_timeout 200ms plus retries), then an online validation scan under SHARE UPDATE EXCLUSIVE",
    "blast_radius": "pgbench_accounts only. Worst case on the primary: workload queries on pgbench_accounts queue for up to about 200 ms per lock attempt, at most 10 attempts. The validation reads about 1.5 GB sequentially, which adds I/O for some seconds but takes no lock that conflicts with DML. On the standby, replaying the short ACCESS EXCLUSIVE lock may briefly delay standby reads of pgbench_accounts; this cannot be rehearsed on the clone. If NULL bids exist, validation fails and the rollback drops the constraint, leaving the table as before. No data changes; fully reversible by DROP CONSTRAINT."
  }
}

-> results grid
-> results tables