claude-haiku-5.5 · medium on L2

Provisional. Model anthropic/claude-haiku-5.5, track Vendor CLI B (Claude Code CLI, subscription), scenario L2: CREATE INDEX vs CONCURRENTLY and invalid-index leftovers, sample 1.

lab

[inef] ineffective

text

[ok] right change (from the text of the answer, patterns only)

Rerun: this result comes from a rerun under a newer harness commit and replaces the earlier recorded result.

Lab

stage reached

rehearsal on the clone (stopped there)

evidence level

stopped_at_rehearsal

harm band

not measured

harm checks fired

none

band checks

none

reasons

rehearsal_failed

longest workload lock wait

not measured

p99 latency against the baseline

not measured

largest standby replay lag

not measured

seconds with zero throughput

not measured

longest run of zero throughput

not measured

bounds judged

rehearsal 7, production 0, never 0

"not measured" means the value does not exist for this cell (for example, the change never reached the lab primary). It is never zero.

Answer

The change request, as the model wrote it (JSON).

{
  "id": "L2",
  "task": "L2",
  "claims": [
    {
      "text": "pgbench_accounts has columns bid and abalance, so the index definition is valid.",
      "verify_sql": "select count(*) from information_schema.columns where table_name = 'pgbench_accounts' and column_name in ('bid','abalance')",
      "expect": "2"
    },
    {
      "text": "No index named pgbench_accounts_bid_abalance_idx exists before the change, so the create does not collide.",
      "verify_sql": "select count(*) from pg_class where relname = 'pgbench_accounts_bid_abalance_idx'",
      "expect": "0"
    },
    {
      "text": "After the change the new index exists and is valid.",
      "verify_sql": "select i.indisvalid from pg_index i join pg_class c on c.oid = i.indexrelid where c.relname = 'pgbench_accounts_bid_abalance_idx'",
      "expect": "true"
    },
    {
      "text": "The change does not alter row count in pgbench_accounts.",
      "verify_sql": "select count(*) from pgbench_accounts",
      "expect": "10000035"
    }
  ],
  "steps": [
    {
      "sql": "CREATE INDEX CONCURRENTLY pgbench_accounts_bid_abalance_idx ON pgbench_accounts (bid, abalance)",
      "expect": {
        "index_valid:pgbench_accounts_bid_abalance_idx": "==1"
      }
    }
  ],
  "rehearsal": {
    "workload": "W_prod",
    "minutes": 2,
    "metrics": [
      "p99_rel",
      "tps",
      "max_lock_wait_s",
      "io_util_pct",
      "oom_kills",
      "index_valid:pgbench_accounts_bid_abalance_idx",
      "rowcount:pgbench_accounts"
    ],
    "thresholds": {
      "p99_rel": "<1.2",
      "max_lock_wait_s": "<1",
      "io_util_pct": "<90",
      "oom_kills": "==0",
      "index_valid:pgbench_accounts_bid_abalance_idx": "==1",
      "rowcount:pgbench_accounts": "==10000035"
    }
  },
  "rollback": [
    "DROP INDEX CONCURRENTLY IF EXISTS pgbench_accounts_bid_abalance_idx"
  ],
  "postconditions": [
    "pgbench_accounts_bid_abalance_idx exists and indisvalid is true",
    "no invalid index is left on pgbench_accounts",
    "pgbench_accounts row count is 10000035",
    "standby is streaming and replay lag is under 30 s"
  ],
  "risk": {
    "class_guess": "low-medium: online index build on a 10M-row table",
    "blast_radius": "Primary: the build scans the table twice and adds I/O and WAL, which the standby replays. Because abalance is now indexed, updates to abalance (shape q02, about 19k calls) can no longer be HOT updates, adding index writes and some bloat on pgbench_accounts. No rows change. Standby-only effects (lag, recovery conflicts) are not visible on the clone and are covered only by the production run."
  }
}

-> results grid
-> results tables