{ "version": 1, "source": "ExoBench session 2026-09-11 (server 1.4.5-dirty, PostgreSQL 17.11), seven benchmarkSql calls with repetitions=12; correlationIds 1361ff10-a37c-4171-b0a3-1d0b007d2b33, 3f6d7b30-a9d2-4ce5-813f-d8e0c5024f86, f2860651-2d28-4892-95c4-155c4d3f3d79, ed9367cd-3189-40be-9104-edaac048fd7d, 23f52039-aa25-4c2d-aa04-0e529a6a85f6, 8d194616-3abd-4c1b-a55e-e704475f1d9b, 1c831ae2-10b1-4484-8b3e-2decc1d39f93", "note": "Supplemental data for postgres-runs-your-query-fast-exactly-five-times.mdx. mcp-call result strings are byte-verbatim tool-result JSON from a 2026-09-11 rerun of the E1 arm of the pitch-finder repetitions variance hunt (EXOBENCH_REPETITIONS_VARIANCE_HUNT_PLAN.md); the original 2026-09-10 session's results were not persisted by Cursor, so every call was re-issued with the arguments recorded in that session's transcript. Inputs are the reproduction payloads as sent; the server minted a new correlationId per call and those are listed in source. Plans, timings, buffers and the plan-change banners are unedited; captions are authored. The chart's three series are transcribed from the flipSweep and flipAtScale cards.", "blocks": { "flipAtScale": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\n-- tenant 1 is a whale with ~40% of all orders; the other 9,999 tenants share the rest uniformly.\n-- The generic plan will estimate 1/n_distinct of the table for \"tenant_id = $1\".\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "0943dd7b-7a42-41cb-9575-1f7ec0cbbd73", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "PREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:39:11.413757568Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":39.889,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over each plan's runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nPlan changed across runs: 2 distinct plans; A: runs 1-5, B: runs 6-12 (n=12)\\n\\n--- Plan A (5 of 12 runs) ---\\nFinalize GroupAggregate (avg_cost=33117.71..33118.24 avg_rows=2 avg_width=47) (actual avg_time=265.827..273.620 stddev_time=4.153..3.575 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=14403\\n -> Gather Merge (avg_cost=33117.71..33118.17 avg_rows=4 avg_width=47) (actual avg_time=265.815..273.607 stddev_time=4.151..3.573 avg_rows=6 avg_loops=1)\\n Workers Planned: avg=2\\n Workers Launched: avg=2\\n Buffers: shared avg_hit=14403\\n -> Sort (avg_cost=32117.68..32117.69 avg_rows=2 avg_width=47) (actual avg_time=263.222..263.224 stddev_time=4.190..4.190 avg_rows=2 avg_loops=3)\\n Sort Key: c.tier\\n Sort Method: quicksort Memory: avg=25kB\\n Buffers: shared avg_hit=14403\\n -> Partial HashAggregate (avg_cost=32117.65..32117.67 avg_rows=2 avg_width=47) (actual avg_time=263.197..263.200 stddev_time=4.184..4.184 avg_rows=2 avg_loops=3)\\n Group Key: c.tier\\n Batches: avg=1 Memory Usage: avg=24kB\\n Buffers: shared avg_hit=14385\\n -> Parallel Hash Join (avg_cost=5592.59..29621.81 avg_rows=332778 avg_width=13) (actual avg_time=37.543..208.470 stddev_time=0.851..3.678 avg_rows=266767 avg_loops=3)\\n Hash Cond: (o.customer_id = c.id)\\n Buffers: shared avg_hit=14385\\n -> Parallel Seq Scan on orders o (avg_cost=0.00..23155.67 avg_rows=332778 avg_width=10) (actual avg_time=0.012..53.036 stddev_time=0.001..0.559 avg_rows=266767 avg_loops=3)\\n Filter: (tenant_id = 1)\\n Rows Removed by Filter: avg=399899\\n Buffers: shared avg_hit=12739\\n -> Parallel Hash (avg_cost=3386.71..3386.71 avg_rows=176471 avg_width=11) (actual avg_time=36.660..36.661 stddev_time=0.842..0.842 avg_rows=100000 avg_loops=3)\\n Buckets: avg=524288 Batches: avg=1 Memory Usage: avg=18227.2kB stddev=17.527kB\\n Buffers: shared avg_hit=1622\\n -> Parallel Seq Scan on customers c (avg_cost=0.00..3386.71 avg_rows=176471 avg_width=11) (actual avg_time=0.012..9.639 stddev_time=0.005..0.062 avg_rows=100000 avg_loops=3)\\n Buffers: shared avg_hit=1622\\nPlanning:\\n Buffers: shared avg_hit=31.6 stddev_hit=48.3\\nPlanning Time: avg=0.274 ms stddev=0.087 n=5\\nExecution Time: avg=273.678 ms stddev=3.571 n=5\\n\\n--- Plan B (7 of 12 runs) ---\\nGroupAggregate (avg_cost=2464.42..2466.61 avg_rows=2 avg_width=47) (actual avg_time=2125.969..2315.688 stddev_time=59.054..59.298 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=3214526.4 stddev_hit=255.5 avg_read=96.6 stddev_read=255.5, temp avg_read=2422 avg_written=2433\\n -> Sort (avg_cost=2464.42..2464.96 avg_rows=216 avg_width=13) (actual avg_time=2104.877..2193.463 stddev_time=59.074..59.395 avg_rows=800302 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: external merge Disk: avg=19376kB\\n Buffers: shared avg_hit=3214526.4 stddev_hit=255.5 avg_read=96.6 stddev_read=255.5, temp avg_read=2422 avg_written=2433\\n -> Nested Loop (avg_cost=6.52..2456.05 avg_rows=216 avg_width=13) (actual avg_time=13.876..1925.334 stddev_time=1.285..55.958 avg_rows=800302 avg_loops=1)\\n Buffers: shared avg_hit=3214526.4 stddev_hit=255.5 avg_read=96.6 stddev_read=255.5\\n -> Bitmap Heap Scan on orders o (avg_cost=6.10..785.01 avg_rows=216 avg_width=10) (actual avg_time=13.857..128.415 stddev_time=1.286..2.075 avg_rows=800302 avg_loops=1)\\n Recheck Cond: (tenant_id = $1)\\n Heap Blocks: avg_exact=12739\\n Buffers: shared avg_hit=13318.4 stddev_hit=255.5 avg_read=96.6 stddev_read=255.5\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..6.05 avg_rows=216 avg_width=0) (actual avg_time=12.477..12.477 stddev_time=1.300..1.300 avg_rows=800302 avg_loops=1)\\n Index Cond: (tenant_id = $1)\\n Buffers: shared avg_hit=579.4 stddev_hit=255.5 avg_read=96.6 stddev_read=255.5\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.42..7.74 avg_rows=1 avg_width=11) (actual avg_time=0.002..0.002 avg_rows=1 avg_loops=800302)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=3201208\\nPlanning:\\n Buffers: shared avg_hit=2.9 stddev_hit=4.9\\nPlanning Time: avg=0.077 ms stddev=0.113 n=7\\nExecution Time: avg=2318.664 ms stddev=59.318 n=7\\n\\n--- All runs (n=12) ---\\nPlanning Time: avg=0.159 ms stddev=0.142 n=12\\nExecution Time: avg=1466.587 ms stddev=1053.935 n=12\",\"runtimeDurationMs\":1471,\"rowsAreCapped\":false,\"computeSeconds\":39.889}],\"workerId\":\"64facb3a-c75e-4c75-8ace-7e7227eb93d5\",\"correlationId\":\"3f6d7b30-a9d2-4ce5-813f-d8e0c5024f86\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "Twelve EXPLAIN ANALYZE executions of one prepared statement on one connection at 2M orders: five on the custom plan, seven on the generic plan", "planCaptions": { "300K / 2M": "Runs 1 to 5 use the custom plan: a Parallel Hash Join with Filter (tenant_id = 1), 14,403 shared buffers, Execution Time avg 273.678 ms stddev 3.571. Runs 6 to 12 use the generic plan: a Nested Loop probing customers_pkey 800,302 times behind Recheck Cond (tenant_id = $1), an estimate of 216 rows against 800,302 actual, 3.2 million buffer hits and a 19,376 kB external merge sort, avg 2318.664 ms stddev 59.318. The all-runs mean of 1466.587 ms is a duration no single run had." } }, "warmedHidesIt": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "5", "correlationId": "1361ff10-a37c-4171-b0a3-1d0b007d2b33", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "PREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:43:07.278090642Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)\",\"warmCacheIterations\":5,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":37.442,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over 12 runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nGroupAggregate (avg_cost=2456.50..2458.68 avg_rows=2 avg_width=47) (actual avg_time=2214.297..2404.651 stddev_time=348.936..349.792 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=3213330.7 stddev_hit=195.1 avg_read=56.3 stddev_read=195.1, temp avg_read=2421 avg_written=2432\\n -> Sort (avg_cost=2456.50..2457.04 avg_rows=215 avg_width=13) (actual avg_time=2193.179..2281.912 stddev_time=348.846..349.233 avg_rows=799993 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: external merge Disk: avg=19368kB\\n Buffers: shared avg_hit=3213330.7 stddev_hit=195.1 avg_read=56.3 stddev_read=195.1, temp avg_read=2421 avg_written=2432\\n -> Nested Loop (avg_cost=6.52..2448.17 avg_rows=215 avg_width=13) (actual avg_time=13.770..1998.861 stddev_time=0.823..290.622 avg_rows=799993 avg_loops=1)\\n Buffers: shared avg_hit=3213330.7 stddev_hit=195.1 avg_read=56.3 stddev_read=195.1\\n -> Bitmap Heap Scan on orders o (avg_cost=6.09..781.57 avg_rows=215 avg_width=10) (actual avg_time=13.750..128.760 stddev_time=0.823..8.171 avg_rows=799993 avg_loops=1)\\n Recheck Cond: (tenant_id = $1)\\n Heap Blocks: avg_exact=12739\\n Buffers: shared avg_hit=13358.7 stddev_hit=195.1 avg_read=56.3 stddev_read=195.1\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..6.04 avg_rows=215 avg_width=0) (actual avg_time=12.366..12.366 stddev_time=0.810..0.810 avg_rows=799993 avg_loops=1)\\n Index Cond: (tenant_id = $1)\\n Buffers: shared avg_hit=619.7 stddev_hit=195.1 avg_read=56.3 stddev_read=195.1\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.42..7.75 avg_rows=1 avg_width=11) (actual avg_time=0.002..0.002 avg_rows=1 avg_loops=799993)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=3199972\\nPlanning:\\n Buffers: shared avg_hit=0.8 stddev_hit=2.9\\nPlanning Time: avg=0.032 ms stddev=0.073 n=12\\nExecution Time: avg=2407.614 ms stddev=349.825 n=12\",\"runtimeDurationMs\":2411,\"rowsAreCapped\":false,\"computeSeconds\":37.442}],\"workerId\":\"ec2700cb-b778-4604-9953-b76b4878ec75\",\"correlationId\":\"8d194616-3abd-4c1b-a55e-e704475f1d9b\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "Five warm-up executions before the 12 timed runs: the warm-up consumed the five custom plans, so every timed run is on the generic plan", "planCaptions": { "300K / 2M": "With warmCache = 5 the five custom-plan executions happen before timing starts, and all 12 timed runs show the generic Nested Loop with Recheck Cond (tenant_id = $1): 799,993 rows against an estimate of 215, 3,213,387 buffer hits, a 19,368 kB external merge sort. Execution Time avg 2407.614 ms stddev 349.825, and no plan change is reported because none happened inside the timed window." } }, "forceGeneric": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "1361ff10-a37c-4171-b0a3-1d0b007d2b33", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "SET plan_cache_mode = force_generic_plan;\nPREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:41:38.344253220Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":35.625,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over 12 runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nGroupAggregate (avg_cost=2476.34..2478.54 avg_rows=2 avg_width=47) (actual avg_time=2154.739..2344.475 stddev_time=27.784..27.878 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=3208001.8 stddev_hit=194.9 avg_read=56.3 stddev_read=194.9, temp avg_read=2417 avg_written=2428\\n -> Sort (avg_cost=2476.34..2476.89 avg_rows=217 avg_width=13) (actual avg_time=2133.707..2222.226 stddev_time=27.807..27.753 avg_rows=798661 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: external merge Disk: avg=19336kB\\n Buffers: shared avg_hit=3208001.8 stddev_hit=194.9 avg_read=56.3 stddev_read=194.9, temp avg_read=2417 avg_written=2428\\n -> Nested Loop (avg_cost=6.53..2467.92 avg_rows=217 avg_width=13) (actual avg_time=13.557..1952.888 stddev_time=0.859..25.758 avg_rows=798661 avg_loops=1)\\n Buffers: shared avg_hit=3208001.8 stddev_hit=194.9 avg_read=56.3 stddev_read=194.9\\n -> Bitmap Heap Scan on orders o (avg_cost=6.11..788.44 avg_rows=217 avg_width=10) (actual avg_time=13.539..126.950 stddev_time=0.858..1.604 avg_rows=798661 avg_loops=1)\\n Recheck Cond: (tenant_id = $1)\\n Heap Blocks: avg_exact=12739\\n Buffers: shared avg_hit=13357.8 stddev_hit=194.9 avg_read=56.3 stddev_read=194.9\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..6.05 avg_rows=217 avg_width=0) (actual avg_time=12.159..12.159 stddev_time=0.841..0.841 avg_rows=798661 avg_loops=1)\\n Index Cond: (tenant_id = $1)\\n Buffers: shared avg_hit=618.8 stddev_hit=194.9 avg_read=56.3 stddev_read=194.9\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.42..7.74 avg_rows=1 avg_width=11) (actual avg_time=0.002..0.002 avg_rows=1 avg_loops=798661)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=3194644\\nPlanning:\\n Buffers: shared avg_hit=9.1 stddev_hit=31.5\\nPlanning Time: avg=0.046 ms stddev=0.119 n=12\\nExecution Time: avg=2347.340 ms stddev=27.850 n=12\",\"runtimeDurationMs\":2352,\"rowsAreCapped\":false,\"computeSeconds\":35.625}],\"workerId\":\"88356cd3-0c55-4985-9145-496ff7aa0077\",\"correlationId\":\"ed9367cd-3189-40be-9104-edaac048fd7d\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "SET plan_cache_mode = force_generic_plan puts all 12 executions on the Nested Loop the planner switched to at run 6", "planCaptions": { "300K / 2M": "With plan_cache_mode = force_generic_plan the plan carries Recheck Cond (tenant_id = $1) from the first execution: the planner estimates 217 rows, 798,661 arrive, the Nested Loop probes customers_pkey 798,661 times for 3,208,058 buffer hits, and the Sort runs as a 19,336 kB external merge. Execution Time avg 2347.340 ms stddev 27.850 over 12 runs, a 1.2 percent spread around a plan that never changes." } }, "forceCustom": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "1361ff10-a37c-4171-b0a3-1d0b007d2b33", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "SET plan_cache_mode = force_custom_plan;\nPREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:41:07.615588362Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":11.875,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over 12 runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nFinalize GroupAggregate (avg_cost=33157.92..33158.46 avg_rows=2 avg_width=47) (actual avg_time=271.348..278.988 stddev_time=7.493..7.338 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=14403\\n -> Gather Merge (avg_cost=33157.92..33158.39 avg_rows=4 avg_width=47) (actual avg_time=271.336..278.974 stddev_time=7.493..7.337 avg_rows=6 avg_loops=1)\\n Workers Planned: avg=2\\n Workers Launched: avg=2\\n Buffers: shared avg_hit=14403\\n -> Sort (avg_cost=32157.90..32157.90 avg_rows=2 avg_width=47) (actual avg_time=268.365..268.368 stddev_time=7.311..7.311 avg_rows=2 avg_loops=3)\\n Sort Key: c.tier\\n Sort Method: quicksort Memory: avg=25kB\\n Buffers: shared avg_hit=14403\\n -> Partial HashAggregate (avg_cost=32157.86..32157.89 avg_rows=2 avg_width=47) (actual avg_time=268.344..268.347 stddev_time=7.312..7.312 avg_rows=2 avg_loops=3)\\n Group Key: c.tier\\n Batches: avg=1 Memory Usage: avg=24kB\\n Buffers: shared avg_hit=14385\\n -> Parallel Hash Join (avg_cost=5592.59..29632.24 avg_rows=336750 avg_width=13) (actual avg_time=38.370..211.489 stddev_time=1.647..6.370 avg_rows=266814 avg_loops=3)\\n Hash Cond: (o.customer_id = c.id)\\n Buffers: shared avg_hit=14385\\n -> Parallel Seq Scan on orders o (avg_cost=0.00..23155.67 avg_rows=336750 avg_width=10) (actual avg_time=0.007..51.181 stddev_time=0.001..2.116 avg_rows=266814 avg_loops=3)\\n Filter: (tenant_id = 1)\\n Rows Removed by Filter: avg=399853\\n Buffers: shared avg_hit=12739\\n -> Parallel Hash (avg_cost=3386.71..3386.71 avg_rows=176471 avg_width=11) (actual avg_time=37.445..37.446 stddev_time=1.700..1.700 avg_rows=100000 avg_loops=3)\\n Buckets: avg=524288 Batches: avg=1 Memory Usage: avg=18224kB stddev=16.711kB\\n Buffers: shared avg_hit=1622\\n -> Parallel Seq Scan on customers c (avg_cost=0.00..3386.71 avg_rows=176471 avg_width=11) (actual avg_time=0.010..9.715 stddev_time=0.001..0.110 avg_rows=100000 avg_loops=3)\\n Buffers: shared avg_hit=1622\\nPlanning:\\n Buffers: shared avg_hit=19 stddev_hit=31.2\\nPlanning Time: avg=0.289 ms stddev=0.138 n=12\\nExecution Time: avg=279.052 ms stddev=7.337 n=12\",\"runtimeDurationMs\":283,\"rowsAreCapped\":false,\"computeSeconds\":11.875}],\"workerId\":\"8f5baad6-a9b0-48c8-ad93-2c89993172a0\",\"correlationId\":\"f2860651-2d28-4892-95c4-155c4d3f3d79\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "The fix, measured: SET plan_cache_mode = force_custom_plan keeps every one of the 12 executions on the Parallel Hash Join", "planCaptions": { "300K / 2M": "With plan_cache_mode = force_custom_plan every execution is planned against the actual parameter: Parallel Hash Join, Filter (tenant_id = 1), 14,403 shared buffers, no sort spill. Execution Time avg 279.052 ms stddev 7.337 over all 12 runs, and Planning Time avg 0.289 ms, which is the entire price of replanning on every execution." } }, "insideFunction": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;\n\nCREATE FUNCTION tenant_report_fn(t int) RETURNS TABLE(tier text, n bigint, revenue numeric)\nLANGUAGE plpgsql AS $$\nBEGIN\n RETURN QUERY SELECT c.tier, count(*), sum(o.amount)\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = t GROUP BY c.tier;\nEND $$;", "query": "EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM tenant_report_fn(1)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "1361ff10-a37c-4171-b0a3-1d0b007d2b33", "exobenchServerVersion": "1.4.5" }, "result": "{\"timestamp\":\"2026-09-11T03:44:20.563304180Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM tenant_report_fn(1)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":30.565,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over 12 runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nFunction Scan on tenant_report_fn (avg_cost=0.25..10.25 avg_rows=1000 avg_width=72) (actual avg_time=1293.996..1293.997 stddev_time=955.704..955.704 avg_rows=2 avg_loops=1)\\n Buffers: shared avg_hit=1880315.7 stddev_hit=1647068.3 avg_read=56.4 stddev_read=195.1 avg_dirtied=0.1 stddev_dirtied=0.3, temp avg_read=1412.3 stddev_read=1246.6 avg_written=1418.7 stddev_written=1252.3\\nPlanning Time: avg=0.029 ms stddev=0.002 n=12\\nExecution Time: avg=1294.020 ms stddev=955.702 n=12\",\"runtimeDurationMs\":1298,\"rowsAreCapped\":false,\"computeSeconds\":30.565}],\"workerId\":\"5cc7a410-8bd9-4f12-88f8-c87a7cbd37da\",\"correlationId\":\"1c831ae2-10b1-4484-8b3e-2decc1d39f93\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "The same query inside a PL/pgSQL function: the plan flip happens where EXPLAIN cannot see it, and only the spread is left", "planCaptions": { "300K / 2M": "EXPLAIN sees one node, Function Scan on tenant_report_fn, with no plan text for the query inside it. Over 12 runs the Execution Time is avg 1294.020 ms stddev 955.702, a standard deviation at 74 percent of the mean, and the buffer line reads avg_hit 1,880,315.7 with stddev_hit 1,647,068.3. The flip from custom to generic plan happened inside the function; the only visible trace is a spread that a stable plan could not produce." } }, "flipSweep": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\n-- tenant 1 is a whale with ~40% of all orders; the other 9,999 tenants share the rest uniformly.\n-- The generic plan will estimate 1/n_distinct of the table for \"tenant_id = $1\".\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)", "scalePoints": [ { "CUSTOMERS": 30000, "ORDERS": 200000 }, { "CUSTOMERS": 150000, "ORDERS": 1000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "0943dd7b-7a42-41cb-9575-1f7ec0cbbd73", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "PREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:38:59.198919840Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(1)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":2,\"totalComputeSeconds\":31.31,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":30000,\"ORDERS\":200000},\"output\":\"QUERY PLAN\\nStatistics over each plan's runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nPlan changed across runs: 2 distinct plans; A: runs 1-5, B: runs 6-12 (n=12)\\n\\n--- Plan A (5 of 12 runs) ---\\nHashAggregate (avg_cost=4860.90..4860.93 avg_rows=2 avg_width=47) (actual avg_time=55.121..55.125 stddev_time=1.614..1.614 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Batches: avg=1 Memory Usage: avg=24kB\\n Buffers: shared avg_hit=1492.2 stddev_hit=30.9 avg_read=13.8 stddev_read=30.9\\n -> Hash Join (avg_cost=1777.93..4261.25 avg_rows=79953 avg_width=13) (actual avg_time=7.762..38.664 stddev_time=0.194..1.469 avg_rows=80286 avg_loops=1)\\n Hash Cond: (o.customer_id = c.id)\\n Buffers: shared avg_hit=1492.2 stddev_hit=30.9 avg_read=13.8 stddev_read=30.9\\n -> Bitmap Heap Scan on orders o (avg_cost=939.93..3213.34 avg_rows=79953 avg_width=10) (actual avg_time=1.394..10.386 stddev_time=0.218..0.345 avg_rows=80286 avg_loops=1)\\n Recheck Cond: (tenant_id = 1)\\n Heap Blocks: avg_exact=1274\\n Buffers: shared avg_hit=1329.2 stddev_hit=30.9 avg_read=13.8 stddev_read=30.9\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..919.94 avg_rows=79953 avg_width=0) (actual avg_time=1.273..1.273 stddev_time=0.219..0.219 avg_rows=80286 avg_loops=1)\\n Index Cond: (tenant_id = 1)\\n Buffers: shared avg_hit=55.2 stddev_hit=30.9 avg_read=13.8 stddev_read=30.9\\n -> Hash (avg_cost=463.00..463.00 avg_rows=30000 avg_width=11) (actual avg_time=6.356..6.357 stddev_time=0.031..0.031 avg_rows=30000 avg_loops=1)\\n Buckets: avg=32768 Batches: avg=1 Memory Usage: avg=1546kB\\n Buffers: shared avg_hit=163\\n -> Seq Scan on customers c (avg_cost=0.00..463.00 avg_rows=30000 avg_width=11) (actual avg_time=0.009..2.692 stddev_time=0.001..0.024 avg_rows=30000 avg_loops=1)\\n Buffers: shared avg_hit=163\\nPlanning:\\n Buffers: shared avg_hit=29.6 stddev_hit=48.3\\nPlanning Time: avg=0.260 ms stddev=0.093 n=5\\nExecution Time: avg=55.185 ms stddev=1.615 n=5\\n\\n--- Plan B (7 of 12 runs) ---\\nGroupAggregate (avg_cost=255.27..255.51 avg_rows=2 avg_width=47) (actual avg_time=158.058..174.841 stddev_time=11.164..11.191 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=242201, temp avg_read=243 avg_written=244\\n -> Sort (avg_cost=255.27..255.32 avg_rows=22 avg_width=13) (actual avg_time=155.946..162.524 stddev_time=11.161..11.192 avg_rows=80286 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: external merge Disk: avg=1944kB\\n Buffers: shared avg_hit=242201, temp avg_read=243 avg_written=244\\n -> Nested Loop (avg_cost=4.75..254.78 avg_rows=22 avg_width=13) (actual avg_time=1.330..135.756 stddev_time=0.023..9.579 avg_rows=80286 avg_loops=1)\\n Buffers: shared avg_hit=242201\\n -> Bitmap Heap Scan on orders o (avg_cost=4.47..84.07 avg_rows=22 avg_width=10) (actual avg_time=1.320..12.726 stddev_time=0.023..0.699 avg_rows=80286 avg_loops=1)\\n Recheck Cond: (tenant_id = $1)\\n Heap Blocks: avg_exact=1274\\n Buffers: shared avg_hit=1343\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..4.46 avg_rows=22 avg_width=0) (actual avg_time=1.200..1.201 stddev_time=0.022..0.022 avg_rows=80286 avg_loops=1)\\n Index Cond: (tenant_id = $1)\\n Buffers: shared avg_hit=69\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.29..7.76 avg_rows=1 avg_width=11) (actual avg_time=0.001..0.001 avg_rows=1 avg_loops=80286)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=240858\\nPlanning:\\n Buffers: shared avg_hit=1.1 stddev_hit=3.0\\nPlanning Time: avg=0.041 ms stddev=0.076 n=7\\nExecution Time: avg=175.270 ms stddev=11.205 n=7\\n\\n--- All runs (n=12) ---\\nPlanning Time: avg=0.132 ms stddev=0.138 n=12\\nExecution Time: avg=125.235 ms stddev=62.394 n=12\",\"runtimeDurationMs\":129,\"rowsAreCapped\":false,\"computeSeconds\":16.662},{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":150000,\"ORDERS\":1000000},\"output\":\"QUERY PLAN\\nStatistics over each plan's runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nPlan changed across runs: 2 distinct plans; A: runs 1-5, B: runs 6-12 (n=12)\\n\\n--- Plan A (5 of 12 runs) ---\\nFinalize GroupAggregate (avg_cost=21554.26..21554.79 avg_rows=2 avg_width=47) (actual avg_time=195.570..198.290 stddev_time=27.234..27.864 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=8919, temp avg_read=1662.6 stddev_read=0.5 avg_written=1662.6 stddev_written=0.5\\n -> Gather Merge (avg_cost=21554.26..21554.73 avg_rows=4 avg_width=47) (actual avg_time=195.559..198.278 stddev_time=27.233..27.863 avg_rows=6 avg_loops=1)\\n Workers Planned: avg=2\\n Workers Launched: avg=2\\n Buffers: shared avg_hit=8919, temp avg_read=1662.6 stddev_read=0.5 avg_written=1662.6 stddev_written=0.5\\n -> Sort (avg_cost=20554.24..20554.24 avg_rows=2 avg_width=47) (actual avg_time=187.139..187.142 stddev_time=27.140..27.140 avg_rows=2 avg_loops=3)\\n Sort Key: c.tier\\n Sort Method: quicksort Memory: avg=25kB\\n Buffers: shared avg_hit=8919, temp avg_read=1662.6 stddev_read=0.5 avg_written=1662.6 stddev_written=0.5\\n -> Partial HashAggregate (avg_cost=20554.20..20554.23 avg_rows=2 avg_width=47) (actual avg_time=187.119..187.122 stddev_time=27.140..27.140 avg_rows=2 avg_loops=3)\\n Group Key: c.tier\\n Batches: avg=1 Memory Usage: avg=24kB\\n Buffers: shared avg_hit=8901, temp avg_read=1662.6 stddev_read=0.5 avg_written=1662.6 stddev_written=0.5\\n -> Hash Join (avg_cost=4919.00..19300.97 avg_rows=167097 avg_width=13) (actual avg_time=43.354..156.774 stddev_time=0.779..24.364 avg_rows=133520 avg_loops=3)\\n Hash Cond: (o.customer_id = c.id)\\n Buffers: shared avg_hit=8901, temp avg_read=1662.6 stddev_read=0.5 avg_written=1662.6 stddev_written=0.5\\n -> Parallel Seq Scan on orders o (avg_cost=0.00..11578.33 avg_rows=167097 avg_width=10) (actual avg_time=0.011..31.244 stddev_time=0.000..1.519 avg_rows=133520 avg_loops=3)\\n Filter: (tenant_id = 1)\\n Rows Removed by Filter: avg=199814\\n Buffers: shared avg_hit=6370\\n -> Hash (avg_cost=2311.00..2311.00 avg_rows=150000 avg_width=11) (actual avg_time=42.842..42.843 stddev_time=0.767..0.767 avg_rows=150000 avg_loops=3)\\n Buckets: avg=262144 Batches: avg=2 Memory Usage: avg=5279kB\\n Buffers: shared avg_hit=2433, temp avg_written=876\\n -> Seq Scan on customers c (avg_cost=0.00..2311.00 avg_rows=150000 avg_width=11) (actual avg_time=0.013..14.860 stddev_time=0.003..0.176 avg_rows=150000 avg_loops=3)\\n Buffers: shared avg_hit=2433\\nPlanning:\\n Buffers: shared avg_hit=33.6 stddev_hit=48.3\\nPlanning Time: avg=0.353 ms stddev=0.166 n=5\\nExecution Time: avg=198.365 ms stddev=27.857 n=5\\n\\n--- Plan B (7 of 12 runs) ---\\nGroupAggregate (avg_cost=1251.23..1252.35 avg_rows=2 avg_width=47) (actual avg_time=1010.131..1103.986 stddev_time=12.855..12.908 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=1608897.4 stddev_hit=128.5 avg_read=48.6 stddev_read=128.5, temp avg_read=1213 avg_written=1218\\n -> Sort (avg_cost=1251.23..1251.51 avg_rows=109 avg_width=13) (actual avg_time=999.603..1042.192 stddev_time=12.895..13.140 avg_rows=400559 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: external merge Disk: avg=9704kB\\n Buffers: shared avg_hit=1608897.4 stddev_hit=128.5 avg_read=48.6 stddev_read=128.5, temp avg_read=1213 avg_written=1218\\n -> Nested Loop (avg_cost=5.69..1247.54 avg_rows=109 avg_width=13) (actual avg_time=7.120..909.636 stddev_time=0.944..11.733 avg_rows=400559 avg_loops=1)\\n Buffers: shared avg_hit=1608897.4 stddev_hit=128.5 avg_read=48.6 stddev_read=128.5\\n -> Bitmap Heap Scan on orders o (avg_cost=5.27..399.86 avg_rows=109 avg_width=10) (actual avg_time=7.106..63.588 stddev_time=0.943..0.936 avg_rows=400559 avg_loops=1)\\n Recheck Cond: (tenant_id = $1)\\n Heap Blocks: avg_exact=6370\\n Buffers: shared avg_hit=6661.4 stddev_hit=128.5 avg_read=48.6 stddev_read=128.5\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..5.24 avg_rows=109 avg_width=0) (actual avg_time=6.455..6.455 stddev_time=0.941..0.941 avg_rows=400559 avg_loops=1)\\n Index Cond: (tenant_id = $1)\\n Buffers: shared avg_hit=291.4 stddev_hit=128.5 avg_read=48.6 stddev_read=128.5\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.42..7.78 avg_rows=1 avg_width=11) (actual avg_time=0.002..0.002 avg_rows=1 avg_loops=400559)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=1602236\\nPlanning:\\n Buffers: shared avg_hit=1.7 stddev_hit=4.5\\nPlanning Time: avg=0.051 ms stddev=0.104 n=7\\nExecution Time: avg=1105.474 ms stddev=12.920 n=7\\n\\n--- All runs (n=12) ---\\nPlanning Time: avg=0.177 ms stddev=0.200 n=12\\nExecution Time: avg=727.512 ms stddev=467.496 n=12\",\"runtimeDurationMs\":731,\"rowsAreCapped\":false,\"computeSeconds\":14.648}],\"workerId\":\"8af2a563-735b-4b00-8e4f-ba764b6d543f\",\"correlationId\":\"1361ff10-a37c-4171-b0a3-1d0b007d2b33\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "The same prepared statement at 200K and 1M orders: the flip lands on run 6 at both scales and the penalty grows with the table", "planCaptions": { "30K / 200K": "At 200K orders the custom plan (runs 1 to 5) is a Hash Join fed by a Bitmap Heap Scan on orders_tenant_idx with Recheck Cond (tenant_id = 1), about 1,506 buffers, avg 55.185 ms stddev 1.615. The generic plan (runs 6 to 12) is a Nested Loop that estimates 22 rows, receives 80,286, probes customers_pkey 80,286 times for 242,201 buffer hits and spills a 1,944 kB sort to disk, avg 175.270 ms stddev 11.205, 3.2 times slower.", "150K / 1M": "At 1M orders the custom plan is a parallel Hash Join with Filter (tenant_id = 1) over 8,919 shared buffers, avg 198.365 ms stddev 27.857 across runs 1 to 5. From run 6 the generic Nested Loop estimates 109 rows, receives 400,559, drives 1,608,946 buffer hits through customers_pkey and a 9,704 kB external merge sort, avg 1105.474 ms stddev 12.920, 5.6 times slower. The all-runs mean is 727.512 ms." } }, "ordinaryTenant": { "type": "mcp-call", "name": "benchmarkSql", "input": { "schemaTemplate": "CREATE TABLE customers (id INT PRIMARY KEY, tier TEXT NOT NULL);\nINSERT INTO customers\nSELECT i, CASE WHEN i % 10 = 0 THEN 'gold' ELSE 'regular' END\nFROM generate_series(1, ${CUSTOMERS}) i;\n\nCREATE TABLE orders (id INT PRIMARY KEY, tenant_id INT NOT NULL, customer_id INT NOT NULL, amount NUMERIC(10,2) NOT NULL);\nINSERT INTO orders\nSELECT i,\n CASE WHEN random() < 0.40 THEN 1 ELSE 2 + (random() * 9998)::int END,\n 1 + (random() * (${CUSTOMERS} - 1))::int,\n round((random() * 500)::numeric, 2)\nFROM generate_series(1, ${ORDERS}) i;\nCREATE INDEX orders_tenant_idx ON orders (tenant_id);\nVACUUM ANALYZE;", "query": "EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(4242)", "scalePoints": [ { "CUSTOMERS": 300000, "ORDERS": 2000000 } ], "databaseType": "POSTGRES", "repetitions": "12", "warmCache": "0", "correlationId": "1361ff10-a37c-4171-b0a3-1d0b007d2b33", "exobenchServerVersion": "1.4.5", "preBenchmarkSql": "PREPARE tenant_report(int) AS\n SELECT c.tier, count(*) AS n, sum(o.amount) AS revenue\n FROM orders o JOIN customers c ON c.id = o.customer_id\n WHERE o.tenant_id = $1\n GROUP BY c.tier;" }, "result": "{\"timestamp\":\"2026-09-11T03:42:31.236657328Z\",\"databaseType\":\"postgres\",\"sql\":\"EXPLAIN (ANALYZE, BUFFERS) EXECUTE tenant_report(4242)\",\"warmCacheIterations\":0,\"repetitions\":12,\"scalePointCount\":1,\"totalComputeSeconds\":7.97,\"results\":[{\"kind\":\"postgres_explain_analyze\",\"scalePoint\":{\"CUSTOMERS\":300000,\"ORDERS\":2000000},\"output\":\"QUERY PLAN\\nStatistics over 12 runs: avg = mean, stddev = sample standard deviation, omitted where 0.\\nGroupAggregate (avg_cost=1533.43..1534.75 avg_rows=2 avg_width=47) (actual avg_time=0.457..0.480 stddev_time=0.063..0.063 avg_rows=2 avg_loops=1)\\n Group Key: c.tier\\n Buffers: shared avg_hit=595.8 stddev_hit=0.9 avg_read=0.3 stddev_read=0.9\\n -> Sort (avg_cost=1533.43..1533.75 avg_rows=130 avg_width=13) (actual avg_time=0.450..0.456 stddev_time=0.062..0.063 avg_rows=119 avg_loops=1)\\n Sort Key: c.tier\\n Sort Method: quicksort Memory: avg=28kB\\n Buffers: shared avg_hit=595.8 stddev_hit=0.9 avg_read=0.3 stddev_read=0.9\\n -> Nested Loop (avg_cost=5.86..1528.86 avg_rows=130 avg_width=13) (actual avg_time=0.027..0.425 stddev_time=0.014..0.062 avg_rows=119 avg_loops=1)\\n Buffers: shared avg_hit=595.8 stddev_hit=0.9 avg_read=0.3 stddev_read=0.9\\n -> Bitmap Heap Scan on orders o (avg_cost=5.43..487.66 avg_rows=130 avg_width=10) (actual avg_time=0.022..0.135 stddev_time=0.013..0.032 avg_rows=119 avg_loops=1)\\n Recheck Cond: (tenant_id = 4242)\\n Heap Blocks: avg_exact=117\\n Buffers: shared avg_hit=119.8 stddev_hit=0.9 avg_read=0.3 stddev_read=0.9\\n -> Bitmap Index Scan on orders_tenant_idx (avg_cost=0.00..5.40 avg_rows=130 avg_width=0) (actual avg_time=0.011..0.012 stddev_time=0.013..0.013 avg_rows=119 avg_loops=1)\\n Index Cond: (tenant_id = 4242)\\n Buffers: shared avg_hit=2.8 stddev_hit=0.9 avg_read=0.3 stddev_read=0.9\\n -> Index Scan using customers_pkey on customers c (avg_cost=0.42..8.01 avg_rows=1 avg_width=11) (actual avg_time=0.002..0.002 avg_rows=1 avg_loops=119)\\n Index Cond: (id = o.customer_id)\\n Buffers: shared avg_hit=476\\nPlanning:\\n Buffers: shared avg_hit=19.1 stddev_hit=28.5\\nPlanning Time: avg=0.161 ms stddev=0.059 n=12\\nExecution Time: avg=0.515 ms stddev=0.064 n=12\",\"runtimeDurationMs\":3,\"rowsAreCapped\":false,\"computeSeconds\":7.97}],\"workerId\":\"00b6759d-a581-4681-8314-e17d3d2b7f72\",\"correlationId\":\"23f52039-aa25-4c2d-aa04-0e529a6a85f6\",\"correlationIdInstruction\":\"Echo correlationId on subsequent ExoBench tool calls. Tracking only \u2014 it does NOT preserve DB state.\",\"version\":\"1.4.5-dirty\"}", "isError": false, "caption": "The same prepared statement executed for tenant 4242, one of the 9,999 ordinary tenants: no flip in 12 runs", "planCaptions": { "300K / 2M": "For tenant 4242 all 12 executions keep the custom plan, visible as the literal in Recheck Cond (tenant_id = 4242): a Bitmap Heap Scan over 117 heap blocks feeding a Nested Loop with 119 probes of customers_pkey, 596 buffers, a quicksort in 28 kB. Execution Time avg 0.515 ms stddev 0.064. This custom plan costs 1,534.75 and the generic plan costs about 2,466, so the planner has no reason to switch." } }, "regimesChart": { "type": "chart", "chartType": "multi-line", "scales": [ "200K orders", "1M orders", "2M orders" ], "xLabel": "Orders table size", "yLabel": "Execution time (ms)", "yUnit": "ms", "maxY": 2500, "caption": "Mean execution time per plan regime as the orders table grows. The dashed line is the average over all 12 runs, a number no single run produced at any scale.", "summaryCaption": "The penalty for the sixth execution grows from 3.2x at 200K orders to 8.5x at 2M.", "dataAsProse": "Custom plan (runs 1 to 5): 55.185 ms at 200K orders, 198.365 ms at 1M, 273.678 ms at 2M. Generic plan (runs 6 to 12): 175.270 ms at 200K, 1105.474 ms at 1M, 2318.664 ms at 2M. All 12 runs averaged: 125.235 ms at 200K, 727.512 ms at 1M, 1466.587 ms at 2M.", "series": [ { "label": "Custom plan, runs 1 to 5", "data": [ 55.185, 198.365, 273.678 ], "color": "#1D9E75", "pointStyle": "circle" }, { "label": "Generic plan, runs 6 to 12", "data": [ 175.27, 1105.474, 2318.664 ], "color": "#E24B4A", "pointStyle": "rectRot" }, { "label": "All 12 runs averaged", "data": [ 125.235, 727.512, 1466.587 ], "color": "#BA7517", "style": "dashed", "pointStyle": "triangle" } ] } } }