Getting Started

Setup & Troubleshooting

Connection, configuration, access levels, and troubleshooting

For basic setup instructions, see the 30-second quickstart on the homepage. It covers Claude, ChatGPT, Gemini, Cursor, and Windsurf.

This section covers what the quickstart doesn't: edge cases, parameters, and what to do when things go wrong.

ChatGPT: The ?short=true Parameter

If ChatGPT tells you there's a character limit when connecting ExoBench, use this URL instead:

https://connect.exobench.ai/mcp?short=true

This returns shorter tool descriptions that fit within ChatGPT's tool definition size limits. The tools work similarly, but the shorter descriptions prompt ChatGPT to consult the ExoBench documentation pages first before constructing tool calls, rather than relying solely on inline tool descriptions.

You only need ?short=true if ChatGPT explicitly reports a character limit. If it works without it, don't add it.

Windsurf: Why npx Is Required

The Windsurf quickstart tells you to use npx mcp-remote in your config. Here's why.

ExoBench uses OAuth (via GitHub) to authenticate users. Some MCP clients, including Windsurf, don't natively support OAuth for remote MCP servers. This isn't an ExoBench limitation, it affects all OAuth-protected MCP servers on Windsurf.

The npx mcp-remote package bridges the gap. It runs locally on your machine, handles the OAuth flow by opening your browser, and forwards authenticated requests to ExoBench. Once Windsurf adds native OAuth support, you won't need it anymore.

Requires: Node.js 18+. Check with node --version. npx comes bundled with Node.js.

Note that Windsurf's "Add custom server" UI just opens the raw JSON config file — there's no form-based flow for arbitrary remote URLs. Two paths depending on auth model:

  • OAuth required (ExoBench): Use npx mcp-remote https://connect.exobench.ai/mcp in the config.
  • No OAuth / static token: Use "serverUrl": "https://connect.exobench.ai/mcp" directly.

Account Tiers & Access

  • Unauthenticated: 0 benchmarks. Connect and authenticate to get started.
  • Authenticated: 5 one-time benchmarks (subject to availability). Enough to get a taste. To seriously evaluate ExoBench, sign up for the trial.
  • Trial: Unlimited benchmarks. Request access.

Check your current usage anytime:

"What's my ExoBench account status?"

Troubleshooting

403 Forbidden on connect The MCP URL must be https://connect.exobench.ai/mcp — not https://connect.exobench.ai (without /mcp). The missing path causes the MCP client to receive a non-MCP response (likely a redirect or HTML page), which cascades into a malformed OAuth challenge and a 403. Your AI assistant may misdiagnose this as a credential formatting issue (e.g., suggesting key=<token> format) — that's wrong. ExoBench uses OAuth 2.1 Bearer tokens, not API keys.

"Tool call failure" or connection hangs (any client) Disconnect the ExoBench connector, restart your AI client completely, and reconnect. OAuth tokens can get stale, and some clients cache MCP connections aggressively.

Tools don't appear after connecting Start a new chat. Some clients only load MCP tools at chat creation time. Verify the URL is exactly https://connect.exobench.ai/mcp with no trailing slashes or typos.

"Authentication required" error in tool results Your OAuth session has expired. Disconnect and reconnect the connector.

ChatGPT says character limit exceeded Use https://connect.exobench.ai/mcp?short=true instead. See ChatGPT: The ?short=true Parameter above.

OAuth redirect doesn't complete (Windsurf / npx users) mcp-remote listens on port 3334 by default for the OAuth callback. If that port is already in use, it falls back to a random port, which can cause a redirect_uri mismatch error because the registered redirect URI no longer matches the actual callback port. Pop-up blockers can also prevent the browser from opening at all.

To fix:

  • Check that port 3334 is free, or specify a different port explicitly: "args": ["-y", "mcp-remote", "https://connect.exobench.ai/mcp", "9696"]
  • To clear stale tokens and force a fresh OAuth flow: rm -rf ~/.mcp-auth
  • Ensure your browser is not blocking pop-ups for localhost

References: mcp-remote README (default port, custom port argument, clearing tokens), GitHub Issue #72 (redirect_uri mismatch from port conflict).

Node.js version issues (Windsurf / npx users) mcp-remote requires Node.js 18+. MCP clients like Claude Desktop use your system-level Node.js, not the version managed by nvm or other version managers. So even if nvm shows Node 22, the client may be invoking Node 16 from /usr/local/bin/node.

Verify with which node && node --version. If it's below 18, upgrade the system-level install — don't rely on nvm for this.

References: mcp-remote npm page ("Claude Desktop will use your system version of Node, even if you have a newer version installed elsewhere"), mcp-remote DeepWiki ("MCP clients like Claude Desktop use your system's Node.js installation, not project-specific versions").

VPN / corporate network issues (Windsurf / npx users) Corporate networks often use custom CA certificates for TLS inspection. Node.js does not trust these by default, so HTTPS requests from mcp-remote to ExoBench fail at the TLS handshake. Set NODE_EXTRA_CA_CERTS in your MCP config's env block, pointing to your corporate CA certificate .pem file:

{
  "mcpServers": {
    "exobench": {
      "command": "npx",
      "args": ["mcp-remote", "https://connect.exobench.ai/mcp"],
      "env": {
        "NODE_EXTRA_CA_CERTS": "/path/to/your/corporate-ca.pem"
      }
    }
  }
}

References: mcp-remote npm page (VPN/proxy troubleshooting with NODE_EXTRA_CA_CERTS), mcp-remote README (same section with --enable-proxy flag).

Benchmark takes too long or times out Reduce your scale points. Start with 10K, 50K, 100K instead of jumping to 1M. ExoBench enforces a maximum of 5 scale points per request and 2 million total rows per scale point. Also check that your schema isn't doing something expensive in data generation (e.g., cross joins that produce exponential row counts).

Giving Feedback

If ExoBench gives you surprising results, hits a bug, or you want a feature, tell your AI assistant:

"Submit feedback to ExoBench: [your message]"

This sends your feedback directly to the ExoBench team along with the benchmark context from your conversation. You don't need to re-paste anything. Categories: benchmark_result, bug, feature_request, docs, and general.