Methodology
Pilot scope
Measure ordinary tool-call correctness and end-to-end route overhead against independently maintained MCP reference tools. No fictional business dataset, LLM judge, or preselected winner. This first pilot does not claim broad protocol conformance or enterprise readiness.
Reference and hosting
Use REFERENCE_MCP_URL=https://mcp-bench-reference.vercel.app/mcp for EVERY contender's upstream and for the direct baseline. It serves official @modelcontextprotocol/server-everything 2.3.0 tools with the SDK stateless Streamable HTTP handler. Exact dependencies and hosting wrapper are in reference/. Vercel region iad1, Node 24. It is independent of the Replicas workspace. Do not create a separate upstream for a contender. Preserve reference/, scripts/, evidence/ and SETUP.md when building the site.
Contenders
Speakeasy is hosted. MCPJungle, Docker MCP Gateway and IBM MCP Context Forge use pinned public releases, installed in the benchmark sandbox with the same remote upstream. Record exact version, command, configuration and topology. Verify each gateway makes a REAL routed upstream call. No home-made proxy substituted for a contender. No direct call mislabeled a gateway. If a dependency/runtime is unavailable, record setup failure with null metrics and no ranking; never invent or simulate results. Local versus hosted topology is conspicuous in every table. Added latency includes each route's network distance, proxy behavior and serverless cold starts; it is not pure gateway processing time.
Speakeasy adapter and authentication
SPEAKEASY_GATEWAY_URL is the gateway. SPEAKEASY_OAUTH_JSON is sealed JSON containing access_token, refresh_token, expires_in, received_at, client_id, token_endpoint and resource. Use Authorization: Bearer access_token. Never log, commit or return these values. If the access token expired, stop with setup blocked and request operator renewal; DO NOT consume the rotating refresh token inside the ephemeral sandbox. scripts/refresh-oauth.py is the verified operator renewal/persistence path, using a Letter API key kept OUTSIDE the sandbox. Scheduled unattended renewal is not implemented. Browser consent is required again at the two-week session deadline.
The live gateway exposes list_servers, describe_server, describe_tools, execute_tool, connection_status. Resolve the member mcp-everything-230-reference-396d, describe exact upstream tool schemas, then execute_tool({name: memberSlug+"--"+toolName, arguments: payload}). Consent uses default All tools (frozen); disclose that cache. Do not compare its meta-tool names with upstream tools/list.
Correctness
The eight published cases in scripts/preflight.py cover echo (ordinary text, empty, Unicode), get-sum (positive, negative, zero, fractions), and get-tiny-image. Validate the direct baseline, then compare the gateway's content, structuredContent and isError with it. Ignore only transport request IDs, gateway metadata and documented wrappers; never normalize substantive content. Correctness rate is passed valid cases / valid cases attempted; a missing tool is a failed case, while infrastructure/auth setup failures get null scores. List invalid-input rejection cases separately, with exact schemas and error envelopes. Optional resources/prompts, progress, sampling, elicitation, filesystem access and stateful sessions are outside this pilot.
Timing and evidence
Per contender: 20 warmup pairs then three rounds of 100 measured echo pairs. Alternate pair order and rotate contender order across rounds. Retain all timings and failures. Report median paired added latency over successful pairs and timed call error rate; never count failed or timed-out calls as fast successes. No p99 or throughput headline from this small pilot. Include sample counts, uncertainty/dispersion, warmup policy and infrastructure. Bound the total job below Letter's 20-minute limit; timed-out contenders remain failed/unranked. All trace details must fit the 1MiB results limit.
Publication
Existing evidence/preflight.json is a connection preflight, NOT the competitive results. Only results produced by the real npm run bench are published as comparison data. Include inputs, pinned versions/configs, raw per-case outcomes and per-pair timings in results/latest.json so Letter retains them. Label this a pilot and disclose the private repo prevents independent public code inspection until its owner makes it public. Do not claim autonomous scheduled reruns until OAuth renewal is wired.
Required corrections from the first build
REFERENCE_MCP_URL is public configuration, NOT a secret or credential prerequisite. Default it in code to https://mcp-bench-reference.vercel.app/mcp for all routes. Do not require that environment variable to run Speakeasy. Only SPEAKEASY_GATEWAY_URL and SPEAKEASY_OAUTH_JSON are sealed credentials. Never include those sealed values or OAuth object fields in evidence, logs or generated pages; report the gateway endpoint as [configured hosted gateway].
The previous smoke run stalled after the subject, apparently in MCPJungle setup. Every setup step, download, process readiness loop, tool call, and child cleanup MUST have a deadline. On timeout record failed/unranked with a reason and continue to the next contender. Dispose only tracked child handles, with bounded graceful shutdown then forceful termination of those children if necessary. Do not wait indefinitely for a long-lived server process to exit before returning its handle.
Before finishing, complete a small smoke run showing all four contender outcomes (Speakeasy skipped without credentials is expected), run npm run build and typecheck, and remove generated results/latest.json and results/history/*.json from the submitted change. Letter writes official results afterwards. Use smoke data only locally to verify pages. Report exact outcomes and build checks. Derive displayed counts from actual results. Private repository disclosure is explicitly authorized and accurate. Preserve the hosted reference and operator scripts.
Contenders
- Speakeasy MCP Gateway (
subject): https://www.speakeasy.com. ReadsSPEAKEASY_GATEWAY_URL,SPEAKEASY_OAUTH_JSON. - MCPJungle (
mcpjungle): https://github.com/mcpjungle/MCPJungle - Docker MCP Gateway (
docker-mcp-gateway): https://github.com/docker/mcp-gateway - IBM MCP Context Forge (
mcp-context-forge): https://github.com/IBM/mcp-context-forge
Metrics
| Metric | Field | Unit | Direction |
|---|---|---|---|
| Tool-call correctness (primary) | conformance_pass_rate |
% | higher is better |
| Median paired added latency | added_latency_p50 |
ms | lower is better |
| Timed call error rate | error_rate |
% | lower is better |
Scenarios
- Tool discovery (
discovery): Resolve the same upstream echo, get-sum and get-tiny-image tools through each documented discovery interface. Speakeasy exposes meta-tools, so do not require tools/list names to equal upstream names. - Reference tool calls (
reference_calls): Run the eight transparent cases in scripts/preflight.py against the official server, directly and through each gateway. Compare text, image bytes/mime types, structured content and isError. Upstream errors invalidate a baseline case. - Invalid arguments (
invalid_input): Use each tool schema to derive missing-required-field and wrong-type cases. Record whether each gateway rejects invalid input or returns isError. Report exact error envelopes separately; differing validation layers are not automatically correctness failures. - Paired echo latency (
latency_small_calls): For each gateway: 20 paired echo warmups, then 100 measured pairs in each of three rounds. Alternate direct/gateway call order. Retain every timing and error; report median of successful per-pair gateway-minus-direct durations.
What a result means
Every contender runs every scenario, with the same inputs, in the same environment, on the same
day. A contender that could not run (no access, an outage, a rate limit) is recorded as
skipped or failed with the reason, and never given a number.