Pilot
MCP Gateway Bench
Do MCP tools behave identically through each gateway as they do directly, and what latency does each gateway add?
Speakeasy MCP Gateway against MCPJungle, Docker MCP Gateway, IBM MCP Context Forge. Every gateway routes to the same independently maintained reference server, and every call is compared with the same call made directly.
Leaderboard
Run of . Ranked on tool-call correctness (higher is better).
Pilot run of 2026-11-16, 4 of 4 gateways measured; tool-call results identical to calling the reference server directly: Speakeasy MCP Gateway (hosted) 100%, median added latency 374.6 ms; MCPJungle (local) 100%, median added latency 217.9 ms; Docker MCP Gateway (local) 100%, median added latency -18.1 ms; IBM MCP Context Forge (local) 100%, median added latency 281.2 ms.
| Rank | Gateway | Where it ran | Status | Tool-call correctness higher is better | Median paired added latency lower is better | Timed call error rate lower is better | Notes |
|---|---|---|---|---|---|---|---|
| 1 | Speakeasy MCP Gateway (subject) | Hosted service | Measured | 100% | 374.6 ms | 0% | |
| 1 | MCPJungle | Local, on the bench machine | Measured | 100% | 217.9 ms | 0% | |
| 1 | Docker MCP Gateway | Local, on the bench machine | Measured | 100% | -18.1 ms | 0% | |
| 1 | IBM MCP Context Forge | Local, on the bench machine | Measured | 100% | 281.2 ms | 0% |
Tool-call correctness is the share of valid reference cases whose result through the gateway was identical to the direct call. Added latency is the median, over successful pairs, of gateway time minus direct time for the same echo call; a negative value means the route through the gateway was faster than the bench machine's own route to the reference server. It includes network distance and cold starts, so hosted and local gateways are not on equal footing.
Per gateway
Speakeasy MCP Gateway
Measured Hosted service
- Version
- Speakeasy Gateway 0.0.0
- Region
- hosted; region not reported by the service
- Discovery
- 3 of 3 tools resolved through meta-tools (list_servers, describe_tools, execute_tool)
- Reference calls
- 8 of 8 valid cases identical to the direct call (5 repetitions each)
- Invalid arguments
- 6 of 6 rejected or returned isError; 6 answered the same way as the direct call
- Added latency
- median 374.59 ms (95% CI 371.84 to 377.66 ms), p10 351.01 ms, p90 403.84 ms
- Samples
- 300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
- End to end
- direct median 148.94 ms, through the gateway 527.14 ms
MCPJungle
Measured Local, on the bench machine
- Version
- MCPJungle 0.4.7 (server reports v0.4.7)
- Region
- local: on the benchmark machine
- Discovery
- 3 of 3 tools resolved through tools/list; some input schemas differ from upstream
- Reference calls
- 8 of 8 valid cases identical to the direct call (5 repetitions each)
- Invalid arguments
- 6 of 6 rejected or returned isError; 6 answered the same way as the direct call
- Added latency
- median 217.89 ms (95% CI 216.18 to 220.67 ms), p10 200.15 ms, p90 249.26 ms
- Samples
- 300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
- End to end
- direct median 153.93 ms, through the gateway 372.86 ms
Docker MCP Gateway
Measured Local, on the bench machine
- Version
- Docker MCP Gateway v0.44.3 (reports v0.44.3); server reports Docker AI MCP Gateway 2.0.1
- Region
- local: on the benchmark machine
- Discovery
- 3 of 3 tools resolved through tools/list
- Reference calls
- 8 of 8 valid cases identical to the direct call (5 repetitions each)
- Invalid arguments
- 6 of 6 rejected or returned isError; 6 answered the same way as the direct call
- Added latency
- median -18.11 ms (95% CI -18.99 to -17.16 ms), p10 -31.48 ms, p90 -5.3 ms
- Samples
- 300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
- End to end
- direct median 148.16 ms, through the gateway 130.62 ms
IBM MCP Context Forge
Measured Local, on the bench machine
- Version
- IBM MCP Context Forge 1.1.1 on Python 3.12.16; server reports ContextForge 1.1.1
- Region
- local: on the benchmark machine
- Discovery
- 3 of 3 tools resolved through tools/list
- Reference calls
- 8 of 8 valid cases identical to the direct call (5 repetitions each)
- Invalid arguments
- 6 of 6 rejected or returned isError; 6 answered the same way as the direct call
- Added latency
- median 281.15 ms (95% CI 278.1 to 283.5 ms), p10 257.87 ms, p90 328.08 ms
- Samples
- 300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
- End to end
- direct median 150.07 ms, through the gateway 431.63 ms
What this run does not show
- This is a pilot: eight reference cases and paired echo timings, not broad protocol conformance or enterprise readiness.
- Added latency is end-to-end: it includes each route's network distance, proxy behaviour and serverless cold starts, not pure gateway processing time.
- The harness repository is private, so its code cannot be inspected publicly until its owner makes it public.
- Runs are not yet scheduled unattended: the Speakeasy OAuth token must be renewed by an operator before each run.
Where it ran
runner | linux 6.1.188-maritime1 x64, 2 × Intel(R) Xeon(R) Processor @ 3.20GHz, 8 GB RAM |
|---|---|
node | v24.22.1 |
bench_scale | small: the METHODOLOGY.md pilot (20 warmup pairs, then 3 rounds of 100 measured echo pairs) |
bench_repetitions | 5 (each reference case is called this many times on each route) |
reference_url | https://mcp-bench-reference.vercel.app/mcp |
order | Contenders run one after another in bench.json order (subject first), as the repo contract requires; contender order is not rotated across rounds. Within a contender, each pair alternates direct/gateway order and consecutive rounds start with opposite orders. |
call_timeout | 20 s per call; failed and timed-out calls are errors, never fast successes |
warmup | 3 untimed echo pairs before the reference cases, then 20 untimed pairs before the latency rounds |
reference_server | @modelcontextprotocol/server-everything 2.3.0 (stateless-streamable-http) |
reference_region | Vercel function iad1, reached through edge fra1 |
subject_version | Speakeasy Gateway 0.0.0 |
subject_region | hosted; region not reported by the service |
mcpjungle_version | MCPJungle 0.4.7 (server reports v0.4.7) |
mcpjungle_region | local: on the benchmark machine |
docker-mcp-gateway_version | Docker MCP Gateway v0.44.3 (reports v0.44.3); server reports Docker AI MCP Gateway 2.0.1 |
docker-mcp-gateway_region | local: on the benchmark machine |
mcp-context-forge_version | IBM MCP Context Forge 1.1.1 on Python 3.12.16; server reports ContextForge 1.1.1 |
mcp-context-forge_region | local: on the benchmark machine |
The full file, with every case, envelope and timing: results/latest.json. Earlier runs: all results.
Scenarios
- Tool discovery (
discovery): Resolve the same upstream echo, get-sum and get-tiny-image tools through each documented discovery interface. Speakeasy exposes meta-tools, so do not require tools/list names to equal upstream names. - Reference tool calls (
reference_calls): Run the eight transparent cases in scripts/preflight.py against the official server, directly and through each gateway. Compare text, image bytes/mime types, structured content and isError. Upstream errors invalidate a baseline case. - Invalid arguments (
invalid_input): Use each tool schema to derive missing-required-field and wrong-type cases. Record whether each gateway rejects invalid input or returns isError. Report exact error envelopes separately; differing validation layers are not automatically correctness failures. - Paired echo latency (
latency_small_calls): For each gateway: 20 paired echo warmups, then 100 measured pairs in each of three rounds. Alternate direct/gateway call order. Retain every timing and error; report median of successful per-pair gateway-minus-direct durations.
How it is measured: methodology. Run it yourself: replicate.