Pilot

MCP Gateway Bench

Do MCP tools behave identically through each gateway as they do directly, and what latency does each gateway add?

Speakeasy MCP Gateway against MCPJungle, Docker MCP Gateway, IBM MCP Context Forge. Every gateway routes to the same independently maintained reference server, and every call is compared with the same call made directly.

Leaderboard

Run of . Ranked on tool-call correctness (higher is better).

Pilot run of 2026-11-16, 4 of 4 gateways measured; tool-call results identical to calling the reference server directly: Speakeasy MCP Gateway (hosted) 100%, median added latency 374.6 ms; MCPJungle (local) 100%, median added latency 217.9 ms; Docker MCP Gateway (local) 100%, median added latency -18.1 ms; IBM MCP Context Forge (local) 100%, median added latency 281.2 ms.

RankGatewayWhere it ranStatusTool-call correctness
higher is better
Median paired added latency
lower is better
Timed call error rate
lower is better
Notes
1Speakeasy MCP Gateway (subject)Hosted serviceMeasured100%374.6 ms0%
1MCPJungleLocal, on the bench machineMeasured100%217.9 ms0%
1Docker MCP GatewayLocal, on the bench machineMeasured100%-18.1 ms0%
1IBM MCP Context ForgeLocal, on the bench machineMeasured100%281.2 ms0%

Tool-call correctness is the share of valid reference cases whose result through the gateway was identical to the direct call. Added latency is the median, over successful pairs, of gateway time minus direct time for the same echo call; a negative value means the route through the gateway was faster than the bench machine's own route to the reference server. It includes network distance and cold starts, so hosted and local gateways are not on equal footing.

Per gateway

Speakeasy MCP Gateway

Measured Hosted service

Version
Speakeasy Gateway 0.0.0
Region
hosted; region not reported by the service
Discovery
3 of 3 tools resolved through meta-tools (list_servers, describe_tools, execute_tool)
Reference calls
8 of 8 valid cases identical to the direct call (5 repetitions each)
Invalid arguments
6 of 6 rejected or returned isError; 6 answered the same way as the direct call
Added latency
median 374.59 ms (95% CI 371.84 to 377.66 ms), p10 351.01 ms, p90 403.84 ms
Samples
300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
End to end
direct median 148.94 ms, through the gateway 527.14 ms

MCPJungle

Measured Local, on the bench machine

Version
MCPJungle 0.4.7 (server reports v0.4.7)
Region
local: on the benchmark machine
Discovery
3 of 3 tools resolved through tools/list; some input schemas differ from upstream
Reference calls
8 of 8 valid cases identical to the direct call (5 repetitions each)
Invalid arguments
6 of 6 rejected or returned isError; 6 answered the same way as the direct call
Added latency
median 217.89 ms (95% CI 216.18 to 220.67 ms), p10 200.15 ms, p90 249.26 ms
Samples
300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
End to end
direct median 153.93 ms, through the gateway 372.86 ms

Docker MCP Gateway

Measured Local, on the bench machine

Version
Docker MCP Gateway v0.44.3 (reports v0.44.3); server reports Docker AI MCP Gateway 2.0.1
Region
local: on the benchmark machine
Discovery
3 of 3 tools resolved through tools/list
Reference calls
8 of 8 valid cases identical to the direct call (5 repetitions each)
Invalid arguments
6 of 6 rejected or returned isError; 6 answered the same way as the direct call
Added latency
median -18.11 ms (95% CI -18.99 to -17.16 ms), p10 -31.48 ms, p90 -5.3 ms
Samples
300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
End to end
direct median 148.16 ms, through the gateway 130.62 ms

IBM MCP Context Forge

Measured Local, on the bench machine

Version
IBM MCP Context Forge 1.1.1 on Python 3.12.16; server reports ContextForge 1.1.1
Region
local: on the benchmark machine
Discovery
3 of 3 tools resolved through tools/list
Reference calls
8 of 8 valid cases identical to the direct call (5 repetitions each)
Invalid arguments
6 of 6 rejected or returned isError; 6 answered the same way as the direct call
Added latency
median 281.15 ms (95% CI 278.1 to 283.5 ms), p10 257.87 ms, p90 328.08 ms
Samples
300 successful of 300 measured pairs (3 rounds of 100, after 20 warmup pairs); 0 gateway and 0 direct failures
End to end
direct median 150.07 ms, through the gateway 431.63 ms

What this run does not show

Where it ran

runnerlinux 6.1.188-maritime1 x64, 2 × Intel(R) Xeon(R) Processor @ 3.20GHz, 8 GB RAM
nodev24.22.1
bench_scalesmall: the METHODOLOGY.md pilot (20 warmup pairs, then 3 rounds of 100 measured echo pairs)
bench_repetitions5 (each reference case is called this many times on each route)
reference_urlhttps://mcp-bench-reference.vercel.app/mcp
orderContenders run one after another in bench.json order (subject first), as the repo contract requires; contender order is not rotated across rounds. Within a contender, each pair alternates direct/gateway order and consecutive rounds start with opposite orders.
call_timeout20 s per call; failed and timed-out calls are errors, never fast successes
warmup3 untimed echo pairs before the reference cases, then 20 untimed pairs before the latency rounds
reference_server@modelcontextprotocol/server-everything 2.3.0 (stateless-streamable-http)
reference_regionVercel function iad1, reached through edge fra1
subject_versionSpeakeasy Gateway 0.0.0
subject_regionhosted; region not reported by the service
mcpjungle_versionMCPJungle 0.4.7 (server reports v0.4.7)
mcpjungle_regionlocal: on the benchmark machine
docker-mcp-gateway_versionDocker MCP Gateway v0.44.3 (reports v0.44.3); server reports Docker AI MCP Gateway 2.0.1
docker-mcp-gateway_regionlocal: on the benchmark machine
mcp-context-forge_versionIBM MCP Context Forge 1.1.1 on Python 3.12.16; server reports ContextForge 1.1.1
mcp-context-forge_regionlocal: on the benchmark machine

The full file, with every case, envelope and timing: results/latest.json. Earlier runs: all results.

Scenarios

How it is measured: methodology. Run it yourself: replicate.