Improve HTTP Headers: Server Compatibility Issues Due to Missing/Wrong Headers

[bettercallsaulj]

Issue Details

Problem Description

The MCP client's HTTP requests were rejected or handled incorrectly by some MCP servers due to missing or incompatible headers:

1. Wrong Accept header for POST requests: Client sent Accept: text/event-stream only, but many MCP servers (especially Streamable HTTP servers) require application/json in the Accept header to return JSON responses
2. Missing User-Agent header: Requests didn't include a User-Agent, making server-side debugging and logging difficult, and some servers may reject requests without it

Root Cause

The HTTP codec filter hardcoded headers that were SSE-centric:

// POST request headers (BEFORE FIX)
request << "Content-Type: application/json\r\n";
request << "Content-Length: " << body_length << "\r\n";
request << "Accept: text/event-stream\r\n";  // ❌ Only SSE - servers may need JSON
request << "Connection: keep-alive\r\n";
// ❌ Missing User-Agent header

When a Streamable HTTP server receives this request, it may:

• Return 406 Not Acceptable (doesn't support text/event-stream)
• Return response in wrong format
• Log "unknown client" making debugging difficult

Technical Flow (Before Fix)

Client                                        Server (Streamable HTTP)
  |                                              |
  |-- POST /rpc HTTP/1.1 ---------------------->|
  |   Host: server.example.com                   |
  |   Content-Type: application/json             |
  |   Content-Length: 42                         |
  |   Accept: text/event-stream        ❌        |
  |   Connection: keep-alive                     |
  |   (no User-Agent)                  ❌        |
  |                                              |
  |<-- HTTP 406 Not Acceptable -----------------|  ❌
  |   OR                                         |
  |<-- Server tries to return SSE stream -------|  ❌ Wrong format
  |   event: message                             |
  |   data: {...}                                |
  |                                              |
  |-- Client can't parse SSE response ----------|  ❌

Technical Flow (After Fix)

Client                                        Server (Streamable HTTP)
  |                                              |
  |-- POST /rpc HTTP/1.1 ---------------------->|
  |   Host: server.example.com                   |
  |   Content-Type: application/json             |
  |   Content-Length: 42                         |
  |   Accept: application/json, text/event-stream| ✅ Both types
  |   Connection: keep-alive                     |
  |   User-Agent: gopher-mcp/1.0       ✅        |
  |                                              |
  |<-- HTTP 200 OK -----------------------------|
  |   Content-Type: application/json             |
  |   {"jsonrpc":"2.0","result":{...}}           |  ✅ JSON response
  |                                              |
  |-- Client parses JSON response ------------->|  ✅

Client                                        Server (SSE)
  |                                              |
  |-- GET /sse HTTP/1.1 ----------------------->|
  |   Host: server.example.com                   |
  |   Accept: text/event-stream                  |
  |   Cache-Control: no-cache                    |
  |   Connection: keep-alive                     |
  |   User-Agent: gopher-mcp/1.0       ✅        |
  |                                              |
  |<-- HTTP 200 OK -----------------------------|
  |   Content-Type: text/event-stream            |
  |   event: endpoint                            |  ✅ SSE response
  |   data: http://server/message                |

---

How to Reproduce

Issue 1: Server Returns 406 Not Acceptable

void reproduce406Error() {
  McpClient client;
  client.setUri("http://strict-server:8080/rpc");

  auto result = client.connect();
  auto init_future = client.initializeProtocol();

  // Before fix: Server may return 406 Not Acceptable
  // because it doesn't accept "text/event-stream" for POST requests

  // Server logs: "Client requested text/event-stream but endpoint only serves JSON"

  // After fix: Accept header includes "application/json"
  // Server returns JSON response successfully
}

Issue 2: Server Returns Wrong Response Format

void reproduceWrongResponseFormat() {
  McpClient client;
  client.setUri("http://flexible-server:8080/rpc");

  // Server supports both formats, picks based on Accept header
  auto result = client.connect();
  auto init_future = client.initializeProtocol();

  // Before fix: Server sees "Accept: text/event-stream"
  // Returns SSE format: "event: message\ndata: {...}\n\n"
  // Client expects plain JSON, parsing fails

  // After fix: Server sees "Accept: application/json, text/event-stream"
  // Prefers JSON (listed first), returns: {"jsonrpc":"2.0",...}
  // Client parses successfully
}

Issue 3: Server Can't Identify Client

void reproduceUnknownClient() {
  // Server admin checking logs for debugging
  // Server log entry:
  // [2026-01-21 10:00:00] POST /rpc from 192.168.1.100 - User-Agent: (none)
  //
  // Before fix: No User-Agent, hard to identify which client software
  // After fix: User-Agent: gopher-mcp/1.0, easy to identify
}

Minimal Reproduction Test

TEST(HttpHeaders, PostRequestAcceptHeaderIncludesJson) {
  HttpCodecFilter filter(callbacks, dispatcher, false /* client */);
  filter.setClientEndpoint("/rpc", "localhost:8080");

  OwnedBuffer buffer;
  buffer.add("{\"jsonrpc\":\"2.0\",\"method\":\"test\",\"id\":1}");
  filter.onWrite(buffer, false);

  std::string request = buffer.toString();

  // Before fix: Only "Accept: text/event-stream"
  // After fix: "Accept: application/json, text/event-stream"
  EXPECT_NE(request.find("Accept: application/json, text/event-stream"),
            std::string::npos);
}

TEST(HttpHeaders, PostRequestIncludesUserAgent) {
  HttpCodecFilter filter(callbacks, dispatcher, false /* client */);
  filter.setClientEndpoint("/rpc", "localhost:8080");

  OwnedBuffer buffer;
  buffer.add("{\"test\":true}");
  filter.onWrite(buffer, false);

  std::string request = buffer.toString();

  // Before fix: No User-Agent header
  // After fix: "User-Agent: gopher-mcp/1.0"
  EXPECT_NE(request.find("User-Agent: gopher-mcp/1.0"), std::string::npos);
}

TEST(HttpHeaders, SseGetRequestIncludesUserAgent) {
  HttpCodecFilter filter(callbacks, dispatcher, false /* client */);
  filter.setClientEndpoint("/sse", "localhost:8080");
  filter.setUseSseGet(true);

  OwnedBuffer buffer;
  filter.onWrite(buffer, false);

  std::string request = buffer.toString();

  // Before fix: No User-Agent header
  // After fix: "User-Agent: gopher-mcp/1.0"
  EXPECT_NE(request.find("User-Agent: gopher-mcp/1.0"), std::string::npos);
}

---

Key Changes in the Fix

Component	Change
POST Accept header	Changed from text/event-stream to application/json, text/event-stream
POST User-Agent	Added User-Agent: gopher-mcp/1.0 header
SSE GET User-Agent	Added User-Agent: gopher-mcp/1.0 header

Updated POST Request Generation

// Before
request << "Accept: text/event-stream\r\n";
request << "Connection: keep-alive\r\n";
request << "\r\n";

// After
// MCP servers may require both Accept types
// Always include both to maximize compatibility
request << "Accept: application/json, text/event-stream\r\n";
request << "Connection: keep-alive\r\n";
request << "User-Agent: gopher-mcp/1.0\r\n";
request << "\r\n";

Updated SSE GET Request Generation

// Before
request << "Accept: text/event-stream\r\n";
request << "Cache-Control: no-cache\r\n";
request << "Connection: keep-alive\r\n";
request << "\r\n";

// After
request << "Accept: text/event-stream\r\n";
request << "Cache-Control: no-cache\r\n";
request << "Connection: keep-alive\r\n";
request << "User-Agent: gopher-mcp/1.0\r\n";
request << "\r\n";

Complete Header Comparison

POST Request (Before):

POST /rpc HTTP/1.1
Host: server.example.com
Content-Type: application/json
Content-Length: 42
Accept: text/event-stream
Connection: keep-alive

{"jsonrpc":"2.0","method":"test","id":1}

POST Request (After):

POST /rpc HTTP/1.1
Host: server.example.com
Content-Type: application/json
Content-Length: 42
Accept: application/json, text/event-stream
Connection: keep-alive
User-Agent: gopher-mcp/1.0

{"jsonrpc":"2.0","method":"test","id":1}

SSE GET Request (Before):

GET /sse HTTP/1.1
Host: server.example.com
Accept: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

SSE GET Request (After):

GET /sse HTTP/1.1
Host: server.example.com
Accept: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
User-Agent: gopher-mcp/1.0