fix: bring back an MCP connector whose process died
Nightly Build / build (push) Successful in 7m46s

A stdio connector *is* its child process, and nothing noticed when that
process went away. The handle stayed in the manager's map, so every later
tool call answered `MCP '<name>' disconnected: process exited with 139`,
and the connector's own background work stopped for good — until the user
happened to log in again.

The second half is the quiet one. A per-user connector is typically the
one that *pushes*: Gmail's poll thread produces the `event/new_email`
notifications that feed event triage. After a crash those simply stop,
with no call to fail and nothing in the UI to say so.

So a death is now reconciled, on the same terms as every other
reconciliation here: best-effort, bounded, settling at the next login if
it fails. `McpServerClient::is_alive` makes the death observable (the
read-loop clears the flag before failing the pending calls, so a caller
woken by the disconnect error finds a handle that admits it is dead), the
manager remembers the spec each server was started from, and one seam —
`restart_if_dead` — is driven from two places:

  - `call()`, which repairs the connector in time for the call that
    noticed it, so a crash costs one restart rather than a dead session;
  - a 10s sweep, which is the only thing that can bring back a connector
    nobody is calling.

The restart policy is a pure function so it can be tested without a DB
pool and a runtime. Backoff is enforced as a time gate, never a sleep: a
tool call that finds the gate shut fails immediately instead of parking a
waiting user behind a crash-loop, and the sweep retries later. Five
consecutive failures stop the attempts, and the reset window doubles as
the escape hatch — a box left running recovers from a transient outage
instead of staying dark.

`stop_server`/`stop_all` forget the spec, which is what keeps a stop a
stop: without it the sweep would resurrect a connector an admin had just
revoked, and a container remount would respawn into the container that
was being replaced.
This commit is contained in:
2026-08-07 12:27:20 +01:00
parent 31b4c76f51
commit c1177a934d
5 changed files with 346 additions and 0 deletions
+12
View File
@@ -273,6 +273,18 @@ pub enum McpCallResult {
pub trait McpServerClient: Send + Sync {
fn tools(&self) -> &[McpTool];
async fn call_tool(&self, name: &str, args: Value) -> anyhow::Result<McpCallResult>;
/// Whether this connection is still usable.
///
/// A stdio server *is* its child process: once that exits, the handle stays in
/// the manager's map but every call on it fails with a disconnect error, so
/// something has to be able to ask. The default is `true` for HTTP/SSE, which
/// holds no process and no long-lived connection — a dead remote surfaces per
/// call, and answering `false` here would make the manager "restart" a server
/// that was never running.
fn is_alive(&self) -> bool {
true
}
}
// ── Helpers ───────────────────────────────────────────────────────────────────
+12
View File
@@ -221,6 +221,10 @@ pub struct McpServer {
/// Capabilities the server advertised in its `InitializeResult`. Captured so a
/// future Tasks polling loop can gate on `tasks` support; unused for now.
server_capabilities: Value,
/// Cleared by the read-loop the moment the child process is gone, so the
/// manager can tell "this handle is dead" from "this call failed". Shared with
/// that task, which is the only writer.
alive: Arc<std::sync::atomic::AtomicBool>,
}
impl McpServer {
@@ -324,6 +328,8 @@ impl McpServer {
Arc::new(Mutex::new(HashMap::new()));
let pending_elicitations = Arc::new(AtomicUsize::new(0));
let alive = Arc::new(std::sync::atomic::AtomicBool::new(true));
let alive_bg = Arc::clone(&alive);
let pending_bg = pending.clone();
let server_name_bg = cfg.name.clone();
let notification_tx_bg = notification_tx;
@@ -381,6 +387,10 @@ impl McpServer {
),
_ => "process exited unexpectedly".into(),
};
// Publish the death *before* failing the pending calls: a caller woken
// by the error below must find `is_alive() == false`, or it would
// conclude the call failed on a healthy server and not restart it.
alive_bg.store(false, Ordering::SeqCst);
let error_msg = format!("MCP '{}' disconnected: {exit_info}", server_name_bg);
if let Some(tx) = &log_tx_bg {
let _ = tx.send(McpLogLine::lifecycle(server_name_bg.clone(), format!("disconnected: {exit_info}")));
@@ -401,6 +411,7 @@ impl McpServer {
tools: Vec::new(),
pending_elicitations,
server_capabilities: json!({}),
alive,
};
let init = server.request("initialize", json!({
@@ -632,4 +643,5 @@ impl McpServer {
impl McpServerClient for McpServer {
fn tools(&self) -> &[McpTool] { self.tools() }
async fn call_tool(&self, name: &str, args: Value) -> Result<McpCallResult> { self.call_tool(name, args).await }
fn is_alive(&self) -> bool { self.alive.load(Ordering::SeqCst) }
}