fix: bring back an MCP connector whose process died
Nightly Build / build (push) Successful in 7m46s
Nightly Build / build (push) Successful in 7m46s
A stdio connector *is* its child process, and nothing noticed when that
process went away. The handle stayed in the manager's map, so every later
tool call answered `MCP '<name>' disconnected: process exited with 139`,
and the connector's own background work stopped for good — until the user
happened to log in again.
The second half is the quiet one. A per-user connector is typically the
one that *pushes*: Gmail's poll thread produces the `event/new_email`
notifications that feed event triage. After a crash those simply stop,
with no call to fail and nothing in the UI to say so.
So a death is now reconciled, on the same terms as every other
reconciliation here: best-effort, bounded, settling at the next login if
it fails. `McpServerClient::is_alive` makes the death observable (the
read-loop clears the flag before failing the pending calls, so a caller
woken by the disconnect error finds a handle that admits it is dead), the
manager remembers the spec each server was started from, and one seam —
`restart_if_dead` — is driven from two places:
- `call()`, which repairs the connector in time for the call that
noticed it, so a crash costs one restart rather than a dead session;
- a 10s sweep, which is the only thing that can bring back a connector
nobody is calling.
The restart policy is a pure function so it can be tested without a DB
pool and a runtime. Backoff is enforced as a time gate, never a sleep: a
tool call that finds the gate shut fails immediately instead of parking a
waiting user behind a crash-loop, and the sweep retries later. Five
consecutive failures stop the attempts, and the reset window doubles as
the escape hatch — a box left running recovers from a transient outage
instead of staying dark.
`stop_server`/`stop_all` forget the spec, which is what keeps a stop a
stop: without it the sweep would resurrect a connector an admin had just
revoked, and a container remount would respawn into the container that
was being replaced.
This commit is contained in:
@@ -273,6 +273,18 @@ pub enum McpCallResult {
|
||||
pub trait McpServerClient: Send + Sync {
|
||||
fn tools(&self) -> &[McpTool];
|
||||
async fn call_tool(&self, name: &str, args: Value) -> anyhow::Result<McpCallResult>;
|
||||
|
||||
/// Whether this connection is still usable.
|
||||
///
|
||||
/// A stdio server *is* its child process: once that exits, the handle stays in
|
||||
/// the manager's map but every call on it fails with a disconnect error, so
|
||||
/// something has to be able to ask. The default is `true` for HTTP/SSE, which
|
||||
/// holds no process and no long-lived connection — a dead remote surfaces per
|
||||
/// call, and answering `false` here would make the manager "restart" a server
|
||||
/// that was never running.
|
||||
fn is_alive(&self) -> bool {
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
// ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -221,6 +221,10 @@ pub struct McpServer {
|
||||
/// Capabilities the server advertised in its `InitializeResult`. Captured so a
|
||||
/// future Tasks polling loop can gate on `tasks` support; unused for now.
|
||||
server_capabilities: Value,
|
||||
/// Cleared by the read-loop the moment the child process is gone, so the
|
||||
/// manager can tell "this handle is dead" from "this call failed". Shared with
|
||||
/// that task, which is the only writer.
|
||||
alive: Arc<std::sync::atomic::AtomicBool>,
|
||||
}
|
||||
|
||||
impl McpServer {
|
||||
@@ -324,6 +328,8 @@ impl McpServer {
|
||||
Arc::new(Mutex::new(HashMap::new()));
|
||||
let pending_elicitations = Arc::new(AtomicUsize::new(0));
|
||||
|
||||
let alive = Arc::new(std::sync::atomic::AtomicBool::new(true));
|
||||
let alive_bg = Arc::clone(&alive);
|
||||
let pending_bg = pending.clone();
|
||||
let server_name_bg = cfg.name.clone();
|
||||
let notification_tx_bg = notification_tx;
|
||||
@@ -381,6 +387,10 @@ impl McpServer {
|
||||
),
|
||||
_ => "process exited unexpectedly".into(),
|
||||
};
|
||||
// Publish the death *before* failing the pending calls: a caller woken
|
||||
// by the error below must find `is_alive() == false`, or it would
|
||||
// conclude the call failed on a healthy server and not restart it.
|
||||
alive_bg.store(false, Ordering::SeqCst);
|
||||
let error_msg = format!("MCP '{}' disconnected: {exit_info}", server_name_bg);
|
||||
if let Some(tx) = &log_tx_bg {
|
||||
let _ = tx.send(McpLogLine::lifecycle(server_name_bg.clone(), format!("disconnected: {exit_info}")));
|
||||
@@ -401,6 +411,7 @@ impl McpServer {
|
||||
tools: Vec::new(),
|
||||
pending_elicitations,
|
||||
server_capabilities: json!({}),
|
||||
alive,
|
||||
};
|
||||
|
||||
let init = server.request("initialize", json!({
|
||||
@@ -632,4 +643,5 @@ impl McpServer {
|
||||
impl McpServerClient for McpServer {
|
||||
fn tools(&self) -> &[McpTool] { self.tools() }
|
||||
async fn call_tool(&self, name: &str, args: Value) -> Result<McpCallResult> { self.call_tool(name, args).await }
|
||||
fn is_alive(&self) -> bool { self.alive.load(Ordering::SeqCst) }
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user