fix: bring back an MCP connector whose process died
Nightly Build / build (push) Successful in 7m46s
Nightly Build / build (push) Successful in 7m46s
A stdio connector *is* its child process, and nothing noticed when that
process went away. The handle stayed in the manager's map, so every later
tool call answered `MCP '<name>' disconnected: process exited with 139`,
and the connector's own background work stopped for good — until the user
happened to log in again.
The second half is the quiet one. A per-user connector is typically the
one that *pushes*: Gmail's poll thread produces the `event/new_email`
notifications that feed event triage. After a crash those simply stop,
with no call to fail and nothing in the UI to say so.
So a death is now reconciled, on the same terms as every other
reconciliation here: best-effort, bounded, settling at the next login if
it fails. `McpServerClient::is_alive` makes the death observable (the
read-loop clears the flag before failing the pending calls, so a caller
woken by the disconnect error finds a handle that admits it is dead), the
manager remembers the spec each server was started from, and one seam —
`restart_if_dead` — is driven from two places:
- `call()`, which repairs the connector in time for the call that
noticed it, so a crash costs one restart rather than a dead session;
- a 10s sweep, which is the only thing that can bring back a connector
nobody is calling.
The restart policy is a pure function so it can be tested without a DB
pool and a runtime. Backoff is enforced as a time gate, never a sleep: a
tool call that finds the gate shut fails immediately instead of parking a
waiting user behind a crash-loop, and the sweep retries later. Five
consecutive failures stop the attempts, and the reset window doubles as
the escape hatch — a box left running recovers from a transient outage
instead of staying dark.
`stop_server`/`stop_all` forget the spec, which is what keeps a stop a
stop: without it the sweep would resurrect a connector an admin had just
revoked, and a container remount would respawn into the container that
was being replaced.
This commit is contained in:
@@ -167,6 +167,10 @@ impl Integrations {
|
||||
"data",
|
||||
crate::mcp::EventLog::Discard,
|
||||
));
|
||||
// Supervise the global connectors: a crashed one is restarted rather than
|
||||
// staying dead until the process does. Spawned post-construction because the
|
||||
// sweep reacts through the `Arc` it is watching (see `spawn_respawn_sweep`).
|
||||
mcp.spawn_respawn_sweep(rt.shutdown_token.clone());
|
||||
|
||||
let mut plugin_manager = PluginManager::new(Arc::clone(&rt.db));
|
||||
for plugin in plugins {
|
||||
|
||||
Reference in New Issue
Block a user