fix(mcp): a failed handshake must not strand the child process
Nightly Build / build (push) Successful in 4m5s
Nightly Build / build (push) Successful in 4m5s
An MCP server whose `initialize` answer is an error — a broken or version- mismatched connector — starts fine and then never exits. `McpServer::start` returned `Err` correctly, but the `Child` lives in the read-loop task rather than in the returned value, so `kill_on_drop` followed a task nothing ever drops. Every retry therefore left a live process holding three pipes and a pidfd, and the supervisor's retry ceiling is deliberately not permanent. The end state was not a dead connector but a dead instance: the process hit its 1024-descriptor limit, `accept()` began failing with EMFILE, and incoming connections queued on a socket nobody could accept from — while the process, the port and every other connector still looked healthy. Observed in production at ~5h from the first bad handshake to unreachable, with 229 orphaned interpreters. `stop_server`/`stop_all` had the same hole from the other side: they document the dropped handle as killing the process, but the task holds its end of stdin, so the child stayed blocked on a read that would never return. Both close with one seam. `McpServer` now owns a oneshot sender whose receiver the read-loop selects on; nothing ever sends, so the drop is the message. That covers a deliberate stop, the last `Arc` going away, and a `?` in `start()` unwinding past the local binding before it was ever returned — including the caller's `timeout`, which drops the same future. The loop then kills and, as importantly, reaps: an unreaped child trades the orphan for a zombie holding the same pipes. Both leaks are covered by tests that fail without the fix. Also raise LimitNOFILE to 65536: the installers write it, and update.sh heals an existing unit additively, leaving an admin's own value alone. The leak is the bug, but 1024 for a process sharing descriptors between the listener, every user's SQLite handles and three pipes per connector is thin regardless.
This commit is contained in:
@@ -201,6 +201,47 @@ ensure_linger() {
|
||||
fi
|
||||
}
|
||||
|
||||
# ── File-descriptor limit ─────────────────────────────────────────────────────
|
||||
# Same reasoning as ensure_linger: the installers now write LimitNOFILE into the
|
||||
# unit, and this heals an install that predates them, since an update never
|
||||
# rewrites the unit file.
|
||||
#
|
||||
# Worth the repair rather than leaving it to the next reinstall, because running
|
||||
# out of descriptors does not degrade gracefully. One process shares the default
|
||||
# 1024 between the HTTP listener, every user's SQLite handles and three pipes per
|
||||
# connector; past the ceiling accept() fails with EMFILE and the app stops
|
||||
# answering while the process, the port and the health of every connector all
|
||||
# still look fine.
|
||||
#
|
||||
# Strictly additive: it appends one line to [Service] and touches nothing else,
|
||||
# so a hand-customized unit survives. Skipped entirely if the admin already set
|
||||
# any LimitNOFILE of their own.
|
||||
ensure_fd_limit() {
|
||||
[ "$OS" = "linux" ] || return 0
|
||||
|
||||
local unit="$HOME/.config/systemd/user/skald-circle.service"
|
||||
|
||||
[ -f "$unit" ] || return 0
|
||||
command -v systemctl >/dev/null 2>&1 || return 0
|
||||
grep -q '^[[:space:]]*LimitNOFILE=' "$unit" && return 0
|
||||
grep -q '^\[Service\]' "$unit" || return 0
|
||||
|
||||
# Write through a temp file so an interrupted update can never leave a
|
||||
# half-written unit behind.
|
||||
local tmp="${unit}.tmp.$$"
|
||||
if awk '/^\[Service\]/ && !done { print; print "LimitNOFILE=65536"; done=1; next } { print }' \
|
||||
"$unit" > "$tmp" 2>/dev/null && [ -s "$tmp" ]; then
|
||||
mv "$tmp" "$unit" \
|
||||
&& systemctl --user daemon-reload 2>/dev/null \
|
||||
&& info "✔ Raised the file-descriptor limit to 65536"
|
||||
else
|
||||
rm -f "$tmp"
|
||||
warn "Could not raise the file-descriptor limit; the default 1024 still applies."
|
||||
echo " Add this under [Service] in ${unit}:"
|
||||
echo " LimitNOFILE=65536"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Cleanup + safety net ──────────────────────────────────────────────────────
|
||||
# Runs on every exit. Removes temp files and, if the update died after the
|
||||
# service was stopped but before it came back up, makes a best-effort restart so
|
||||
@@ -349,6 +390,9 @@ main() {
|
||||
|
||||
# ── Restart ────────────────────────────────────────────────────────────────
|
||||
ensure_linger
|
||||
# Before the start, so the new limit applies to the process we are about to
|
||||
# bring up rather than to the one after it.
|
||||
ensure_fd_limit
|
||||
start_service
|
||||
STARTED=1
|
||||
|
||||
|
||||
Reference in New Issue
Block a user