An MCP server whose `initialize` answer is an error — a broken or version-
mismatched connector — starts fine and then never exits. `McpServer::start`
returned `Err` correctly, but the `Child` lives in the read-loop task rather
than in the returned value, so `kill_on_drop` followed a task nothing ever
drops. Every retry therefore left a live process holding three pipes and a
pidfd, and the supervisor's retry ceiling is deliberately not permanent.
The end state was not a dead connector but a dead instance: the process hit
its 1024-descriptor limit, `accept()` began failing with EMFILE, and incoming
connections queued on a socket nobody could accept from — while the process,
the port and every other connector still looked healthy. Observed in
production at ~5h from the first bad handshake to unreachable, with 229
orphaned interpreters.
`stop_server`/`stop_all` had the same hole from the other side: they document
the dropped handle as killing the process, but the task holds its end of
stdin, so the child stayed blocked on a read that would never return.
Both close with one seam. `McpServer` now owns a oneshot sender whose receiver
the read-loop selects on; nothing ever sends, so the drop is the message. That
covers a deliberate stop, the last `Arc` going away, and a `?` in `start()`
unwinding past the local binding before it was ever returned — including the
caller's `timeout`, which drops the same future. The loop then kills and, as
importantly, reaps: an unreaped child trades the orphan for a zombie holding
the same pipes.
Both leaks are covered by tests that fail without the fix.
Also raise LimitNOFILE to 65536: the installers write it, and update.sh heals
an existing unit additively, leaving an admin's own value alone. The leak is
the bug, but 1024 for a process sharing descriptors between the listener,
every user's SQLite handles and three pipes per connector is thin regardless.
Four things found while re-reading the family of scripts around the
logout fix.
Both installers piped curl straight into tar, so a truncated download
half-extracted — and the installer explicitly supports reinstalling over
an existing install, which turned an interrupted download into a tree
mixing old and new files with no error saying so. They now download to a
temp file and verify the archive in a staging dir before writing
anything to the install directory: the ordering update.sh has had since
it was written, for the same reason.
update.sh never removed files deleted upstream. Extracting over the
install dir only adds and overwrites, so a renamed page under docs/ kept
being mounted read-only into every container for the assistant to read,
and a removed command kept being discovered. It now prunes, from the
directories the tarball owns end to end (web, commands, skills, docs),
whatever the already-verified staging copy does not have. Pruning after
the extraction rather than replacing the directory keeps every
intermediate state a complete install. agents/ is deliberately excluded:
dropping in an agent is a documented extension point, so that directory
is not ours alone and pruning it would delete somebody's work.
uninstall.sh fed `docker ps -aq --filter 'name=skald-'` to `docker rm
-f`. Docker's name filter is a regex matched anywhere in the name, not a
prefix, so any unrelated container merely containing "skald-" was
force-removed. Anchored to ^skald-.
uninstall.sh also matched uname's raw Linux/Darwin while its three
siblings normalize to lowercase. It was correct on its own, but being
the odd one out of four copy-paste relatives is precisely how update.sh
acquired its no-op case arms, so it now normalizes like the others.
Finally, the uninstaller reports that lingering is still enabled and how
to turn it off, rather than disabling it: it is a persistent per-user
setting other user services may rely on by now, so taking it back
silently would stop those too.
A `systemctl --user` unit runs under the per-user manager, which systemd
starts at first login and stops when the user's last session ends — so
closing the SSH session that started Skald killed it, and it never came
up at boot. No crash and nothing in the journal: the whole cgroup is
simply torn down. Both installers now enable lingering after installing
the unit, and update.sh carries the same helper so an installation
predating this fix is healed by an ordinary update. A failure to enable
it only ever warns, with the manual command — it must not abort an
install.
Two things found on the way there:
update.sh matched `case "$OS" in Linux) ... Darwin)`, but $OS had already
been normalized to lowercase at the top of the file, so stop_service and
start_service were both silent no-ops. None of the ordering the file
documents at its head was executing: the tarball went over the running
binary (ETXTBSY, aborting the update mid-way) and the safety-net restart
in cleanup() was a no-op too, leaving the box down.
Neither workflow published install.sh / install-nightly.sh to the web
root, so the scripts served by builds.skaldagent.net were hand-copied and
drifting from the repo — an installer fix would reach every existing box
through update.sh but never a new one. Nightly publishes the nightly
installer, release publishes the release one, both with the same atomic
temp-and-rename the tarballs use.
Also on the unit: dropped `After=docker.service`, which a user manager
silently ignores rather than honouring advisorily, and moved
`Restart=on-failure` to `always` — run.sh exits 0 on any graceful
shutdown, including one nobody asked for, which on-failure reads as a
clean stop. That is also what absorbs the boot race against Docker now
that lingering makes us start at boot.
scripts/ held the pre-marketplace MCP servers (gmail, gcal, gmaps, ssh, weather,
google_trends, whatsapp, serpapi_flights). Nothing referenced them any more:
connectors are admin-curated and installed into connectors/ from the
marketplace, and ci/package.sh never shipped scripts/ in the first place — so on
every installed box requirements.txt was pulling google-auth, googlemaps,
paramiko, trendspyg and friends for files that did not exist there.
requirements.txt now states what it is actually for: the two TTS plugins, which
spawn a bare `python3` on an embedded server script and so have no dependency
reconciler of their own. A connector's deps stay with the connector —
`ensure_installed` puts them in .pydeps/node_modules inside the user's
container, `ensure_installed_host` beside the files for a global one.
The venv itself stays load-bearing for those two plugins and for the host pip
that installs a global connector's deps, so the run/install/update scripts keep
creating it — but their "Python MCP servers will be unavailable" warning was
naming the one thing that no longer depends on it, and now says what really
breaks.
CONNECTOR_MANIFEST_GUIDE.md moves to the repo root: it was the one thing in
scripts/ still referenced (CLAUDE.md), and being under a gitignored directory it
had never been committed at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
torch and bitsandbytes pull in hundreds of MB of CUDA libraries
(nvidia_cublas ~423 MB) which are useless on headless servers
without an NVIDIA GPU like the NiPoGi.
- requirements.txt: keep only lightweight essential dependencies
- requirements-optional.txt (new): Orpheus TTS deps (torch, transformers,
bitsandbytes, snac, huggingface_hub)
- Installers and update.sh: show hint about optional deps after install
- ci/package.sh: include requirements-optional.txt in tarball
install.sh / install-nightly.sh:
- Write .release-channel file ('release' or 'nightly') for future updates
- install.sh also writes .release-version (for update --check)
update.sh (new):
- Reads .release-channel to determine which channel to pull from
- Release: checks releases/LATEST vs .release-version, skips if current
- Nightly: always downloads latest
- Stops service before extraction, restarts after
- Rebuilds Python venv on update
ci/package.sh:
- Include update.sh in distribution tarball
Web / mobile:
- Add settings-page component for mobile
- Wire settings page into mobile-app navigation
- Chat page: load current user (/api/auth/me) for sender identity
- Full mobile.css redesign
- i18n: add mobile settings strings (en/fr/it)
uv venv does not bundle pip by default (unlike python3 -m venv), so when the
venv python was on PATH, ensure_installed_host's 'python3 -m pip install' failed
with 'No module named pip'. Apply --seed across the 4 copies of the venv-setup
block (install.sh, install-nightly.sh, run.sh, run-docker.sh).
Daniele
dguiducci