fix: keep the server running after you log out

A `systemctl --user` unit runs under the per-user manager, which systemd
starts at first login and stops when the user's last session ends — so
closing the SSH session that started Skald killed it, and it never came
up at boot. No crash and nothing in the journal: the whole cgroup is
simply torn down. Both installers now enable lingering after installing
the unit, and update.sh carries the same helper so an installation
predating this fix is healed by an ordinary update. A failure to enable
it only ever warns, with the manual command — it must not abort an
install.

Two things found on the way there:

update.sh matched `case "$OS" in Linux) ... Darwin)`, but $OS had already
been normalized to lowercase at the top of the file, so stop_service and
start_service were both silent no-ops. None of the ordering the file
documents at its head was executing: the tarball went over the running
binary (ETXTBSY, aborting the update mid-way) and the safety-net restart
in cleanup() was a no-op too, leaving the box down.

Neither workflow published install.sh / install-nightly.sh to the web
root, so the scripts served by builds.skaldagent.net were hand-copied and
drifting from the repo — an installer fix would reach every existing box
through update.sh but never a new one. Nightly publishes the nightly
installer, release publishes the release one, both with the same atomic
temp-and-rename the tarballs use.

Also on the unit: dropped `After=docker.service`, which a user manager
silently ignores rather than honouring advisorily, and moved
`Restart=on-failure` to `always` — run.sh exits 0 on any graceful
shutdown, including one nobody asked for, which on-failure reads as a
clean stop. That is also what absorbs the boot race against Docker now
that lingering makes us start at boot.

.gitea/workflows/nightly.yml

@@ -64,3 +64,18 @@ jobs:
           done
           echo "[nightly] Deployed:"
           ls -lh "$DEST/"
+
+      - name: Publish the nightly installer
+        run: |
+          cd "${GITHUB_WORKSPACE:-.}"
+          # install-nightly.sh is served straight from the web root
+          # (curl -fsSL https://builds.skaldagent.net/install-nightly.sh | bash),
+          # so without this it stays whatever was copied there by hand and drifts
+          # from the repo — a fix to the installer would reach every existing box
+          # through update.sh but never a new one. Same atomic publish as the
+          # tarballs: a client mid-download never sees a half-written script.
+          ROOT=/var/www/builds.skaldagent.net
+          cp install-nightly.sh "$ROOT/.install-nightly.sh.tmp"
+          chmod 644 "$ROOT/.install-nightly.sh.tmp"
+          mv -f "$ROOT/.install-nightly.sh.tmp" "$ROOT/install-nightly.sh"
+          echo "[nightly] Published install-nightly.sh"

.gitea/workflows/release.yml

@@ -104,3 +104,18 @@ jobs:
           printf '%s\n' "$VERSION" > "$DEST/.LATEST.tmp"
           mv -f "$DEST/.LATEST.tmp" "$DEST/LATEST"
           echo "[release] Updated releases/LATEST → $VERSION"
+
+      - name: Publish the release installer
+        run: |
+          cd "${GITHUB_WORKSPACE:-.}"
+          # install.sh is served straight from the web root
+          # (curl -fsSL https://builds.skaldagent.net/install.sh | bash), so
+          # without this it stays whatever was copied there by hand and drifts
+          # from the repo — a fix to the installer would reach every existing box
+          # through update.sh but never a new one. Published here rather than on
+          # every push so the served installer always matches a real release.
+          ROOT=/var/www/builds.skaldagent.net
+          cp install.sh "$ROOT/.install.sh.tmp"
+          chmod 644 "$ROOT/.install.sh.tmp"
+          mv -f "$ROOT/.install.sh.tmp" "$ROOT/install.sh"
+          echo "[release] Published install.sh"

SKALD.md

@@ -98,6 +98,22 @@ systemd service → ExecStart=run.sh
 
 **Fix**: removed `Requires=docker.service` from the user unit template in both install scripts. Kept `After=docker.service` (advisory, doesn't block if the unit isn't found).
 
+**Follow-up**: `After=docker.service` was dropped too. It never did anything — a _user_ manager has no view of system units, so the ordering was silently ignored rather than merely advisory, and keeping it suggested a guarantee that was not there. What actually handles the boot race is `Restart` (see below): the server fails fast when the Docker daemon is unreachable, and systemd brings it back a few seconds later.
+
+## Bug fix: the server dies when you log out ✅
+
+**Problem**: `systemctl --user start skald-circle` worked, but closing the SSH session killed the server — and it never came up at boot. Not an application bug: a `--user` unit runs under the per-user manager (`user@UID.service`), which systemd starts at first login and **stops when the user's last session ends**, tearing down every user service in the cgroup. No crash, no error in the journal — the whole cgroup is simply killed.
+
+**Fix**: both installers now run `loginctl enable-linger $USER` after installing the unit (helper `enable_linger`, tried unprivileged first, then `sudo -n`, then interactive `sudo`, and only warns if all three fail — a missing linger must never abort an install). `update.sh` carries the same helper so an installation predating this fix is healed by an ordinary update.
+
+**Also**: `Restart=on-failure` → `Restart=always`. `run.sh` exits 0 on _any_ graceful shutdown, including one nobody asked for (a stray SIGTERM to the server), which `on-failure` reads as a clean stop and leaves the box down. An explicit `systemctl --user stop` is unaffected — systemd never restarts after a requested stop. With lingering on, this is also what absorbs the boot race against Docker.
+
+## Bug fix: update.sh never stopped or restarted the service ✅
+
+**Problem**: `stop_service` and `start_service` matched `case "$OS" in Linux) … Darwin)`, but `$OS` had already been normalized to `linux`/`darwin` at the top of the script. Every branch fell through: both functions were no-ops. So the updater extracted the tarball **over the running binary** (`ETXTBSY` on Linux, aborting the update mid-way) and, when extraction did succeed, left the old build running in memory with the safety-net trap firing a restart that was itself a no-op. The careful stop → wait-for-exit → extract ordering the file documents at the top had not been executing at all.
+
+**Fix**: matched the normalized lowercase values, with a comment at the seam saying why the capitalization is load-bearing.
+
 ## Bug fix: skald-setup non interattivo con curl | bash ✅
 
 **Problem**: `skald-setup` controlla `isatty(0)`, ma con `curl ... | bash` stdin è un pipe, quindi saltava senza chiedere username/password. L'installer arrivava fino in fondo ma senza aver creato l'admin.

install-nightly.sh

@@ -127,6 +127,43 @@ stop_existing_service() {
     fi
 }
 
+# ── systemd user lingering ────────────────────────────────────────────────────
+# A `systemctl --user` unit runs under the per-user manager (user@UID.service),
+# which systemd starts at first login and STOPS when the user's last session
+# ends — taking every user service down with it. So without lingering the server
+# dies the moment you close the SSH session that started it, and never comes up
+# at boot. Enabling it is the whole difference between "runs while I'm logged
+# in" and "is a daemon".
+enable_linger() {
+    local target="${USER:-$(id -un)}"
+
+    if ! command -v loginctl >/dev/null 2>&1; then
+        warn "loginctl not found — cannot enable lingering."
+        echo "  The server will stop when you log out of this machine."
+        return 0
+    fi
+
+    case "$(loginctl show-user "$target" --property=Linger 2>/dev/null || true)" in
+        *=yes) info "✔ Lingering already enabled for ${target}"; return 0 ;;
+    esac
+
+    # Enabling linger for yourself is normally allowed without elevation; fall
+    # back to sudo, non-interactive first so `curl | bash` never blocks on a
+    # password prompt it has no terminal to answer.
+    if loginctl enable-linger "$target" 2>/dev/null \
+        || sudo -n loginctl enable-linger "$target" 2>/dev/null \
+        || { [ "$IS_INTERACTIVE" = true ] && sudo loginctl enable-linger "$target"; }; then
+        info "✔ Lingering enabled — the server keeps running after you log out"
+    else
+        warn "Could not enable lingering for ${target}."
+        echo "  Without it, the server stops as soon as your last session ends"
+        echo "  and does not start at boot. Run this once, as an administrator:"
+        echo ""
+        echo "      sudo loginctl enable-linger ${target}"
+        echo ""
+    fi
+}
+
 # ── Docker install helper ─────────────────────────────────────────────────────
 install_docker() {
     if [ "$OS" = "linux" ]; then
@@ -349,13 +386,21 @@ if [ "$OS" = "linux" ] && [ -z "${NOSYSTEMD:-}" ]; then
 [Unit]
 Description=Skald Circle (${DISPLAY_VERSION})
 Documentation=https://skaldagent.net
-After=network.target docker.service
+# No After=docker.service here: this is a *user* unit, and docker.service is a
+# system unit the user manager knows nothing about — the dependency would be
+# silently ignored. Docker may therefore still be starting when we do; the
+# server fails fast when the daemon is unreachable and Restart brings it back a
+# few seconds later, so boot ordering settles itself.
 
 [Service]
 Type=simple
 ExecStart=${INSTALL_DIR}/run.sh
 WorkingDirectory=${INSTALL_DIR}
-Restart=on-failure
+# always, not on-failure: run.sh exits 0 on any graceful shutdown, including one
+# nobody asked for (a stray SIGTERM to the server), which on-failure would treat
+# as a clean stop and leave the box down. An explicit "systemctl --user stop"
+# is unaffected — systemd never restarts after a requested stop.
+Restart=always
 RestartSec=5
 Environment=SKALD_BIN=${INSTALL_DIR}/bin/skald
 Environment=SKALD_SETUP_BIN=${INSTALL_DIR}/bin/skald-setup
@@ -368,6 +413,9 @@ SERVICE
     systemctl --user enable --now skald-circle.service
 
     info "✔ Service installed and started"
+
+    enable_linger
+
     echo ""
     echo "  Status:  systemctl --user status skald-circle"
     echo "  Logs:    journalctl --user -u skald-circle -f"

install.sh

@@ -130,6 +130,43 @@ stop_existing_service() {
     fi
 }
 
+# ── systemd user lingering ────────────────────────────────────────────────────
+# A `systemctl --user` unit runs under the per-user manager (user@UID.service),
+# which systemd starts at first login and STOPS when the user's last session
+# ends — taking every user service down with it. So without lingering the server
+# dies the moment you close the SSH session that started it, and never comes up
+# at boot. Enabling it is the whole difference between "runs while I'm logged
+# in" and "is a daemon".
+enable_linger() {
+    local target="${USER:-$(id -un)}"
+
+    if ! command -v loginctl >/dev/null 2>&1; then
+        warn "loginctl not found — cannot enable lingering."
+        echo "  The server will stop when you log out of this machine."
+        return 0
+    fi
+
+    case "$(loginctl show-user "$target" --property=Linger 2>/dev/null || true)" in
+        *=yes) info "✔ Lingering already enabled for ${target}"; return 0 ;;
+    esac
+
+    # Enabling linger for yourself is normally allowed without elevation; fall
+    # back to sudo, non-interactive first so `curl | bash` never blocks on a
+    # password prompt it has no terminal to answer.
+    if loginctl enable-linger "$target" 2>/dev/null \
+        || sudo -n loginctl enable-linger "$target" 2>/dev/null \
+        || { [ "$IS_INTERACTIVE" = true ] && sudo loginctl enable-linger "$target"; }; then
+        info "✔ Lingering enabled — the server keeps running after you log out"
+    else
+        warn "Could not enable lingering for ${target}."
+        echo "  Without it, the server stops as soon as your last session ends"
+        echo "  and does not start at boot. Run this once, as an administrator:"
+        echo ""
+        echo "      sudo loginctl enable-linger ${target}"
+        echo ""
+    fi
+}
+
 # ── Docker install helper ─────────────────────────────────────────────────────
 install_docker() {
     if [ "$OS" = "linux" ]; then
@@ -354,13 +391,21 @@ if [ "$OS" = "linux" ] && [ -z "${NOSYSTEMD:-}" ]; then
 [Unit]
 Description=Skald Circle (release ${VERSION})
 Documentation=https://skaldagent.net
-After=network.target docker.service
+# No After=docker.service here: this is a *user* unit, and docker.service is a
+# system unit the user manager knows nothing about — the dependency would be
+# silently ignored. Docker may therefore still be starting when we do; the
+# server fails fast when the daemon is unreachable and Restart brings it back a
+# few seconds later, so boot ordering settles itself.
 
 [Service]
 Type=simple
 ExecStart=${INSTALL_DIR}/run.sh
 WorkingDirectory=${INSTALL_DIR}
-Restart=on-failure
+# always, not on-failure: run.sh exits 0 on any graceful shutdown, including one
+# nobody asked for (a stray SIGTERM to the server), which on-failure would treat
+# as a clean stop and leave the box down. An explicit "systemctl --user stop"
+# is unaffected — systemd never restarts after a requested stop.
+Restart=always
 RestartSec=5
 Environment=SKALD_BIN=${INSTALL_DIR}/bin/skald
 Environment=SKALD_SETUP_BIN=${INSTALL_DIR}/bin/skald-setup
@@ -373,6 +418,9 @@ SERVICE
     systemctl --user enable --now skald-circle.service
 
     info "✔ Service installed and started"
+
+    enable_linger
+
     echo ""
     echo "  Status:  systemctl --user status skald-circle"
     echo "  Logs:    journalctl --user -u skald-circle -f"

update.sh

@@ -43,6 +43,13 @@ fi
 
 CHANNEL="$(tr -d '[:space:]' < "$CHANNEL_FILE")"
 
+# ── Detect interactive stdin ──────────────────────────────────────────────────
+if [ -t 0 ]; then
+    IS_INTERACTIVE=true
+else
+    IS_INTERACTIVE=false
+fi
+
 # ── Colours (if terminal) ─────────────────────────────────────────────────────
 if [ -t 1 ]; then
     RED='\033[0;31m'
@@ -96,9 +103,15 @@ STOPPED=0   # set once the service has been stopped
 STARTED=0   # set once it has been (re)started
 
 # ── Stop service ──────────────────────────────────────────────────────────────
+# NOTE: match the *normalized* OS values set above (linux/darwin), not uname's
+# capitalized output. Getting this wrong turns both stop_service and
+# start_service into silent no-ops, and then none of the ordering this file
+# documents at the top actually happens: the tarball is extracted over the
+# running binary (ETXTBSY on Linux, aborting the update mid-way), and the
+# safety-net restart in cleanup() is a no-op too, so the box stays down.
 stop_service() {
     case "$OS" in
-        Linux)
+        linux)
             if command -v systemctl >/dev/null 2>&1; then
                 if systemctl --user is-active skald-circle.service >/dev/null 2>&1; then
                     info "⏹️  Stopping service …"
@@ -106,7 +119,7 @@ stop_service() {
                 fi
             fi
             ;;
-        Darwin)
+        darwin)
             if command -v launchctl >/dev/null 2>&1; then
                 if launchctl list com.skald.circle >/dev/null 2>&1; then
                     info "⏹️  Stopping agent …"
@@ -145,13 +158,13 @@ wait_until_stopped() {
 # ── Start service ─────────────────────────────────────────────────────────────
 start_service() {
     case "$OS" in
-        Linux)
+        linux)
             if command -v systemctl >/dev/null 2>&1; then
                 info "▶ Starting service …"
                 systemctl --user start skald-circle.service
             fi
             ;;
-        Darwin)
+        darwin)
             if command -v launchctl >/dev/null 2>&1; then
                 info "▶ Starting agent …"
                 launchctl load "$HOME/Library/LaunchAgents/com.skald.circle.plist" 2>/dev/null || true
@@ -160,6 +173,34 @@ start_service() {
     esac
 }
 
+# ── systemd user lingering ────────────────────────────────────────────────────
+# Same helper the installers run, repeated here so an install predating it gets
+# healed by an ordinary update: a `systemctl --user` unit lives under the
+# per-user manager, which systemd stops when the user's last session ends —
+# so without lingering the server dies at logout and never starts at boot.
+# Idempotent, and a failure is only ever a warning: the update itself is fine.
+ensure_linger() {
+    [ "$OS" = "linux" ] || return 0
+
+    local target="${USER:-$(id -un)}"
+
+    command -v loginctl >/dev/null 2>&1 || return 0
+
+    case "$(loginctl show-user "$target" --property=Linger 2>/dev/null || true)" in
+        *=yes) return 0 ;;
+    esac
+
+    if loginctl enable-linger "$target" 2>/dev/null \
+        || sudo -n loginctl enable-linger "$target" 2>/dev/null \
+        || { [ "$IS_INTERACTIVE" = true ] && sudo loginctl enable-linger "$target"; }; then
+        info "✔ Lingering enabled — the server now survives logout and starts at boot"
+    else
+        warn "Lingering is not enabled for ${target}."
+        echo "  The server stops when your last session ends. Run this once:"
+        echo "      sudo loginctl enable-linger ${target}"
+    fi
+}
+
 # ── Cleanup + safety net ──────────────────────────────────────────────────────
 # Runs on every exit. Removes temp files and, if the update died after the
 # service was stopped but before it came back up, makes a best-effort restart so
@@ -279,6 +320,7 @@ main() {
     fi
 
     # ── Restart ────────────────────────────────────────────────────────────────
+    ensure_linger
     start_service
     STARTED=1