Skald-Circle/docs/relay/server.md

Relay Server — Implementation

Guide for the coding agent building crates/skald-relay-server. The protocol is in relay-protocol.md; the cryptography (which the relay barely touches) is in crypto.md. Here: internal architecture, persistence, push bridge, deploy, quotas.

---

1. Role (and Non-Role)

The relay is the only centralised component. It does only four things:

1. Authenticates connections (Ed25519 challenge-response) and routes by namespace_id.
2. Forwards opaque messages between agent and clients of the same namespace.
3. Store-and-forward: queues for offline recipients.
4. Push bridge: for offline clients it talks to APNs (Apple) and FCM (Google).

It does nothing else: no business logic, no decryption, no content reading, no user accounts. Deliberately dumb. Its only "truth" is: pubkey → namespace, pubkey → device_token, and a FIFO queue of blobs.

Zero-Trust: What It Means Here (precise)

The relay is content-confidential, not metadata-private (see index.md §4). It sees pubkeys, device_tokens, IPs, the relationship graph, and timing; it does not see content or detailed device_info (which travel E2E). Everything the relay persists is either non-sensitive or E2E-encrypted.

---

2. Stack & Structure

Language: Rust. Static musl binary ~5–7 MB, ~30 MB RAM, cold start < 100 ms.

Crate	Use
axum	HTTP server + WebSocket upgrade, healthcheck
tokio / tokio-tungstenite	async runtime + per-connection WS
prost	protobuf encode/decode (RelayFrame from skald-relay-common)
sqlx (sqlite)	persistence (namespaces, clients, queue)
ed25519-dalek = "2"	challenge-response signature verification
sha2, hex	hashing and encoding
a2	APNs HTTP/2 + JWT
reqwest	FCM HTTP v1 (Android)
tracing	structured logs (metrics only, never content)
clap	CLI flags
governor	per-IP / per-connection rate limiting

crates/skald-relay-server/
├── Cargo.toml
└── src/
    ├── main.rs        # config, init, axum server, graceful shutdown
    ├── ws.rs          # WS handler: challenge → auth(role) → forward loop
    ├── auth.rs        # signature verification, namespace_id derivation, role gating
    ├── routing.rs     # live connection registry (namespace → agent/clients)
    ├── store.rs       # sqlx: namespaces, clients, queue, pairing
    ├── push.rs        # APNs + FCM bridge, content-in-push vs wake
    ├── limits.rs      # quotas, rate-limit, timeouts
    └── types.rs       # serde types for JSON (used only for push payloads / logging)

Shared crate. Frame types and auth crypto (signature verification + namespace_id derivation) are common with the plugin and live in crates/skald-relay-common. The relay depends on that crate. X25519/HKDF/AES-GCM remain in the shared crate for E2E (plugin + app) and for the gen-vectors binary, not used by the relay itself.

---

3. Data Model (SQLite)

Minimal schema. No sensitive data in plaintext: ciphertext is E2E; pubkeys are public identifiers.

CREATE TABLE namespaces (
    namespace_id      TEXT PRIMARY KEY,          -- hex(SHA256(domain‖pub)), immutable
    agent_ed25519_pub BLOB NOT NULL UNIQUE,      -- 32B, binds id to key
    created_at        INTEGER NOT NULL,          -- unix ms
    last_active       INTEGER NOT NULL,          -- for 7-day GC
    -- pairing window (at most one active per namespace):
    pairing_token     BLOB,                      -- 32B random, NULL if closed
    pairing_expiry    INTEGER,                   -- unix ms
    pairing_consumed  INTEGER NOT NULL DEFAULT 0 -- 0/1 single-use
);

CREATE TABLE clients (
    namespace_id       TEXT NOT NULL REFERENCES namespaces(namespace_id) ON DELETE CASCADE,
    client_ed25519_pub BLOB NOT NULL,            -- 32B, routing + auth
    client_x25519_pub  BLOB NOT NULL,            -- 32B, opaque (forwarded to agent)
    device_token       TEXT,                     -- push token (APNs/FCM)
    platform           TEXT NOT NULL,            -- 'ios' | 'android'
    state              TEXT NOT NULL,            -- 'pending' | 'authorized'
    last_seen          INTEGER,                  -- unix ms
    PRIMARY KEY (namespace_id, client_ed25519_pub)
);

CREATE TABLE queue (
    id            INTEGER PRIMARY KEY AUTOINCREMENT,
    namespace_id  TEXT NOT NULL REFERENCES namespaces(namespace_id) ON DELETE CASCADE,
    to_pub        BLOB NOT NULL,                 -- 32B recipient
    from_pub      BLOB NOT NULL,                 -- 32B sender (guaranteed by relay)
    nonce         BLOB NOT NULL,                 -- 12B
    ciphertext    BLOB NOT NULL,                 -- opaque (ciphertext‖tag)
    created_at    INTEGER NOT NULL               -- unix ms (for 7-day TTL)
);
CREATE INDEX idx_queue_dest ON queue(namespace_id, to_pub, id);

Notes:

• client_x25519_pub is persisted for robustness (re-forwarding ClientPaired if the agent missed it), even though the relay does not use it for crypto.
• The relay does NOT store shared_secret/aes_key (it neither has them nor can compute them).
• state='pending' until the agent sends Authorize including the pubkey (→ authorized). A role:"client" connection is only accepted from authorized.

⚠️ Constraint: SQLite on EFS ⇒ Single Instance

In v1 the relay runs as a single Fargate task with SQLite on an EFS volume. SQLite on NFS/EFS does not support concurrent writes from multiple processes (unreliable locking → corruption). Therefore:

• Do NOT scale horizontally with this configuration (no HA, no multi-task).
• Store-and-forward assumes a single writer.
• Scale path (post-v1, if HA is needed): replace store.rs with Postgres (RDS) for the queue and a distributed connection registry (e.g. Redis pub/sub) for cross-instance routing. The store.rs API is designed to make this substitution localised.

---

4. Concurrency & Routing

Model: one Tokio task per WS.

• routing.rs holds in memory DashMap<namespace_id, NamespaceConns> where NamespaceConns { agent: Option<Sender>, clients: HashMap<pubkey, Sender> } and Sender is a tokio::sync::mpsc::Sender<RelayFrame> toward that WS's task.
• Forwarding: on receiving a Message from an authenticated WS, check peer (destination):
  • if the recipient has a live connection in the same namespace → send on its Sender;
  • otherwise → store::enqueue(...) and, for a client, push::notify(...). Unless Message.live=true, in which case → send PeerOffline { peer } back to the sender.
• Single agent: one agent connection per namespace; a new one displaces the old (close old).
• Single client per pubkey: same for devices.
• Keepalive: ping task every 30 s; close on 120 s silence (relay-protocol.md §8).

Store-and-Forward (delivery)

async fn deliver_pending(tx: &Sender<RelayFrame>, store: &Store,
                         ns: &str, to_pub: &[u8;32]) -> anyhow::Result<()> {
    for m in store.fetch_pending(ns, to_pub).await? {     // ORDER BY id ASC (FIFO)
        tx.send(build_message_frame(m.from_pub, m.nonce, m.ciphertext, false)).await?;
        store.delete_pending(m.id).await?;                // delete after delivery
    }
    Ok(())
}

Queue full (> 200 for recipient, relay-protocol.md §10) → reject new messages with queue_full until drained. TTL: a periodic task deletes messages older than 7 days and namespaces inactive for 7 days.

---

5. Push (APNs / FCM Bridge)

When a message is destined for an offline client with a device_token, the relay sends a push. Two modes, decided by the size of the encrypted blob:

5.1 Content-in-Push (preferred, enables "approve from notification")

If len(raw ciphertext) fits within the payload limit (APNs ~4 KiB, FCM ~4 KiB), the relay includes the already E2E-encrypted blob in the push. The device decrypts it in the Notification Service Extension and shows a rich notification with Approve/Reject actions, without opening the app.

APNs payload:

{
  "aps": {
    "alert": { "title": "Skald", "body": "Action required" },
    "badge": 1,
    "sound": "default",
    "mutable-content": 1,
    "category": "skald_inbox"
  },
  "d": {
    "ns": "<namespace_id hex>",
    "from": "<agent_ed25519_pub hex>",
    "n": "<nonce hex 24>",
    "c": "<ciphertext base64>"
  }
}

• mutable-content: 1 activates the Notification Service Extension (decrypts d.c).
• aps.alert is a generic fallback shown if the NSE fails: never sensitive content.
• The relay does NOT know what is in d.c: it copies it as-is from the queue.

Note: inside d, the values use hex/base64 encoding for JSON compatibility (nonce hex 24 chars, ciphertext base64). This is the one context where the encoding conventions from index.md §5 apply outside the E2E JSON payload.

5.2 Wake-Only (fallback when blob exceeds limit)

{
  "aps": { "alert": { "title":"Skald", "body":"Action required" },
           "badge":1, "sound":"default", "content-available":1 },
  "d": { "ns": "<namespace_id hex>", "wake": true }
}

The device wakes, opens a temporary WS, downloads queued messages, and shows the Inbox. No content in the push.

Choice rule (normative): content-in-push if len(raw_ciphertext_bytes) <= 3500 (after base64-encoding into JSON it will be ≤ ~4666 chars), otherwise wake-only. Conservative threshold to leave room for other fields. Keep summary/detail in payloads small to stay in the preferred case.

5.3 FCM (Android)

Use FCM HTTP v1 with a data-only message ("data": { … }) so the app always handles decryption (even in background), avoiding automatic display of an unencrypted notification. Fields ns/from/n/c as above. Priority high.

5.4 Push Key Management

Secret	Where	How
APNs .p8 (Apple)	relay only	AWS Secrets Manager in prod; config/apns-key.json (git-ignored) locally
FCM service account JSON (Google)	relay only	Secrets Manager / local file

Never in the app, never in the plugin. At startup the relay loads secrets and generates:

• APNs JWT (ES256, valid 60 min) held in memory, refreshed every ~30 min (never more than once every 20 min, per Apple's rules). No key on disk beyond the .p8.
• FCM OAuth token (from the service account) with auto-refresh.

Example APNs secret in Secrets Manager:

{ "team_id":"ABC123DEFG", "key_id":"XYZ789ABCD",
  "private_key":"-----BEGIN PRIVATE KEY-----\nMIGTA…\n-----END PRIVATE KEY-----" }

Minimal IAM for the ECS task:

{ "Effect":"Allow", "Action":"secretsmanager:GetSecretValue",
  "Resource":"arn:aws:secretsmanager:REGION:ACCOUNT:secret:skald/push-keys-*" }

---

6. Security Checklist

• WSS mandatory: reject plain ws://.
• Verify Ed25519 signature before any other logic; reject malformed input with bad_request.
• namespace_id recomputed from pubkey, never trusted from client input.
• Pairing token and tag comparisons in constant-time (subtle).
• PairingStart token single-use enforced atomically (UPDATE … WHERE consumed=0).
• Role gating: client only if authorized; pairing only if window open + token valid.
• Rate-limit per-IP on new connections and per-connection on messages (governor).
• Frame size limit 64 KiB (pre-auth + store-and-forward), 512 KiB (live channel post-auth); payload_too_large + close on exceeded.
• No content in logs: log only namespace_id, truncated pubkeys, codes, counts, latencies. Never ciphertext, nonce, full device_token (truncate/hash).
• Authorize shrink → close the revoked client's WS + purge their queue + forget device_token.
• 7-day GC for namespaces/queues.

---

7. Startup & Shutdown

1. main.rs loads config (CLI/env): port, DB path, push key source, thresholds.
2. Load push keys (Secrets Manager or file). Generate APNs JWT + FCM token (refresh task).
3. Initialise SQLite (migrations via sqlx::migrate!).
4. Start axum on 0.0.0.0:{port}; route GET /healthz → 200; GET /v1/ws → upgrade.
5. Graceful shutdown on SIGTERM/SIGINT: stop accepting, drain WS connections, flush queue, close DB.

Logging

main.rs writes logs to both stdout and a file at logs/skald-relay.log (daily rotation via tracing-appender), aligned with the main app. Log level controlled by RUST_LOG; default skald_relay_server=info,info. In development:

RUST_LOG=skald_relay_server=debug   # auth, routing, queue drain
RUST_LOG=skald_relay_server=trace   # frame-level tracing

Invariant: never log content — only namespace_id, truncated pubkeys, codes, counts (see §6).

---

8. Deploy

Aspect	v1 choice
Compute	AWS ECS Fargate, 1 task (§3 constraint)
Container	musl static, FROM scratch, ~7 MB
Storage	SQLite on EFS (persistent across restarts)
Push keys	Secrets Manager (skald/push-keys)
Domain/TLS	relay.skaldagent.net via ALB + ACM (free TLS)
Logs	CloudWatch (metrics only)
Cost	~$5–10/month Fargate + ~$0.40 Secrets Manager

FROM clux/muslrust:stable AS build
COPY . /src
WORKDIR /src
RUN cargo build --release --target x86_64-unknown-linux-musl -p skald-relay-server --bin skald-relay-server

FROM scratch
COPY --from=build /src/target/x86_64-unknown-linux-musl/release/skald-relay-server /skald-relay-server
EXPOSE 8080
ENTRYPOINT ["/skald-relay-server"]

Self-Hosting

Anyone can host their own relay: an Apple Developer Key ($99/year) for APNs (and/or a Firebase project for FCM) is required. docker compose with local SQLite, or deploy on your own cloud. The relay is open source and interoperable with any agent/app conforming to these documents.

---

9. Definition of Done

• cargo build --release produces a musl static binary.
• An agent can authenticate, create the namespace, start/stop pairing, authorize.
• A client can pair, then connect as client only after authorize.
• Messages routed live; store-and-forward + FIFO delivery on reconnect.
• Push content-in-push below threshold, wake-only above; APNs and (at least stub) FCM working.
• Revocation via Authorize shrink closes WS and purges queue.
• Live channel: PeerOffline sent correctly; no queue/push for live=true messages.
• Presence: PresenceList on PresenceRequest; PresenceEvent on connect/disconnect.
• Quotas/rate-limit active; no content in logs.
• 7-day GC verified.
• Relay never alters nonce/ciphertext (test: bytes identical in/out).