Operations

Everything runs on one Linux server: nginx in front, native Postgres, and systemd units for the long-lived services.

Environments

	TEST (live)	Production (planned)
App (sign-in)	test-play.mindrune.xyz	play.mindrune.xyz (a "coming soon" page for now)
Public pages	test.mindrune.xyz: /hiscores, /log, /market, the landing page	mindrune.xyz (for now a beta landing page: sign-in leads into the TEST app, public links to TEST's)
API	test-api.mindrune.xyz (also /api on the app host)	api.mindrune.xyz
Docs	docs.mindrune.xyz (one site for both)
Game world	its own LostCity world	a separate world
Database	its own Postgres database and role	its own

Each environment gets its own database, secrets and game world. Who may sign in is set per environment: TEST uses an allowlist; production is meant to run with open sign-up (below). Test and production sessions can't cross, because cookies are host-only.

Services (TEST)

Unit	What it runs
autorune-test-api	The API. It spawns one agent-runner per running avatar as child processes, plus the shared pathfinding host. Each runner starts its sandbox as a transient systemd scope.
autorune-test-engine	The LostCity game engine from vendor/rs-sdk, bound to loopback by a preload.
autorune-test-gateway	rs-sdk's gateway.
autorune-test-friend	rs-sdk's friends server, needed for private messages.
autorune-test-login	rs-sdk's login server: game accounts, bcrypt password checks, and reading and writing the saves (below).
autorune-console	The operator console API (shared by all environments, own system user).
netdata	Server metrics (loopback only), shown in the console.

The unit files live in infra/systemd/, and the non-secret config in infra/autorune-api/test.env and infra/rs-sdk/test.env. Secrets live in env files outside git. Every variable is declared once with its default (zod schemas in the API, the console API and the runner): a value that doesn't fit, such as a word where a number belongs, stops the service at start with the variable's name in the journal (never its value). A unit test parses the committed env files with the same schemas, so a typo fails the checks before a deploy. Who may sign in to TEST is not in test.env: the allowlist file is managed in the operator console, and the API unit reads it on start. A graceful API stop leaves every avatar's desired state alone, so running avatars come back on the next start.

Deploying

What	Script	Where it goes
App	infra/deploy/test-app.sh	static files to the app's web root
Public sites	infra/deploy/landing.sh test|prod|play	static files: test.mindrune.xyz (landing page and /hiscores/), mindrune.xyz (landing, beta), play.mindrune.xyz (coming soon)
Docs	infra/deploy/docs.sh	static files to the docs web root
Operator console	infra/deploy/console.sh	web app, API bundle, config, sudoers drop-in, unit (details)
API, runner	pull main, run migrations if any (bun run db:migrate, before the restart: the API checks its tables at start), restart autorune-test-api	running avatars resume by themselves (one boot_resume run event each)

The static sites are built with Bun + Vite (VitePress for docs) and copied with rsync --delete. nginx serves them with long-lived caching for hashed assets and no-cache for HTML.

nginx and TLS

• One vhost config per site group in infra/nginx/, installed into the server's nginx sites-available.
• Certificates come from Let's Encrypt via certbot (webroot challenge). Port 80 only serves the ACME challenge and redirects everything else to HTTPS.
• After any change: sudo nginx -t, then sudo systemctl reload nginx (a reload, never a restart).
• The sites moved from autorune.xyz to mindrune.xyz (certs mindrune-xyz-*). The old names answer with a 301 to the same path on the new name (infra/nginx/autorune-xyz-redirects.conf), except the two with Discord sign-in (test-play, console), which keep serving until the Discord application has the new redirect URIs.
• A catch-all default server (infra/nginx/default-drop.conf) drops requests for names no site serves: HTTP gets no reply, HTTPS refuses the handshake (no certificate shown). So a new site needs its own server_name on port 80 and 443, IPv4 and IPv6, or it is dropped too.

Swap (zram)

The server has no disk swap. A compressed swap device in RAM (zram, via systemd-zram-generator) is the safety net, so a memory spike slows processes down instead of waking the OOM killer.

• /etc/systemd/zram-generator.conf: /dev/zram0, a quarter of RAM (about 8 GB), zstd, swap priority 100. It is set up again at every boot.
• /etc/sysctl.d/99-zram.conf: vm.swappiness = 150, vm.page-cluster = 0 (tuned for swap in RAM).
• Check with zramctl, cat /proc/swaps or free -h.
• Avatar sandboxes don't use it: the runner starts each scope with MemorySwapMax=0, so an avatar that crosses its MemoryMax is still killed. Swap only helps the services and everything else outside the sandboxes.

Avatar memory

• Each sandbox scope has a MemoryMax: 2G by default, SANDBOX_MEMORY_MAX=2G on TEST (infra/autorune-api/test.env, read when the API restarts). Check a scope with systemctl --user show -p MemoryMax <scope>; kernel kills show in sudo journalctl -k | grep "Memory cgroup out of memory".
• An OOM kill takes only the biggest process in the sandbox (OOMPolicy=continue): a runaway agent script dies alone and the agent sees it fail; if it was opencode, the sandbox restarts like after a crash; if it was rs-sdk's MCP server, the runner reconnects it after the turn.
• What drives an avatar's memory is the length of its engine session: peaks went from about 0.6 GB (a 16 MB opencode session) to 2.4-2.8 GB (250-380 MB). Sessions are replaced by fresh ones after 24 hours or at 100 MB (SESSION_MAX_AGE_H, SESSION_MAX_MB; 0 turns a limit off).
• Measured on TEST (2026-11-01, 14 avatars, about 70 minutes with session rotation and the shared pathfinding map, systemctl --user show <scope> -p MemoryPeak, sampled every 15 s): opencode sandboxes peaked at 0.56-1.28 GB, Claude Code ones at 0.19-0.52 GB, agent scripts 40-50 MB each; no avatar was OOM-killed. 1G would not fit opencode (its baseline alone is 0.5-0.7 GB). 1.5G clears every measured peak, but those sessions were young; 2G (the current setting) stays until a session has been measured close to its 100 MB limit.
• opencode's archived databases: .run/agent-data/avatars/<id>/opencode-archive/<time>/opencode.db.gz (outside the sandbox, newest 3 kept). To look at one, copy it elsewhere and gunzip it there.
• To remove it: sudo swapoff /dev/zram0, sudo systemctl stop systemd-zram-setup@zram0, delete both files, sudo sysctl vm.swappiness=60 vm.page-cluster=3.

Shared pathfinding map

rs-sdk's pathfinding loads a 27 MB collision map into every process that walks, about 370 MB each. That means each sandbox's MCP server and every running script. The API runs one pathfinding host for all of them instead (Architecture). Its log lines are in the API's journal under [pathfinding].

• Memory (TEST, 14 running avatars, 2026-11-01, before → after):

  	Stock rs-sdk	Shared map
  rs-sdk MCP server, each	436 MB	65 MB
  A script using runScript (whole SDK loaded, map initialized), each	409 MB (peak 425)	38 MB
  Sandbox scope MemoryCurrent, average	744 MB	359 MB
  Sandbox scope MemoryPeak, average	1,089 MB	488 MB
  All 14 scopes together	10.4 GB	5.0 GB
  Pathfinding host (2 workers), once	—	0.8 GB
  Server memory in use	19.5 GB	14.6 GB

  At 15 avatars that is about 4.8 GB less (15 × 371 MB minus the host), and another ~370 MB for each script running at the same time. Check it with systemctl --user show <scope> -p MemoryCurrent -p MemoryPeak and the host's VmRSS.

• Settings (API env, read by the runners too): PATHFINDING=stock turns it off, so every process loads rs-sdk's own map again. PATHFINDING_WORKERS (default 2) sets how many maps the host holds; each is ~400 MB and computes one path at a time. PATHFINDING_DIR (default .run/pathfinding) is the socket dir, mounted read-only in every sandbox.

• When it's down. The API restarts a host that exits after 1 s, then backs off up to a minute. A process that loses its connection waits up to 5 s for the new host, then loads rs-sdk's own map and carries on (one [autorune] shared pathfinding map unavailable line). Its paths stay the same, only its memory grows.

• After an rs-sdk update that changes sdk/pathfinding.ts, sandboxes load stock pathfinding (with a warning) until the shim is verified again and its pin moved. The checks fail until then (see Updating rs-sdk).

The market proxy

The Grand Exchange page is served by the game engine, which listens on loopback only. The app vhost (infra/nginx/autorune-test.conf) forwards one anchored regex of paths to it: /market, /api/market/{status,items,items/<id>,items/<id>/history}, the page's two background images and /sprite/item/<id>.png. Anything else, including the engine's own /, /bot, /rs2.cgi and status pages, falls through to the app.

• Only GET is allowed (anything else gets 403); request bodies, cookies and Authorization are never forwarded.
• That location wins over the app's /api/ prefix, so /api/market/* never reaches the MindRune API. The MindRune API must not add routes under /api/market.
• The page needs inline script and styles, so it gets its own CSP (same-origin data and images only, no framing) in addition to the usual security headers.
• The app's service worker ignores these paths, so the market page is never cached as the app shell.

Login server

The TEST world runs with LOGIN_SERVER=true (infra/rs-sdk/test.env), so every login, the game client's and every bot's, is checked by rs-sdk's own login server, autorune-test-login (src/login.ts in the engine, loopback port 43500). Without it the engine accepts any password for any name.

• What it owns. Accounts (account table, bcrypt hashes) in the engine's db.sqlite, and the saves: it reads data/players/main/<name>.sav at login and writes it on logout and autosave, from the engine's directory, so the files are the same ones as before. WEBSITE_REGISTRATION=false lets a name's first login register it (rs-sdk's default refuses unknown names). The gateway reads the same variables and checks each SDK client's bot password with it.
• Avatars' names. apps/autorune-api/scripts/register-game-accounts.ts registers every avatar row (running or not) with its stored password through the login server, --reserve-orphan-saves also locks the names of saves no avatar owns; it is idempotent and exits non-zero unless every avatar has an account. The API registers a new avatar's account at create time (LOGIN_SERVER_URL). Run the script with the API's env files loaded (set -a; . infra/autorune-api/test.env; . ./.env.test-api; set +a).
• Order. Start autorune-test-login before the engine and the gateway (the units say so), and don't stop it while the engine runs. If it is down, logins fail with "Login server offline" (bots retry), autosaves are skipped, and logout saves stay in the engine, retried every 15 s until it answers; an engine stop waits for them. Restart the engine alone when you can: a running engine reconnects to a restarted login server by itself.
• Its own management port is 8896 (metrics only, loopback); the engine keeps 8898.
• Logs: journalctl -u autorune-test-login (takeovers show as [LOGIN] Account <name> already logged in …); refused passwords are not logged.

This docs site

cd apps/autorune-docs
bun run dev        # local preview with hot reload
bun run build      # static output in src/.vitepress/dist
bun run linkcheck  # checks every internal link in the build
infra/deploy/docs.sh   # build, check links and the CSP, publish

The pages are Markdown in apps/autorune-docs/src/. Diagrams are Mermaid code blocks. The repository's docs/ folder holds the detailed dev notes; this site is the human-readable source of truth.

Keeping it current. The repository's AGENTS.md makes it a rule: any change to user-visible behaviour, architecture, AI providers, the rs-sdk relationship, security or operations updates the matching page here in the same change, then redeploys with infra/deploy/docs.sh. The deploy refuses to publish a build with a broken internal link or an inline script the CSP doesn't allow.

Search engines. Unlike the app, API and hiscores (which send noindex), this site is meant to be found: it serves a robots.txt that allows everything and a sitemap.xml generated by the build.

Checks before a push

From the repository root, bun run check runs, and stops at the first failure: the type check in every workspace, every unit test, Biome (lint + format), knip (no unused files, exports or dependencies) and this site's link check. Postgres tests run only against a throwaway test database. The app and console end-to-end tests (Playwright) compare every screen with committed screenshots; those baselines change only for an intended change of look, never to make a test pass. Details: the repository's docs/checks.md.

Updating rs-sdk

vendor/rs-sdk is a pinned, unmodified submodule. Updating means moving the pin. The full procedure and a history of bumps are in the repo's docs/rs-sdk-updates.md. In short:

1. Bump: in vendor/rs-sdk, fetch, review the new upstream commits and check out the new one. Never edit files there. git -C vendor/rs-sdk status must stay clean.
2. Read the changes that touch us: engine login/logout and takeover (the lite ↔ browser handover), the MCP server's bot manager and list_bots (the siblings preload), sdk/pathfinding.ts (the shared pathfinding map is pinned to it), and the SDK API, CLAUDE.md and learnings/ the agents read.
3. Rebuild what we build from it: the engine's content pack and the webclient bundle, plus bun install and engine migrations where lockfiles or schemas changed.
4. Test: bun run check (below) and the app e2e. The tests include an rs-sdk contract test that imports the real vendored modules MindRune relies on and points at the code that would break.
5. Commit the new pin with the list of upstream commits and their impact.
6. Deploy to TEST: note a running bot's state, restart the engine (and the friends server or gateway if they changed), then the API. Check that bots reconnect by themselves, keep their goals and state, and that PMs still arrive.

Avatars pick up the new rs-sdk (and their regenerated instructions) on their next start. If the wiki's pages changed, rebuild the LostHQ pages and the tags (below). Always rebuild the world object pages (below).

Rebuilding the LostHQ pages

The LostHQ snapshot (data/losthq/raw/, see Game data) is a one-time fetch and is not re-run. The pages in data/losthq/md/ are built from it and can be rebuilt any time, for example after an rs-sdk update changed the wiki's slugs:

bun scripts/losthq/build.ts            # data/losthq/md from the raw snapshot (deterministic)
bun scripts/build-tag-vocabulary.ts    # the topic tags' wiki and LostHQ parts

Commit the changed pages. Tests fail when the checked-in pages or tags differ from what the raw snapshot and the wiki give. Every sandbox mounts data/losthq/md read-only at /work/losthq; the mount is added when an avatar starts, so a new mount needs an API restart. The build rewrites the pages in place (never a new directory), so changed pages reach running avatars at once.

Rebuilding the world object pages

The world/ pages (see Game data) are generated from rs-sdk's server content: the map files, the object id list and the object configs. Rebuild them with every rs-sdk update:

bun scripts/world/build.ts             # data/world/md from vendor/rs-sdk/server/content (deterministic)

Commit the changed pages with the bump. A test fails while the checked-in pages differ from what the vendored content gives, so a bump can't ship stale pages. Every sandbox mounts data/world/md read-only at /work/world; the mount is added when an avatar starts (a new mount needs an API restart). The build rewrites the files in place and never replaces the directory, so changed pages reach running avatars at once. Never delete and recreate data/world/md (or data/losthq/md) while avatars run: the sandboxes mount the directory itself and would keep an empty, deleted one until their next start.

Open sign-up

By default an environment's API lets in only the Discord accounts on its allowlist. To let anyone sign up (production), set these in that environment's API env file (non-secret, next to test.env's equivalent), then restart its API unit once:

AUTH_SIGNUP_MODE=open                    # allowlist (default) | open
AUTH_SIGNUPS_PER_HOUR=50                 # new accounts per rolling hour; then "Sign-ups are paused, try later"
AUTH_MIN_DISCORD_ACCOUNT_AGE_DAYS=0      # e.g. 7 to refuse brand-new Discord accounts; 0 = off
AUTH_DENIED_DISCORD_IDS=                 # comma list of Discord IDs that may never sign in

• The allowlist is ignored in open mode; locked accounts stay locked, and the avatar and rate limits are unchanged.
• A mistyped value (say AUTH_SIGNUP_MODE=yes) stops the API at start, naming the variable.
• Check: the API logs AUTH_SIGNUP_MODE=open: any Discord user can sign in (…) at start, and the console's Users & access shows the mode and how many accounts were created in the last hour and 24 hours.
• Back to invite-only: remove AUTH_SIGNUP_MODE (or set allowlist) and restart. Existing accounts keep working in both directions; lock one in the console to shut it out.

Who started or stopped an avatar

Every change of an avatar's run state is recorded with its cause and who made it (run history): in the console (Avatars & world → Run history), and as one line in the API's journal:

journalctl -u autorune-test-api --since today | grep '\[api\] run:'
# [api] run: <avatar id> stopped_limit→running by owner <user id> (owner_play)
# [api] run: <avatar id> running→running by system (boot_resume)
# [api] run: <avatar id> running→stopped_by_user by operator:<console user> (operator_stop) "Stopped by the operator"

Causes: owner_play, owner_stop, operator_start, operator_stop, account_locked, boot_resume, boot_skipped, crash_restart, crashed, crash_loop, restart_failed, provider_limit, provider_error, deleted, restored. The table (avatar_run_events) keeps 180 days (RUN_EVENTS_RETENTION_DAYS); the API prunes it hourly. A restart records one boot_resume per avatar that was running and nothing else: if an avatar's state differs after a restart, its last events say why.

Research data

The API keeps internal history for analysis (what and how long; details in the repository's docs/research-data.md): ai_usage (the model, tokens, cost and latency of every model call, or for Claude Code of every turn per model, next to the model the avatar was set to; Claude Code reports running totals for its process, and the runner stores each turn's difference), avatar_snapshots (hourly hiscores and wealth per avatar), ge_book_snapshots (the GE order book every 5 minutes), all without a time limit. The script reviewer's calls are in ai_usage too, with purpose = 'review', the owner's user_id, the avatar whose file it reviewed and the review_id (engine review_http for a direct API call, claude_code or codex on a subscription), and every review is a row in script_reviews. The removed Coach's stored data (its runs, settings and usage rows) was deleted by migration 0023_drop_coach. After a restart:

journalctl -u autorune-test-api --since "-10 min" | grep '\[research\]'
# [research] snapshots on: avatars hourly, GE book every 5 min

The first GE book follows a minute after the start, the first avatar snapshot five minutes after it, ai_usage rows as soon as avatars run. A failed write is logged (ai usage write failed, [research] … snapshot failed) and never stops an avatar.

The script reviewer

The script reviewer runs inside the API, replacing the removed Coach. It checks every 5 minutes, per owner with active avatars (on unless the owner switched it off), with at most 2 reviews at once and one per owner. Migration 0022_script_reviewer adds script_reviewer_settings, script_reviews and ai_usage.review_id; it is additive. Run bun run db:migrate before you restart an API with it: the API refuses to start without script_reviews. Migration 0023_drop_coach deletes the removed Coach's ai_usage rows (purpose = 'coach') and drops ai_usage.coach_run_id, coach_runs and coach_settings (back up those rows before you run it; nothing reads them). Migration 0020_snippet_runs holds the inline-run counts that the waste detector reads; bun apps/autorune-api/scripts/backfill-snippet-runs.ts fills the last 7 days once. Its log lines start with [reviewer]:

journalctl -u autorune-test-api --since "-1 h" | grep '\[reviewer\]'
# [reviewer] owner <id>: reviewed bots/<bot>/fish.ts (failures), 1 unknown call(s) flagged

A provider refusal (bad key, quota, plan limit) makes the reviewer rest for that owner for 6 hours, or until they change its settings. It never touches the avatars. To see what it did: select created_at, file, trigger, outcome, reason, summary, model, cost_usd from script_reviews order by created_at desc limit 20;. The reviewer only writes <name>.review.md files next to scripts, and an avatar may delete them.

Personality

• Starting personality. Each avatar's runner writes a random starting personality into its personality.md once, at its first start, and records it in <agent data>/avatars/<id>/personality-seed.json (version, seed, hashes; never text). The runner log says only personality: seeded (version 1) or reseeded. A second seed for everyone needs a code change (PERSONALITY_SEED_VERSION) and the owner's say-so; back up every personality.md first (as for the 2026-11-05 reset: ~/autorune-backups/personality-reset-<stamp>/, directory 700, files 600, one file per avatar).
• Compaction nudges. After an engine compacts an avatar's conversation, one optional personality line goes to it at most every 3 hours (.state/compaction-nudge.json in its workspace); the runner logs compaction: nudge pending and autonomy: compaction nudge queued into the running turn.
• Drives (removed 2026-11-05). The drives' routes, hourly job and the PERSONALITY_DRIVES_PROMPT / EXPERIENCE_WEIGHT_ENABLED switches are gone. The tables from migration 0024_personality_drives (family_drives, avatar_drives, avatar_experience_events) stay in the database, unused; nothing reads or writes them, and no migration drops them.

Deleted avatars

Deleting an avatar (by its owner in the app, or in the console) moves everything it had into a private backup instead of erasing it:

~/autorune-backups/deleted-avatars/<date>/<game name>/
  manifest.json   who, when, by whom, purge date
  avatar.json     its database row (the game password stays encrypted)
  skills.json     its removed scripts (personal skills stay with the family)
  chat.json       its chat history
  market.json     its Grand Exchange account checkpoint and offers
  files/          agent workspace (goal, task notes), sandbox dir, bot dir, game save

The API removes backups older than 7 days (checked hourly). The avatar is stopped and hidden at once; the move happens once the engine has logged the character out, which for a bot whose client just disconnected takes the engine's idle timeout (about 3–4 minutes). The API retries until then, and a delete that a restart interrupted is finished when the API starts again. Until then the avatar can't be started and its name stays taken.

The game save can be moved while the world runs because the character is logged out first and the world (through its login server) reads a save only when someone logs in. The game account stays, so the name stays taken and a restored avatar logs in with its old password. Its Grand Exchange rows go too, otherwise the engine would bring the old character back from its exchange checkpoint. Other players' saves are never touched.

Restoring one (within the 7 days, and only if its name is still free):

set -a; . infra/autorune-api/test.env; . ./.env.test-api; set +a
sudo systemctl stop autorune-test-api
bun apps/autorune-api/scripts/restore-avatar.ts ~/autorune-backups/deleted-avatars/<date>/<game name>
sudo systemctl start autorune-test-api

It comes back stopped, with its character, progress, scripts (rescored from new runs: their recorded uses are not in the backup), chat and workspace; its owner presses Play. Its personal skills stayed with the family but are not re-linked to it.

Updating the engines

opencode, Claude Code and Codex are installed as pinned, integrity-checked binaries next to older versions (infra/opencode/install.sh, infra/claude-code/install.sh, infra/codex/install.sh). Bumping one means installing the new version, bumping the pinned version in the runner, re-checking the event and stream shapes the glue relies on, running the tests, and restarting the API. For Codex a contract test checks the installed binary: every config override and disabled feature must still be accepted, and the app-server protocol must be the one the glue was written against.

Master key rotation

The master key that wraps AI keys and game passwords can be rotated without downtime. Add the new key, keep the old one as a "previous" key, restart, re-wrap every row with the rotation script, check, then drop the old key. Keep the master key backed up: without it, the stored AI keys and avatar game passwords can't be decrypted.