# Hosting languid on Fly.io

A hosted setup keeps agents running while your own computers are off. This guide sets one up on [Fly.io](https://fly.io) with three apps, all built from the repo root with configs in `deploy/fly/`. The configs are the ones [languid.ai](https://languid.ai) runs; change the app names, domains and region to your own. (The build context is filtered by `/.dockerignore`, an allowlist.)

| config | role | public? |
|---|---|---|
| `pds.toml` + `pds.Dockerfile` | the Spaces-alpha PDS: owner and agent accounts and their spaces | yes, HTTPS |
| `host.toml` | the languid service: runs agents, owns their records | optional: tailnet only, or public with sign-in |
| `runner.toml` | a computer that runs tools (pi's `bash`, `read`, `edit`, …) for the service | no: it dials out to the service |

`host.toml` and `runner.toml` share one image (`deploy/fly/Dockerfile`). `deploy/fly/languid.sh` joins the tailnet (`TS_AUTHKEY`), then runs the role set by `LANGUID_ROLE`.

```
 owners (pi, languid, browser) ──HTTPS or tailnet──▶ service ◀──long poll── runners (Fly, owners' computers)
                                                        │  HTTPS
                                                        ▼
                                                       PDS ──▶ plc.directory
                                                        │
                                                 model provider
```

Making this one image and one command is on the [roadmap](https://tangled.org/cameron.stream/languid/blob/main/ROADMAP.md).

## Secrets

Never in the repo, never in records. Keep one source for every app: a Bitwarden Secrets Manager project (below), or the file `~/.config/languid/hosted.env` (mode 0600, `KEY=value` lines). `just secrets <pds|host|runner> <app>` stages the keys each app needs, so every value is entered once. Fly secrets are per-app and can't be read back, so apps can't share them directly.

| secret | app | what |
|---|---|---|
| `PDS_JWT_SECRET`, `PDS_ADMIN_PASSWORD`, `PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX` | PDS | generate them yourself (see Bluesky's PDS docs) |
| `LANGUID_PDS_ADMIN_PASSWORD` (= `PDS_ADMIN_PASSWORD`), `LANGUID_SERVER_TOKEN` | service | the service creates accounts with the PDS admin password; the server token is the installation's own token (random, 32 bytes or more) |
| `TS_AUTHKEY` | service, runners | a Tailscale OAuth client secret (below) |
| `*_API_KEY` (`OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, `RADIUS_API_KEY`, …) | service | model provider keys |
| `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET` | service | optional: selling credit and plans on `/billing` (see [Credit and billing](#credit-and-billing)) |
| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | service | optional: cloud sandboxes on Modal |
| `LANGUID_SERVICE_URL`, `LANGUID_JOIN_CODE` | runners | set by hand per runner (see Runners); the code works once |

To add a key without it passing through anyone else: `echo 'OPENROUTER_API_KEY=…' >> ~/.config/languid/hosted.env`.

### Bitwarden

With Bitwarden Secrets Manager set up, the project is the source and `hosted.env` is only the fallback. Install `bws` from Bitwarden's [releases](https://github.com/bitwarden/sdk-sm/releases) (it isn't in Homebrew), make a machine account with access to one project, then:

```sh
security add-generic-password -U -s languid-bws -a "$USER" -w    # paste the access token at the prompt
echo 'BWS_PROJECT_ID=<project id>' > ~/.config/languid/bws.env && chmod 600 ~/.config/languid/bws.env
just secrets import    # hosted.env and inkling-modal.key into the project; rerun any time, it only changes what differs
just secrets check     # key names in Bitwarden, hosted.env and Fly side by side
```

| command | does |
|---|---|
| `just secrets host languid-nimbus [--dry-run]` | stage the host's keys on Fly (also `pds`, `runner`); they take effect on the next deploy |
| `just secrets modal [--dry-run]` | set the Modal secret `inkling-api-key` from `INKLING_API_KEY` |
| `just secrets run -- <cmd>` | run a command with every secret in its environment, instead of pasting `hosted.env` |
| `just secrets names` | which source is in use, and its key names |

Every command says when it fell back to `hosted.env`. If Bitwarden is configured but unreachable, it stops rather than using a file that may be stale. `LANGUID_SECRETS_SOURCE=hosted.env` forces the file. Fly-only keys set by hand (`NIMBUS_*`, `OPENROUTER_API_KEY`, `RADIUS_API_KEY`, `TS_AUTHKEY`, `LANGUID_SERVER_TOKEN` on `languid-nimbus`) can't be read back from Fly; add them to the project with `bws secret create` to make it complete.

The PLC rotation key can rewrite every account's DID document. Keep a copy somewhere safe, such as a password manager: Fly secrets can't be read back.

## Setting it up

### 1. Tailscale

The configs put the service and runners on a tailnet: the installation token is accepted there, and runners can reach a service that has no public address. They join as tagged nodes using one OAuth client secret as `TS_AUTHKEY`, which doesn't expire, and keep their Tailscale state on a volume, so they log in once.

1. Tailnet policy file: add a tag (the configs use `tag:nimbus`) with `"tagOwners": { "tag:nimbus": ["autogroup:admin"] }`. If the policy restricts traffic beyond the default allow-all, allow your devices and the tag to reach the tag on tcp 2590 (the service). Runners need no inbound port.
2. Admin console → Settings → OAuth clients → Generate: scope **Auth Keys** (write), with that tag.
3. Add the `tskey-client-…` secret to `hosted.env` as `TS_AUTHKEY`.

Without `TS_AUTHKEY`, the service listens on all interfaces at `LANGUID_URL`, for setups that put it behind TLS themselves. A plain auth key (`tskey-auth-…`) also works if you clear `TS_TAGS` in the toml.

### 2. PDS

The PDS uses the real PLC directory (`plc.directory`), so DIDs are public and permanent. Handles are `<name>.<your domain>`. It's invite-only with crawlers off, so nothing is announced to relays. Fly takes daily volume snapshots and keeps five. `pds.Dockerfile` hands the volume to the `node` user and drops root before starting the PDS.

```sh
fly apps create <pds-app>
fly volumes create pds_data -a <pds-app> -r <region> -s 1
deploy/secrets.sh pds <pds-app>
fly deploy . -a <pds-app> --config deploy/fly/pds.toml --ha=false
fly certs add pds.<domain> -a <pds-app>
fly certs add '*.<domain>' -a <pds-app>     # handles
```

Set `PDS_HOSTNAME` and `PDS_SERVICE_HANDLE_DOMAINS` in `pds.toml` to your domain first. Then add the DNS records below and wait for both certificates before creating accounts: DIDs on plc.directory can't be deleted.

If a deploy fails with "insufficient resources … existing volume", the volume's host is full: create a new empty volume with the same name and destroy the old one (only while it's empty).

### 3. Service

```sh
fly apps create <service-app>
fly volumes create nimbus_data -a <service-app> -r <region> -s 1
deploy/secrets.sh host <service-app>
fly deploy . -a <service-app> --config deploy/fly/host.toml

# the installation owner, and a first agent
fly ssh console -a <service-app> -C "languid init you.<domain>"
fly ssh console -a <service-app> -C "languid admin publish-lexicon"
fly ssh console -a <service-app> -C "languid agent create scout --model openrouter/deepseek/deepseek-v4-flash"
```

`publish-lexicon` publishes the agent space type from the owner's repo and prints the TXT record to add (`_lexicon.space.languid.ai` for languid's NSIDs). Without it, every OAuth sign-in is refused. After that, the service republishes it itself whenever it starts with a changed collection list, so a deploy that adds or renames a collection needs no step of its own. `publish-lexicon` needs the owner's password, so it runs on the service's host, not from a client signed in with `languid login`. `publish-lexicon eval` publishes the eval lexicons the same way (`_lexicon.eval.languid.ai`); the service doesn't publish those itself.

**Models.** The service calls models with its own keys: any provider pi-ai supports, from the `*_API_KEY` secrets. `languid models` lists what's available. Subscription logins work only where pi's `auth.json` is, so not on the host.

### 4. A public address and sign-in

Optional: without it, the service is reachable only over the tailnet with the installation token. With it, owners sign in with their atproto accounts and you can invite people.

1. In `host.toml`, set `LANGUID_URL=https://<host>` and keep the `[http_service]` section. atproto authorization servers fetch `https://<host>/oauth/client-metadata.json`, so it must be HTTPS on port 443. With an `https://` URL, `languid.sh` starts the service on every interface (tailnet included) so Fly's proxy can reach it.
2. Set `LANGUID_CLIENT_IP_HEADER=fly-client-ip`, so rate limits see each client's address rather than the proxy's, and requests through the proxy are never taken as local.

To choose which models run on your keys, set `LANGUID_HOUSE_MODELS` to comma-separated `provider/model` values, in order. The pickers show them first, under **Picked by Languid**, and they become the only models anyone runs on your keys and credit, you included. Every other model needs the owner's own connection (ChatGPT, Grok, Anthropic or OpenRouter). `LANGUID_GUEST_MODEL` sets the model new agents get (`together/thinkingmachines/Inkling-2-Small` on languid.ai), and stands in for `LANGUID_HOUSE_MODELS` when that's unset. With neither set, your keys run any model.
3. `fly certs add <host> -a <service-app>` and point DNS at the service's addresses (`fly ips list`).
4. Invite owners: see [Owners, invites and credit](#owners-invites-and-credit).
5. Optional, for [agents' sites](/docs/sites): set `LANGUID_SITES_HOST=sites.<domain>`, run `fly certs add sites.<domain> -a <service-app>`, and add the `sites` record below. Use a name of its own, never the service's host: sites run other people's HTML and scripts. Without it, sites are served only by a service on loopback, at `/_sites/`.

The installation token then works only from the service's host and the tailnet, so `fly ssh console … languid …` and tailnet clients keep working. `https://<host>/` is the landing page, so the host can be the site's own name; reserved names (`www`, `pds`, `api`, …) can't be taken as handles.

### 5. Clients

```sh
curl -fsSL https://<host>/install | sh   # installs the CLI and signs in to this service
languid login --url https://<host>   # with a public address; the browser asks which account
languid chat scout
languid computer setup                             # let agents run tools on this computer
```

Without a public address, a client on the tailnet uses the installation token instead: `~/.languid/state.json` with `{"pds": "https://pds.<domain>", "agents": {}, "runners": {}, "server": {"url": "http://<service node>.<tailnet>.ts.net:2590", "token": "<LANGUID_SERVER_TOKEN>"}}`.

### 6. Runners on Fly

```sh
languid computer invite --agent scout --ttl 60   # prints a join code
fly apps create <runner-app>
fly volumes create runner_data -a <runner-app> -r <region> -s 1
deploy/secrets.sh runner <runner-app>
fly secrets set -a <runner-app> --stage LANGUID_SERVICE_URL=<service URL> LANGUID_JOIN_CODE=<code>
fly deploy . -a <runner-app> --config deploy/fly/runner.toml
```

The runner joins once: its registration and tailnet node live on its volume, so restarts and redeploys need no new invite. To re-join (after `computer remove`, say), delete `/data/nimbus/state.json` on it and set a new code.

## Owners, invites and credit

Sign-up is by invite. An invite is a link to the service's `/signup` page, where the invitee picks a handle and a password. The service makes their account on its own PDS and lets them sign in.

Make invites on `/admin`, signed in as the installation owner, or from the service's host:

```sh
languid admin invite-owner --budget 10 --uses 1 --days 7   # prints the link
languid admin owners                                        # who signed up, their agents, and what they've spent
languid admin budget alice.<domain> 20                      # set someone's credit; none makes it unlimited
```

- `--budget` is the credit, in US dollars, that the invite gives each person. Leave it out for unlimited use of your keys.
- `--uses` is how many people one link signs up. For a room, make one link with `--uses 30`; `/admin` also shows its QR code, sized to project.
- `--days` is how long the link works, 7 by default. `--model provider/id` picks the model their new agents start on.
- The service keeps only a hash of each code. If making the account fails (a taken handle, a PDS outage), the link still works.
- The PDS has no email, so nobody can reset a password by email. `com.atproto.admin.updateAccountPassword` on the PDS is the way back in.

`/admin` lists every owner with what they've spent and their credit, and can add credit, set it, make it unlimited, and give or take back a cloud sandbox and volume. It also shows the sign-up funnel.

### Credit and billing

Each model call on your keys costs the provider's price plus 5%, and the service adds that to the owner's spend when the call ends. An owner whose credit is used up gets `402` on calls to your keys until they have more. Calls on their own connections never count.

To let owners buy credit on `/billing`, set `STRIPE_SECRET_KEY` and `STRIPE_WEBHOOK_SECRET`, and point a Stripe webhook at `https://<host>/stripe/webhook` with these events: `checkout.session.completed`, `checkout.session.async_payment_succeeded`, `charge.refunded`, `charge.dispute.created`, and `customer.subscription.created`, `.updated` and `.deleted`. Credit comes in packs of $10, $25 and $50, or any amount from $10 to $500. `STRIPE_TAX=on` adds Stripe Tax, once you've set it up in Stripe. Without the Stripe keys, `/billing` shows credit and says buying isn't open yet. [payments.md](https://tangled.org/cameron.stream/languid/blob/main/docs/payments.md) has the full setup.

With `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` set too, `/billing` sells a cloud sandbox ($10 a month, 100 hours) and a volume for it ($3 a month, 10 GB).

## What has to stay running

- **The service has to stay on.** Running turns, runner connections, the wakeup clock and the Discord bots all live in its one process. If it stops, runs end, and every computer and bot looks offline. It runs on one machine, because its queues and write locks are in memory and `state.json` is on one disk.
- **The PDS is on, and could idle.** All of its state is on disk, so stopping it would lose nothing. In practice it would rarely sleep: the service writes to it on every step, and other atproto services look the agents' accounts up through it.
- **Runners are up to their owners.** A runner works only while it's running. It connects out, so nothing can wake it: a stopped runner shows as offline, and calls for it fail. Cloud sandboxes are the exception: the service starts one on Modal when a call needs it.
- **Model providers bill per call**, with nothing to keep running.

## DNS

For a domain whose apex is the service and whose subdomains are handles, as languid.ai's is:

| type | host | value |
|---|---|---|
| A / AAAA | `pds` | the PDS app's addresses |
| A / AAAA | `*` | the PDS app's addresses (for `name.<domain>` handles) |
| CNAME | `_acme-challenge` | from `fly certs setup '*.<domain>' -a <pds-app>` (wildcard certificate) |
| A / AAAA | `@` | the service app's addresses |
| CNAME | `www` | `<service-app>.fly.dev.` (the service redirects it to the apex) |
| CNAME | `sites` | `<service-app>.fly.dev.` (agents' sites, with `LANGUID_SITES_HOST`) |
| TXT | `_lexicon.space` | `did=<owner DID>`, from `publish-lexicon` |

Explicit records win over the `*` wildcard, so every other name under the domain goes to the PDS. Check certificates with `fly certs check <name> -a <app>`.

## Upgrading the PDS image

Dev (`dev/compose.yaml`) and hosted (`deploy/fly/pds.Dockerfile`) pin the same image digest. A new Spaces alpha release usually needs code changes too, so the image and the code move in one commit, and a hosted service gets both close together. The PDS's database migrations are one-way: there's no going back to the old image on migrated data, only to a snapshot.

1. Snapshot the PDS volume: `fly volumes snapshots create <volume-id>`, and note its id.
2. Deploy the PDS. The account database migrates at start; each account's store migrates when it's first opened. Check that `https://pds.<domain>/xrpc/_health` reports the new version.
3. Deploy the service right after, so it runs matching code.
4. Check an agent's records (`languid records <agent>`) and run a turn.

To go back: restore the snapshot into a new volume, pin the old digest again, and redeploy the PDS and the old code. Anything written after the snapshot is lost. Rehearse each upgrade on the dev stack with existing data first.

## Day to day

The `justfile` at the repo root has recipes for languid.ai's apps (`just` lists them); set `host_app`, `runner_app` and `pds_app` at its top to yours. Deploys refuse to run from anything but a clean, committed `main`.

| task | command |
|---|---|
| test, then deploy the service, the Fly runner and this computer's runner | `just deploy` |
| only the service / a Fly runner / the PDS | `just deploy-host` / `just deploy-runner [app]` / `just deploy-pds` |
| the service, runners and computers at a glance | `just status` |
| logs | `just logs host` (or `runner`, `pds`, `local`) |
| a languid command on the service host | `just remote agent list` |
| stage an app's secrets | `just secrets host <service-app>` |
| invite someone | `languid admin invite-owner --budget 10`, or `/admin` |
| tear down a runner | `languid computer remove <name>`, then `fly apps destroy <app>` |

Costs on Fly: a shared-cpu-1x 512 MB machine and a 1 GB volume per app, a few dollars a month each.
