Verde infrastructure

The NixOS configuration for gron, the EC2 host that runs Verde's services.

nixos-rebuild switch --flake ~/git/verde-infra#gron \
  --target-host gron --build-host gron --sudo --ask-sudo-password

The fish aliases newGron (deploy to gron) and newNix (rebuild locally) are defined in gron/default.nix.

What runs here

Service	Address	Notes
Caddy	:80, :443	TLS via Cloudflare DNS-01
PostgreSQL	unix socket	Planka, Forgejo and the ad servers
MySQL	unix socket	Ghost, one database per instance
Planka	planka.verdelehti.fi	container
Authelia	auth.verdelehti.fi	identity for every Verde service
Forgejo	git.verdelehti.fi	the Verde repositories
LLDAP	users.verdelehti.fi	the directory Authelia authenticates against
Ghost	www.verdelehti.fi, staging.verdelehti.fi	production is the public site; staging is gated
Ad server	/ghost/ads/ on both Ghost sites	manager behind Authelia, serving API public

Secrets

sops-nix, with the file at gron/secrets.yaml and the recipients in .sops.yaml. Read or edit it with sops gron/secrets.yaml; a value can be read with:

sops -d gron/secrets.yaml | yq '.ghost.prod'

Every secret declares the units that read it, so editing one and deploying restarts what needs restarting. A declared secret that is missing from the file fails activation before any service starts, so the declaration and the value always change together.

Authelia

Authelia is the gate; LLDAP is where the people are. Adding or removing a colleague, resetting someone's password, or changing which groups they are in is done at users.verdelehti.fi — a form, not a sops edit and a deploy. Authelia binds to it as authelia, a member of lldap_password_manager: it decides who may pass and can set passwords, and administers nothing else. Sessions are held in memory, so a restart signs everyone out.

Forgotten passwords are reset through Authelia's "Reset password" link, which mails a link to the address in LLDAP; the link lives 30 minutes. This does not work for members of lldap_admin. LLDAP does not let a password manager change an administrator's password, so for them the mail arrives and the link opens, and then Authelia answers "Unable to reset your password". That is accepted rather than fixed. Giving Authelia's account full administrator rights would let anyone holding its credential rewrite any password. An lldap_admin member who forgets theirs asks another one to set a new password at users.verdelehti.fi, or signs in there as admin, whose password is the lldap/admin-password secret and is reset to it on every start of LLDAP.

Authelia also limits reset requests per client address: a burst of five, after which they come back one per ten minutes at best and one per thirty once fifteen have gone. A person never meets this; a script that loops does.

Groups drive everything downstream: admins and editors are what the Ghost admin gate, the staging gate and the ad server's authorisation all read, arriving as Remote-Groups.

admins also decides how many factors the production Ghost admin asks for: that group gets two_factor, everyone else one_factor. It is meant to hold the people who are Administrators in Ghost as well, so the second factor stands where the privilege is — keep the two in step when someone's Ghost role changes. Note that this is now the only factor between a person and Ghost's admin: the session Ghost's SSO adapter mints is created already verified, so Ghost never sends its own sign-in code.

admins is not an LLDAP permission. LLDAP's own rights come from lldap_admin, lldap_password_manager and lldap_strict_readonly; being in admins gets you as far as users.verdelehti.fi, where an ordinary user sees only their own profile.

Per-application roles are not here and cannot be. Ghost and Forgejo both take the identity from Authelia — Forgejo over OIDC, Ghost from the Remote-Email its own gate asserts — and both keep their permissions to themselves: Ghost's Editor/Author roles live in Ghost's admin, Forgejo's repository permissions in Forgejo. Both have perfectly good interfaces for that.

Ghost's half of that is authelia.sso in gron/ghost.nix, and it provisions nothing. An address Authelia asserts that no Ghost staff account carries simply lands on Ghost's own sign-in form, so adding a colleague is still two steps — LLDAP for the identity, a Ghost invitation for the account — while removing one is a single step, because the gate stops them before Ghost is reached. The address has to match: mail in LLDAP and the email on the Ghost account are what is compared.

Signing out of Ghost ends the Authelia session too. It has to: Ghost destroys its own session and then reloads the admin, which the exchange would otherwise answer by signing you straight back in.

The circularity is deliberate but worth knowing. LLDAP's interface sits behind Authelia, which authenticates against LLDAP. If that ever traps you, reach LLDAP's own login directly:

ssh -L 17170:127.0.0.1:17170 gron
# then http://127.0.0.1:17170 in a browser, with LLDAP's own admin account

The admin account's password is lldap/admin-password in sops. It is LLDAP's own, not an Authelia account, and exists for exactly this.

Mail goes out over Brevo SMTP, sharing the credentials Ghost uses for its transactional mail. The whole notifier block lives in sops (authelia/notifier) rather than here because it carries the password; Authelia merges every settings file it is given. Password resets and enrolment links arrive by email, and nothing is written to disk any more.

Authelia checks the SMTP connection at startup, so bad credentials surface as a failed authelia-main.service rather than as mail that silently never arrives.

Forgejo

Repositories are anonymously readable so nix flake update works on any machine without credentials. Sign-in is delegated to Authelia over OIDC and there is no local signup; a unit registers the provider with Forgejo's CLI on every activation, because the NixOS module cannot express it.

Git over SSH uses this host's sshd, with Forgejo maintaining authorized_keys for the git user, so clone URLs are git@git.verdelehti.fi:….

Forgejo does not promote its first user to administrator and has no command to promote one later, so the same unit creates the janne account up front with the admin flag. ACCOUNT_LINKING=auto attaches the Authelia identity to it on first sign-in, matched by email — which means the address in gron/forgejo.nix and the one in Authelia's user file have to agree.

Bringing these up for the first time

Steps 1 to 6 are done. They are kept here because they are what a rebuild from scratch would need.

1. Point auth.verdelehti.fi and git.verdelehti.fi at gron in Cloudflare, DNS-only. A proxied record breaks git over SSH, since Cloudflare's proxy carries only HTTP and HTTPS.
2. Deploy. Watch Caddy obtain both certificates.
3. Add a bootstrap password to sops as authelia/bootstrap-password, declare it in sops.nix, and sign in at auth.verdelehti.fi.
4. Enrol a TOTP device from the emailed link.
5. Change the password away from the bootstrap one, then delete the secret from both the sops file and sops.nix — sops-nix fails activation for a declared secret that is missing from the file.
6. Sign in to git.verdelehti.fi with the Authelia button. Confirm you land on the janne account and that it is an administrator.
7. Create the repositories, push them, and add a GitHub push mirror to each under Settings → Repository → Mirror settings. That needs a GitHub token with write access.
8. Repoint the flake inputs — this repository's own, and ghost-verde's theme input — at git.verdelehti.fi, then nix flake update.

Keep the verde organisation public as well as its repositories. Forgejo hides a private one by answering 404, and that reaches Nix as a fetch failure rather than a permissions error, so nix flake update fails somewhere far from the cause.

What reaches production

Every input this flake deploys is pinned to a branch called production, so nix flake update cannot carry anything onto the server that nobody put there on purpose. Work happens on main and reaches production when somebody merges it:

git -C ~/git/reporter-verde push origin main:production   # or a pull request in Forgejo
nix flake update reporter-verde-production                # in this repository
newGron

The theme is pinned twice, because it is the only thing the two Ghost instances can differ in: staging runs the theme's main and production its production. That is what staging is for — a theme change is seen there first, and nix flake update reporter-verde-staging && newGron is how it gets there, leaving production where it was.

Everything else is shared by construction. One NixOS module builds both instances' units, so a change to ../ghost-verde or ../verde-adserver arrives at staging and production together and only after a merge to their production branch. To rehearse one before merging, point its input at the branch in flake.nix for as long as that takes, and remember that production is running it too.

Protect the production branches in Forgejo (Settings → Branches) so they take pull requests rather than pushes. The pinning above decides what a deploy picks up; branch protection is what makes the merge itself deliberate.

Ghost and the ad server

Both come from ../ghost-verde and ../verde-adserver as flake inputs; see those repositories for how the modules work. Two things about them are decided here rather than there:

• Where the site's readers are counted. Production runs Ghost's own analytics — the tracker Ghost injects, the ingest container beside it, and the store behind that — and staging does not, because its readers are us. See ../ghost-verde's README for the parts.
• Which ad server paths are public. The serving API, the render beacon and the click redirect answer anonymous readers; everything else goes through Authelia. Ghost's admin gate excludes /ghost/ads and /ghost/ads/* for the same reason. Staging needs none of that separation, since Caddy applies its site-wide gate before reaching those handlers.

Testing sign-in end to end

sso-test on gron is a new colleague done by a script: it creates a user in LLDAP with a mailbox it can read, sets the password through Authelia's forgotten-password mail, enrols a TOTP device through the one-time-code mail, opens Ghost's admin through the gate on both instances, signs into Planka over OIDC twice — with the portal session open, and from a cold browser the way a person does — and deletes the user again. Every step is reported, with how long each mail took to arrive, and a failure names which of Brevo, Authelia, LLDAP, Caddy, Ghost or Planka answered wrongly.

ssh gron sso-test                  # the whole journey as an editor in planka-users; non-zero on any failure
ssh gron sso-test matrix           # every group combination in turn, one user each, one table at the end
ssh gron sso-test --case planka    # one combination: planka, ghost, ghost+planka, each +lldap-admin, admin
ssh gron sso-test --groups editors # any other combination; the expectations follow from the groups
ssh gron sso-test --no-totp        # a colleague without a second factor: Planka and staging refuse
ssh gron sso-test --keep           # leave the user in LLDAP and print its password and TOTP secret
ssh gron sso-test --ghost prod -v  # one instance, every request logged
ssh gron sso-test cleanup          # delete whatever sso-test-* users an interrupted run left

What a user should reach is read off the groups, the same way the access rules read them: anyone the portal knows gets through Ghost's gate (one factor on production, two on staging and for admins), but only editors and admins are Ghost staff, so a user outside those is given a tagged address (verde-sso-test+nostaff@…) that Ghost has never seen and must land on Ghost's own form; only planka-users get into Planka, and the test expects Authelia's refusal for everyone else; only admins reach the directory UI at users.verdelehti.fi. lldap_admin is tested by putting the user in it after the first reset and resetting again. That second reset is expected to be refused, as described under Authelia above, and the step passes when it is. If it ever succeeds, the step fails: Authelia's account would then have more than password-manager rights. Every case also signs into LLDAP itself with the password in force.

Authelia rate-limits password resets per client address. The default buckets read as five starts in ten minutes, ten in fifteen and fifteen in thirty, but each refills only one request per period: once the burst is spent, one reset start gets through every thirty minutes. A refused request still uses up the other buckets, so retrying early only lengthens the wait. The test retries once, at the time Authelia's Retry-After header names. The matrix makes eleven reset starts from gron, so a full run takes about an hour when the limiter is rested and several hours straight after another run. Run it in the background:

ssh gron 'nohup sso-test matrix > /root/sso-test-matrix.log 2>&1 &'

The same limit applies to everyone behind one office address, but a colleague's handful of attempts stays well inside the burst.

The test address is verde-sso-test@sturmfrei.fi, a catch-all on Fastmail, read over JMAP with the token in sops at sso-test/fastmail-token. The value there is a placeholder until a token is made (Fastmail → Settings → Privacy & Security → Integrations → API tokens, read-only access to mail) and stored:

sops set gron/secrets.yaml '["sso-test"]["fastmail-token"]' '"fmu1-…"'

Without a mailbox, sso-test --mail stdin asks for the link and the code to be pasted, and --mail dir:/some/path reads them from files dropped there.

Two things are left behind on purpose: a Contributor staff account for the test address in each Ghost instance — created from an invitation row the test writes and then accepts through Ghost's API, since only an administrator's session could create the invitation — and the Planka accounts the first OIDC sign-in creates, one for each of the two addresses. Later runs adopt both by address, which is the case that matters: someone whose Ghost or Planka account predates their directory identity.

gron/sso-test.nix has a scheduled switch for a daily run; a failed run is a failed unit, so the alert sweep mails the report. It is off until the token is real.

Backups

Everything stateful on this host goes into a local Borg repository at /var/backup/borg at 02:30, and that repository is copied to S3 straight afterwards. Borg provides retention and deduplication; the S3 copy is what survives losing the machine.

Databases are dumped first, because a database's files on disk are not a backup of it: PostgreSQL at 01:30 via services.postgresqlBackup, and Ghost's MySQL by its own module around midnight. Borg then takes those dumps together with the files no dump contains — Ghost's uploaded images and themes, the ad creative, the git repositories, Authelia's enrolled 2FA devices, and Planka's uploads.

Retention is 7 daily, 4 weekly and 3 monthly. The S3 sync uses --delete, because a Borg repository is a set of chunks: leaving pruned ones behind would grow without limit and leave the copy inconsistent with its own index. It runs only after Borg has finished, never alongside it, and a 05:00 timer repeats it in case the network was down.

The repository is encrypted, so the passphrase is part of the backup. It is borg/passphrase in sops, encrypted to the admin age key. Losing that key loses every archive in S3 as completely as losing the disk would, so keep it somewhere other than this machine.

Restoring:

export BORG_REPO=/var/backup/borg
export BORG_PASSCOMMAND="cat /run/secrets/borg/passphrase"
borg list                       # what is there
borg extract ::archive-name     # into the current directory

To rehearse a restore from the S3 copy rather than the local repository, give Borg a security directory of its own:

aws s3 sync s3://verdelehti-backup-gron/borg /tmp/restore/repo
export BORG_REPO=/tmp/restore/repo BORG_SECURITY_DIR=/tmp/restore/security
export BORG_PASSCOMMAND="cat /run/secrets/borg/passphrase"
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
borg list

Without BORG_SECURITY_DIR, Borg records the temporary path as the repository's location for this user, and the scheduled job then refuses to touch the real repository — it sees a repository that has apparently moved and stops for confirmation nobody is there to give. Recovering from that means accepting the relocation once by hand.

Still to do

• Move this host to t4g.large. 3.86 GiB is not enough — see gron/memory.nix for what it cost on 19 and 21 September, and MIGRATION-t4g.md for the checklist that doubles it. The one thing to settle before anything else is whether 13.60.120.148 is an Elastic IP, because if it is not, stopping this instance releases it and takes all six hostnames with it.
• /var/lib/lldap is in neither the Borg paths nor the PostgreSQL dumps, and LLDAP keeps its users in SQLite there. Every colleague's account and password hash is currently unbacked-up.
• The remaining work otherwise is the ghost.org migration itself, in ../ghost-verde/MIGRATION.md.