Ghost for Verde

Tooling to run Ghost for the online publication Verde, currently hosted at ghost.org. See APP.md for the spec and the decisions behind it, and MIGRATION.md for the cutover checklist.

Development

ghost-dev runs the whole site, not only Ghost: Caddy answers on port 2368 and splits /ghost/ads off to ../verde-adserver, exactly as it does on the server, with Ghost itself published behind it. Ghost is the same container image the server uses, against a MySQL 8.4 the dev shell runs natively; the ad server is run from its own working tree against a PostgreSQL of its own. Everything lives under .state/ and can be thrown away.

nix develop
ghost-dev start     # the site at http://localhost:2368, admin at /ghost/, ads at /ghost/ads/
ghost-dev seed      # load the development dataset (see dataset/README.md)
ghost-dev status
ghost-dev logs      # follow the Ghost container log
ghost-dev mysql     # a MySQL shell on the dev database
ghost-dev ads       # re-seed the ad server with a placeholder advert per placement
ghost-dev ads-psql  # a psql shell on the ad server's database
ghost-dev stop
ghost-dev recreate  # replace the container, keeping the database and content
ghost-dev reset     # delete the container, database and content

Container settings are fixed when the container is created, so after changing any of them use recreate rather than restart.

start is idempotent: it initialises MySQL, creates the database, creates the owner account, activates the theme and draws the placeholder adverts only when those things are missing, so it is also the way to bring a stopped instance back up.

Requires rootless podman on the host — the dev shell deliberately does not ship its own, because rootless containers need the system's setuid helpers.

The owner account is dev@verdelehti.fi / verde-development.

Adverts

The theme's ad slots are filled here rather than left collapsed, so the front page looks like the front page and the whole serving path — placement, draw, snippet, impression, click — is exercised on a laptop instead of first on the server.

start runs ../verde-adserver from its working tree (GHOST_DEV_ADSERVER_DIR, default ../verde-adserver) and seeds it with nix/ads-seed.py: a placement for each of the theme's four slots, and one advert per placement, drawn to the placement's exact size — a grey box with AD across it. The manager is at http://localhost:2368/ghost/ads/ and needs no sign-in, because the dev shell sets VERDE_AUTH_BYPASS; on the server that page is behind Authelia.

The placement list lives in nix/ads-seed.py and mirrors the theme's assets/css/components/ads.css, which is where the slots are sized. Add a placement to the theme and add it there, or the new slot is the one thing on the page that stays collapsed. ghost-dev ads re-seeds without restarting anything; the seed creates what is missing and leaves the rest alone, so adverts uploaded by hand in the manager survive it.

Without the ../verde-adserver checkout the front door still stands, /ghost/ads answers nothing and the slots collapse — which is what the theme does on a page with no adverts anyway.

Signing in

The development instance runs the same nix/sso-adapter.js the server does, so a Remote-Email header signs you in as that staff user:

curl -i http://localhost:2368/ghost/ -H 'Remote-Email: dev@verdelehti.fi' -H 'Accept: text/html'

which comes back with a ghost-admin-api-session cookie good for the admin API. The Accept matters: the exchange only runs for a page someone is opening, never for the assets that page then loads, and a request that looks like neither gets no session. On the server the header comes from Authelia by way of Caddy, which strips whatever a client sent and requires a shared secret alongside it; here there is no proxy doing either, so the header is believed on its own — safe only because the instance listens on the loopback interface. The password form still works, and is what a browser uses.

Theme

The theme is ../reporter-verde, bind-mounted into the container's content directory, so the running site always reflects your working tree. ghost-dev start builds it once if assets/built is missing; run npm run dev in the theme repo to rebuild on change.

Activation is written straight into Ghost's settings table, which skips gscan. Run npm test in the theme repo when you want the theme actually validated.

Configuration

The defaults come from nix develop; override any of them in the environment.

Variable	Default
GHOST_DEV_STATE_DIR	$PWD/.state
GHOST_DEV_IMAGE	pinned in nix/image.nix
GHOST_DEV_PORT	2368 — the front door, and the port the site is visited on
GHOST_DEV_GHOST_PORT	2369 — where Ghost itself is published, behind the front door
GHOST_DEV_ADS_PORT	2370 — where the ad server listens, behind the front door
GHOST_DEV_URL	http://localhost:$GHOST_DEV_PORT
GHOST_DEV_THEME_DIR	../reporter-verde
GHOST_DEV_ADSERVER_DIR	../verde-adserver
GHOST_DEV_ADS_SEED, GHOST_DEV_ADS_DB_NAME	nix/ads-seed.py, verde
GHOST_DEV_DB_NAME, GHOST_DEV_DB_USER, GHOST_DEV_DB_PASSWORD	ghost_dev, ghost, ghost
GHOST_DEV_ADMIN_NAME, GHOST_DEV_ADMIN_EMAIL, GHOST_DEV_ADMIN_PASSWORD, GHOST_DEV_SITE_TITLE	see nix/ghost-dev.sh

Layout

• flake.nix — dev shell, the ghost-dev package, and packages.theme.
• nix/image.nix — the pinned Ghost image, shared by the dev instance and the NixOS module.
• nix/ghost-dev.sh — the development instance: Ghost, MySQL, the ad server and the front door.
• nix/ads-seed.py — draws a placeholder advert for every placement, for that instance.
• nix/module.nix — services.verde-ghost.instances.<name> for the server.
• nix/analytics-image.nix — the pinned traffic analytics image, the ingest half of Ghost's stats.
• nix/most-read.sh — tags the most read articles, for the theme's front page column.
• nix/stats-api.js — answers Ghost's analytics queries from ClickHouse, when the store is local.
• nix/stats-schema.sql — the one table those queries read.
• nix/sso-adapter.js — turns an Authelia identity into a Ghost staff session.
• nix/theme.nix — builds a Ghost theme from its source tree.
• nix/tinybird.sh — verde-tinybird: signing in to Tinybird, installing its pipes, reading tokens.
• dataset/ — the development dataset and the script that builds it.

Deployment

nixosModules.default provides services.verde-ghost.instances.<name>. Each instance is the pinned Ghost container published on a loopback port, behind Caddy, against the host's MySQL over its unix socket. In ../verde-infra:

imports = [ inputs.ghost-verde.nixosModules.default ];

services.verde-ghost.instances = {
  prod = {
    url = "https://www.verdelehti.fi";            # the canonical host; the apex redirects
    port = 2368;
    environmentFile = config.sops.secrets."ghost/prod".path;
    theme.package = themes.prod;                    # see the theme note below
    authelia = {
      endpoint = "http://127.0.0.1:9091";           # gates /ghost
      portalUrl = "https://auth.verdelehti.fi";     # where signing out goes
      sso = {
        enable = true;                              # and signing in comes from
        proofFile = config.sops.secrets."ghost/sso-proof".path;
      };
    };
  };

  staging = {
    url = "https://staging.verdelehti.fi";
    port = 2369;
    environmentFile = config.sops.secrets."ghost/staging".path;
    theme.package = themes.staging;
    mail.enable = false;                            # SMTP points at the discard port
    authelia = {
      endpoint = "http://127.0.0.1:9091";
      protect = "site";
      portalUrl = "https://auth.verdelehti.fi";
      sso.enable = true;
    };
  };
};

The theme is per instance, and verde-infra uses that: it pins reporter-verde twice — staging at main, production at the production branch — and builds each with lib.mkTheme, so a theme change is seen on staging before a merge takes it to readers.

mkTheme = src: inputs.ghost-verde.lib.mkTheme {
  inherit pkgs src;
  version = (builtins.fromJSON (builtins.readFile "${src}/package.json")).version;
  npmDepsHash = "sha256-...";                       # of the theme's package-lock.json
};

packages.theme builds the same thing from this flake's own pin. Nothing deployed uses it; it is what nix build checks and what a single-theme deployment would consume.

Each instance gets a MySQL database and user, a Caddy virtual host, a daily database dump, and its theme copied into the content directory on every start. MySQL itself is enabled by the module, shared by all instances, and pinned to 8.4 — Ghost 6 supports it and nixpkgs no longer ships 8.0.

An npmDepsHash — the one in flake.nix, and verde-infra's — changes whenever the theme's package-lock.json does; recompute it with nix run nixpkgs#prefetch-npm-deps -- path/to/package-lock.json.

Secrets

The environmentFile is a sops-managed env file. podman reads it literally, so values may contain spaces and angle brackets; the only constraint is that the database password must not contain a single quote, because it is interpolated into SQL. Generate it with openssl rand -hex 32.

Transactional mail is plain nodemailer, so any SMTP provider works — Brevo below:

database__connection__password=...
mail__transport=SMTP
mail__options__host=smtp-relay.brevo.com
mail__options__port=587
mail__options__auth__user=...
mail__options__auth__pass=...
mail__from=Verde <noreply@verdelehti.fi>

Newsletters are a separate system and only work with Mailgun. Ghost's bulk sender instantiates mailgun-email-provider and nothing else, over Mailgun's API rather than SMTP, so no SMTP provider can stand in for it. Add all three to the same file:

bulkEmail__mailgun__apiKey=...
bulkEmail__mailgun__domain=mg.verdelehti.fi
bulkEmail__mailgun__baseUrl=https://api.eu.mailgun.net/v3

baseUrl is not optional — Ghost calls new URL() on it with no fallback — and it selects the region, so a domain in Mailgun's EU region needs api.eu.mailgun.net or every send fails to authenticate. Only the origin is used, so the path is cosmetic.

Configuring it here rather than in Ghost's admin matters for more than tidiness. Ghost prefers config over the mailgun_* values in its database, and keeping the credentials out of the database keeps them out of the dumps — so restoring production into staging cannot carry a working newsletter sender to an instance holding real members. A mail.enable = false instance pins these at the discard port for the same reason.

Staging's file needs only the database password; its mail is disabled in the module rather than left unconfigured, so a stray send fails instead of reaching real subscribers.

Filling a non-production instance

Two manual units, neither on a timer, because both replace whatever is there:

systemctl start verde-ghost-staging-seed       # the development dataset
systemctl start verde-ghost-staging-restore    # production, members scrubbed

seed = {
  enable = true;
  credentialsFile = config.sops.secrets."ghost/staging-seed".path;  # SEED_ADMIN_*
};
restore.from = "prod";

seed creates the owner if the instance has not been set up, then imports dataset/. restore takes the named instance's most recent dump from backupDir and a copy of its content directory, then rewrites every member address and name — every column the schema keeps one in, staff accounts excepted, since rewriting those would lock everyone out of the instance being restored.

The dump is written with --databases, so it names its source; the restore strips those lines, without which it would write production over itself rather than filling the target.

Images, media and files are hardlinked rather than copied, so a restore costs no disk however large the image store is — production's runs to tens of gigabytes. Both names point at one inode, which is safe because Ghost only adds image files: an upload on staging does not appear in production, and wiping these directories on the next restore drops a name rather than the data. Anything that rewrote an image in place would change both, which Ghost does not do. The two instances have to share a filesystem, which they do while they live under the same stateDir.

Web analytics

Ghost 6 counts readers itself, through Tinybird, and analytics turns that on:

analytics = {
  enable = true;
  port = 3000;
  store = "local";                                              # or "tinybird"
  tokenFile = config.sops.secrets."ghost/analytics-prod".path;
  local.secretFile = config.sops.secrets."ghost/stats-prod".path;
  local.port = 3002;
};

Three parts, only one of them Ghost's:

• The tracker. Ghost injects it into every page by itself once analytics.enable is set — the theme needs no change, and no third-party script is involved. It posts to /.ghost/analytics/api/v1/page_hit on this site, which is the point: the same request to someone else's domain is filtered by every blocklist a reader is likely to be running, and the views that go missing are not a random sample. Expect higher numbers here than Plausible or Umami ever reported.
• The ingest service, a second container from nix/analytics-image.nix. It parses the user agent and the referrer, hashes the reader's address together with a salt that rotates daily, and posts the result to Tinybird. The address itself goes no further: what is forwarded holds the user agent, language, the country the browser's own timezone implies, the referrer, the UTM parameters and the path — and a signature that cannot be turned back into a person, or matched to the same person tomorrow.
• The store, which is either Tinybird or this machine — see below.

The store, in detail:

Keeping the page views here

analytics.store = "local" replaces the account with a ClickHouse and a small service that answers the endpoints Ghost queries:

analytics = {
  enable = true;
  port = 3000;
  store = "local";
  tokenFile = config.sops.secrets."ghost/analytics-prod".path;   # TINYBIRD_TRACKER_TOKEN
  local = {
    port = 3002;                                                 # free on the host, not just here
    secretFile = config.sops.secrets."ghost/stats-prod".path;    # VERDE_STATS_*
  };
};

Only the reading half of Tinybird is Tinybird-shaped — the ingest container posts wherever it is told — so nix/stats-api.js speaks those endpoints over ClickHouse and Ghost cannot tell the difference. It is a reimplementation of the pipes Ghost ships in its own image, column for column; the ones it does not implement (the UTM breakdowns) answer empty, which is how Ghost draws a panel with nothing in it. The approach and several of the queries follow kaperkunde/CoolGhost (MIT), which did this first.

Three things are worth knowing about how it is wired:

• Ghost reads its own figures over its own public name. Its request client refuses a private address unless the host matches the site url, so tinybird__stats__endpoint is https://<site>/.ghost/stats and the container gets that name in its hosts file pointing at the gateway. The request leaves the container, reaches Caddy, and comes back — without leaving the machine.
• Only the pipes are public. The admin reads them from a browser, so they have to be. The same service also accepts page views, and that endpoint stays on the loopback interface where only the ingest container reaches it: a stranger who could post there would be choosing what the front page calls most read. Both halves check a token regardless.
• The schema is applied on every start, and every statement is an IF NOT EXISTS, so a column added to nix/stats-schema.sql reaches an instance that already has a table.

ClickHouse is shared by every instance that keeps its views here, as MySQL is, and gets a database each. Its own defaults assume the machine belongs to it — most of the memory, five gigabytes of mark cache — so the module caps it at analytics.local.memoryMax, a gigabyte by default, with a 256 MB mark cache and two threads a query. Raise it if the Stats page feels slow; watch it with systemctl status clickhouse. Rows older than retentionDays (400) are dropped by ClickHouse itself.

Nothing about the reader ever reaches it: what the ingest service forwards holds a user agent, a language, the country the browser's own timezone implies, a referrer and a path — and a signature salted daily, which cannot be turned back into a person or matched to the same person tomorrow.

Keeping them in Tinybird

analytics.store = "tinybird" is the arrangement Ghost's own documentation assumes, and the store is then the one part not on this machine:

analytics = {
  enable = true;
  port = 3000;
  store = "tinybird";
  endpoint = "https://api.europe-west3.gcp.tinybird.co";   # regional; see below
  tokenFile = config.sops.secrets."ghost/analytics-prod".path;
};

A page view spends one of the free plan's thousand daily requests, which was measured here rather than read out of the documentation — the plan page says "1,000 requests per day per organization", the limits page calls the same thousand queries and lists the Events API only under separate per-second limits, and Tinybird's own forum answers contradict each other. So the free plan tops out at a thousand page views a day, minus whatever the Stats page spends, and past that the views are dropped rather than the dashboard merely erroring. That is what the local store above is for; a paid plan is the other answer.

Caddy routes /.ghost/analytics to the service. /.ghost rather than /ghost means it falls outside the admin gate and outside the ad server's prefix, so nothing has to be excluded from anything.

Two of Ghost's four settings are secret and belong in the instance's environmentFile, spelled the way Ghost spells them:

tinybird__workspaceId=...
tinybird__adminToken=...

Ghost signs its own short-lived, site-scoped read tokens from those, so nothing long-lived reaches a browser. The service's own TINYBIRD_TRACKER_TOKEN goes in analytics.tokenFile, a second file: that container has no business holding Ghost's database password, and Ghost none holding an append token.

Setting up the workspace

Only with the Tinybird store; a local one has no workspace and nothing to deploy. The datasources and pipes Ghost queries ship inside the Ghost image, and verde-tinybird installs them from the image this flake pins — so what Tinybird holds is what this Ghost asks for, not whatever the vendor's repository looked like that day. It runs from a workstation, never the server:

nix develop
verde-tinybird login      # once, in a browser; pick an EU region
verde-tinybird deploy     # installs the datasources and pipes
verde-tinybird tokens     # prints the values, grouped by the file they go in

tokens prints the tracker token, the workspace id and admin token, and the regional API host that analytics.endpoint wants — Tinybird's default api.tinybird.co is not necessarily yours.

A Ghost upgrade can change those datafiles. Nothing breaks loudly if it does: the Stats page and the most-read job simply see less than they should. Run verde-tinybird deploy again after an upgrade that moves the analytics schema.

The most read articles

The theme's front page has a column of three articles beside the hero. It used to hold the newest three; it now holds the most read, and mostRead is what keeps that true:

mostRead = {
  enable = true;
  credentialsFile = config.sops.secrets."ghost/most-read".path;   # GHOST_ADMIN_KEY, TINYBIRD_TOKEN
};

Hourly, verde-ghost-<name>-most-read.service asks Tinybird for the most read posts of the last seven days and puts the internal tag #luetuimmat on the top three, taking it off whatever was carrying it before. The theme selects on that tag. It needs analytics.enable, since that is what puts anything in Tinybird to read. Development has no readers to rank, so the fixture carries the tag on three articles instead — see dataset/README.md.

It asks the same pipe Ghost's own Stats page asks, api_top_pages, with the site uuid and the timezone it reads from Ghost — so neither is configured here and neither can drift from what Ghost believes.

The ranking goes into Ghost rather than into a JSON file the browser fetches because a theme can ask Ghost and nothing else. Going through a tag means the column is rendered server-side by the theme's own post card: no second copy of that markup in JavaScript to drift from the first, nothing to lay out after the page has painted, and a reader with no JavaScript sees what everyone else sees. Internal tags are invisible to readers — Ghost's foreach filters a post's tags to the public ones — though they do appear in post_class, as #import-* already does on every imported post.

Three things follow from where the ranking lives:

• It reads Tinybird directly, not Ghost's stats API. That API is closed to integration tokens: an allowlist in Ghost's admin middleware lets a key reach posts, tags and site and nothing else, whatever an endpoint's own permissions say — /ghost/api/admin/stats/* answers 403. Reaching it would take a staff user's token, which can do everything that member of staff can. Two narrow credentials are the better trade.
• So it carries two. GHOST_ADMIN_KEY, a custom integration's Admin API key in id:secret form, which edits posts and tags and reads no member; and TINYBIRD_ADMIN_TOKEN, the same secret Ghost signs its own read tokens with. The job signs one too — five minutes, one pipe, one site — which is why nothing long-lived has to be issued for it and why the same code works against either store. Ghost is reached on the loopback port, so Authelia is never in the way.
• An incomplete answer changes nothing. Fewer than three of the most read posts still being published, or Tinybird being unreachable, fails the unit rather than tagging a short set. The front page then keeps the ranking it has — or falls back to the newest three, which is what the theme does whenever the tag is not on exactly three posts. gron's failed-unit sweep turns the failure into mail.

The tag name is in two places, mostRead.tag here and the theme's filter="tag:hash-luetuimmat"; change them together. window, count and interval are options too.

The posts themselves are edited, so each run bumps updated_at on the handful of posts that enter or leave the ranking, and the update carries the updated_at it read — an editor saving the same post at the same moment collides instead of being overwritten.

What the module does not do

• Activate the theme on an ordinary deploy. Ghost stores the active theme in its database, so it arrives with the migration import or is set once in the admin UI; deployment only installs the files. The seed unit is the exception — it activates the theme through the API, because Ghost's content importer does not apply active_theme.
• Ship backups off the machine. Dumps land in /var/backup/verde-ghost; backing that up, along with the content directories under /var/lib/verde-ghost, is ../verde-infra's job.
• Run Authelia. authelia.endpoint only emits the Caddy forward_auth block, pointing at the Authelia ../verde-infra runs. Leave it null and the instance is ungated.

Sharing a domain with the ad server

The ad server is served from /ghost/ads on the same host, so an instance gating /ghost would authenticate its serving API too — an advert nobody can see. authelia.excludePaths leaves paths out of the gate:

authelia.excludePaths = [ "/ghost/ads" "/ghost/ads/*" ];

Both forms are needed. Caddy's /ghost/* matches the bare /ghost/ads, so excluding only the subtree leaves that one path falling through to Ghost, which answers 404.

Signing in through Authelia

authelia.sso.enable installs nix/sso-adapter.js into the instance's content directory and points Ghost at it. Ghost then exchanges the address in Remote-Email for a staff session on every request to /ghost, and creates it already verified — which is also what stops Ghost emailing a six-digit code, so an instance with mail.enable = false becomes usable rather than locked.

Nothing is provisioned: an identity with no Ghost account of that address just reaches the ordinary sign-in form. Invite staff in Ghost as before, with the address they have in Authelia.

Two things hold the trust up, and the option asserts on both:

• Caddy strips every Remote-* a client sends before forward-auth sets the real ones. Without that, the paths in excludePaths reach Ghost without meeting the gate, and anyone could name themselves the owner. This is why sso.enable requires caddy.enable.
• sso.proofFile adds a secret in a header only Caddy sets, read by Caddy and the container from the same file so it lives in neither the Nix store nor a command line. It covers the loopback port the container publishes, which every process on the host can reach. Set it on anything real; the file has to be readable by the caddy user as well as by root.

Caddy also strips Referer under /ghost. Ghost binds a session to the origin of the request that created it, and works that origin out from the Referer when there is no Origin header — which a top-level navigation has none of. Arriving from the Authelia portal, the session would be bound to the portal, and the admin's first API call refused as coming from the wrong place.

authelia.portalUrl is what makes signing out mean something. Ghost destroys its session and then reloads the admin, which the exchange answers with a new one; Caddy marks the sign-out and sends the reload that follows to Authelia's /logout. Leave it null and the sign-out button only flickers.