Run GitLab CI jobs on exe.dev, a fresh VM per job

How it works#

• my-runner-base is an ordinary VM. It never runs jobs; it's what each job's VM is copied from. It has the gitlab-runner binary, which jobs use for artifacts and caches, but it isn't registered with GitLab.
• my-runner runs GitLab Runner as a service, registered with your project or group. It asks GitLab for jobs (outgoing HTTPS only: no webhook, nothing public), and runs each one through exe-executor, a short shell script that GitLab Runner calls at each stage of a job:
  1. config names the job's VM, my-runner-job-<job id>-<random>;
  2. prepare runs ssh exe.dev cp my-runner-base my-runner-job-… --sandbox --cpu=2 --memory=4GB and waits for the copy to answer over SSH;
  3. run sends each script GitLab Runner generates (fetch the sources, restore caches, your script:, upload artifacts) to the VM over SSH and runs it there, as exedev;
  4. cleanup deletes the VM, whether the job passed, failed, or was cancelled. A timer on my-runner deletes any job VM still around after six hours.
• The runner reaches exe.dev and the job VMs with one SSH key, scoped to VMs tagged gitlab-ci. The base carries that tag, and so does every copy of it. The key stays on my-runner: the job VMs have no exe.dev key, and no GitLab token beyond the job's own CI_JOB_TOKEN.

Before you start#

• An exe.dev account that can create sandboxes. Personal accounts can. On a Work team, an admin turns them on with ssh exe.dev team settings standalone all-users.
• A GitLab project or group, on GitLab.com or your own GitLab, where you can create runners: the Maintainer role for a project runner, Owner for a group runner. The steps use a project, you/app.
• About 20 minutes.

Prepare the base VM#

ssh exe.dev new --name=my-runner-base --tag=gitlab-ci

ssh my-runner-base.exe.xyz 'curl -fsSL https://exe-staging.dev/deploy/gitlab-runner/base-setup.sh | bash'

The base already has what the default exe.dev image ships: Docker, Go, Python with uv, git, and build tools. Install whatever else your jobs need, the way you would on any Ubuntu 24.04 machine:

ssh my-runner-base.exe.xyz 'sudo apt-get install -y postgresql-client libvips-dev'

Then warm the caches by building your project there once. Jobs run as the same exedev user, so they find ~/go/pkg/mod, ~/.cache/go-build, ~/.npm, ~/.cache/pip, or ~/.cargo already full. From a checkout on your computer:

rsync -a --exclude=.git ./ my-runner-base.exe.xyz:warm/
ssh my-runner-base.exe.xyz 'cd warm && go test ./... ; cd && rm -rf warm && sync'

(npm ci && npm test, uv sync && uv run pytest, cargo test: whatever your CI runs.) Docker images work the same way: docker pull them on the base, and every job has them.

Don't put secrets on the base. Everything on it is on every job's VM.

Create the runner VM and its key#

On your computer, create the runner's VM, without the tag, and set it up with runner-setup.sh, giving it the base's name:

ssh exe.dev new --name=my-runner
ssh my-runner.exe.xyz 'curl -fsSL https://exe-staging.dev/deploy/gitlab-runner/runner-setup.sh | bash -s my-runner-base'

It installs GitLab Runner from GitLab's package repository as the gitlab-runner service, puts exe-executor in /usr/local/bin, makes an SSH key in /etc/gitlab-runner/exe/, pins exe.dev's host key, writes the settings to /etc/gitlab-runner/exe.env, and starts the timer that sweeps up leftover job VMs. It ends by printing the public key.

Add that key to your exe.dev account, scoped to the gitlab-ci tag and to the commands the executor uses:

ssh my-runner.exe.xyz sudo cat /etc/gitlab-runner/exe/id_ed25519.pub \
  | ssh exe.dev ssh-key add --tag=gitlab-ci --cmds=cp,ls,rm,ssh --exp=1y

The key can copy, list, reach, and delete VMs tagged gitlab-ci, and nothing else. Check it from the runner VM:

ssh my-runner.exe.xyz 'sudo ssh -i /etc/gitlab-runner/exe/id_ed25519 -o UserKnownHostsFile=/etc/gitlab-runner/exe/known_hosts exe.dev ls'

It lists my-runner-base, and nothing that isn't tagged gitlab-ci. To revoke the key, find it in ssh exe.dev ssh-key list and pass its name or fingerprint to ssh exe.dev ssh-key remove.

Warning

Don't tag my-runner itself gitlab-ci. The key would then reach the VM that holds it and the GitLab runner token, and so would anything else with that tag.

Register the runner with GitLab#

GitLab now creates runners in its UI and hands you a runner authentication token, which starts with glrt-; the old registration tokens are deprecated and go away in GitLab 20.0. Tags and the other settings belong to the runner in GitLab, not to the command that registers it.

In your project on GitLab, open Settings → CI/CD, expand Runners, and select Create project runner (for a group: Build → Runners → Create group runner):

• Tags: exe. Leave Run untagged jobs off, so only jobs that ask for exe come here.
• Runner description: my-runner, or anything you'll recognise.
• Optional: Protected, to run only pipelines for protected branches and tags, and Lock to current projects.

Select Create runner, then Linux. GitLab shows the token once, in a gitlab-runner register command. Copy the token, then on the runner VM (ssh my-runner.exe.xyz) register it with the custom executor. This reads the token without echoing it or saving it in your shell history:

printf 'Runner token (glrt-…): '; read -rs T; echo
sudo gitlab-runner register --non-interactive \
  --url https://gitlab.com --token "$T" \
  --executor custom --description "$(hostname)" \
  --builds-dir /home/exedev/builds --cache-dir /home/exedev/cache \
  --custom-config-exec /usr/local/bin/exe-executor --custom-config-args config \
  --custom-prepare-exec /usr/local/bin/exe-executor --custom-prepare-args prepare \
  --custom-run-exec /usr/local/bin/exe-executor --custom-run-args run \
  --custom-cleanup-exec /usr/local/bin/exe-executor --custom-cleanup-args cleanup
unset T

For self-managed GitLab, put its address in --url, such as https://gitlab.example.com. It says Runner registered successfully, and the runner shows as Online under Settings → CI/CD → Runners within a few seconds.

Last, let it run several jobs at once. concurrent is the most job VMs at any time; further jobs wait in GitLab's queue. Keep it under your plan's VM limit (50 on Personal) minus the VMs you already have:

sudo sed -i 's/^concurrent = .*/concurrent = 5/' /etc/gitlab-runner/config.toml

GitLab Runner reloads its config on its own.

Move a job#

In each job you want on exe.dev, add the tag, or set it for every job with default::

default:
  tags: [exe]

test:
  script:
    - go test ./...

Drop image: for jobs that move: the base is the image. Push, and the job's log starts:

Using Custom executor with driver exe.dev sandbox 1...
Running on my-runner-job-1234-a3f0, a copy of my-runner-base with --cpu=2 --memory=4GB (3s)

A parallel: 4 job gets four VMs at once, or concurrent of them with the rest started as the first ones finish. Moving one job at a time is fine: jobs without the tag stay on GitLab's runners, and needs: and artifacts work across the two.

A job that needs more room asks for it with a variable; it gets 2 GiB of memory per vCPU:

build:
  variables:
    EXE_CPU: "8"
  script:
    - make -j"$(nproc)"

EXE_CPU takes 2, 4, 8, and so on up to MAX_CPU in /etc/gitlab-runner/exe.env (8); anything else fails the job before a VM is made. To change every job's size, set CP_FLAGS there instead. Leave --cpu and --memory in it: a copy otherwise takes the base's memory, which may be more than the 2 GiB per vCPU a sandbox is billed for.

Caches: the base or an S3 cache#

GitLab Runner keeps cache: on the machine that ran the job unless you give it an S3 bucket, and here that machine is deleted after every job. Two kinds of cache work, and they combine.

• The base holds whatever was on its disk when a job's VM was copied: toolchains, dependency caches, build caches, Docker images. Nothing a job writes comes back to it. It costs no time per job, but it's only as fresh as your last warm-up. Re-run the warm-up when dependencies change a lot, or weekly. To update the base, change it and end with sync: new job VMs are copies of its disk, and anything not yet written to disk isn't in them.
• cache: with an S3 bucket works as on GitLab's runners: the runner hands each job short-lived signed URLs, and the job's VM downloads and uploads cache.zip directly. Any S3-compatible bucket works; set it in the [runners.cache] section of /etc/gitlab-runner/config.toml.

sudo mkdir -p /srv/s3/runner-cache
echo "RCLONE_AUTH_KEY=\"cache,$(openssl rand -hex 24)\"" | sudo tee /etc/gitlab-runner/s3.env >/dev/null
sudo chmod 600 /etc/gitlab-runner/s3.env
sudo docker run -d --name runner-cache --restart unless-stopped \
  --env-file /etc/gitlab-runner/s3.env -p 8000:8080 -v /srv/s3:/data \
  rclone/rclone serve s3 --addr :8080 /data

ssh exe.dev share set-public my-runner

  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "my-runner.exe.xyz"
      AccessKey = "cache"
      SecretKey = "the hex secret from s3.env"
      BucketName = "runner-cache"
      BucketLocation = "us-east-1"

Variables and secrets#

CI/CD variables, masked and protected ones included, and the job's CI_JOB_TOKEN reach the job's VM the way they reach any runner, and are gone with it.

exe.dev can keep some keys off job VMs altogether. An integration attached by the gitlab-ci tag is usable from every job VM, without a key on it: an HTTP proxy to a private package registry or API, or the LLM gateway for tests that call a model. A job calls https://<name>.int.exe.xyz/ and exe.dev adds the credential. Remember that any job can use it, including one from a merge request.

Merge requests from forks#

A merge request from a fork runs its pipeline in the fork, with the fork's runners: GitLab doesn't enable your project runners there. A pipeline in your project for a fork's merge request runs only when a member of your project starts it from the merge request, after reading the change, as GitLab's merge request pipelines docs describe. Throwaway VMs with no keys limit what such code can do; what's left is your bill and anything attached by the gitlab-ci tag. Tick Protected on the runner if only protected branches and tags should run here.

What it costs#

exe.dev bills the sandboxes by the second, from the copy to the delete: $0.105 an hour per 2 vCPU / 4 GiB, so $0.00175 a minute for the default size, $0.0035 for 4 vCPU, $0.007 for 8. Each job VM lives about 10 seconds longer than its scripts. my-runner and my-runner-base are ordinary VMs that share your plan's capacity with your other VMs and cost nothing extra (on Personal, plans start at $15 a month).

On GitLab.com, compute minutes are a job's duration times the runner's cost factor, not rounded: 1 for the default Linux runner (2 vCPU, 8 GB), 2 for medium (4 vCPU, 16 GB), 3 for large (8 vCPU, 32 GB, Premium and Ultimate only). A namespace gets 400 a month on Free, 10,000 on Premium, 50,000 on Ultimate, and extra minutes are $10 per 1,000, paid once and valid for 12 months. Your own runners, which this is, use no compute minutes on any plan, and need no card on file for Free accounts. Self-managed GitLab has no hosted runners: there, every runner is one you run.

Some months on GitLab.com, on 30 days:

• A small team on Free. 40 jobs a day, 4 minutes each: 4,800 minutes, 4,400 over the quota. GitLab: $50 in minute packs (the leftover 600 carry over). exe.dev, 2 vCPU: 1,200 VMs × 4⅙ minutes ≈ 83 hours ≈ $8.75, or $17.50 on 4 vCPU / 8 GiB.
• A team on Premium. 100 jobs a day, 5 minutes each: 15,000 minutes, 5,000 over the quota. GitLab: $50. exe.dev, 2 vCPU: 3,000 VMs × 5⅙ minutes ≈ 258 hours ≈ $27.10, or $54.25 on 4 vCPU. Here the included minutes win unless the warm base makes the jobs shorter.
• Big builds on Premium. 2,000 builds, 12 minutes each, on large runners: 72,000 compute minutes, 62,000 over. GitLab: $620. exe.dev, 8 vCPU / 16 GiB ($0.42 an hour): ≈ 406 hours ≈ $170.

GitLab's runners stay the better deal for jobs that fit in the included minutes, and for macOS, Windows, arm64, and GPU jobs, which exe.dev doesn't run. Mind the memory, too: GitLab's default runner has 8 GB, and a 2 vCPU sandbox 4 GiB, so a job that needs more takes EXE_CPU: "4" here.

Just one runner#

For one small project whose merge requests all come from you, one persistent runner on one VM may be all you need: no base, no executor script, no exe.dev key, and nothing to pay beyond your plan. With GitLab Runner's Docker executor, each job runs in a fresh container from its image:, with services: such as a database next to it, and cache: kept on the VM between jobs. The containers share the VM, though, and jobs see what earlier ones left in the cache and in Docker. Keep it to your own code.

ssh exe.dev new --name=my-docker-runner
ssh my-docker-runner.exe.xyz

sudo systemctl enable --now docker
curl -fsSL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install -y gitlab-runner
printf 'Runner token (glrt-…): '; read -rs T; echo
sudo gitlab-runner register --non-interactive \
  --url https://gitlab.com --token "$T" \
  --executor docker --docker-image alpine:3 --description "$(hostname)"
unset T
sudo sed -i 's/^concurrent = .*/concurrent = 2/' /etc/gitlab-runner/config.toml

test:
  tags: [exe-docker]
  image: node:22-alpine
  services:
    - name: postgres:17-alpine
      alias: db
      variables:
        POSTGRES_PASSWORD: test
  script:
    - npm ci && npm test   # Postgres is at db:5432

Update#

• GitLab Runner on my-runner. It comes from GitLab's apt repository: sudo apt-get update && sudo apt-get install -y gitlab-runner. GitLab releases a new minor version every month.
• gitlab-runner on the base. Jobs use it for artifacts and caches; keep it at the runner's version by re-running ssh my-runner-base.exe.xyz 'curl -fsSL https://exe-staging.dev/deploy/gitlab-runner/base-setup.sh | bash'.
• The executor. Re-run runner-setup.sh as in Create the runner VM. It keeps the key and /etc/gitlab-runner/exe.env; running jobs are unaffected.
• The exe.dev key expires in a year. Make a new one with sudo rm /etc/gitlab-runner/exe/id_ed25519*, re-run runner-setup.sh, remove the old key, and add the new one as above.

Troubleshooting#

Start with the job's log in GitLab, then the runner's on my-runner: journalctl -u gitlab-runner -n 50 --no-pager.

• A job is stuck, "waiting for a runner". Its tags must all be among the runner's (exe), or the runner must allow untagged jobs. A runner marked Protected takes only protected branches and tags. If concurrent job VMs are running, it waits its turn.
• cp my-runner-base my-runner-job-… failed with exe.dev's reason: sandboxes aren't enabled for your team, the VM limit is reached, or the base isn't tagged gitlab-ci. GitLab retries prepare three times, then fails the job as a system failure.
• Permission denied (publickey) in the runner's log. The key isn't on your exe.dev account, has expired, or isn't scoped to gitlab-ci; ssh exe.dev ssh-key list shows it.
• gitlab-runner: command not found or git-lfs errors while a job fetches its sources or handles artifacts. The base lacks them: re-run base-setup.sh on it.
• A job can't find a tool or a cached package. It wasn't on the base's disk when the job's VM was copied. Install it, sync, and re-run.
• Failed to extract cache every time. Without an S3 cache, that's expected: see Caches. With one, the runner's log shows the S3 error; check ServerAddress and that my-runner is public.
• Leftover job VMs. exe-sweep.timer deletes them after MAX_HOURS (6). To see them: ssh exe.dev ls --json | jq -r '.vms[].vm_name | select(startswith("my-runner-job-"))'.