Inspecting CI

Inspecting CI

After a push, CI’s verdict is a fact about your change and you should be able to read it without opening a browser and without asking anyone. This page is how.

The CI is gitea Actions with a self-hosted act_runner. Everything below is read-only: listing runs, reading logs. Re-running and cancelling are deliberately not covered — an agent inspecting a push should be looking at an untouched picture.

The tool

scripts/ci_runs.sh — POSIX sh, curl, and jq (or python3). Two verbs.

scripts/ci_runs.sh                      # last 20 runs across all branches
scripts/ci_runs.sh dev -n 5             # last 5 runs on dev
scripts/ci_runs.sh logs 420             # log tail for every failed job of run 420
scripts/ci_runs.sh logs 420 macos       # just that job, pass or fail
scripts/ci_runs.sh logs 420 macos --full  # whole log, not the last 200 lines

The listing gives you RUN BRANCH SHA RESULT WORKFLOW FAILED_JOBS. Take the number in the RUN column and hand it to logs. That is the whole workflow: list, spot the red row, read its jobs.

Log output has the per-line RFC3339 timestamp stripped, and any line over 1000 characters clipped (--raw to keep them whole). That clipping is not cosmetic: the compile telemetry blob is a single 40KB JSON line, and a handful of them will bury the actual error.

Credentials

The repo is private, so every call needs auth. In order:

1. GITEA_TOKEN — a personal access token, sent as Authorization: token …. Make one at gitea → Settings → Applications → Generate Token; read:repository scope is enough. This is the path to prefer.
2. git credential fill — whatever git already uses to push to gitea.pockle.world. On macOS that is the osxkeychain entry, and it is normally an account password, not a token. Gitea rejects a password on the Authorization: token header but accepts it via HTTP basic auth, so the script uses basic auth for this path. It works, but a scoped read-only token is the better answer.

If neither exists the script says so by name and exits 1. It never prompts.

The API underneath

If the script rots, this is enough to do the job by hand. Base is https://gitea.pockle.world/api/v1, and the server must be gitea ≥ 1.25 (it is currently 1.25.4; check with GET /api/v1/version, which needs no auth).

GET /repos/{owner}/{repo}/actions/tasks also exists and returns a flat list of tasks rather than runs; it is occasionally handy but the runs → jobs → logs path is the one to know.

Where things run

Five runner labels. Four of them are served by a single self-hosted runner named pit-server, which advertises pit-ci, pit-console-ci, ubuntu-latest, image-builder, pit-pages-ci; macos is served by a separate mac-runner.

The runner list at GET /repos/john/cell/actions/runners comes back empty even when runners are working — they are registered at instance level, not on the repo. Do not read that emptiness as “no runners”. Read the job’s runner_name field instead, or the Runner labels: line at the top of any job log.

Bucketing a failure

When you have a red job, the question is always which of these it is. The first three mean the server is not provisioned; only (d) means your change is wrong.

• (a) missing runner label — no runner advertises the label, so the job is never picked up. It sits queued; runner_name is empty and started_at is null. Note this is not the same as a job that ran and failed fast.
• (b) missing secret — CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, REGISTRY_USER, REGISTRY_TOKEN, GITEA_TOKEN. Workflows mostly guard with test -n "$SECRET", so the symptom is an immediate failure on a one-line step.
• (c) missing or stale CI image — the log says could not update image '…' …: manifest unknown, and then act_runner continues with whatever local copy the host happens to have. That fallback is the dangerous part: the job proceeds on a stale image and fails much later for a confusing reason (a missing node, a missing SDK). Always grep for manifest unknown before believing any other error in a containerized job.
• (d) real failure in the repo’s code — a test, a lint, or a build genuinely failing on the committed tree. This is the only bucket that means the push is bad.
• (e) infrastructure flake — runner offline, disk full, timeout, network.
• (f) stale workflow expectation — the workflow’s hand-written command no longer matches the repo. The class case: a workflow that invokes clang directly with its own flags, which then trips a #error guard the repo added later. The repo is right and the workflow is out of date.

Buckets (c) and (d) are the pair most often confused, and (c) masquerading as (d) is what wastes an afternoon. The discriminator: if the thing that broke is part of the image (a compiler, an SDK, node, an apt package) it is (c); if it is the repo’s own tests or a lint naming repo files, it is (d).

Traps

• Run id is not run number. The API’s id and the run_number in the web UI’s URL are different integers — run id 424 is run_number 398. Every API route takes the id. A wrong id gives you a 200 with an empty job list, not a 404, so it looks like an empty run rather than a mistake.
• limit is silently ignored without page. ?limit=20 alone returns every run the server has (hundreds). You must send ?page=1&limit=20. The same is true of status.
• Image bootstrap ordering. A pit-ci job cannot pass until build-ci-image.yml has run and pushed pit-ci:latest, and that builder only fires on pushes to its listed branches that touch its listed paths. Add a branch to the consumer without adding it to the builder and you get a branch whose jobs can never be green. Check what actually exists with GET /packages/john?type=container before believing a job failure.
• GITEA_TOKEN missing is a green deploy, not a red one. scripts/fetch_bench_trend.sh falls back to the checked-in sample when it cannot fetch the published trendline, and only logs perf-data trendline is not available yet; using checked-in sample. The Pages deploy then succeeds and publishes a site with frozen sample benchmark data. A green preview job does not mean the perf page is real — grep its log for that line.
• A fast failure is usually not your code. A pit-ci job that dies in under a minute almost never got as far as the tests. Read the failing step name before assuming a regression.