Logs and diagnostics

This page says where the worker writes its logs and state on your machine, and how to use pidgen-worker doctor and the worker's other commands to find out what is wrong. For fixes to particular failures, see Troubleshooting.

Where the logs are

The worker keeps everything under its home, ~/.pidgen unless your setup chose another directory (the PIDGEN_HOME environment variable names it).

Path under the home	What it holds
`logs/worker-manager.log`	The manager's log, when the worker was started in the background (the usual case after setup). New lines are appended; follow it with `tail -f`.
`logs/`	The logs of the prompt, script and workspace workers the manager starts. Set `PIDGEN_WORKER_LOG_DIR` in `worker.env` to put them somewhere else.
`spool/`	Events waiting to be sent to Pidgen. It grows while the worker cannot reach Pidgen and shrinks again once it can.
`staging/`	Files a step produced, waiting to be uploaded.
`run/manager.sock`	The socket the worker's processes use to reach the manager. It exists only while the manager runs.
`config/`	`node.yaml` and `worker-policy.yaml`, and the `.bak` copies an upgrade keeps.
`worker.env`	The worker's credential and settings. Never share it, or paste it into a log or a ticket.

When a service manager runs the worker instead, the manager's own log goes where that service manager sends it; on Linux under systemd, that is the journal:

journalctl -n 50 -u pidgen-worker

To follow the background log:

tail -f ~/.pidgen/logs/worker-manager.log

Leave the spool and staging alone

Do not delete files in spool/ or staging/ by hand. The worker sends and uploads them in order between steps, and a step whose events are missing cannot finish. A spool that keeps growing means the worker cannot keep its connection to Pidgen open; see Events pile up in the spool. To see the backlog, oldest first:

ls -ltr ~/.pidgen/spool

Log lines worth knowing

Log line	What it means
`could not ship a batch; the spool keeps it for the next pass`	The worker could not send events just now. It keeps them and tries again. Repeated for minutes, it is a connection problem.
`worker node credential revoked by the authority`	The worker was removed from your Account, or its credential withdrawn. The manager stops and does not reconnect. Set the worker up again with the Pidgen CLI.
`cannot confine the harness CLI environment`	A prompt worker could not limit a coding agent's environment, so it refuses to run agent steps. Update the worker.

Check the machine with `pidgen-worker doctor`

pidgen-worker doctor checks everything the worker depends on and names each problem with what to do about it. It starts nothing, holds nothing and changes nothing (except creating the home if it does not exist yet), so it is safe to run while the worker is serving work. Run it as the operating-system user the worker runs as:

pidgen-worker doctor
pidgen-worker doctor --json
pidgen-worker doctor --check-service

Option	What it adds
(none)	Every check of this machine.
`--json`	The same report as JSON, for a script to read.
`--check-service`	Also asks Pidgen whether it can be reached, and which worker protocol versions it accepts. This is the only check that uses the network.

Its exit status is the answer: 0 when every check passed, 1 when any failed, and 2 for an option it does not know.

Reading the report

After a line naming the worker's version, each check prints one line: ok or FAIL, the check's name, and what it found. A failed check is followed by a line starting -> that says what to do. The last line counts the failures and names them. For example (shortened):

  ok    python          python 3.12.7
  ok    tool:git        git 2.43.0
  FAIL  harness:codex   codex is on but this host has no codex on PATH, so it is not advertised
                        -> install codex on this host, or switch codex off with `pidgen-worker capacity set codex --max 0`

1 defect(s): harness:codex

--json prints {"ok": ..., "checks": [...]}, where each check has name, ok, summary, defect (a short code for the kind of failure, or null) and remedy.

The checks

They run in this order, and none stops the others: one run reports every problem.

Check	What it verifies	`defect` when it fails
`python`	The worker's Python is 3.12 or newer.	`below_floor`
`tool:git`, `tool:node`, `tool:npm`	Git 2.25 or newer, Node.js 20 or newer and npm 10 or newer are installed and on `PATH`.	the kind of problem, for example a missing command or a version below the floor
`home`	The worker's home exists and the worker can write in it.	`not_writable`
`worker_env`	`worker.env` is in the home.	`missing`
`worker_policy`	`config/worker-policy.yaml` exists and is valid.	`missing`, `invalid`
`harness:<harness>`	Each coding agent that `node.yaml` switches on has its command on `PATH`. An agent switched off passes without one.	`cli_unresolvable`
`subscription:<harness>/<label>`	Each subscription `node.yaml` lists is logged in, and to a different provider account from the ones before it.	`not_logged_in`, `same_account`
`node_capacity`	`node.yaml` is valid. An older `version: 1` file passes: the worker upgrades it when it starts.	the code of the first problem, with the key at fault in the summary
`lane:process`, `lane:workspace`	The script workers and the workspace worker can start on this machine; the workspace worker needs `git`.	the reason, with the missing command
`service`	Only with `--check-service`: the worker's Pidgen address answers.	`unconfigured`, `unreachable`, `unhealthy`
`worker_protocol`	Only with `--check-service`: Pidgen accepts a worker protocol version this worker speaks.	`unconfigured`, `refused`, `probe_failed`

A coding agent with no command, or a kind of worker that cannot start, is not an error while the worker runs: the worker simply does not offer that work to Pidgen. doctor is how you find out. Install what is missing and the worker offers the work again within a minute, with no restart.

Some things doctor cannot judge for you: that the SSH agent holds the key you meant (ssh-add -l), that SSH will not stop to ask about a host key, and that the coding agents switched on are the ones you want this machine to run.

Other commands that show the worker's state

Command	What it shows
`pidgen-worker capacity show`	Whether the running worker has applied the `node.yaml` you saved, and if it refused a change, why. See Configure the worker.
`pidgen-worker subscriptions show`	Each coding agent's subscriptions and whether each is logged in. See Coding agents and subscriptions.
`pidgen-worker --version`	The worker's version.

Whether the worker is running and connected is shown by the Pidgen CLI; see Workers.