Monitoring

Watch health, metrics, integrity, and the limits that grow with use

By the end you will know which probe to point at the hub, which numbers to scrape, and where the store's own limits are. The backup that pairs with integrity checking is in Back up and restore.

Before you start

Liveness and readiness

Two probes answer different questions:

• GET /healthz is liveness only: the process is up.
• GET /readyz is readiness, and it answers 200 only when the store is usable. The schema version is read, is supported, and matches what the process opened; the data directory and hub.db still have the identity they had at open, so a removed or replaced store is caught; a content read confirms it is still a hub store; and the volume has free space above a safety margin.

Any leg failing is a 503 problem, so a supervisor can take the node out of rotation instead of holding it there while every call fails.

agent-hub health [--url URL] asks /readyz and exits 0 when ready, non-zero otherwise. It is what the container healthcheck runs, because the runtime image is distroless and has no shell or curl.

Metrics

GET /metrics reports the hub's own counters in Prometheus text:

• HTTP requests, by method and status class;
• failed responses, by hub error code;
• events appended, by kind;
• MCP tool calls, by tool;
• sends to the notify target, by result (agenthub_notify_sends_total, delivered or failed).

It also carries storage gauges, read fresh for each scrape: free bytes on the data volume, the hub.db size, and the write-ahead log size. A time-series system can alert before the readiness margin trips rather than only when the node is nearly out of room. The route is admin-gated like the rest of the control surface, so the scraper carries the admin token in Authorization: Bearer. There is no separate scrape credential.

Set integrity_sample_secs (HUB_INTEGRITY_SAMPLE_SECS, default 0, disabled) to a number of seconds to have the hub sample the store's own integrity in the background, on the sweeper's cadence. Each sample runs PRAGMA quick_check on its own connection, never inside a request, bounded by a wall-clock cap and skipped while the previous one has not finished, so a large or damaged store cannot pin the loop. It raises two series: agenthub_integrity_ok, a gauge that is 1 for a passing sample and 0 for a failing one (absent until the first sample runs), and agenthub_integrity_failures_total, a counter of samples that timed out, errored, or reported a problem. A failed sample is what tells you a zeroed page has appeared between offline check runs.

Offline integrity

Two read-only commands walk the store with the full PRAGMA integrity_check:

agent-hub check --data-dir /path/to/data
agent-hub doctor --data-dir /path/to/data

check verifies any backup manifest present, runs the integrity check on the store and every brain and knowledge file, and cross-checks artifact rows against the blobs on disk. doctor prints one report: the data directory and its identity, the schema version and the most this binary supports, free space, the write-ahead log size, the id high-water mark, and any artifact blob the store names but the tree is missing. Both refuse while a hub holds the store, and exit non-zero on any problem.

Practical limits

The node is sized for one operator and their agents, not a fleet of tenants, so a handful of things grow with use rather than being capped. Knowing where they are is how you decide when to prune.

• Sessions. No count cap. A session lives until the human prunes it. One session brain file warns past 256 MiB and is refused at 1 GiB, and a listing returns at most 200 sessions.
• Events. A project's feed holds events_per_project (HUB_EVENTS_PER_PROJECT, default one million). A write past it is refused and names the cap; promote durable work to the knowledge base or prune the feed to make room.
• Knowledge base. One page is capped at 1 MiB and one path at 512 bytes. A project's knowledge base file warns past 256 MiB and is refused at 1 GiB, so its page count follows that file's size rather than a count of its own. History and last-write scan the write log newest first, which is the brain engine's own call table; the hub wraps it and does not index it, so those reads cost more as the log grows. A history request carries at most 200 rows and reports the real total.
• Artifacts. One blob is capped at 50 MiB. A blob moves on the Tokio blocking pool, so a large transfer does not hold an async worker and the hub keeps answering other requests while it lands. Many large transfers at once are felt in memory (roughly 50 MiB each in flight) and disk bandwidth rather than in the workers; past the blocking pool's ceiling they queue.
• Search. A filtered search reads at most 5000 rows before it stops, and returns at most 100 hits a page. A scope that matches more than the fetch cap is reported as truncated rather than scanned whole.

The feed ceiling

A project's feed is bounded by events_per_project. A write past it is refused with the cap named, so a runaway agent cannot fill the node; pruning a session removes its lifecycle records and makes room. The ceiling bounds every agent-surface writer: signals, questions, answers, artifact publish and update, and the knowledge base's lifecycle signal. Session lifecycle records and the hub's audit trail are exempt, so a full feed can never refuse session_start; a knowledge base write itself still succeeds, because its feed signal is best-effort and is dropped when the feed is full.

Shutdown

On SIGTERM or SIGINT the hub stops accepting, lets in-flight requests finish for a bounded window, then checkpoints the store and exits 0. docker stop and a service restart therefore drain rather than cut. This is why the service stop timeout in Deploy as a service sits above the ten-second drain.

You should now see

curl -sS -H "Authorization: Bearer $HUB_ADMIN_TOKEN" \
  http://127.0.0.1:8080/metrics | head

Prometheus text with the counters and the three storage gauges.

See also

• Back up and restore - the offline check
• Notifications - what a notify failure is counted as
• Configuration - events_per_project, integrity_sample_secs