Observability

Four things to watch: the metrics at /metrics, traces over OTLP when enabled, the log on stdout and in the in-memory buffer, and the probes. This chapter names every metric the code emits and what each flag does.

What changed in 2.6.0

Two webhook metrics: sanshain_webhook_deliveries_total{outcome} and sanshain_webhook_outbox_pending.

Prometheus metrics#

GET /metrics answers the text exposition (text/plain; version=0.0.4). The route is unauthenticated; restrict it at the ingress if the instance is reachable from outside. The HTTP layer (axum-prometheus) records every request with method, endpoint and status labels:

axum_http_requests_total
axum_http_requests_duration_seconds (a histogram)
axum_http_requests_pending

The service's own metrics, as the code names them:

Metric	Type, labels	Counts
`sanshain_version_rejected_total`	counter; `reason`, `producer`	Provides refused by a version rule (`immutable_ga`, `snapshot_owned`, `breaking_without_major`, `ga_requires_releaser`, …).
`sanshain_background_job_failures_total`	counter; `job`	Failed runs of a periodic job: `snapshot_cleanup`, `dependency_cleanup`, `trunk_cleanup`, `credential_cleanup`, `webhook_cleanup`, `webhook_delivery`. Each failure also logs a WARN line; a counter that keeps rising is a job failing every run. Alert on `increase(sanshain_background_job_failures_total[2h]) > 0`.
`sanshain_catalog_pushes_total`	counter	Catalog pushes handled.
`sanshain_branch_updates_total`	counter; `branch`	Tagged provides and requires recorded against a sanshain-branch.
`sanshain_branches`	gauge	Sanshain-branches that exist, refreshed on create and delete.
`sanshain_webhook_deliveries_total`	counter; `outcome` = `delivered`, `failed`, `dead`	Webhook attempts by outcome.
`sanshain_webhook_outbox_pending`	gauge	Deliveries waiting in the outbox, refreshed after every worker pass.
`sanshain_mcp_tool_calls_total`	counter; `tool`, `outcome`	MCP tool calls.
`sanshain_mcp_result_bytes_total`	counter; `tool`	Bytes answered by MCP tools.

Counters appear once they have been incremented, so a fresh instance does not expose sanshain_webhook_deliveries_total until the first attempt. The observability page's request and failure counters (below) are separate in-process numbers, not Prometheus series.

Traces over OTLP#

OTEL_ENABLED=true exports spans over OTLP/gRPC to OTEL_EXPORTER_OTLP_ENDPOINT (http://localhost:4317), batched, with the resource attributes service.name=sanshain_service and service.version=2.6.0, the always-on sampler and W3C trace-context propagation, so a traceparent header from a client continues its trace. Spans come from the instrumented handlers and services (the HTTP layer, provide, require, webhook passes, LDAP calls). An exporter that cannot be built stops the start with ERROR: Failed to initialize OpenTelemetry tracing: …; a collector that is merely unreachable at runtime is logged by the exporter and does not affect requests. The exporter's TLS does not use EXTRA_CA_CERTS_DIR. Spans are flushed on shutdown.

Logs#

stdout is filtered by RUST_LOG (sanshain_service=info,tower_http=warn), as text or, with LOG_FORMAT=json, one JSON object per line. Independently, an in-memory buffer keeps the last LOG_BUFFER_SIZE (100) lines per level that pass CAPTURE_LOG_FILTER (sanshain_service=debug,tower_http=debug), so debug lines are available on the observability page without restarting or flooding stdout. Lines carry the service and version fields when the code logged them, which the viewer shows. Secrets, tokens and passwords are never logged; startup prints SECURITY WARNING lines for developer mode and for binding all interfaces, and the initial admin password once.

The observability page#

/observability.html (System Observability) needs a signed-in session; everything on it but the debug toggles and the audit tab is open to any signed-in user (the view_observability permission exists but no route checks it in 2.6.0).

Section	Route	Shows
System stats	`GET /admin/observability/stats`	`cpu_usage`, `memory_used`, `memory_total`, `system_uptime`, `process_uptime`, `requests_total`, `failures_total` (failed requests), counted since the process started.
Log viewer	`GET /admin/observability/logs`	The in-memory buffer by level, with copy and download. Each line shows the date and time in the reader's time zone; the Rust module that wrote it appears on hover. Copied and downloaded lines carry the exact UTC instant and the module, as stdout does.
Debug Tracing	`GET /admin/observability/debug-config`, `POST /admin/observability/debug-config-update` (`manage_settings`)	Two runtime flags: detailed logs for the service layer and OpenAPI processing (business logic), and for the presentation and auth layer (admin and user activity). Not persisted.
Audit logs	`GET /admin/observability/audit-logs`, `GET /admin/observability/audit-logs/export` (`view_audit`)	The newest audit entries, and Export CSV of the newest 1000.

The audit log itself, with filters and the diff viewer, is /audit.html.

Health, readiness and version#

Route	Answer
`GET /health`	`200` `OK`, without the database. Liveness.
`GET /ready`	`200 {"status":"ready"}` when a trivial query succeeds, `503 {"status":"unavailable"}` otherwise. Readiness.
`GET /version`	`{"version":"2.6.0","instance_id":"…"}`.

What to alert on: /ready answering 503; axum_http_requests_total with status 5xx rising; sanshain_webhook_outbox_pending climbing while sanshain_webhook_deliveries_total{outcome="failed"} climbs (a receiver down or a host removed from the allow-list); and sanshain_version_rejected_total by reason, which is the pipelines' health rather than the service's.

Checked against v2.6.0.