Operator

Troubleshooting

The messages below are quoted as the service writes them, so a search for the text you see lands here. Each one has its cause and its fix. Messages that a developer meets in a pipeline (409 with a proposed version, 404 unknown, 410 absent) are explained at the end and in depth in CI integration.

What changed in 2.6.0

The Webhooks section is new, as is the feature.
The LDAP stall message (accepted the connection but did not respond within 10 s) replaces the ingress's 502 Bad Gateway; see LDAP.

Startup#

ERROR: Address 0.0.0.0:3000 already in use. Another instance might be running.: The port is taken. Stop the other process, or set BIND_ADDRESS (BIND_ADDRESS=0.0.0.0:3080). There is no PORT variable.
CONFIGURATION ERROR: LOG_BUFFER_SIZE="1OO" is not a valid number. Correct it or unset it to use the default (100).: A numeric variable holds something that is not a number. The service refuses to run with a setting it did not apply; fix or unset it. The same shape names any numeric variable.
CONFIGURATION ERROR: LOG_FORMAT="yaml" is not recognised. Valid values: json, text. Unset it to use the default (text).: A fixed-vocabulary variable (LOG_FORMAT, OTEL_ENABLED, MCP_ENABLED, WEBHOOKS_ENABLED) holds another value. Values are case-insensitive and trimmed.
CONFIGURATION ERROR: MAX_SPEC_BODY_BYTES=0 would reject every request., … MCP_CALLS_PER_MINUTE=0 would refuse every MCP call., … MCP_MAX_RESULT_BYTES=0 would summarise every MCP answer., … WEBHOOK_WORKER_INTERVAL_SECS and WEBHOOK_DELIVERY_TIMEOUT_SECS must be greater than 0.: A zero that would make a feature useless. Unset the variable or give it a positive value.
CONFIGURATION ERROR: PUBLIC_BASE_URL="sanshain.corp" must start with http:// or https://.: Give the full origin: https://sanshain.corp.
ERROR: Invalid bind address '…': …: BIND_ADDRESS must be host:port with a numeric port; IPv6 in brackets.
ERROR: Can't build the TLS trust store: CA certificate directory '/certs' does not exist: EXTRA_CA_CERTS_DIR points at a path that is not mounted. Mount it or unset the variable. Variants: Could not read CA certificate directory '…': … (permissions) and Could not read CA certificate file '…': … (a file with the right extension that is not a PEM bundle, or holds no certificate). See Additional CA certificates.
ERROR: Failed to initialize OpenTelemetry tracing: …: OTEL_ENABLED=true and the exporter could not be built, usually a malformed OTEL_EXPORTER_OTLP_ENDPOINT. The collector being down is not this error.
SANSHAIN_DEV_MODE is ignored: dev mode is the Dev auth mode, picked on the admin page or with INITIAL_AUTH_MODE=dev: The server still sets SANSHAIN_DEV_MODE, which no longer does anything. Remove it; see Developer mode.
SECURITY WARNING: binding to all interfaces (0.0.0.0:3000). Set BIND_ADDRESS=127.0.0.1:3000 to restrict access to localhost.: Informational. In a container this is what you want.

Database#

ERROR: Can't connect to SQLite database: …: The directory of the file in DATABASE_URL does not exist or is not writable by the service's user (UID 1000 in the manifests). mode=rwc creates the file, not the directory.
ERROR: Can't connect to PostgreSQL database: …: Host, port, credentials or network. Check the URL and that the pod can reach the server; /ready answers 503 for the same reason at runtime.
ERROR: Invalid database URL: …: A SQLite URL the driver cannot parse. The shape is sqlite:/data/sanshain.db?mode=rwc.
ERROR: Can't run SQLite migrations: … / … PostgreSQL migrations: …: A migration failed, or the database was touched by a newer version (migrations are forward-only). Restore the backup from before the upgrade, see Backup and restore.
ERROR: Can't normalize stored spec text: …: The one-time rewrite of stored spec text (2.4.0, ADR-0008) failed (a database error mid-way). The next start repeats it; every step is idempotent.
ERROR: Can't claim root usernames: …: Creating the placeholder accounts for unclaimed SANSHAIN_ROOT_USERS names failed; a database error.

Logins, sessions and tokens#

Lost the initial root password: There is no password reset, and SANSHAIN_ROOT_USERS does not help: a name added there is claimed at startup as a placeholder account with a random, unknown password (only a directory login under that name can use it), and registering it is refused. The initial admin is created only while the users table is empty, so the options are: restore a backup taken while the password was known; or delete every row of users (DELETE FROM users; with sqlite3 on the stopped service's file, or psql), which removes every account, session and API token, so the next start creates the initial admin again with INITIAL_ADMIN_PASSWORD or a new random password on stderr. The Danger Zone's Nuke Users is not a way out: it spares every effective administrator, root included, and needs a working admin session anyway.
POST /auth/login answers 401: Wrong username or password, or an empty password, which is refused before any check; with LDAP, the directory refused and the local check refused too.
POST /auth/login answers 403: The account is pending approval (approve it under Users), or the mode is Off or Dev, where only administrators sign in: the answer then names the mode (Sanshain is in maintenance; only administrators can sign in).
503 Service is in Maintenance Mode / Disabled: Every API call while the authentication mode is Off (maintenance), which is also where a fresh instance starts unless INITIAL_AUTH_MODE is set. An administrator signs in and selects another mode in the Authentication section or with PUT /admin/auth-config.
409 This username is reserved: Registration of a name in SANSHAIN_ROOT_USERS. See Root users.
409 User already exists: Registration of a taken name, also one that only exists as an LDAP or OIDC shadow account.
An API call answers a bare 403 or 401: 403 without a body: no Authorization: Bearer header at all. 401: the token is invalid, expired or revoked. Mint one on /account.html.
403 this API token has scope 'read_only' and may not change the catalog; create a token with scope 'full' for provide, require and catalog pushes: A read-only token on a write route. The scope is fixed at creation; mint a full one.
403 this API token has scope 'read_only' and may not call administrative routes; use a token with scope 'full' of an account holding the permission: A read-only token on the backup download. Every other admin route answers 401 to any API token: they take sessions.
403 publishing GA for 'orders' requires the 'releaser' role — publish as a snapshot, or ask an administrator to grant the role: Grant releaser to the CI user (Roles and permissions).
A user_manager granting admin or releaser gets 403: Both roles are admin-guarded; an admin or root grants them.
400 Role 'maintainer' is scoped to specific Producers and cannot be granted instance-wide.: Assign the user or group as a maintainer of a Producer instead.

LDAP#

400 LDAP connection failed: Connection failed: LDAP server at ldaps://ldap.example.com:389 accepted the connection but did not respond within 10 s. Check that ldaps:// uses the TLS port and StartTLS the plain one.: Shown by Test Connection; a login that stalls the same way answers a bare 401 after the local password check. The directory took the TCP connection and never answered: almost always a scheme and port mismatch (ldaps:// on 389, StartTLS on 636), otherwise a firewall that accepts and drops. The table in Authentication lists the combinations. Test the port from inside the pod, not from a laptop.
Test Connection shows 502 Bad Gateway: Sanshain never answers 502; the ingress gave up before Sanshain's 10 s did. Raise the ingress timeout above LDAP_OPERATION_TIMEOUT_SECS, or lower the variable.
400 LDAP connection failed: Could not connect to LDAP server at ldap://…: … followed by DNS:, Resolvers: and one line per address: The directory was never reached. The message names the step: the hostname could not be resolved, this container has no route to …, the network of … is unreachable from this container, nothing listens on … (host up, port closed) or no answer from … within 5 s (host down or a firewall that drops). The lines under it are the container's own diagnosis; read them as described in When the directory is unreachable. No route to host (os error 113) in the brackets is the kernel saying the same as the first line: the pod has no route to that address today, which is a network change, not a directory one.
400 LDAP connection failed: Connection failed: Could not connect to LDAP server at ldaps://…: the server certificate is not trusted: …: A private CA without EXTRA_CA_CERTS_DIR. The sibling messages name a certificate that is not valid for the URL's host, one that has expired or is not valid yet (check the clocks), a plain LDAP port answering an ldaps:// URL, a refused handshake, or a connection closed during StartTLS (a TLS port; use ldaps://). The rustls text follows in brackets.
400 LDAP connection failed: Connection failed: LDAP bind failed (check DN/Password): …: The service account's DN or password. LDAP bind request failed for cn=…: … is the same step failing at the protocol level.
A directory user logs in but has no admin rights: Roles come from the group mapping on every check: the user's memberOf values must equal the name of a directory group that has a role attached (the Bootstrap Admin Group is such a group). A change takes up to SANSHAIN_DIRECTORY_GROUP_TTL_SECS (300) to show. The log line Could not read directory groups for 'alice'; continuing without directory-derived roles: … means the directory was unreachable at the check; root and local grants still work.

OpenID Connect#

400 OIDC is not configured: The mode is oidc or the SSO button was used without an oidc_config. Store one with PUT /admin/auth-config.
500 OIDC discovery failed: …: <issuer_url>/.well-known/openid-configuration could not be fetched or parsed from inside the pod, or its certificate is not trusted: a private CA goes through EXTRA_CA_CERTS_DIR. Url must use TLS: '…' means the provider serves its key set (jwks_uri) over plain http, which is refused.
400 OIDC issuer URL: …, 400 OIDC redirect URL: …: One of the two is not a valid URL.
400 no OIDC login in progress, 400 missing code, 400 missing state: The callback arrived without the flow cookie (older than 10 minutes, another browser, or a cookie blocked because the site is not served over HTTPS; the cookie is Secure) or without the provider's parameters. Start again from the SSO button.
The callback answers 401: The state did not match, the code exchange failed, or the ID token did not verify (nonce, issuer, audience, signature, expiry). Check the client secret and that the redirect URL registered with the provider is exactly the configured one.
400 OIDC config required for oidc mode, 400 LDAP config required for ldap mode, 400 Invalid auth mode: …: PUT /admin/auth-config switching to a mode without its configuration, or with a mode name outside disabled, dev, local, ldap, oidc.

Webhooks#

403 no hosts are allowed for webhooks yet — an administrator has to add the receiver's host to the allow-list (admin page, Webhooks) first: A maintainer registering a hook on an instance whose allow-list is empty. An administrator adds the host, see The allow-list.
403 the host 'hooks.example.com' is not in the webhook allow-list — ask an administrator to allow it, or use one of: hooks.slack.com: The URL's host matches no pattern. For plain http:// the message adds (plain http:// needs a pattern that says http:// itself).
400 '…' is not a host pattern: use a host name with optional `*`/`?` wildcards, prefixed `http://` only to allow plain HTTP to it: A line of the allow-list with a path, a port, a scheme other than http:// or another character.
400 the hook URL must be an http:// or https:// URL with a host, 400 the hook URL is not a valid URL: …: The receiver URL itself.
400 webhook.test is sent by the test button only and cannot be subscribed to: webhook.test in event_types. Leave the set empty for every event.
409 webhooks are disabled on this instance (WEBHOOKS_ENABLED=false): a test event would never be delivered: Send test with the worker off. Set WEBHOOKS_ENABLED=true (the default) and restart.
Delivery log: the hook's host is not in the allow-list (webhook_allowed_hosts) any more: The host was removed after registration. Re-add it; the retries continue until the 24-hour window closes.
Delivery log: the receiver answered HTTP 404, or a connection error: The receiver's answer or the network; the body of a non-2xx answer is not read. Fix the receiver and redeliver, or wait for the next retry.
Delivery log: given up: 24 hours have passed since the event: Dead. A hook paused for a day finds its deliveries dead on resume. Redeliver sends one more attempt with the same id.
409 delivery 12 is in flight — a worker is attempting it right now; try again after that attempt: Redeliver while a worker holds the row. Wait for the attempt.
Nothing arrives and the delivery log stays pending: No worker pass is running (WEBHOOKS_ENABLED=false writes nothing, so pending rows with no attempts mean the worker cannot reach the database or the process is wedged), or the hook is paused. The log line Webhook worker enabled: every 5 s, 10 s delivery timeout, deep links on https://… confirms the worker started.
The Slack message shows no link, the Teams card no button: PUBLIC_BASE_URL is unset and there is no OIDC redirect URL to borrow the origin from.

Backup#

409 This instance runs PostgreSQL, which has no snapshot download; take backups with pg_dump (see the deployment guide, Backup and restore): Use pg_dump; the Helm chart switches the CronJob for you. See PostgreSQL.
409 A database snapshot is already being produced; retry when it has finished: Two downloads at once. Retry.
The backup CronJob fails with 401 or 403: The token in backup.tokenSecret expired or was revoked (401), is read_only or belongs to an account without run_destructive_operations (403). Mint a full token of an admin and replace the Secret.
500 VACUUM INTO failed: … or snapshot file: …: The data volume has no room for a second copy, or the directory of the database is not writable.

Settings and administration#

400 days must be at most 36500: Every *_max_age_days setting caps at a hundred years.
400 days must be greater than 0: dependency_max_age_days cannot be disabled; the snapshot and trunk ages accept 0.
400 Invalid confirmation: A Danger Zone route without its exact phrase, see The Danger Zone.
409 'orders' is part of the recorded graph of sanshain-branch(es) release-2026-10 — deleting it would remove those edges from release cuts that already happened. Delete the branch(es) first if the record is no longer needed.: Deleting a Producer or Consumer a branch still references. Delete the named branches first, if their record is no longer needed.
400 Only Sanshain's own groups can be renamed; a directory group's name is the directory's., 400 Group '…' comes from the directory, so its membership is the directory's to change.: A directory group is edited in the directory; only its roles are Sanshain's.
400 Supply exactly one of user_id or group_id.: A maintainer assignment with both or neither.

Provides and requires#

409 with proposed_version: Refused by the version rules: the number is GA with different content, the change is breaking without a major bump, a snapshot for a number that went GA. Set the spec's version to the proposal and republish. If nothing changed, the difference is whitespace or a comment: content is compared byte-for-byte after line endings and a leading BOM are normalised. See API lifecycle.
404 Producer 'orders' has no openapi version 2.1.0 (in either stability) — a missing pinned version is a configuration error, nothing waits for it to appear: Unknown: the pin names a version that does not exist, expired unused (snapshot) or was deleted. GET /producers/{name}/versions lists what exists.
410 openapi version 2.1.0 of 'orders' does not include GET /orders: Absent: the version exists and lacks the endpoint. Check path and method, or pin a version that has it.
400 on a provide: Invalid YAML, a version that is not MAJOR[.MINOR[.PATCH]] (optionally v-prefixed, no suffix such as -SNAPSHOT), OpenAPI outside 3.0 and 3.1, AsyncAPI outside 2.x and 3.x, a proto file without exactly one // sanshain-version: comment.
413 on a provide: The body exceeds MAX_SPEC_BODY_BYTES (4 MiB). Raise it.

Web pages#

The dependency graph is empty: Dependencies are recorded only by a successful /require or /require-bundle without dry_run. scripts/demo.sh populates an instance.
Download PNG of the graph does nothing: The export renders in the browser; a reverse proxy that adds a Content-Security-Policy must allow blob: URLs. Sanshain's own headers do.
The audit page shows Error loading timeline: Session expired: The session is older than 7 days or was revoked. Sign in again.

Checked against v2.6.1.