Reference — hort-server and hort-worker configuration

Canonical, exhaustive reference for the command-line surface and environment variables of the two v2 runtime binaries:

• hort-server — the HTTP service + operational subcommands (crates/hort-server/)
• hort-worker — the multi-kind job dispatcher (crates/hort-worker/)

Scope: the v2 rewrite in crates/ only. The backend/ prototype and the prototype-migration how-tos are out of scope here.

Authority. The ground truth for every value below is the parsing code: crates/hort-server/src/config.rs (+ composition.rs, telemetry.rs) for hort-server and crates/hort-worker/src/config.rs (+ composition.rs, extra_ca.rs, telemetry.rs) for hort-worker. Where this page and that code disagree, the code wins and this page is the bug — fix it in the same change. The Helm-values mapping lives in how-to/deploy/values-reference.md; this page documents the binary surface the chart renders into.

Conventions used in the tables:

• Default is the literal value the binary uses when the variable is unset, quoted exactly as coded. "unset → …" describes the unset-path behaviour when there is no static default.
• Required? — "Yes" means the binary refuses to start without it; "Cond." means required only under a condition stated in § Validation & interlocks.
• Durations are seconds unless the name says otherwise.
• A 0/non-integer value is rejected at startup for every positive-integer knob (ConfigError::ValueNotPositive for 0, ConfigError::InvalidInt for a non-integer) except the two explicitly noted kill-switch knobs.

hort-server — command-line surface

hort-server is a multi-subcommand binary (clap). The subcommand is optional: a bare hort-server invocation is exactly equivalent to hort-server serve. Both ExecStart=/usr/local/bin/hort-server (systemd) and args: ["serve"] (k8s) are therefore correct and identical.

--help / --version work at the top level and on every subcommand. Unknown subcommand → clap InvalidSubcommand (non-zero exit). On error each subcommand prints Error: {err:?} (full anyhow context chain) and exits non-zero; a Tokio-runtime build failure prints error: building tokio runtime: {err}.

Why migrate/admin/reconcile-groups use MinimalConfig (ADR 0009): the runtime DSN is least-privilege (DML only) and these DB-only subcommands must not require the storage / public-base-url / OIDC surface. MinimalConfig parses exactly the five shared variables in § Core (shared). serve and scrub parse the full surface and enforce every interlock. Migrations are owned only by hort-server migrate — the serve path calls migrate::assert_current (a read-only check on _sqlx_migrations) and refuses to run migrate::run.

scrub flags

Exit code: 0 when there are no hash mismatches (even with missing / read-error blobs); 1 when any mismatch is found (so a CronJob can escalate); non-zero failure on config/connection error. The mismatch-handling behaviour itself is selected by HORT_CAS_SCRUB_ACTION_ON_MISMATCH.

admin subcommands

admin issue-svc-token — mint a service-account token.

The prior admin bootstrap subcommand is removed (it seeded the now-deleted HTTP-Basic-against- local-admin-row identity path — see docs/auth-catalog.md Entry 8). Minimal-setup bring-up uses issue-svc-token plus hort-cli auth login --paste; see crates/hort-server/README.md.

reconcile-groups flags

validate-config

Offline gitops-config validation: runs the static subset of the boot-time apply pass — parse + cross-validate plus the snapshot-free linter rules, via the same StaticConfigValidator (crates/hort-app/src/lint/static_validate.rs) the apply path delegates to — over the tree in HORT_CONFIG_DIR, with no database and no running server. Implementation: crates/hort-server/src/cli/validate_config.rs. The CI-recipe walkthrough (deriving the env from Helm values) is in declare-gitops-config.md §5a.

Synchronous and DSN-free: it builds no Tokio runtime, parses neither Config nor MinimalConfig, and never reads HORT_DATABASE_URL / DATABASE_URL or connects to anything.

Environment inputs (read directly, not via Config):

Exit codes:

Checks run vs. not run:

A directory that exists but contains zero YAML files validates clean — exit 0 — with the warning validated 0 config files — is HORT_CONFIG_DIR correct?; under --strict it is exit 1 (catches a typo'd-but-existing path in CI). The validator and its static facts (the provenance-capable format set, the linter defaults) are compiled into the binary, so the invoked image tag is exactly the version being validated against; there is no --version selector.

hort-server — environment variables

Core (shared — MinimalConfig)

Parsed by every subcommand except validate-config, which is DSN-free and reads its own env directly (see § validate-config).

¹ Exactly one of HORT_DATABASE_URL / DATABASE_URL is required; HORT_DATABASE_URL is the canonical operator var and is tried first, with bare DATABASE_URL as the documented compat fallback. With neither set the error surfaces the name "DATABASE_URL". This is identical to the worker.

Logging & tracing

HTTP server & binding (serve)

See tune HTTP transport timeouts for the timeout knobs in depth.

Trust, TLS & public URL

One of HORT_PUBLIC_BASE_URL or a non-empty HORT_TRUSTED_PROXY_CIDRS is mandatory — with neither set the binary fails at startup (ConfigError::TrustUnconfigured).

Authentication provider (OIDC)

Native tokens, OCI token signing & token exchange

Rate limiting, lockout & load shedding

The HORT_AUTH_LOCKOUT_* env vars are removed — they powered the now-deleted authenticate_local HTTP-Basic-against-local-admin-row path. The HORT_PAT_LOCKOUT_* env vars (a distinct mechanism — bearer-path per-IP / per-token-prefix brute-force protection inside PatValidationUseCase) are unchanged.

Storage

Ephemeral store (Redis)

Pull-through deduplication

Pull-through coalescing degradation alarm (recommended)

Pull-through stampede suppression is TTL-best-effort, not durable: the pulldedup: keyspace is registered in the Evictable class, so under the sustained pull-burst the negative cache exists to absorb, allkeys-lru eviction (or a Redis outage) can degrade cluster-wide coalescing toward per-replica or fully un-coalesced upstream retries. Correctness is never affected (the CAS + path-conflict short-circuit absorbs the fan-out idempotently); only backpressure efficacy degrades. This is an audited, accepted residual. There is no new metric — alarm on the existing signals only:

Operator action when the alarm fires: isolate the pulldedup: / evictable keyspace onto its own Redis instance via HORT_REDIS_URL_EVICTABLE (per-class routing — see the Ephemeral store table above) and/or raise that instance's maxmemory so the short-lived Failed records survive their TTL; alternatively, accept the degraded-to-per-replica backpressure posture (correctness holds either way — the trade-off is only upstream-amplification risk during a burst). Routing the negative-cache Failed records to the Durable class is the stronger structural fix; it is deferred pending a KEYSPACE_REGISTRY class-assignment review (load-bearing, user-coordination-gated).

Scan-queue backlog alarm (recommended)

The scan queue is fed by four trigger sources (ingest, cron, advisory, manual). The cron and manual paths can amplify enqueue volume faster than the worker drains it: a cron-rescan tick batches up to 1000 stale artifacts per invocation, and an unthrottled manual-rescan caller can enqueue repeatedly. This is an audited, accepted residual (see the open-items register in docs/adr/0000-historical-decisions-index.md). The security half is already neutralised — the fail-closed release predicate (ADR 0007) makes release contingent on a successful scan, so a deep scan queue can only delay availability of newly-ingested or just-rescanned artifacts, never weaken the quarantine gate. The residual is therefore an availability concern only; alarm on the existing signal — there is no new metric:

Metric-name note. Older audit prose referred to hort_jobs_pending_count{kind="scan"}; that metric name does not exist. Its shipped equivalent is hort_scan_queue_depth (see docs/metrics-catalog.md — a no-label gauge emitted by the worker heartbeat tick reading count(*) FROM jobs WHERE kind='scan' AND status='pending', which is exactly the "pending scan jobs" quantity the audit named). Treat hort_scan_queue_depth as the canonical name when wiring the alarm.

Operator action when the alarm fires: scale out worker replicas (the claim loop is FOR UPDATE SKIP LOCKED, so additional replicas drain cooperatively without coordination), and/or raise priority on the latency-sensitive trigger sources via admin so ingest first-scans stay ahead of routine cron rescans. Lengthening the cron-rescan schedule (scheduledTasks.cronRescanTick.schedule) reduces the batch-injection rate. The structural fix — a per-(principal, repository) manual-rescan enqueue rate cap with an HTTP 429 response — is deferred (it requires a KEYSPACE_REGISTRY durable prefix and an hort-http-core error-contract 429 change, both cross-cutting); the open item is recorded in the open-items register (docs/adr/0000-historical-decisions-index.md). Because the security half is already closed by the release predicate, this deferral carries no security risk — it is purely an availability-hardening follow-on.

Anomalous advisory-diff-volume alarm (recommended)

The advisory-watch tick (AdvisoryWatchTickHandler) ingests the OSV bulk feed over TLS only; OSV publishes no signed manifest or per-zip hash for the bulk all.zip archives, so the feed is trusted-but-unauthenticated. This is an audited, accepted residual (see the open-items register in docs/adr/0000-historical-decisions-index.md) with this alarm as one of two compensating controls (the other is the reject-requires-scanner invariant — a poisoned advisory only ever enqueues a rescan that a real scanner adjudicates, so a poisoned advisory alone cannot reject a clean artifact). This alarm is the feed-poisoning / feed-suppression tripwire; it also serves the NIS2 21(2)(f) vulnerability-handling-efficacy signal. There is no new metric — alarm on the existing signals only:

Metric-name note. The shipped metrics are hort_advisory_diff_processed_total{ecosystem,result}, hort_advisory_diff_duration_seconds{ecosystem}, and hort_advisory_ingest_count{category} (the under-floor efficacy metric, not a per-tick diff counter — all documented in docs/metrics-catalog.md). No metric is added; the suppression tripwire deliberately reuses the existing hort_advisory_ingest_count under-floor alert rather than introducing a parallel signal.

Operator action when the alarm fires: treat it as a possible feed-poisoning or feed-suppression event, not a routine queue blip. Do not trust the next rescan wave at face value: cross-check OSV's published advisory volume out-of-band (the OSV web UI / a second mirror), inspect the affected ecosystem's recent hort_advisory_diff_* samples, and confirm whether a real upstream advisory drop explains the volume before letting the enqueued rescans proceed to operator attention. On a confirmed injection, the reject-requires-scanner invariant already bounds the blast radius to scan-queue amplification (no false rejections) — the queue-amplification mitigations in the Scan-queue backlog alarm above apply. On a confirmed suppression, escalate as a vulnerability-handling-efficacy incident (NIS2 21(2)(f)): the advisory DB is stale and the cron-rescan safety net re-runs the same poisoned/suppressed data, so it is not an independent recovery path. The stronger structural control — a second, authenticated feed (GitHub Advisory) — is deferred (see the open-items register in docs/adr/0000-historical-decisions-index.md); it is referenced, not scheduled here.

Notifications & webhooks

Metrics

CAS scrubber

Operational / gitops / misc

hort-server — validation & interlocks

These cross-field rules are enforced at startup by serve / scrub (full Config). Each is fail-fast — the binary refuses to start rather than run misconfigured.

1. Trust unconfigured (unconditional). HORT_PUBLIC_BASE_URL unset and HORT_TRUSTED_PROXY_CIDRS empty → boot fails (TrustUnconfigured). One of the two is mandatory.
2. HORT_REQUIRE_HTTPS=true. Fails if HORT_PUBLIC_BASE_URL is http and HORT_TRUSTED_PROXY_CIDRS is empty (no positive TLS evidence). An https:// base URL or a non-empty CIDR list satisfies it.
3. Metrics public-bind guard. HORT_METRICS_BIND on an unspecified IP while HORT_METRICS_PUBLIC_BIND=false → boot fails (MetricsPublicBindRefused).
4. Storage required fields. filesystem requires HORT_STORAGE_FILESYSTEM_PATH; s3 requires HORT_STORAGE_S3_BUCKET, a region (AWS_REGION/AWS_DEFAULT_REGION), AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY.
5. S3 allow-http ↔ endpoint scheme. http:// endpoint without HORT_STORAGE_S3_ALLOW_HTTP=true → error; the flag set with an https:// endpoint, or set with no endpoint (real AWS S3) → error.
6. S3 SSE-KMS requires the ARN. HORT_S3_SSE_MODE=sse-kms without HORT_S3_SSE_KMS_KEY_ARN → error (never silently downgrades).
7. OCI signing key source exclusivity. Setting both HORT_OCI_TOKEN_SIGNING_KEY_FILE and HORT_OCI_TOKEN_SIGNING_KEY (non-empty) → error; _FILE wins. Same rule for the _PREV pair.
8. Native tokens need a key. HORT_NATIVE_TOKENS_ENABLED=true with no resolved active signing key → error.
9. Token exchange dependency set. HORT_TOKEN_EXCHANGE_ENABLED=true requires HORT_NATIVE_TOKENS_ENABLED=true always (the exchange mints hort_* native tokens the server must be able to validate). Under HORT_AUTH_PROVIDER=oidc it additionally requires HORT_OIDC_ISSUER_URL, HORT_OIDC_CLI_CLIENT_ID, and HORT_PUBLIC_BASE_URL (they back the /.well-known/hort-client-config discovery doc + the interactive device flow). Under HORT_AUTH_PROVIDER=disabled none of those three are required — the federated-JWT branch validates against the gitops OidcIssuer rows, not the interactive IdP config (federation-only / no-Keycloak mode). The error names every missing variable.
10. Redis URL under the Redis backend. HORT_EPHEMERAL_STORE_BACKEND=redis requires HORT_REDIS_URL (the per-class overrides fall back to it at composition time).
11. HORT_CONFIG_DIR must be a directory if set non-empty.
12. HORT_UPSTREAM_RESOLVER_REFRESH_SECS floor is 5.
13. Auth-enabled startup gate. HORT_AUTH_PROVIDER=disabled is rejected unless a local admin row exists or HORT_NATIVE_TOKENS_ENABLED=true. oidc always passes without a DB query.
14. HORT_EXTRA_CA_BUNDLE fail-closed. Set-but-unreadable / invalid PEM / zero certs aborts boot — never degrades silently to public-CA-only trust.

hort-worker — command-line surface

hort-worker claims jobs from the shared jobs table and dispatches each row to the TaskHandler registered for its kind (scan, cron-rescan-tick, advisory-watch-tick, staging-sweep, service-account-rotation, noop).

healthcheck deliberately stays cheap (it runs every few seconds): it (1) parses WorkerConfig::from_env, then (2) opens a one-connection pool (acquire timeout 2000 ms) and runs a single SELECT 1 under an outer 2500 ms budget, then exits. It does not start the dispatcher, probe scanners, or initialise tracing/Prometheus. Exit 0 on success, 1 on any failure (hort-worker fatal: … to stderr). Unknown subcommand → clap InvalidSubcommand.

The worker is otherwise driven by Helm values and an HTTP admin-task endpoint (CronJob → POST /api/v1/admin/tasks/{kind}), not by additional CLI flags — see rotating-service-account-tokens.md.

hort-worker — environment variables

The scanner adapters (hort-adapters-scanner-trivy / -osv) and the advisory adapter (hort-adapters-advisory-osv) read no env vars themselves — every value reaches them through WorkerConfig fields wired in composition.rs. The worker uses the hort_app_role runtime DSN only (it asserts, never runs, migrations) and does not read HORT_NATS_URL, HORT_EPHEMERAL_STORE_BACKEND, a non-evictable HORT_REDIS_URL, or any AWS SDK credential-chain variable.

Core, identity & logging

¹ Exactly one of HORT_DATABASE_URL / DATABASE_URL is required; with neither set the error surfaces the name "DATABASE_URL".

Dispatcher tuning

Scanner & advisory adapters

Storage (for staging-sweep + scanner content reads)

Same names and semantics as the hort-server Storage table — HORT_STORAGE_BACKEND (default filesystem), HORT_STORAGE_FILESYSTEM_PATH, HORT_STORAGE_S3_BUCKET, AWS_REGION / AWS_DEFAULT_REGION, AWS_ENDPOINT_URL_S3 / AWS_ENDPOINT_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, HORT_STORAGE_S3_FORCE_PATH_STYLE, HORT_STORAGE_S3_ALLOW_HTTP. The filesystem CAS root must match the server's. HORT_STATEFUL_UPLOAD_STAGING_DIR is read by the staging-sweep task with the same derivation rule as the server.

Service-account rotation

Extra CA / subprocess trust

HORT_EXTRA_CA_BUNDLE propagation (worker)

extra_ca::read_and_propagate runs once at boot, before composition:

• Unset/empty → no-op; scanner subprocesses use their default OS trust store; SSL_CERT_FILE is left untouched.
• Set & non-empty: the bundle is read and parsed (ExtraTrustAnchors), merged with the system store (/etc/ssl/certs/ca-certificates.crt), and the merged PEM is written atomically to /tmp/hort-worker-ca-bundle.pem. The parsed anchors flow into the OSV advisory adapter's reqwest client; the merged-bundle path flows into each scanner subprocess as the per-process SSL_CERT_FILE. Any read / parse / write failure aborts boot (hort-worker fatal: …) — a configured-but-broken bundle fails fast rather than degrading to public-root-only TLS. The unreadable-file fatal names the missing path and points at the absent CA-bundle mount, so a half-wired manual recipe (env set but the worker volume forgotten) is an actionable error, not an opaque crashloop. The chart only sets HORT_EXTRA_CA_BUNDLE on the worker when it also auto-mounts the bundle there (extraCaBundle.configMapName / secretName); see extra-ca-bundle.md. The chart must provide a writable /tmp (an emptyDir).

Not consumed by the v2 binaries

Operators occasionally carry these forward from the prototype or the target-architecture docs. The v2 binaries do not read them — they have no effect:

See also

• Helm values reference — the chart keys that render into these env vars.
• Security hardening checklist — chart-vs-binary defaults for the security-relevant knobs.
• Tune HTTP transport timeouts
• Trust internal or corporate CAs
• Wire secrets
• Provision the two Postgres roles
• Declare configuration via $HORT_CONFIG_DIR