project-hort.de

Federate a k8s workload to hort-server via projected SA tokens

This guide is for platform engineers running Flux (or any other gitops controller) who want their k8s workloads to authenticate to hort without a long-lived PAT pasted into a Secret. The workload's pod fetches a projected ServiceAccount token from the cluster's API server, exchanges it at POST /api/v1/auth/exchange, and gets back a short-lived bearer whose cap snapshots the matched kind: ServiceAccount's PermissionGrants at exchange time.

For the design rationale see ADR 0018 and the federation entries in docs/auth-catalog.md.

1. What this gives you

The 2026 industry-standard pattern for non-human identities in k8s is "no long-lived tokens at rest." Every pod that needs to talk to hort-server fetches a short-lived JWT from its own cluster's API server, exchanges it for an hort-server bearer, and either repeats the exchange when the bearer expires or simply re-fetches the projected token on each call. Nothing persists across pod restarts.

hort-server treats the cluster as a generic OIDC issuer: the JWKS endpoint is the cluster's /openid/v1/jwks (served by the API server itself when --service-account-issuer is configured), the JWT's iss claim is the cluster's issuer URL, and the sub claim encodes the pod's ServiceAccount as system:serviceaccount:<namespace>:<name>. The operator declares trust by writing one kind: OidcIssuer envelope per cluster + one kind: ServiceAccount envelope per workload identity.

The two artefacts that change in your gitops repository:

  1. kind: OidcIssuer — names the trusted cluster and binds the audience the projected token must carry.
  2. kind: ServiceAccount — declares the non-human identity in hort-server and lists which JWT claim shapes are allowed to assume it. Its authority is the PermissionGrants declared alongside (ADR 0044).

Existing PATs continue to work — federation is additive. Operators adopt this path workload-by-workload at their own pace.

2. Prerequisites

Useful upstream documentation: ServiceAccount token volume projection and Manage Service Accounts (issuer flags).

3. Step 1 — declare the OidcIssuer

One envelope per cluster. The issuerUrl must match the cluster's --service-account-issuer flag exactly — hort-server matches on the iss claim string with no normalisation.

apiVersion: project-hort.de/v1
kind: OidcIssuer
metadata:
  name: cluster-prod
spec:
  issuerUrl: https://kubernetes.default.svc.cluster.local
  audiences: [hort-server]
  jwksRefreshInterval: 1h
  allowedAlgorithms: [RS256]

Field-by-field:

kubectl apply -f oidc-issuer-cluster-prod.yaml is the wrong tool — kind: OidcIssuer is an hort-server gitops envelope, not a k8s CRD. Place the file in $HORT_CONFIG_DIR and restart hort-server (or push to your gitops repo and let Flux roll the chart). See declare-gitops-config.md §5 for the boot sequence.

4. Step 2 — declare the ServiceAccount

One envelope per workload identity. The federatedIdentities[] block lists which (issuer, claims) shapes may assume this SA. Multiple shapes are an OR — any single match suffices; multiple matches across SAs are a multiple_sa_match deny.

The envelope binds the identity; the PermissionGrant(s) beside it are the SA's authority — the exchanged bearer's cap snapshots them at exchange time:

apiVersion: project-hort.de/v1
kind: ServiceAccount
metadata:
  name: prod-pypi-pusher
spec:
  federatedIdentities:
    - issuer: cluster-prod
      claims:
        sub: system:serviceaccount:apps:pypi-publisher
---
apiVersion: project-hort.de/v1
kind: PermissionGrant
metadata:
  name: prod-pypi-pusher-write
spec:
  subject:
    kind: serviceAccount
    name: prod-pypi-pusher
  permission: write
  repository: pypi-internal

The claims: map is exact-match — every (key, value) in the map must equal the corresponding field in the JWT payload. The canonical k8s SA claim shape is sub: system:serviceaccount:<namespace>:<name>; that single claim pins one principal — the example above is a validation-clean, complete federatedIdentities[] entry on its own, no additional claim required. Newer clusters also expose kubernetes.io/serviceaccount/namespace and …/name as separate claims; add either alongside sub if you want the entry to read more explicitly, but keep sub in the map — namespace/name alone are not enough (see the subject-identifying-claim rule below).

Validation rules the apply pipeline enforces:

See declare-gitops-config.md kind: ServiceAccount for the canonical reference.

5. Step 3 — configure the pod

Mount a projected SA token with the right audience. The token's aud must be one of the values in OidcIssuer.spec.audiences — hort-server in this example.

apiVersion: v1
kind: Pod
metadata:
  name: pypi-publisher
  namespace: apps
spec:
  serviceAccountName: pypi-publisher
  containers:
    - name: app
      image: my-org/pypi-publisher:1.2.3
      volumeMounts:
        - name: hort-token
          mountPath: /var/run/secrets/hort-server
          readOnly: true
  volumes:
    - name: hort-token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              audience: hort-server
              expirationSeconds: 3600

expirationSeconds: 3600 is the lower bound k8s honours; the API server may issue a longer-lived token but kubelet refreshes the file inside the pod well before expiry (kubelet starts refreshing at 80% of token lifetime). Treat the token file as a moving target — read it on every exchange, do not cache its contents in process memory.

The pod's serviceAccountName: pypi-publisher must exist in the namespace (kubectl create serviceaccount pypi-publisher -n apps). That k8s ServiceAccount is the sub claim source — the JWT will carry sub: system:serviceaccount:apps:pypi-publisher, which is exactly what the kind: ServiceAccount envelope above matches on.

6. Step 4 — fetch and exchange

Inside the pod, read the projected token and post it to /api/v1/auth/exchange. The exchange returns a short-lived bearer.

JWT=$(cat /var/run/secrets/hort-server/token)
RESPONSE=$(curl -sS -X POST \
  https://hort.example.com/api/v1/auth/exchange \
  -d "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \
  -d "subject_token=${JWT}" \
  -d "subject_token_type=urn:ietf:params:oauth:token-type:jwt")
HORT_TOKEN=$(echo "${RESPONSE}" | jq -r .access_token)

The response body is RFC 8693 §2.2.1 standard:

{
  "access_token": "hort_svc_…",
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "token_type": "Bearer",
  "expires_in": 3600
}

expires_in is the lesser of 1 hour and the JWT's remaining exp — the bearer cannot outlive the source token. No refresh token is issued; when ${HORT_TOKEN} expires, re-read the projected token (still fresh, kubelet keeps it that way) and repeat the exchange.

7. Step 5 — verify it works

A twine upload is a tight end-to-end check: it exercises auth, authorization, repository scoping, and the ingest path. Substitute your repository's URL and any tiny package you have at hand.

twine upload \
  --repository-url https://hort.example.com/pypi/pypi-internal/ \
  --username __token__ \
  --password "${HORT_TOKEN}" \
  dist/*.whl

A successful upload returns 200 OK from twine and prints the artifact path. To confirm the federation path was actually taken, check hort-server logs for the TokenIssued event matching this exchange:

kubectl logs -n hort-server deploy/hort-server | grep TokenIssued | tail -1

Look for source_issuer = "cluster-prod", source_sub = "system:serviceaccount:apps:pypi-publisher", and source_jti populated with the projected token's jti. If those three fields appear, the federation branch handled the exchange — not a fallback path.

The metric hort_token_exchange_total{kind="federated_jwt", result="success"} increments on every successful exchange. Plot it alongside kind="federated_jwt", result!="success" to alert on broken trust policies.

8. Troubleshooting

Every deny returns 403 with a error_description body. The table maps the deny reason to its likely cause; the second column quotes the deny hint hort-server emits in the response body so operator-side automation can match exactly.

ReasonDeny hintLikely cause
unknown_issuer"no OidcIssuer matches \iss\ — declare one or fix the JWT"The iss claim of the projected token does not equal OidcIssuer.spec.issuerUrl. Compare the JWT payload's iss to the envelope. The cluster's --service-account-issuer may have changed (e.g. after a managed-control-plane upgrade).
aud_mismatch"aud not in OidcIssuer.audiences"The projected token's aud does not match any entry in spec.audiences. Common cause: the serviceAccountToken.audience: on the pod is missing or differs from hort-server.
no_sa_match(handler-emitted) "no ServiceAccount.federatedIdentities matched"The token validated cryptographically but the claims: map in every candidate SA failed exact-match against the JWT payload. The pod's sub claim is the usual mismatch site — typo in the namespace or k8s SA name.
multiple_sa_match(handler-emitted) "multiple ServiceAccount matches"Two ServiceAccount envelopes are overly broad. Tighten one envelope's claims: map (add a discriminator). The full SA-name candidates appear in hort-server's INFO log.
signature_invalid"signature verification failed"JWKS cache is stale relative to the issuer, or the projected token was minted by a different cluster. Lowering jwksRefreshInterval to 5m temporarily, then back, is a fast way to confirm staleness.

For the full taxonomy see the FederationDenyReason enum in crates/hort-domain/src/ports/federated_jwt_validator.rs.

9. What's NOT covered

10. See also