project-hort.de

Configure OCI pull-through with verified upstream

This guide is for operators who want a Remote OCI repository in hort that proxies Docker Hub, GHCR, Quay, Harbor, or any upstream OCI Distribution Spec registry, and serves manifests and blobs to docker pull / skopeo copy / podman pull / containerd. It covers verified pull-through, the 503 quarantine response, and the digest-pin mitigation for the moved-tag case under quarantine-by-default.

For the underlying design see ADR 0006 — mandatory upstream verification, ADR 0007 — fail-closed quarantine release, and the prefetch pipeline explanation.

1. What pull-through verification means

Every blob and manifest that an OCI client fetches through a Remote OCI repository is SHA-256-verified before it is admitted to local CAS. The verification target is the protocol-native digest each side of an OCI transaction sends:

Verification produces a ChecksumVerified event on the artifact stream, in the same append batch as ArtifactIngested, with algorithm = HashAlgorithm::Sha256. A tampered blob (one whose bytes do not hash to the requested digest) produces a 502 Bad Gateway, a ChecksumMismatch event on the repository stream, and never reaches local CAS.

2. Quarantine-by-default + the 503 response

Quarantine is the default for every freshly-ingested artifact (configurable per repo via ScanPolicy.quarantineDuration — see ADR 0007). The settled OCI read-path response for a quarantined manifest:

A pull of a tag whose currently-referenced manifest is in quarantine returns 503 Service Unavailable with a Retry-After: <seconds> header — NOT a substitute manifest.

The response body is the canonical OCI error envelope:

{
  "errors": [{
    "code": "UNAVAILABLE",
    "message": "manifest is quarantined; retry after the indicated interval",
    "detail": { "retry_after_seconds": 3540 }
  }]
}

The Retry-After value is the computed seconds-until-quarantine- deadline (clamped to ≥ 1; default 1 hour when no deadline is set).

Why not a silent substitution to the prior tag target?

The design weighed and explicitly rejected the "deferred move" alternative — hold the new manifest as a pending_target, serve the previous target until quarantine clears, swap on ArtifactReleased. The rejection rationale, reinforced under quarantine-by-default:

The 503 + Retry-After is the right signal: "a new image was published but hasn't cleared verification yet — wait and retry."

3. The moved-tag mitigation: pin by digest

A client pulling :latest (or any tag) during a genuine new-image quarantine window sees the 503. The operationally-correct mitigation is digest pinning:

# Floating-tag pull — vulnerable to the quarantine window.
docker pull hort.example.com/oci-mirror/library/nginx:latest

# Digest-pinned pull — resolves to a specific manifest forever.
docker pull hort.example.com/oci-mirror/library/nginx@sha256:abc...

Once a manifest is ingested AND released (its quarantine window has elapsed and its scan has cleared), pulls by @sha256:<digest> resolve to those exact bytes for the lifetime of the cache — no subsequent quarantine flip is possible because the digest IS the content. This is the standard OCI supply-chain hygiene recommendation (reproducibility, sigstore signatures, attested SBOMs all rest on digest pinning); the hort quarantine layer simply makes the cost of ignoring it immediate and visible rather than latent.

Where this matters most:

4. Prefetch shrinks the exposure window

The OCI prefetch wires OnDistTagMove into the manifest-fetch hot path: when a tag pull resolves a new upstream digest that differs from hort's previously-held digest for that tag, the proxy spawns background pull-throughs for the new manifest's referenced blobs (config + every layer). The blobs ingest + quarantine + scan in parallel with whatever the client is doing, so by the time anyone actually pulls the new image's bytes the quarantine window has either closed or is very close to closing.

Prefetch never skips the window — it only moves it earlier. The worst-case still requires a window to elapse before the first client pull succeeds; the typical case shifts that wait off the build's critical path.

Operator opt-in (per repository, declared via gitops):

# repositories/oci-public.yaml
apiVersion: project-hort.de/v1
kind: ArtifactRepository
metadata:
  name: oci-mirror
spec:
  format: oci
  type: proxy
  prefetchPolicy:
    enabled: true
    triggers: [on_dist_tag_move]
    depth: 5
    transitive_depth: 5
    max_age_days: 30

The on_dist_tag_move trigger fires only on detected tag-target changes (held digest != upstream digest, or first-time pull of a tag). It rides PullDedup inside the existing blob pull-through, so a racing client pull for the same blob collapses to a single upstream fetch.

Observability — the prefetch metrics fire with format="oci" / trigger="on_dist_tag_move" labels:

5. What does NOT apply to OCI

IndexMode::ReleasedOnly is meaningless for OCI

Repository.index_mode has two settings: ReleasedOnly (the default, hides non-released versions from the served index) and IncludePending (advertises upstream's full catalog minus known-quarantined). These apply to formats whose clients resolve a range to a concrete version — npm install foo@^1.2.0, pip install foo, cargo build, mvn install — because filtering the catalog rewrites what the resolver picks.

An OCI tag is not a range. docker pull foo:latest is an exact pointer; an OCI registry CAN'T answer it with anything other than the manifest that tag points at (substitution is precisely the hide-the-gate rejection from §2). So IndexMode::ReleasedOnly is explicitly excluded from OCI by design. There is no operator knob to enable it for OCI — this is settled, and the [tests::oci_manifest_serve_path_must_not_consult_index_mode] regression guard in crates/hort-http-oci/src/prefetch.rs enforces it at test time. The index_mode column on an OCI repository is present for schema uniformity but inert.

If a future operational need genuinely requires range-style filtering on OCI (e.g. a curated multi-version mirror), the design document must be amended FIRST and the guardrail test retired in the same change. Do not paper over this with a quiet code patch.

Per-artifact quarantine opt-out → quarantineDuration: 0s

Operators who run OCI under quarantine-by-default but cannot tolerate the :latest-pull 503 for a specific upstream can set quarantineDuration: 0s on a ScanPolicy scoped to that repository:

# policies/oci-trusted-upstream.yaml
apiVersion: project-hort.de/v1
kind: ScanPolicy
metadata:
  name: oci-trusted-upstream
spec:
  appliesTo:
    repositories: ["oci-trusted-mirror"]
  quarantineDuration: 0s          # bypass the observation window
  scanBackends: [trivy, osv]      # scan is NOT bypassed

This bypasses the observation window (the time-based aging) but NOT the scan gate — a freshly-published image with a known-CVE layer is still rejected on import. See declare-gitops-config.md → kind: ScanPolicy → "Hosted-repo recommendation" (which applies to proxy too) for the full operator surface.

The quarantineDuration: 0s knob is the documented per-repo operator opt-out from quarantine-window friction. The repo-wide Permissive posture is a more invasive sledgehammer.

6. Declare a Remote OCI repository

# repositories/oci-public.yaml
apiVersion: project-hort.de/v1
kind: ArtifactRepository
metadata:
  name: oci-mirror
spec:
  format: oci
  type: proxy
  isPublic: true
  storageBackend: filesystem
  storagePath: /data/repos/oci-mirror
  # Optional but recommended — see §4.
  prefetchPolicy:
    enabled: true
    triggers: [on_dist_tag_move]
    depth: 5
# upstreams/oci-public.yaml
apiVersion: project-hort.de/v1
kind: UpstreamMapping
metadata:
  name: oci-public
spec:
  repository: oci-mirror
  pathPrefix: dockerhub/
  upstreamUrl: https://registry-1.docker.io
  # upstreamNamePrefix omitted — Docker Hub needs no outbound path
  # splice. Set it only when the upstream's path layout requires one
  # (Zot multi-storage, Artifactory rewrites, GitLab per-project URLs
  # — see declare-gitops-config.md). An empty string is rejected at
  # apply time; omit the field entirely instead.
  auth:
    type: anonymous
  # Opt-in publish-time anchoring. When true, the
  # quarantine window is anchored on the upstream's Last-Modified
  # rather than hort's ingest time. Useful for high-latency mirrors of
  # already-aged images.
  trustUpstreamPublishTime: false

Apply via the standard gitops flow — declare-gitops-config.md §5 covers the boot sequence.

7. Point an OCI client at the proxy

Docker

docker pull hort.example.com/oci-mirror/dockerhub/library/nginx:1.27

The path components after the host map to: <oci-repo-key>/<upstream-mapping-prefix>/<upstream-image-name>:<tag>.

For automated builds, prefer digest pins:

FROM hort.example.com/oci-mirror/dockerhub/library/nginx@sha256:abc...

skopeo / podman / containerd

All three speak OCI Distribution Spec verbatim. Replace the host component with hort.example.com/<oci-repo-key>/... in whatever reference your tool accepts.

Authentication

Anonymous pulls work on public OCI repositories. For private repos:

docker login hort.example.com
# Username: <your-username>
# Password: <PAT issued via hort-cli>

The OCI client's Authorization: Basic <base64> is decoded by hort's oci_bearer_auth middleware; the password slot carries an hort PAT (the same twine-compatible flow the other formats use).

8. Diagnose 503 and 502 from the proxy

503 Service Unavailable + Retry-After

The manifest is quarantined. Either wait the indicated number of seconds and retry, OR switch to a digest-pinned pull (§3) for a previously-released digest, OR adjust the repository's ScanPolicy (§5).

502 Bad Gateway

Two failure modes; check the response body detail.reason:

The metric hort_upstream_checksum_total{format="oci", result=<reason>} ticks on every verification failure; alerting on a non-zero rate is recommended.

9. See also