kc/keycloak-hybrid-x509-spi/README.md

# Keycloak Hybrid X509 Certificate Lookup SPI

Custom `x509cert-lookup` provider that lets a single Keycloak instance authenticate
x509/CAC users arriving via **two paths at once**:

1. **F5 path (header):** F5 BigIP does L7 break-and-inspect, validates the user's cert,
   re-encrypts to Keycloak (through the L4-passthrough NLB), and forwards the user cert
   in an HTTP header — like the built-in `nginx` provider handles today.
2. **Direct path (TLS):** TGW/brokerage traffic that bypasses the F5 performs mTLS
   directly with Keycloak (which terminates TLS itself behind the passthrough NLB with
   `https-client-auth=request`), and the cert is read straight off the handshake.

The critical addition over stock Keycloak: **the header is only honored when the request
provably came from the F5** (by the F5's own TLS client cert fingerprint/DN, or source IP).
Without this, any client on the TGW could forge the header and impersonate any user.

Per-request decision logic:

| Cert header | Peer is trusted proxy | `trust-mode` | Result |
|---|---|---|---|
| present | yes | any | header cert used |
| present | no | `log` | header cert used + **loud warning** |
| present | no | `enforce` | header **ignored**; TLS peer cert used if present |
| absent | yes | any | no user cert (the F5's own cert is never a user) |
| absent | no | any | TLS peer cert used if present |

No changes are needed to your realm's X509 browser authentication flow — the
authenticator receives the cert the same way regardless of source.

> **Deploying this?** Follow the step-by-step operator guide in [RUNBOOK.md](RUNBOOK.md).
> This README is the reference (config options, security rationale, troubleshooting).

---

## 1. Build

```bash
mvn package          # or: podman run --rm -v $PWD:/src -w /src maven:3.9-eclipse-temurin-17 mvn package
```

Produces `target/keycloak-hybrid-x509-spi-1.0.0.jar` (a few KB — contains only these
classes; all Keycloak APIs are `provided`). Set `<keycloak.version>` in `pom.xml` to match
your deployed version (any 24.x–26.x works; the APIs used are stable across that range).

## 2. Deploy (custom image)

Add to your existing Keycloak Dockerfile:

```dockerfile
FROM quay.io/keycloak/keycloak:26.0 AS builder            # match your current base/tag
COPY keycloak-hybrid-x509-spi-1.0.0.jar /opt/keycloak/providers/

# https-client-auth is a BUILD-time option in Quarkus Keycloak — it must be present
# when kc.sh build runs, not just at start.
ENV KC_HTTPS_CLIENT_AUTH=request
# ...plus whatever build options your image already sets (KC_DB, KC_FEATURES, ...)
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:26.0
COPY --from=builder /opt/keycloak/ /opt/keycloak/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
CMD ["start", "--optimized"]
```

If your image doesn't use `--optimized`, dropping the jar into `/opt/keycloak/providers/`
and setting `KC_HTTPS_CLIENT_AUTH=request` at runtime is enough — Keycloak re-augments on
start. **`request` means the client cert is OPTIONAL**: connections without one (OIDC app
backchannels, admin console, the F5 before it's given a cert) proceed exactly as before.
Do NOT use `required`.

## 3. Runtime configuration (env vars on the Keycloak pod)

```bash
# Switch the lookup provider from nginx to hybrid
KC_SPI_X509CERT_LOOKUP_PROVIDER=hybrid

# Header name: NOTHING TO ADD. The hybrid provider inherits ssl-client-cert,
# ssl-cert-chain-prefix and certificate-chain-length from your existing
# KC_SPI_X509CERT_LOOKUP_NGINX_* settings in the Helm values — leave them in place.
# (Set KC_SPI_X509CERT_LOOKUP_HYBRID_SSL_CLIENT_CERT only to override; the startup
# log line "inherited from the nginx provider config" confirms what was picked up.)

# Day-one: permissive mode (headers accepted from anyone, like today, but everything logged)
KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=log

# After harvesting values from the logs (see rollout plan), add trust rules:
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CERT_SHA256=<sha256hex>[,<sha256hex-next-rotation>]
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CIDRS=10.x.y.0/28
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_SUBJECT_DN=CN=f5.internal.example,OU=Infra,O=YourOrg
# ...then flip:
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=enforce
```

> Keycloak 26 prefers a double-dash format (`KC_SPI_X509CERT_LOOKUP__HYBRID__TRUST_MODE`);
> the single-dash form above works on 24–26 (26 logs a deprecation warning). Use one style
> consistently.

All options (prefix `spi-x509cert-lookup-hybrid-`):

| Option | Default | Meaning |
|---|---|---|
| `ssl-client-cert` | inherits nginx setting, else `ssl-client-cert` | Header with the user cert (PEM, URL-encoded PEM, or base64 DER — auto-detected) |
| `ssl-cert-chain-prefix` | inherits nginx setting, else `ssl-cert-chain` | Optional chain headers `<prefix>-0..n` |
| `certificate-chain-length` | inherits nginx setting, else `1` | Max chain headers to read |
| `trust-mode` | `log` | `log` = honor all headers, warn on untrusted; `enforce` = ignore headers from untrusted peers |
| `trusted-proxy-cert-sha256` | — | Comma-separated SHA-256 fingerprints of the F5's client cert(s). **Strongest check.** List old+new during rotations. |
| `trusted-proxy-subject-dn` | — | `\|`-separated exact subject DNs (RFC2253). Safe because TLS already validated the chain. Survives rotation. |
| `trusted-proxy-cidrs` | — | Comma-separated CIDRs/IPs for the F5's egress addresses. Weakest; requires NLB client-IP preservation. |
| `header-cert-enabled` | `true` | Master switch for the header path |
| `direct-cert-enabled` | `true` | Master switch for the direct-TLS path |
| `rebuild-chain-from-truststore` | `true` | Rebuild issuer chain for header certs from the Keycloak truststore (matches nginx-provider behavior) |
| `truststore-chain-depth` | `4` | Max issuers appended during rebuild |
| `verbose` | `true` | Per-request INFO logging of the decision (peer fingerprint, source IP, path taken). Set `false` after rollout. |

**Truststore:** your existing truststore (already working for header-cert validation) must
also be reachable by the TLS layer so Keycloak can request/validate browser certs on the
direct path. On KC 25+ point `KC_TRUSTSTORE_PATHS` at your CA bundle; on KC 24 use the
`spi-truststore-file-*` options you likely already have. The CAs listed there are also what
browsers use to filter the cert-picker dialog.

## 4. Rollout / test plan (one 30-min cycle each, in order)

**Cycle 0 — no deploy, info gathering.** From the current deployment grab: truststore
config, base image tag, and whether `start --optimized` is used. (The header name is
inherited automatically from the existing `KC_SPI_X509CERT_LOOKUP_NGINX_*` values — no
need to look it up.) Ask the F5 team to start on the one-pager (section 6) in parallel.

**Cycle 1 — deploy in log mode (zero behavior change expected).**
Jar + `KC_HTTPS_CLIENT_AUTH=request` + provider=hybrid + `trust-mode=log`; keep all
existing `KC_SPI_X509CERT_LOOKUP_NGINX_*` values in the Helm values file untouched.
Verify: existing F5 CAC login still works. Then grep logs for `x509-hybrid`:
- startup lines show the parsed config, including which nginx settings were inherited;
- each F5 login logs `remoteAddr=` (→ your CIDR value) and `peer=` (→ `no-tls-client-cert`
  until the F5 presents one);
- `header cert decoded using strategy '...'` (DEBUG) confirms the F5's encoding.
If a TGW-side test client exists already, hit Keycloak directly: expect a browser cert
prompt and `using DIRECT TLS peer cert` in the logs.

**Cycle 2 — F5 presents its client cert.** After the F5 change, each F5 login logs
`peer=subject=[...] sha256=<fingerprint>` — that fingerprint is your
`trusted-proxy-cert-sha256` value. Confirm logins still work (`request` mode tolerates the
new cert automatically, provided the F5 cert's CA is in the truststore).

**Cycle 3 — enforce.** Set the trust rules + `trust-mode=enforce`. Verify: F5 login works,
direct TGW login works, and the negative test — from a TGW-side box,
`curl -k https://keycloak.../realms/<realm>/account -H "ssl-client-cert: <any user PEM>"` —
produces `IGNORED cert header from untrusted peer` in the logs and no authentication.

**Cycle 4 — quiet down.** `verbose=false`, optionally keep DEBUG off. Done.

Rollback at any cycle: set `KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx` and restart — the
built-in provider and all its config are untouched by this deployment.

## 5. Security notes

- **Never** leave `trust-mode=log` long-term once TGW routes are open: it preserves the
  legacy trust-any-header behavior. The provider logs a warning at startup to this effect.
- Fingerprint pinning breaks when the F5 rotates its client cert — list the next cert's
  fingerprint alongside the current one before rotation, or rely on `subject-dn` (which
  survives rotation and is chain-validated by the TLS layer in `request` mode).
- Source-IP trust requires the NLB target group to have **client IP preservation** enabled
  (targets registered by instance ID have it on by default; by IP it's configurable). The
  `remoteAddr=` log line from Cycle 1 tells you definitively what Keycloak sees.
- The F5 pre-screens users against an allowlist and (presumably) checks revocation; the
  direct path has no such screen. Enable **OCSP/CRL revocation checking** in the realm's
  X509 authenticator config (Authentication → your browser x509 flow → config) if it isn't
  already on, and confirm the cert-to-user mapping attribute is strict enough that only
  provisioned brokerage users resolve.
- `https-client-auth=request` sends a TLS CertificateRequest on every connection. Browsers
  without a matching cert and server-to-server OIDC clients simply continue certless; users
  hitting Keycloak directly get the platform cert-picker (expected CAC behavior).

## 6. One-pager for the F5 team

> Keycloak's listener will request (not require) a TLS client certificate. We need the
> BigIP virtual server that fronts Keycloak to authenticate itself on its **server-side
> (re-encrypt) SSL profile**:
> 1. Issue a client certificate for the BigIP from an internal CA (or reuse an existing
>    device cert). Any subject is fine, e.g. `CN=bigip-keycloak-proxy,OU=Infra,O=<org>`.
> 2. On the **Server SSL profile** used for the Keycloak pool: set this certificate/key so
>    the BigIP presents it during the TLS handshake with the backend.
> 3. Send us: the certificate's SHA-256 fingerprint
>    (`openssl x509 -in cert.pem -noout -fingerprint -sha256`), its exact subject DN, the
>    issuing CA chain (PEM), and the self-IP/SNAT addresses the BigIP uses toward the
>    Keycloak NLB.
> 4. No iRule/header changes needed — keep injecting the client cert header exactly as today.
> Timing note: this can be deployed before or after our Keycloak change; the Keycloak side
> is backward compatible either way (`https-client-auth=request` is optional-cert).

## 7. Troubleshooting via logs (category `com.example.keycloak.x509`)

| Log line | Meaning |
|---|---|
| `initialized. trust-mode=...` | Config as parsed at startup — check this first |
| `using HEADER cert from trusted proxy` | F5 path, healthy (enforced) |
| `honoring cert header from UNTRUSTED peer` | Log-mode: would fail in enforce — fix trust rules before flipping |
| `IGNORED cert header from untrusted peer` | Enforce-mode rejection: forgery attempt, or your F5 trust rule is wrong/stale |
| `using DIRECT TLS peer cert` | TGW path, healthy |
| `trusted proxy connection without usable cert header` | F5 request with no/blank header (e.g. health checks) — normal |
| `failed to parse cert header` | Unexpected header encoding — the logged prefix shows what arrived |
| `no certificate found` | Certless request (normal for non-CAC flows) |

Set `KC_LOG_LEVEL=INFO,com.example.keycloak.x509:debug` during rollout for
decode-strategy and chain-rebuild details.