kc/keycloak-hybrid-x509-spi

Keycloak Hybrid X509 Certificate Lookup SPI

Custom x509cert-lookup provider that lets a single Keycloak instance authenticate x509/CAC users arriving via two paths at once:

1. F5 path (header): F5 BigIP does L7 break-and-inspect, validates the user's cert, re-encrypts to Keycloak (through the L4-passthrough NLB), and forwards the user cert in an HTTP header — like the built-in nginx provider handles today.
2. Direct path (TLS): TGW/brokerage traffic that bypasses the F5 performs mTLS directly with Keycloak (which terminates TLS itself behind the passthrough NLB with https-client-auth=request), and the cert is read straight off the handshake.

The critical addition over stock Keycloak: the header is only honored when the request provably came from the F5 (by the F5's own TLS client cert fingerprint/DN, or source IP). Without this, any client on the TGW could forge the header and impersonate any user.

Per-request decision logic:

Cert header	Peer is trusted proxy	trust-mode	Result
present	yes	any	header cert used
present	no	log	header cert used + loud warning
present	no	enforce	header ignored; TLS peer cert used if present
absent	yes	any	no user cert (the F5's own cert is never a user)
absent	no	any	TLS peer cert used if present

No changes are needed to your realm's X509 browser authentication flow — the authenticator receives the cert the same way regardless of source.

Deploying this? Follow the step-by-step operator guide in RUNBOOK.md. This README is the reference (config options, security rationale, troubleshooting).

---

1. Build

mvn package          # or: podman run --rm -v $PWD:/src -w /src maven:3.9-eclipse-temurin-17 mvn package

Produces target/keycloak-hybrid-x509-spi-1.0.0.jar (a few KB — contains only these classes; all Keycloak APIs are provided). Set <keycloak.version> in pom.xml to match your deployed version (any 24.x–26.x works; the APIs used are stable across that range).

2. Deploy (custom image)

Add to your existing Keycloak Dockerfile:

FROM quay.io/keycloak/keycloak:26.0 AS builder            # match your current base/tag
COPY keycloak-hybrid-x509-spi-1.0.0.jar /opt/keycloak/providers/

# https-client-auth is a BUILD-time option in Quarkus Keycloak — it must be present
# when kc.sh build runs, not just at start.
ENV KC_HTTPS_CLIENT_AUTH=request
# ...plus whatever build options your image already sets (KC_DB, KC_FEATURES, ...)
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:26.0
COPY --from=builder /opt/keycloak/ /opt/keycloak/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
CMD ["start", "--optimized"]

If your image doesn't use --optimized, dropping the jar into /opt/keycloak/providers/ and setting KC_HTTPS_CLIENT_AUTH=request at runtime is enough — Keycloak re-augments on start. request means the client cert is OPTIONAL: connections without one (OIDC app backchannels, admin console, the F5 before it's given a cert) proceed exactly as before. Do NOT use required.

3. Runtime configuration (env vars on the Keycloak pod)

# Switch the lookup provider from nginx to hybrid
KC_SPI_X509CERT_LOOKUP_PROVIDER=hybrid

# Header name: NOTHING TO ADD. The hybrid provider inherits ssl-client-cert,
# ssl-cert-chain-prefix and certificate-chain-length from your existing
# KC_SPI_X509CERT_LOOKUP_NGINX_* settings in the Helm values — leave them in place.
# (Set KC_SPI_X509CERT_LOOKUP_HYBRID_SSL_CLIENT_CERT only to override; the startup
# log line "inherited from the nginx provider config" confirms what was picked up.)

# Day-one: permissive mode (headers accepted from anyone, like today, but everything logged)
KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=log

# After harvesting values from the logs (see rollout plan), add trust rules:
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CERT_SHA256=<sha256hex>[,<sha256hex-next-rotation>]
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CIDRS=10.x.y.0/28
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_SUBJECT_DN=CN=f5.internal.example,OU=Infra,O=YourOrg
# ...then flip:
#KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=enforce

Keycloak 26 prefers a double-dash format (KC_SPI_X509CERT_LOOKUP__HYBRID__TRUST_MODE); the single-dash form above works on 24–26 (26 logs a deprecation warning). Use one style consistently.

All options (prefix spi-x509cert-lookup-hybrid-):

Option	Default	Meaning
ssl-client-cert	inherits nginx setting, else ssl-client-cert	Header with the user cert (PEM, URL-encoded PEM, or base64 DER — auto-detected)
ssl-cert-chain-prefix	inherits nginx setting, else ssl-cert-chain	Optional chain headers <prefix>-0..n
certificate-chain-length	inherits nginx setting, else 1	Max chain headers to read
trust-mode	log	log = honor all headers, warn on untrusted; enforce = ignore headers from untrusted peers
trusted-proxy-cert-sha256	—	Comma-separated SHA-256 fingerprints of the F5's client cert(s). Strongest check. List old+new during rotations.
trusted-proxy-subject-dn	—	|-separated exact subject DNs (RFC2253). Safe because TLS already validated the chain. Survives rotation.
trusted-proxy-cidrs	—	Comma-separated CIDRs/IPs for the F5's egress addresses. Weakest; requires NLB client-IP preservation.
header-cert-enabled	true	Master switch for the header path
direct-cert-enabled	true	Master switch for the direct-TLS path
rebuild-chain-from-truststore	true	Rebuild issuer chain for header certs from the Keycloak truststore (matches nginx-provider behavior)
truststore-chain-depth	4	Max issuers appended during rebuild
verbose	true	Per-request INFO logging of the decision (peer fingerprint, source IP, path taken). Set false after rollout.

Truststore: your existing truststore (already working for header-cert validation) must also be reachable by the TLS layer so Keycloak can request/validate browser certs on the direct path. On KC 25+ point KC_TRUSTSTORE_PATHS at your CA bundle; on KC 24 use the spi-truststore-file-* options you likely already have. The CAs listed there are also what browsers use to filter the cert-picker dialog.

4. Rollout / test plan (one 30-min cycle each, in order)

Cycle 0 — no deploy, info gathering. From the current deployment grab: truststore config, base image tag, and whether start --optimized is used. (The header name is inherited automatically from the existing KC_SPI_X509CERT_LOOKUP_NGINX_* values — no need to look it up.) Ask the F5 team to start on the one-pager (section 6) in parallel.

Cycle 1 — deploy in log mode (zero behavior change expected). Jar + KC_HTTPS_CLIENT_AUTH=request + provider=hybrid + trust-mode=log; keep all existing KC_SPI_X509CERT_LOOKUP_NGINX_* values in the Helm values file untouched. Verify: existing F5 CAC login still works. Then grep logs for x509-hybrid:

• startup lines show the parsed config, including which nginx settings were inherited;
• each F5 login logs remoteAddr= (→ your CIDR value) and peer= (→ no-tls-client-cert until the F5 presents one);
• header cert decoded using strategy '...' (DEBUG) confirms the F5's encoding. If a TGW-side test client exists already, hit Keycloak directly: expect a browser cert prompt and using DIRECT TLS peer cert in the logs.

Cycle 2 — F5 presents its client cert. After the F5 change, each F5 login logs peer=subject=[...] sha256=<fingerprint> — that fingerprint is your trusted-proxy-cert-sha256 value. Confirm logins still work (request mode tolerates the new cert automatically, provided the F5 cert's CA is in the truststore).

Cycle 3 — enforce. Set the trust rules + trust-mode=enforce. Verify: F5 login works, direct TGW login works, and the negative test — from a TGW-side box, curl -k https://keycloak.../realms/<realm>/account -H "ssl-client-cert: <any user PEM>" — produces IGNORED cert header from untrusted peer in the logs and no authentication.

Cycle 4 — quiet down. verbose=false, optionally keep DEBUG off. Done.

Rollback at any cycle: set KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx and restart — the built-in provider and all its config are untouched by this deployment.

5. Security notes

• Never leave trust-mode=log long-term once TGW routes are open: it preserves the legacy trust-any-header behavior. The provider logs a warning at startup to this effect.
• Fingerprint pinning breaks when the F5 rotates its client cert — list the next cert's fingerprint alongside the current one before rotation, or rely on subject-dn (which survives rotation and is chain-validated by the TLS layer in request mode).
• Source-IP trust requires the NLB target group to have client IP preservation enabled (targets registered by instance ID have it on by default; by IP it's configurable). The remoteAddr= log line from Cycle 1 tells you definitively what Keycloak sees.
• The F5 pre-screens users against an allowlist and (presumably) checks revocation; the direct path has no such screen. Enable OCSP/CRL revocation checking in the realm's X509 authenticator config (Authentication → your browser x509 flow → config) if it isn't already on, and confirm the cert-to-user mapping attribute is strict enough that only provisioned brokerage users resolve.
• https-client-auth=request sends a TLS CertificateRequest on every connection. Browsers without a matching cert and server-to-server OIDC clients simply continue certless; users hitting Keycloak directly get the platform cert-picker (expected CAC behavior).

6. One-pager for the F5 team

Keycloak's listener will request (not require) a TLS client certificate. We need the BigIP virtual server that fronts Keycloak to authenticate itself on its server-side (re-encrypt) SSL profile:

1. Issue a client certificate for the BigIP from an internal CA (or reuse an existing device cert). Any subject is fine, e.g. CN=bigip-keycloak-proxy,OU=Infra,O=<org>.
2. On the Server SSL profile used for the Keycloak pool: set this certificate/key so the BigIP presents it during the TLS handshake with the backend.
3. Send us: the certificate's SHA-256 fingerprint (openssl x509 -in cert.pem -noout -fingerprint -sha256), its exact subject DN, the issuing CA chain (PEM), and the self-IP/SNAT addresses the BigIP uses toward the Keycloak NLB.
4. No iRule/header changes needed — keep injecting the client cert header exactly as today. Timing note: this can be deployed before or after our Keycloak change; the Keycloak side is backward compatible either way (https-client-auth=request is optional-cert).

7. Troubleshooting via logs (category com.example.keycloak.x509)

Log line	Meaning
initialized. trust-mode=...	Config as parsed at startup — check this first
using HEADER cert from trusted proxy	F5 path, healthy (enforced)
honoring cert header from UNTRUSTED peer	Log-mode: would fail in enforce — fix trust rules before flipping
IGNORED cert header from untrusted peer	Enforce-mode rejection: forgery attempt, or your F5 trust rule is wrong/stale
using DIRECT TLS peer cert	TGW path, healthy
trusted proxy connection without usable cert header	F5 request with no/blank header (e.g. health checks) — normal
failed to parse cert header	Unexpected header encoding — the logged prefix shows what arrived
no certificate found	Certless request (normal for non-CAC flows)

Set KC_LOG_LEVEL=INFO,com.example.keycloak.x509:debug during rollout for decode-strategy and chain-rebuild details.