kc/keycloak-hybrid-x509-spi/RUNBOOK.md

Runbook: Rolling out the Hybrid X509 Lookup SPI

Step-by-step deployment procedure for operators. Background, config reference, and troubleshooting tables live in README.md — this document is the "what do I actually type" guide.

What we're changing and why (30 seconds): Today all user traffic passes through the F5 BigIP, which validates user CAC/x509 certs and forwards the cert to Keycloak in an HTTP header. New Transit Gateway (TGW) connectivity from brokerage AWS accounts bypasses the F5, so Keycloak must additionally accept certs directly from the TLS handshake, while making sure the F5's cert header can't be forged by TGW clients. This SPI does both.

Today:            user ──mTLS──> F5 (validates cert, adds header) ──TLS──> NLB(L4) ──> Keycloak
New TGW path:     user ────────────────────mTLS───────────────────────────> NLB(L4) ──> Keycloak

---

Roles and coordination

Who	Needed for	Task
Keycloak/EKS operator	every phase	image build, Helm values, log checks
F5/BigIP team	Phase 3	add client cert to the Server SSL profile (see README §6 one-pager)
TGW-side tester	Phases 2 & 4	a browser + CAC on a machine that reaches Keycloak without the F5

Each phase is designed to fit one ~30-minute test window and to be independently verifiable and reversible. Do not combine phases — the log output of each phase is the input to the next.

---

Phase 0 — Prep (no deployment)

1. Collect from the current deployment:
   • Keycloak version → set <keycloak.version> in pom.xml to match (24.x–26.x).
   • Base image tag and whether the entrypoint uses start --optimized.
   • Truststore config: KC_TRUSTSTORE_PATHS (KC 25+) or spi-truststore-file-* env vars. The user-CA bundle must be present (it already is if header login works).
   • Confirm the existing KC_SPI_X509CERT_LOOKUP_NGINX_* env vars are in the Helm values (the new provider inherits them automatically — do NOT remove them).
2. Hand the F5 team the one-pager (README §6) so their change request runs in parallel. Ask them for: cert SHA-256 fingerprint, subject DN, issuing CA chain (PEM), and the self-IP/SNAT CIDR the BigIP uses toward the Keycloak NLB.
3. Confirm with networking whether the NLB target group preserves client IPs (needed only if you plan to use the CIDR trust rule; Phase 2 logs verify it either way).
4. Brokerage-user readiness (should already be true): users exist/are federated in Keycloak, and their certs chain to a CA in the truststore.

Phase 1 — Build the jar and image

cd keycloak-hybrid-x509-spi
mvn package                        # or the podman one-liner in README §1
ls target/keycloak-hybrid-x509-spi-1.0.0.jar

Add to the Keycloak image (see Dockerfile.example for the full picture):

COPY keycloak-hybrid-x509-spi-1.0.0.jar /opt/keycloak/providers/
ENV KC_HTTPS_CLIENT_AUTH=request   # BUILD-time option: must precede `kc.sh build`
RUN /opt/keycloak/bin/kc.sh build

request = certificate optional. Never use required — it would break every certless connection (OIDC backchannels, admin console, health checks).

Phase 2 — Deploy in log mode (zero expected behavior change)

Helm values — add only these (leave every existing ..._NGINX_... var untouched):

extraEnv:                          # or however your chart names it
  - name: KC_SPI_X509CERT_LOOKUP_PROVIDER
    value: hybrid
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
    value: log
  - name: KC_LOG_LEVEL
    value: INFO,com.example.keycloak.x509:debug

Deploy, then verify in this order:

1. Startup config was parsed correctly:

   kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid: initialized"
   kubectl logs -n <ns> <keycloak-pod> | grep "inherited from the nginx provider config"
   

   The header= value must match what the F5 injects. A WARN about "no trusted proxy rules" is expected at this phase.
2. Regression check: a normal user logs in through the F5 with their CAC. Must work identically to before.
3. Harvest values for later phases from the login's log lines:

   kubectl logs -n <ns> <keycloak-pod> | grep "x509-hybrid:"
   

   • remoteAddr=<ip> → the F5's egress IP (→ TRUSTED_PROXY_CIDRS value; also tells you whether client-IP preservation is on — if you see NLB private IPs here, the CIDR rule is unusable and you must rely on the cert fingerprint).
   • peer=no-tls-client-cert → expected until Phase 3.
   • header cert decoded using strategy '...' (DEBUG) → confirms the F5's encoding.
4. If a TGW-side tester is available: browse to https://<keycloak-host>/realms/<realm>/account, expect the browser cert-picker, pick the CAC cert, expect login. Log shows using DIRECT TLS peer cert.

Rollback: set KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx, redeploy. (Keep the image; https-client-auth=request is harmless on its own.)

Phase 3 — F5 presents its client certificate

After the F5 team applies their Server SSL profile change:

1. Ensure the CA that issued the F5's client cert is in Keycloak's truststore (append to the bundle in KC_TRUSTSTORE_PATHS if it's a different CA). In request mode a client cert that fails validation aborts the handshake — coordinate cert + truststore in the same window if the CA is new.
2. Regression check: F5 CAC login still works.
3. Harvest the fingerprint (compare with what the F5 team reported):

   kubectl logs -n <ns> <keycloak-pod> | grep -o 'peer=subject=\[[^]]*\] sha256=[a-f0-9]*' | sort -u
   

Rollback: F5 team removes the client cert from the profile; everything else keeps working.

Phase 4 — Enforce

Add the trust rules and flip the mode:

  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CERT_SHA256
    value: "<sha256-from-phase-3>"          # comma-separate old,new during F5 cert rotations
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUSTED_PROXY_CIDRS
    value: "<f5-egress-cidr>"               # optional second factor; omit if IPs weren't preserved
  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE
    value: enforce

Verify all three, in order:

1. F5 path: CAC login through the F5 works; log shows using HEADER cert from trusted proxy (cert-fingerprint).
2. Direct path: TGW-side CAC login works; log shows using DIRECT TLS peer cert.
3. Negative test (the point of this whole exercise): from a TGW-side box, send a forged header and confirm it is rejected:

   curl -vk "https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth?client_id=account-console&response_type=code&redirect_uri=https://<keycloak-host>/realms/<realm>/account/" \
        -H "<header-name>: $(cat any-user-cert.pem | python3 -c 'import sys,urllib.parse;print(urllib.parse.quote(sys.stdin.read()))')"
   

   Expected: NO authenticated session, and the log shows IGNORED cert header from untrusted peer. If instead you see the login succeed, stop — the trust rules are wrong; re-check the fingerprint and revert to log.

Rollback: KC_SPI_X509CERT_LOOKUP_HYBRID_TRUST_MODE=log (keeps both paths working while you investigate), or provider back to nginx (F5 path only, pre-project behavior).

Phase 5 — Quiet down (housekeeping)

  - name: KC_SPI_X509CERT_LOOKUP_HYBRID_VERBOSE
    value: "false"
  - name: KC_LOG_LEVEL
    value: INFO

Also confirm (once, not per-deploy):

• OCSP/CRL revocation checking is enabled in the realm's X509 authenticator config (Authentication → browser x509 flow → config). The F5 pre-screens users on its path; the direct path relies entirely on this setting.
• A calendar/reminder exists for the F5 client-cert expiry: add the next cert's fingerprint to TRUSTED_PROXY_CERT_SHA256 (comma-separated) before rotation.

---

If something breaks

Symptom	Likely cause	Action
F5 logins fail right after Phase 2 deploy	header name mismatch (inheritance picked wrong value)	check the initialized/inherited startup lines; set KC_SPI_X509CERT_LOOKUP_HYBRID_SSL_CLIENT_CERT explicitly
F5 logins fail right after Phase 3	F5 client cert fails TLS validation (CA missing from truststore)	add issuing CA to the bundle, restart; or F5 team pulls the cert
F5 logins fail right after Phase 4	fingerprint/CIDR wrong or F5 rotated its cert	revert TRUST_MODE=log, re-harvest fingerprint from logs
TGW users get no cert prompt	truststore CAs don't cover their issuing CA (browser filters the picker), or TLS isn't reaching Keycloak at L4	verify CA bundle; verify NLB listener is TCP passthrough, not TLS
TGW users get a prompt but login fails	user mapping — cert attribute doesn't resolve to a Keycloak user	check the X509 authenticator's user-identity mapping config
Everything on fire	—	KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx + redeploy = exact pre-project behavior

Full log-line reference: README §7. Every decision the provider makes is logged with the x509-hybrid: prefix while verbose=true.