Building a Kubernetes Operator for Workload Identity Federation Across AWS and GCP

The problem: federation is easy to explain and tedious to wire

Most clusters I've worked on have at least one pod with an AWS access key or a GCP service-account JSON key sitting in a Kubernetes Secret. Someone minted it, someone copied it in, and nobody is sure when it last rotated.

Clouds providers have had a better answer for years. AWS has AssumeRoleWithWebIdentity and GCP has Workload Identity Federation. The workload presents a JWT from an issuer the cloud trusts and gets back credentials that expire in minutes. The catch is the setup.

For one pod in namespace payments that needs S3 in one AWS account and GCS in one GCP project, you need:

1. An issuer that will mint a token for that specific workload. Usually that's the cluster's own ServiceAccount token issuer (its OIDC discovery endpoint), or an external IdP such as Keycloak, Auth0 or SPIRE.
2. That issuer registered as an IAM OIDC provider in the AWS account, with the right values in its ClientIDList.
3. A statement in the IAM role's trust policy that matches this workload's claims and nobody else's.
4. A Workload Identity Pool and Provider in the GCP project, with an attributeMapping that turns a token claim into google.subject.
5. A roles/iam.workloadIdentityUser binding on the target GCP service account for that workload's principal.
6. Teardown of every one of those when the workload goes away.

Now multiply that by namespaces, clusters and accounts. I tried Terraform first. It's a poor fit here because the inputs (ServiceAccounts) are created by app teams at deploy time, not by the platform pipeline. And the drift I care about, like someone editing a trust policy in the console, happens outside any pipeline. The cluster already knows which workloads exist, so I made the cluster reconcile everything else.

Myceliam is a Kubernetes operator that makes the federated path the easy one.

Three systems that don't know about each other

Kubernetes trusts its own ServiceAccount tokens. IDPs (Identiy providers) Keycloak, Auth0, and SPIRE each issue JWTs they're willing to vouch for. Public clouds like AWS and GCP will happily accept a federated token — from an identity provider they've already been told to trust. Each of those three systems already knows how to trust something. None of them know about each other by default.

Myceliam is named after mycelium, the hidden underground fungal network that connects separate organisms and lets them trade nutrients. In the same way, Myceliam is a Kubernetes operator that weaves identity threads between clusters, identity providers and clouds. It watches namespace and ServiceAccount events and creates the matching resources in IdPs and cloud IAM on demand, so workloads can exchange short-lived credentials to access cloud resources.

The design: map Kubernetes objects onto IdP and cloud objects

The core idea is a fixed mapping from Kubernetes constructs to identity-provider and cloud constructs. Everything is derived from names, so the operator never needs a database:

Myceliam operator supports SPIFFE, Keycloak and Auth0 for IdPs, AWS and GCP for cloud providers.

M

How it actually works

The user-facing interface is deliberately small: labels on a ServiceAccount, plus a namespaced CRD per cloud that lists which roles or service accounts the namespace may federate into.

apiVersion: myceliam.io/v1 kind: AutoidpAllowlist # CRD 1: the namespaces the operator may provision metadata: name: default namespace: myceliam-system namespaces: - payments - inventory --- apiVersion: myceliam.io/v1 kind: AwsAccessProfile # CRD 2: AWS accounts, and the IAM roles in each metadata: name: awsap-payments namespace: payments accounts: "123456789012": - s3-reader --- apiVersion: myceliam.io/v1 kind: GcpAccessProfile # CRD 3: GCP projects, and the service accounts in each metadata: name: gcpap-payments namespace: payments projects: "project-2f04607": - app-default@project-2f04607.iam.gserviceaccount.com --- apiVersion: v1 kind: ServiceAccount metadata: name: checkout namespace: payments labels: myceliam.io/autoidp: "true" myceliam.io/oidc-provider: "keycloak" # or auth0; omit for SPIFFE myceliam.io/client-type: "secret" # or signedjwt myceliam.io/aws-access-profile: "awsap-payments" # roles this workload may assume myceliam.io/gcp-access-profile: "gcpap-payments" # service accounts it may impersonate 

Four decisions shape everything after this:

A label alone is not enough. A single AutoidpAllowlist object (myceliam-system/default) lists the namespaces the operator may provision. If it doesn't exist, every namespace is allowed, so a first install works with zero config. Once it exists, only listed namespaces get new provisioning, and removing a namespace from it never blocks cleanup of what's already there. Who can write that one object, controlled with ordinary RBAC, is the real security boundary. Without it, anyone who can create a ServiceAccount could trigger provisioning in an external IdP and cloud account.

Trust, not authorization. The operator registers issuers and writes trust statements. It never attaches a permissions policy or grants an IAM role. If a role or GCP service account listed in a profile doesn't exist yet, the operator creates it with no permissions, as long as MYCELIAM_AUTO_CREATE_PRINCIPAL is true (the default). Auto-creation exists because AWS won't create a role with an empty trust policy, so pre-provisioning meant hand-writing exactly the statement the operator writes anyway. Teams that want to own role creation can set the flag to false, and a missing principal then fails that ServiceAccount's reconcile with a terminal error instead. GCP's default compute service accounts can't be created this way, so those must already exist. Either way, the operator's own credentials need permission to create roles and service accounts. Keeping the operator out of "what can this identity do" limits its blast radius to "who can assume what".

No per-SA role selection inside a profile. Every ServiceAccount that references a profile gets the same targets. If you need finer access, you split the profile. That keeps the CRD trivial to read in a code review.

An issuer in the middle, not just the cluster's own. Every Kubernetes cluster already runs an OIDC issuer for its ServiceAccount tokens, and you can register it directly with AWS and GCP. That works, but the cluster signs every namespace's tokens with one key set, and every cluster's issuer has to be registered in every target cloud account, which grows into a mesh. Putting a Keycloak realm per namespace, an Auth0 tenant or a SPIRE trust domain in between gives per-namespace signing keys (with Keycloak) and turns the mesh into hub and spoke: the clouds trust a handful of issuers instead of every cluster.

From there, the operator takes over — and keeps taking over, on a timer, not just once:

That reconcile loop is the part that matters most. The CRDs aren't a setup script you run once — they're the desired state, continuously enforced. Someone rotates a client secret by hand in the Keycloak console, or edits a trust policy directly in the AWS console? The next reconcile (default: every 300 seconds) puts it back. There's no drift to notice three weeks later during an audit.

One pattern, three identity sources today

The detail worth sitting with: Myceliam is only in steps one and two. It never proxies, mints, or even sees the workload's actual cloud credentials — once the trust is registered, the workload talks to the identity provider and to AWS directly, and asks for its own short-lived token at request time. The operator's job stops at making sure that handshake can happen.

Bootstrap the operator

The operator needs admin access to Keycloak or Auth0 and credentials in every cloud account. It would be ironic to solve that with a static key, but there's a real chicken-and-egg problem.

• IdP admin: the operator never generates its own credential. It reads one of two pre-created secrets, a client secret or a kubernetes.io/tls keypair (the keypair wins if both exist). With Keycloak's single-certificate signed-JWT mode, sending a kid actually breaks authentication ("Unable to load public key"), so the operator omits it. Auth0 is the opposite: it assigns its own kid, and you have to supply it in config.
• Cloud access: the operator exchanges its own IdP token (or a SPIRE JWT-SVID) for credentials. On AWS it assumes a pre-provisioned myceliam-operator role in each account. On GCP it does an STS exchange and then impersonates a hub service account. Those bootstrap trusts are created once by an admin, and the operator never modifies them.
• An accepted trade-off: with Auth0, one Application holds both the Management API audience and the cloud-federation audience. Because AWS keys off azp, the two tokens are interchangeable as far as STS is concerned, so they can't be revoked independently. I accepted this because compromising that credential already means full Auth0 tenant admin, but I documented it rather than pretending it's not there.

Declare identity in Kubernetes, reconcile them as identity resources in Keycloak and AWS

Establish trust in AWS IAM

Workload gets AWS credentials using a Keycloak-issued access token

Drift correction and teardown

Building the reconcile loop on controller-runtime

The operator is written in Go on controller-runtime, with one reconciler each for ServiceAccounts, Namespaces, the two AccessProfile CRDs, credential Secrets and the allowlist. Four pieces of plumbing ended up mattering more than I expected.

1. Retryable vs. terminal errors. Every call to Keycloak, Auth0, IAM or GCP can fail in one of two ways. A missing config or a malformed label is permanent: retrying won't fix it. A 503 or "client not created yet" is temporary. I defined two error types, operr.Permanent and operr.Temporary (which carries its own retry delay), and translate them into a ctrl.Result in one place:

func classify(ctx context.Context, err error, periodicRequeue time.Duration) (ctrl.Result, error) { if err == nil { return ctrl.Result{RequeueAfter: periodicRequeue}, nil // re-arm drift check } if temp, ok := operr.AsTemporary(err); ok { return ctrl.Result{RequeueAfter: temp.Delay}, nil } if _, ok := operr.AsPermanent(err); ok { // don't hot-loop; try again on the next change or periodic pass return ctrl.Result{RequeueAfter: periodicRequeue}, nil } return ctrl.Result{}, err } 

2. Finalizers. Without a finalizer, a deleted ServiceAccount is gone before you can clean up its Keycloak client and trust statements. ServiceAccounts, Namespaces and both AccessProfile CRDs get myceliam.io/finalizer, and cleanup runs before it's removed.

3. Knowing what changed. If someone removes the aws-access-profile label, the operator has to remove that SA's trust statements. But a reconcile only sees the current object, not the previous one. So after each successful reconcile, the operator stores the last-applied state in an annotation. It's scoped to the four fields that matter, so unrelated label changes don't churn it:

const lastReconciledAnnotation = "myceliam.io/last-reconciled-state" type reconciledState struct { ClientType string `json:"clientType,omitempty"` AwsProfile string `json:"awsProfile,omitempty"` GcpProfile string `json:"gcpProfile,omitempty"` SpiffeJWTIssuer string `json:"spiffeJwtIssuer,omitempty"` } 

4. Periodic resync. Drift on the IdP or cloud side produces no Kubernetes event. Every successful reconcile requeues itself after MYCELIAM_RECONCILE_INTERVAL_SECONDS (default 300), so a trust policy edited in the console or a secret rotated by hand in Keycloak gets put back within five minutes.

One interface for identity providers

Keycloak and Auth0 model the world very differently, so I pushed the differences behind one interface. This is trimmed from the real one:

type Provider interface { EnsureTenant(ctx context.Context, tenantID string) error DeleteTenant(ctx context.Context, tenantID string) error EnsureSecretClient(ctx context.Context, spec *models.ClientSpec) (clientID, clientSecret string, err error) EnsureSignedjwtClient(ctx context.Context, spec *models.ClientSpec, certPEM []byte, jwks, kid string) (clientID, actualKid string, err error) DeleteClient(ctx context.Context, tenantID, clientName string) error IssuerURL(tenantID string) (string, error) // Returns the value this client's tokens actually carry in `azp`. EnsureAccessScope(ctx context.Context, tenantID, clientName, scopeName, audience string) (azpValue string, err error) DeleteAccessScope(ctx context.Context, tenantID, clientName, scopeName string) error GetOperatorToken(ctx context.Context) (string, error) } 

tenantID is a neutral stand-in for "whatever this IdP isolates a namespace with". The odd return value on EnsureAccessScope exists because of an AWS behavior covered below.

Keycloak: make azp mean something

Each namespace gets a realm, and each ServiceAccount gets a client whose client_id is forced to the SA name. Keycloak always sets azp to the client_id, so every token's azp claim reliably names the ServiceAccount. I didn't need a custom mapper for that, and it turned out to be the most useful property in the whole system.

Cloud audiences come from client scopes. The operator creates one scope per (cloud, SA), named <cloud>-<sa>, with an audience mapper, and attaches it as an optional scope. The name doesn't need the namespace in it, because a realm already is one namespace.

A token carries the AWS audience only when the workload explicitly asks for it, and the example app fetches one token per cloud:

awsScope := fmt.Sprintf("aws-%s", clientID) // client_id == the SA name gcpScope := fmt.Sprintf("gcp-%s", clientID) form := url.Values{ "grant_type": {"client_credentials"}, "client_id": {clientID}, "client_secret": {clientSecret}, "scope": {awsScope}, // a second request uses gcpScope } 

Auth0: no realms and no predictable client IDs

Auth0 has one tenant per deployment, assigns its own opaque client_id, and Resource Servers (APIs) aren't something you want one of per ServiceAccount. So the Auth0 provider uses one shared Resource Server per (cloud, cluster, namespace), and the per-SA Client Grants are what actually scope access.

That leaves a problem: nothing in an Auth0 token says which ServiceAccount it belongs to. The fix is a tenant-wide Action on the credentials-exchange trigger that copies values the operator stores in each Application's metadata into the token:

exports.onExecuteCredentialsExchange = async (event, api) => { const md = event.client.metadata || {}; const claims = { service_account: 'myceliam_service_account', namespace: 'myceliam_namespace', cluster_id: 'myceliam_cluster_id', cloud: 'myceliam_cloud', }; const principalTags = {}; for (const key of Object.keys(claims)) { if (md[key]) { api.accessToken.setCustomClaim(claims[key], md[key]); principalTags[claims[key]] = [md[key]]; } } if (Object.keys(principalTags).length > 0) { api.accessToken.setCustomClaim('https://aws.amazon.com/tags', { principal_tags: principalTags, }); } }; 

The operator manages the Action like any other resource. It creates it if missing, compares the deployed code to the embedded script on every reconcile, and re-deploys if they differ. The aws.amazon.com/tags block looks odd here, and the AWS section explains it.

Signed-JWT clients and cert-manager

For client-type: signedjwt, the workload authenticates with private_key_jwt (RFC 7523) instead of a shared secret. If a kubernetes.io/tls secret already exists (for example from cert-manager), the operator reads tls.crt. Otherwise it generates an RSA keypair and a self-signed cert. Either way it pushes the public key to the IdP, and an edited tls.crt is re-uploaded on the next reconcile.

Two details cost me time. First, Keycloak takes a JWKS, but Auth0's credentials endpoint takes a raw PEM and assigns its own kid (an RFC 7638 thumbprint). The interface therefore returns actualKid, and callers persist that rather than the value they computed. Second, I write client_id and kid to a separate -metadata secret so the TLS secret stays a pure kubernetes.io/tls secret that cert-manager is happy to own.

AWS: trust policies, and what STS actually checks

Registering the IdP is a get-or-create on the IAM OIDC provider, followed by adding whichever ClientIDList entries are missing. The interesting part is the role trust policy.

Editing someone else's trust policy safely

A role may already trust ec2.amazonaws.com, or another namespace's workload. The operator must touch only its own statement. I model just enough of the policy document to find statements by Sid, and I derive the Sid deterministically, because IAM only allows alphanumerics there and namespaces and SPIFFE IDs are full of hyphens and slashes:

func roleTrustStatementID(issuerURL, subjectClaim, subjectValue string) string { sum := sha256.Sum256([]byte(issuerURL + "|" + subjectClaim + "|" + subjectValue)) return "myceliam" + hex.EncodeToString(sum[:])[:16] } 

upsertStatement replaces the matching Sid in place, or appends it, and returns false if nothing changed, so no write happens.

Bug I shipped: that "nothing changed" check used reflect.DeepEqual against the decoded existing policy. A JSON array decoded into any comes back as []any, but I was building conditions as []string. They never compared equal, so every reconcile rewrote every trust policy. The fix was to build the statement with []any throughout. It's an easy bug to miss because everything still works. It just isn't idempotent.

Gotcha 1: when azp is present, AWS ignores aud

This one cost the most debugging time. When a token carries an azp claim, AssumeRoleWithWebIdentity checks the OIDC provider's ClientIDList against azp, not aud. Inside the trust policy, the <issuer>:aud condition key also resolves to azp's value. I confirmed live that this happens even when aud is a single string. Get it wrong and you get a bare InvalidIdentityToken or AccessDenied with no hint, even when the signature, issuer and thumbprint are all correct.

Keycloak and Auth0 client-credentials tokens always carry azp. SPIRE JWT-SVIDs never do. So the operator writes an OR-match on :aud that accepts either shape. For Auth0 and SPIFFE it also adds a separate identity condition, because the namespace-shared audience doesn't prove which ServiceAccount is calling:

audMatches := []any{audience} if azpValue != "" && azpValue != audience { audMatches = append(audMatches, azpValue) } condition := map[string]map[string]any{ "ForAnyValue:StringEquals": {hostAndPath + ":aud": audMatches}, } if subjectClaim != "azp" { // Keycloak is covered by azp-in-:aud; see below subjectKey := hostAndPath + ":" + subjectClaim if subjectClaimViaSessionTag { subjectKey = "aws:RequestTag/" + subjectClaim // see gotcha 3 } condition["ForAnyValue:StringEquals"][subjectKey] = subjectValue } 

Gotcha 2: a multi-valued aud without azp is rejected

With no azp, AWS falls back to aud and refuses a token with more than one audience (Token audience contains more than one audience while authorized party is not present). GCP would accept one combined token, but AWS won't. A SPIFFE workload that needs both clouds therefore fetches two single-audience JWT-SVIDs. A Keycloak token could safely carry both audiences, because azp is always there, but all three example apps fetch one token per cloud so they work the same way.

Gotcha 3: AWS can't see claims an Auth0 Action adds

A condition on <issuer>:myceliam_service_account never matched, even though the claim was in the JWT with exactly the right value. :sub and :aud conditions on the same token worked. My best explanation is that AWS only exposes claims listed in the provider's discovery document (claims_supported), and a claim injected at runtime never appears there.

The workaround is AWS session tags. The Action writes the same values into the reserved https://aws.amazon.com/tags claim. The trust policy then matches aws:RequestTag/myceliam_service_account and grants sts:TagSession alongside sts:AssumeRoleWithWebIdentity. I verified it in both directions: a matching value succeeds, a mismatched one fails, and a Deny keyed on the tag really blocks calls. Two traps here:

• Every tag value must be an array with exactly one string. A genuine multi-element array makes AWS reject the whole token with InvalidIdentityToken.
• Forgetting sts:TagSession produces a generic AccessDenied, not anything that mentions tagging.

Gotcha 4: trailing slashes are part of the issuer

Auth0's iss is https://{domain}/ with a trailing slash. The IAM OIDC provider URL must match exactly, and AWS doesn't normalize it.

One audience scheme for every provider and cloud

My first version gave each Keycloak ServiceAccount its own AWS audience (aws-<namespace>-<sa>), a namespace-wide one for GCP, and a third convention for Auth0. Three schemes for one concept were hard to reason about and harder to explain. Audiences are now always <cloud>-<cluster-id>-<namespace>: shared by every SA in the namespace, and never shared across clouds or clusters, because both are part of the string. That only works because the audience is no longer what proves identity. On AWS, the per-SA check is azp folded into :aud for Keycloak, the session-tag condition for Auth0, and sub for SPIFFE. On GCP it's google.subject plus the per-principal binding.

Known sharp edge: trust-policy size

Every SA adds a statement to each role it targets. A popular role eventually hits the trust-policy size quota (LimitExceeded: Cannot exceed quota for ACLSizePerRole). The operator doesn't prune stale statements yet. That's next on my list.

GCP: pools, providers, and a 32-character ID

I call the IAM REST API directly rather than using the Go SDK, mainly to avoid Application Default Credentials. The operator already has its own short-lived GCP token and just uses it as a bearer token.

Each GCP project gets one shared Workload Identity Pool, and each namespace gets one provider per identity source inside it. Provider IDs must be 4–32 lowercase characters, and the cluster ID is part of the name, so truncation needs care:

func providerID(namespace, kind, workloadProvider string, s *config.Settings) string { suffix := "-" + kind if kind == KindOIDC && workloadProvider != "" { suffix = "-" + workloadProvider // "-keycloak", "-auth0" } base := s.ClusterID + "-" + namespace if maxBaseLen := 32 - len(suffix); len(base) > maxBaseLen { base = base[:maxBaseLen] // reserve room for the suffix first } return strings.TrimRight(base, "-") + suffix } 

GCP details that mattered:

• Mapping identity. attributeMapping sets google.subject from assertion.azp for Keycloak, assertion.myceliam_service_account for Auth0 (GCP can read Action-injected claims, unlike AWS), and assertion.sub for SPIFFE. The IAM binding then targets principal://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<pool>/subject/<value>. Note that this needs the project number, so there's a lookup step.
• Audience doesn't narrow identity here. GCP narrows identity through google.subject and the binding, not aud, so the shared gcp-<cluster-id>-<namespace> audience costs nothing.
• Embedded JWKS. For Keycloak and Auth0 providers, the operator fetches the issuer's discovery document and JWKS and embeds the keys in the provider (jwksJson). This avoids depending on GCP reaching the issuer itself, and because the provider is PATCHed on every reconcile, rotated keys propagate within one interval. SPIFFE providers rely on normal discovery.
• Soft deletes. A deleted pool or provider lingers in DELETED state and blocks re-creation under the same ID. The operator calls :undelete, which is an asynchronous long-running operation. Rather than chain the next call onto it, the operator returns a temporary error with a 15-second requeue and lets the next pass continue.

SPIFFE: the path with nothing to create

If SPIRE already issues identities, there's no IdP object to create. The operator only registers the SPIRE issuer with AWS and GCP and uses sub, the full SPIFFE ID such as spiffe://example.org/ns/payments/sa/checkout, as the identity claim. It's the simplest path in the codebase and the best illustration of the operator's actual job: it doesn't issue identity, it makes existing identity legible to the clouds.

What the workload does with all this

The operator is out of the picture once trust exists. It never proxies, mints or sees cloud credentials. The workload gets a token from its IdP and exchanges it directly. On AWS, no ambient credentials are needed for the STS call:

stsClient := sts.NewFromConfig(aws.Config{ Region: "us-east-1", Credentials: aws.AnonymousCredentials{}, }) assumed, err := stsClient.AssumeRoleWithWebIdentity(ctx, &sts.AssumeRoleWithWebIdentityInput{ RoleArn: aws.String(roleARN), RoleSessionName: aws.String("checkout"), WebIdentityToken: aws.String(jwt), }) 

On GCP it's two hops: an RFC 8693 token exchange at sts.googleapis.com, then generateAccessToken to impersonate the target service account:

payload, _ := json.Marshal(map[string]any{ "audience": gcpSTSAudience, // the WLI provider's full resource name "grantType": "urn:ietf:params:oauth:grant-type:token-exchange", "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token", "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt", "subjectToken": jwt, "scope": "https://www.googleapis.com/auth/cloud-platform", }) 

The repo has two small example apps, one OIDC and one SPIFFE, that do exactly this and list real S3 and GCS buckets. I used them as my end-to-end smoke test after every change.

What I'd tell someone building something similar

• Decode your tokens before you debug your policies. Almost every AWS failure I hit came down to which claims the token actually carried (azp present or not, aud single or multi-valued) and how STS maps them to condition keys. The error messages won't tell you.
• Change one variable at a time against the real service. InvalidIdentityToken and AccessDenied cover a dozen different causes. Fakes in unit tests are necessary but they encode your assumptions, which is exactly what was wrong.
• Make every external write idempotent, and compare in the types you decode into. The []string vs []any bug is a reminder that "it works" and "it converges" are different properties.
• Know which external objects are shared before you delete anything. Most of my worst bugs were cleanup bugs, not creation bugs.
• Put a cluster ID in every name and stamp ownership. It's cheap to add on day one and painful to add later.
• Keep the controller out of authorization. Registering trust is a narrow, auditable job. "What can this identity do" already has a mature home in each cloud's IAM.
• Single-instance operators need Recreate. Or use leader election. Either way, don't assume replicas: 1 means one process.

Limitations and what's next

This is early-stage software. Known gaps: no automatic pruning of stale trust-policy statements, no expiry-based rotation for operator-generated signed-JWT keypairs (rotation happens only when tls.crt changes), single-instance operation only, and SPIRE must already be installed. Cloud providers are still parallel packages rather than an interface like the IdP side. Tightening that is the prerequisite for adding Azure and OCI.

Myceliam on GitHub.

Demo: