One model gateway across clouds: LiteLLM on Cloud Run and Azure with zero API keys

As of November 2026. All identifiers in this post are examples.

We built a proof of concept with two AI agents: one on Google Cloud, in Agent Runtime, and one on Azure, in Microsoft Foundry. Each agent has its own identity in Microsoft Entra Agent ID and stores no secret.

Agents call models. This post covers how ours reach them: a model gateway that runs in both clouds, serves Gemini and Azure OpenAI from either side, and holds no API key for any of it.

During a demo we switched off the gateway in one cloud. The agent's next answer came through the other one, and we barely noticed the switch.

Why put a gateway between agents and models

An agent can call a model directly. With two agents that is fine. With fifty it gets uncomfortable:

• Every agent needs credentials for every model provider it uses.
• There is no single place that says which agent may use which model.
• Usage is spread across two clouds and two billing consoles.

A model gateway fixes the three at once. Agents talk to one endpoint, and the rules and the usage records live there. We used LiteLLM because it speaks to both Vertex AI and Azure OpenAI behind one OpenAI-compatible API.

The catch is that a gateway usually introduces two new kinds of secret: the keys it holds for the model providers, and the keys it hands out to its callers. We wanted neither.

The design

There is one LiteLLM in each cloud, with the same configuration:

• On Google Cloud it runs on Cloud Run.
• On Azure it runs on Container Apps.
• Each one has a Redis sidecar in the same service, for its own short-lived state.
• Both use one shared Postgres database, on Cloud SQL, for agents, allowed models and usage.

Each gateway is connected to both model families. The one on Azure can call Gemini, and the one on Google Cloud can call Azure OpenAI. An agent uses the gateway of its own cloud and gets the same catalogue either way.

The model list is the same file on both sides:

model_list: - model_name: gpt-4.1-mini litellm_params: model: azure/gpt-4.1-mini api_base: https://<your-resource>.openai.azure.com/ - model_name: gemini-3.1-flash-lite litellm_params: model: vertex_ai/gemini-3.1-flash-lite vertex_project: <your-project> vertex_location: global 

There is no api_key line anywhere in it.

Agents sign in as themselves

LiteLLM normally gives each caller a virtual key. That is one more secret per agent, which is exactly what we wanted to avoid.

Each agent already has an identity in Microsoft Entra Agent ID and can get a token for any audience it has a role on. So the gateway is one more audience, a2a-llm-gateway, with one role, LLM.Invoke. The agent sends its own Entra token as the bearer.

On the LiteLLM side, a custom auth hook replaces the key lookup. Its core is short:

def _verify(token: str) -> dict: key = _JWKS.get_signing_key_from_jwt(token).key return jwt.decode(token, key, algorithms=["RS256"], audience=AUDIENCES, issuer=ISSUER, options={"require": ["exp", "iss", "aud"]}) async def user_api_key_auth(request, api_key): claims = await asyncio.to_thread(_verify, api_key) if "LLM.Invoke" not in (claims.get("roles") or []): raise Exception("forbidden: missing app role LLM.Invoke") team = await get_team_object(team_id=claims["azp"], ...) # the agent's ID model = await _requested_model(request) if model and team.models and model not in team.models: raise Exception(f"forbidden: model '{model}' not allowed for this agent") return UserAPIKeyAuth(team_id=claims["azp"], team_models=team.models, ...) 

It is wired in with one line of configuration:

general_settings: custom_auth: custom_auth.user_api_key_auth 

The caller ID in the token is the agent's identity, and LiteLLM's "team" is keyed by that same ID. So usage is recorded per agent without anyone inventing a second naming scheme.

Which agent may use which model

The allow-list comes from the inventory file we use to onboard agents:

- name: gcp-agent llm: gateway: gcp models: [gemini-3.1-flash-lite] 

The onboarding pipeline assigns the LLM.Invoke role in Entra and creates the team in LiteLLM with those models. An agent that asks for a model outside its list is refused before any provider is called.

The gateway holds no model keys either

This is the part that makes the design hold together. Every link from the gateway outwards uses the identity of the workload.

From Google Cloud to Gemini. The Cloud Run service account has the Vertex AI role. Nothing to configure.

From Google Cloud to Azure OpenAI. LiteLLM asks Google for an OIDC token and presents it to Entra, where a federated credential on an app registration accepts it. In the model entry it is one line:

 azure_ad_token: oidc/google/api://AzureADTokenExchange 

From Azure to Azure OpenAI. The Container App's managed identity.

From Azure to Gemini. This is the reverse federation. The managed identity's token is exchanged at a Google workload identity pool, which lets it act as the same service account the Cloud Run side uses. Google's client libraries do the exchange when they find this credential file:

{ "type": "external_account", "audience": "//iam.googleapis.com/projects/<NUMBER>/locations/global/workloadIdentityPools/azure-workloads/providers/<PROVIDER>", "subject_token_type": "urn:ietf:params:oauth:token-type:jwt", "token_url": "https://sts.googleapis.com/v1/token", "credential_source": { "file": "/var/run/fed/azure-token" }, "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/litellm-runtime@<PROJECT>.iam.gserviceaccount.com:generateAccessToken" } 

A small sidecar keeps the Azure token in that file fresh. The pool accepts only that one managed identity as a subject.

To the database. Both gateways reach Cloud SQL through the Cloud SQL Auth Proxy, as a sidecar, with IAM database authentication:

cloud-sql-proxy --auto-iam-authn --port=5432 <project>:<region>:<instance> 

The database user is the service account. There is no password in the connection string.

When a gateway goes down

Once every agent in a cloud depends on that cloud's gateway, the gateway is a single point of failure for all of them. So one thing has to be guaranteed: an agent keeps getting answers when the gateway of its own cloud is unavailable.

Two properties of this design make that possible:

• The same token works on both gateways. They share the audience, the role and the database. An agent needs nothing new to be served by the other cloud.
• Either gateway reaches both model families. A model provider being down and a gateway being down are separate failures, and neither one strands an agent.

There is also a rule worth stating: a refusal is not an outage. A 401, a 403 or a 429 means the gateway is working and said no. Moving to the other cloud must never be a way around that answer.

In our test, with the Azure gateway switched off, the Azure agent's next answer came through the Google Cloud gateway. Nobody touched anything, and the switch was barely noticeable.

What we learned the hard way

• Let one side own the schema. Two LiteLLM instances migrating the same database at startup is a race. One runs the migration. The other starts with DISABLE_SCHEMA_UPDATE=true.
• Custom auth failures come back as 401. LiteLLM maps exceptions from the hook that way, even for what is logically a 403. Clients should read the message, not only the code.
• Pin the image version. The custom auth hook imports LiteLLM internals. An unpinned upgrade can break sign-in for every agent at once.
• Turn off message logging. With turn_off_message_logging: true the database keeps who called what and how much, and never the prompts.

What this design asks of you

• Per-minute limits are per cloud. Rate counters live in Redis, and each cloud has its own. A limit of 60 requests a minute is 60 on each side.
• The shared database lives in one cloud. Both gateways depend on it. It is the first thing to look at when taking this to production.
• The gateway still has an admin key. LiteLLM needs a master key for its own administration. It stays in each cloud's secret manager, read by the gateway's identity. No agent ever sees it, and it is not part of any request between an agent and a model.

Wrapping up

• One gateway per cloud, same configuration, same catalogue.
• Agents authenticate with the identity they already have.
• The gateway reaches both model providers and its database with workload identity.
• An agent can be served from either cloud with the same token.

Between an agent and a model there is no API key to rotate.