Deploying LiteLLM on Kubernetes

Introduction

I have access to LLM APIs from several different providers. I use Hermes for all kinds of tasks, and I do not want to edit the Hermes configuration every time I switch models. So I set up LiteLLM and put the models from those different providers behind one OpenAI-compatible API. Hermes only needs to be configured once. After that, when I want to switch the underlying model, I just change the model mapping in the LiteLLM admin UI.

I first got everything working locally and confirmed that the models could actually be called. Then I moved it to Kubernetes and connected PostgreSQL, Redis, Traefik, and ArgoCD. This post documents the full setup and a few problems I ran into along the way.