feat(litellm): install LiteLLM

2025-12-03 23:05:23 +09:00
parent 46fdff720f
commit 5055a36d87
7 changed files with 993 additions and 0 deletions
--- a/litellm/.gitignore
+++ b/litellm/.gitignore
@@ -0,0 +1,3 @@
+litellm-values.yaml
+apikey-external-secret.yaml
+models.yaml
--- a/litellm/README.md
+++ b/litellm/README.md
@@ -0,0 +1,349 @@
+# LiteLLM
+
+Unified LLM gateway and proxy for accessing multiple LLM providers through a single OpenAI-compatible API:
+
+- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, Cohere, Azure, Bedrock, Vertex AI
+- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK
+- **Load Balancing & Fallback**: Automatic failover between providers
+- **Virtual Keys**: Generate API keys for users with usage tracking
+- **Cost Tracking**: Monitor spending across providers
+- **Rate Limiting**: Control usage per key/user
+
+## Prerequisites
+
+- Kubernetes cluster (k3s)
+- External Secrets Operator (required)
+- PostgreSQL cluster (CloudNativePG)
+- Vault for secrets management
+
+## Configuration Overview
+
+LiteLLM requires two types of configuration:
+
+1. **Environment variables** (`.env.local`): Host, namespace, chart version
+2. **Model definitions** (`models.yaml`): LLM providers and models to expose
+
+This separation allows flexible model configuration without modifying environment files.
+
+## Installation
+
+### Step 1: Create Model Configuration
+
+Copy the example configuration and customize:
+
+```bash
+cp litellm/models.example.yaml litellm/models.yaml
+```
+
+Edit `litellm/models.yaml` to configure your models:
+
+```yaml
+# Anthropic Claude
+- model_name: claude-sonnet
+  litellm_params:
+    model: anthropic/claude-3-7-sonnet-latest
+    api_key: os.environ/ANTHROPIC_API_KEY
+
+# OpenAI
+- model_name: gpt-4o
+  litellm_params:
+    model: openai/gpt-4o
+    api_key: os.environ/OPENAI_API_KEY
+
+# Ollama (local models - no API key required)
+- model_name: llama3
+  litellm_params:
+    model: ollama/llama3.2
+    api_base: http://ollama.ollama:11434
+```
+
+### Step 2: Set API Keys
+
+For each provider that requires an API key:
+
+```bash
+just litellm::set-api-key anthropic
+just litellm::set-api-key openai
+```
+
+Or interactively select the provider:
+
+```bash
+just litellm::set-api-key
+```
+
+API keys are stored in Vault and synced to Kubernetes via External Secrets Operator.
+
+### Step 3: Install LiteLLM
+
+```bash
+just litellm::install
+```
+
+You will be prompted for:
+
+- **LiteLLM host (FQDN)**: e.g., `litellm.example.com`
+- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed
+
+## Model Management
+
+### Add a Model Interactively
+
+```bash
+just litellm::add-model
+```
+
+This guides you through:
+
+1. Selecting a provider
+2. Choosing a model
+3. Setting a model alias
+
+### Remove a Model
+
+```bash
+just litellm::remove-model
+```
+
+### List Configured Models
+
+```bash
+just litellm::list-models
+```
+
+### Example Output
+
+```text
+Configured models:
+  - claude-sonnet: anthropic/claude-3-7-sonnet-latest
+  - claude-haiku: anthropic/claude-3-5-haiku-latest
+  - llama3: ollama/llama3.2
+```
+
+## API Key Management
+
+### Set API Key for a Provider
+
+```bash
+just litellm::set-api-key anthropic
+```
+
+### Get API Key (from Vault)
+
+```bash
+just litellm::get-api-key anthropic
+```
+
+### Verify All Required Keys
+
+```bash
+just litellm::verify-api-keys
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+| -------- | ------- | ----------- |
+| `LITELLM_NAMESPACE` | `litellm` | Kubernetes namespace |
+| `LITELLM_CHART_VERSION` | `0.1.825` | Helm chart version |
+| `LITELLM_HOST` | (prompt) | External hostname (FQDN) |
+| `OLLAMA_NAMESPACE` | `ollama` | Ollama namespace for local models |
+| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor |
+
+## API Usage
+
+LiteLLM exposes an OpenAI-compatible API at `https://your-litellm-host/`.
+
+### Get Master Key
+
+```bash
+just litellm::master-key
+```
+
+### Generate Virtual Key for a User
+
+```bash
+just litellm::generate-virtual-key user@example.com
+```
+
+### OpenAI SDK Example
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="https://litellm.example.com",
+    api_key="sk-..."  # Virtual key or master key
+)
+
+response = client.chat.completions.create(
+    model="claude-sonnet",  # Use your model alias
+    messages=[{"role": "user", "content": "Hello!"}]
+)
+print(response.choices[0].message.content)
+```
+
+### curl Example
+
+```bash
+curl https://litellm.example.com/v1/chat/completions \
+  -H "Authorization: Bearer sk-..." \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "claude-sonnet",
+    "messages": [{"role": "user", "content": "Hello!"}]
+  }'
+```
+
+## Supported Providers
+
+| Provider | Model Prefix | API Key Required |
+| -------- | ------------ | ---------------- |
+| Anthropic | `anthropic/` | Yes |
+| OpenAI | `openai/` | Yes |
+| Ollama | `ollama/` | No (uses `api_base`) |
+| Mistral | `mistral/` | Yes |
+| Groq | `groq/` | Yes |
+| Cohere | `cohere/` | Yes |
+| Azure OpenAI | `azure/` | Yes |
+| AWS Bedrock | `bedrock/` | Yes |
+| Google Vertex AI | `vertexai/` | Yes |
+
+## Architecture
+
+```plain
+External Users/Applications
+      |
+Cloudflare Tunnel (HTTPS)
+      |
+Traefik Ingress (HTTPS)
+      |
+LiteLLM Proxy (HTTP inside cluster)
+  |-- PostgreSQL (usage tracking, virtual keys)
+  |-- Redis (caching, rate limiting)
+  |-- External Secrets (API keys from Vault)
+      |
+      +-- Anthropic API
+      +-- OpenAI API
+      +-- Ollama (local)
+      +-- Other providers...
+```
+
+## Upgrade
+
+After modifying `models.yaml` or updating API keys:
+
+```bash
+just litellm::upgrade
+```
+
+## Uninstall
+
+```bash
+just litellm::uninstall
+```
+
+This removes:
+
+- Helm release and all Kubernetes resources
+- Namespace
+- External Secrets
+
+**Note**: The following resources are NOT deleted:
+
+- PostgreSQL database (use `just postgres::delete-db litellm`)
+- API keys in Vault
+
+### Full Cleanup
+
+To remove everything including database and Vault secrets:
+
+```bash
+just litellm::cleanup
+```
+
+## Troubleshooting
+
+### Check Pod Status
+
+```bash
+kubectl get pods -n litellm
+```
+
+Expected pods:
+
+- `litellm-*` - LiteLLM proxy
+- `litellm-redis-master-0` - Redis instance
+
+### View Logs
+
+```bash
+kubectl logs -n litellm deployment/litellm --tail=100
+```
+
+### API Key Not Working
+
+Verify the ExternalSecret is synced:
+
+```bash
+kubectl get externalsecret -n litellm
+kubectl get secret apikey -n litellm -o yaml
+```
+
+### Model Not Found
+
+Ensure the model is configured in `models.yaml` and the deployment is updated:
+
+```bash
+just litellm::list-models
+just litellm::upgrade
+```
+
+### Provider API Errors
+
+Check if the API key is set correctly:
+
+```bash
+just litellm::get-api-key anthropic
+```
+
+If empty, set the API key:
+
+```bash
+just litellm::set-api-key anthropic
+```
+
+### Database Connection Issues
+
+Check PostgreSQL connectivity:
+
+```bash
+kubectl exec -n litellm deployment/litellm -- \
+  psql -h postgres-cluster-rw.postgres -U litellm -d litellm -c "SELECT 1"
+```
+
+## Configuration Files
+
+| File | Description |
+| ---- | ----------- |
+| `models.yaml` | Model definitions (user-created, gitignored) |
+| `models.example.yaml` | Example model configuration |
+| `litellm-values.gomplate.yaml` | Helm values template |
+| `apikey-external-secret.gomplate.yaml` | ExternalSecret for API keys |
+
+## Security Considerations
+
+- **Pod Security Standards**: Namespace configured with **baseline** enforcement
+  (LiteLLM's Prisma requires write access to `/.cache`, which prevents `restricted` level)
+- **Secrets Management**: API keys stored in Vault, synced via External Secrets Operator
+- **Virtual Keys**: Generate scoped API keys for users instead of sharing master key
+- **TLS/HTTPS**: All external traffic encrypted via Traefik Ingress
+- **Database Credentials**: Unique PostgreSQL user with minimal privileges
+
+## References
+
+- [LiteLLM Documentation](https://docs.litellm.ai/)
+- [LiteLLM GitHub](https://github.com/BerriAI/litellm)
+- [LiteLLM Helm Chart](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm)
+- [Supported Models](https://docs.litellm.ai/docs/providers)
+- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys)
--- a/litellm/apikey-external-secret.gomplate.yaml
+++ b/litellm/apikey-external-secret.gomplate.yaml
@@ -0,0 +1,29 @@
+{{- $models := (datasource "models") -}}
+{{- $providerMap := dict -}}
+{{- range $models -}}
+{{- if has .litellm_params "api_key" -}}
+{{- $parts := strings.Split "/" .litellm_params.model -}}
+{{- $provider := index $parts 0 -}}
+{{- $providerMap = merge $providerMap (dict $provider true) -}}
+{{- end -}}
+{{- end -}}
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: apikey-external-secret
+  namespace: {{ .Env.LITELLM_NAMESPACE }}
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: vault-secret-store
+    kind: ClusterSecretStore
+  target:
+    name: apikey
+    creationPolicy: Owner
+  data:
+{{- range $provider, $_ := $providerMap }}
+  - secretKey: {{ $provider | strings.ToUpper }}_API_KEY
+    remoteRef:
+      key: litellm/{{ $provider }}
+      property: apikey
+{{- end }}
--- a/litellm/justfile
+++ b/litellm/justfile
@@ -0,0 +1,440 @@
+set fallback := true
+
+export LITELLM_NAMESPACE := env("LITELLM_NAMESPACE", "litellm")
+export LITELLM_CHART_VERSION := env("LITELLM_CHART_VERSION", "0.1.825")
+export LITELLM_HOST := env("LITELLM_HOST", "")
+export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
+export OLLAMA_NAMESPACE := env("OLLAMA_NAMESPACE", "ollama")
+export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring")
+export MONITORING_ENABLED := env("MONITORING_ENABLED", "")
+
+[private]
+default:
+    @just --list --unsorted --list-submodules
+
+# Create LiteLLM namespace
+create-namespace:
+    kubectl get namespace ${LITELLM_NAMESPACE} &>/dev/null || \
+        kubectl create namespace ${LITELLM_NAMESPACE}
+
+# Delete LiteLLM namespace
+delete-namespace:
+    kubectl delete namespace ${LITELLM_NAMESPACE} --ignore-not-found
+
+# Check prerequisites
+check-prerequisites:
+    #!/bin/bash
+    set -euo pipefail
+    if ! helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
+        echo "Error: External Secrets Operator is required but not installed."
+        echo "Please install External Secrets Operator first:"
+        echo "  just external-secrets::install"
+        exit 1
+    fi
+    if [ ! -f models.yaml ]; then
+        echo "Error: models.yaml not found."
+        echo "Please create models.yaml from the example:"
+        echo "  cp models.example.yaml models.yaml"
+        echo "Then edit models.yaml to configure your models."
+        exit 1
+    fi
+    echo "Prerequisites check passed."
+
+# Extract required providers from models.yaml
+[private]
+get-required-providers:
+    #!/bin/bash
+    set -euo pipefail
+    # Extract providers that require API keys (exclude ollama)
+    yq -r '.[] | select(.litellm_params.api_key != null) | .litellm_params.model' models.yaml | \
+        cut -d'/' -f1 | sort -u
+
+# Verify all required API keys are set in Vault
+verify-api-keys:
+    #!/bin/bash
+    set -euo pipefail
+    providers=$(just get-required-providers)
+    missing=()
+    for provider in $providers; do
+        if ! just vault::get "litellm/${provider}" apikey &>/dev/null; then
+            missing+=("$provider")
+        fi
+    done
+    if [ ${#missing[@]} -gt 0 ]; then
+        echo "Error: Missing API keys for the following providers:"
+        for p in "${missing[@]}"; do
+            echo "  - $p"
+        done
+        echo ""
+        echo "Please set the API keys:"
+        for p in "${missing[@]}"; do
+            echo "  just litellm::set-api-key provider=$p"
+        done
+        exit 1
+    fi
+    echo "All required API keys are configured."
+
+# Set API key for a provider
+set-api-key provider='':
+    #!/bin/bash
+    set -euo pipefail
+    provider="{{ provider }}"
+    if [ -z "${provider}" ]; then
+        available=$(just get-required-providers 2>/dev/null || echo "anthropic openai mistral groq cohere")
+        provider=$(echo "$available" | tr ' ' '\n' | gum choose --header="Select provider:")
+    fi
+    apikey=$(gum input --prompt="${provider} API key: " --password --width=80)
+    if [ -z "${apikey}" ]; then
+        echo "Error: API key cannot be empty"
+        exit 1
+    fi
+    just vault::put "litellm/${provider}" apikey="${apikey}"
+    echo "API key for ${provider} has been stored in Vault location 'litellm/${provider}'."
+
+# Get API key for a provider
+get-api-key provider='':
+    #!/bin/bash
+    set -euo pipefail
+    provider="{{ provider }}"
+    if [ -z "${provider}" ]; then
+        echo "Usage: just litellm::get-api-key provider=<provider>"
+        exit 1
+    fi
+    just vault::get "litellm/${provider}" apikey
+
+# Add a model interactively
+add-model:
+    #!/bin/bash
+    set -euo pipefail
+    if [ ! -f models.yaml ]; then
+        echo "Creating models.yaml from example..."
+        cp models.example.yaml models.yaml
+    fi
+
+    echo "Add a new model to LiteLLM"
+    echo ""
+
+    provider=$(gum choose --header="Select provider:" \
+        "anthropic" "openai" "ollama" "mistral" "groq" "cohere" "azure" "bedrock" "vertexai")
+
+    model_name=$(gum input --prompt="Model alias (e.g., claude-sonnet): " --width=60)
+    if [ -z "${model_name}" ]; then
+        echo "Error: Model name is required"
+        exit 1
+    fi
+
+    case $provider in
+        anthropic)
+            model=$(gum choose --header="Select Anthropic model:" \
+                "claude-sonnet-4-20250514" \
+                "claude-haiku-4-20251015" \
+                "claude-opus-4-20250514")
+            api_key_line="    api_key: os.environ/ANTHROPIC_API_KEY"
+            ;;
+        openai)
+            model=$(gum choose --header="Select OpenAI model:" \
+                "gpt-4o" \
+                "gpt-4o-mini" \
+                "o3" \
+                "o4-mini")
+            api_key_line="    api_key: os.environ/OPENAI_API_KEY"
+            ;;
+        ollama)
+            model=$(gum input --prompt="Ollama model name: " --width=60 --placeholder="qwen3:8b")
+            api_key_line="    api_base: http://ollama.${OLLAMA_NAMESPACE}:11434"
+            ;;
+        mistral)
+            model=$(gum choose --header="Select Mistral model:" \
+                "mistral-large-latest" \
+                "ministral-8b-latest" \
+                "codestral-latest")
+            api_key_line="    api_key: os.environ/MISTRAL_API_KEY"
+            ;;
+        groq)
+            model=$(gum choose --header="Select Groq model:" \
+                "meta-llama/llama-4-scout-17b-16e-instruct" \
+                "llama-3.3-70b-versatile" \
+                "llama-3.1-8b-instant")
+            api_key_line="    api_key: os.environ/GROQ_API_KEY"
+            ;;
+        cohere)
+            model=$(gum choose --header="Select Cohere model:" \
+                "command-r-plus" \
+                "command-r" \
+                "command-light")
+            api_key_line="    api_key: os.environ/COHERE_API_KEY"
+            ;;
+        *)
+            model=$(gum input --prompt="Model identifier: " --width=60)
+            api_key_line="    api_key: os.environ/${provider^^}_API_KEY"
+            ;;
+    esac
+
+    echo "" >> models.yaml
+    echo "- model_name: ${model_name}" >> models.yaml
+    echo "  litellm_params:" >> models.yaml
+    echo "    model: ${provider}/${model}" >> models.yaml
+    echo "${api_key_line}" >> models.yaml
+
+    echo ""
+    echo "Model '${model_name}' added to models.yaml"
+    if [ "$provider" != "ollama" ]; then
+        echo ""
+        echo "Don't forget to set the API key if not already done:"
+        echo "  just litellm::set-api-key provider=${provider}"
+    fi
+    echo ""
+    echo "Run 'just litellm::install' or 'just litellm::upgrade' to apply changes."
+
+# Remove a model interactively
+remove-model:
+    #!/bin/bash
+    set -euo pipefail
+    if [ ! -f models.yaml ]; then
+        echo "Error: models.yaml not found"
+        exit 1
+    fi
+    models=$(yq -r '.[].model_name' models.yaml)
+    if [ -z "$models" ]; then
+        echo "No models configured."
+        exit 0
+    fi
+    model=$(echo "$models" | gum choose --header="Select model to remove:")
+    if gum confirm "Remove model '${model}'?"; then
+        yq -i "del(.[] | select(.model_name == \"${model}\"))" models.yaml
+        echo "Model '${model}' removed from models.yaml"
+        echo "Run 'just litellm::upgrade' to apply changes."
+    else
+        echo "Cancelled."
+    fi
+
+# List configured models
+list-models:
+    #!/bin/bash
+    set -euo pipefail
+    if [ ! -f models.yaml ]; then
+        echo "No models.yaml found. Create one with:"
+        echo "  cp models.example.yaml models.yaml"
+        exit 0
+    fi
+    echo "Configured models:"
+    yq -r '.[] | "  - \(.model_name): \(.litellm_params.model)"' models.yaml
+
+# Create API key external secret
+create-api-key-external-secret:
+    #!/bin/bash
+    set -euo pipefail
+    gomplate -d models=models.yaml -f apikey-external-secret.gomplate.yaml -o apikey-external-secret.yaml
+    kubectl apply -f apikey-external-secret.yaml
+    echo "Waiting for API key secret to be ready..."
+    kubectl wait --for=condition=Ready externalsecret/apikey-external-secret \
+        -n ${LITELLM_NAMESPACE} --timeout=60s
+
+# Delete API key external secret
+delete-api-key-external-secret:
+    kubectl delete externalsecret apikey-external-secret -n ${LITELLM_NAMESPACE} --ignore-not-found
+    kubectl delete secret apikey -n ${LITELLM_NAMESPACE} --ignore-not-found
+
+# Create Postgres user and database
+create-postgres-user-and-db:
+    #!/bin/bash
+    set -euo pipefail
+    if just postgres::user-exists litellm &>/dev/null; then
+        echo "PostgreSQL user 'litellm' already exists"
+    else
+        echo "Creating PostgreSQL user and database..."
+        PG_PASSWORD=$(just utils::random-password)
+        just postgres::create-user-and-db litellm litellm "${PG_PASSWORD}"
+        just vault::put litellm/db username=litellm password="${PG_PASSWORD}"
+        echo "PostgreSQL user and database created."
+    fi
+
+# Delete Postgres user and database
+delete-postgres-user-and-db:
+    #!/bin/bash
+    set -euo pipefail
+    if gum confirm "Delete PostgreSQL user and database 'litellm'?"; then
+        just postgres::delete-user-and-db litellm litellm || true
+        just vault::delete litellm/db || true
+        echo "PostgreSQL user and database deleted."
+    else
+        echo "Cancelled."
+    fi
+
+# Create Postgres secret
+create-postgres-secret:
+    #!/bin/bash
+    set -euo pipefail
+    if kubectl get secret postgres-auth -n ${LITELLM_NAMESPACE} &>/dev/null; then
+        echo "Postgres auth secret already exists"
+        exit 0
+    fi
+    PG_USERNAME=$(just vault::get litellm/db username)
+    PG_PASSWORD=$(just vault::get litellm/db password)
+    kubectl create secret generic postgres-auth \
+        --from-literal=username="${PG_USERNAME}" \
+        --from-literal=password="${PG_PASSWORD}" \
+        -n ${LITELLM_NAMESPACE}
+
+# Delete Postgres secret
+delete-postgres-secret:
+    kubectl delete secret postgres-auth -n ${LITELLM_NAMESPACE} --ignore-not-found
+
+# Install LiteLLM
+install:
+    #!/bin/bash
+    set -euo pipefail
+
+    just check-prerequisites
+    just verify-api-keys
+
+    while [ -z "${LITELLM_HOST}" ]; do
+        LITELLM_HOST=$(gum input --prompt="LiteLLM host (FQDN): " --width=80 \
+            --placeholder="e.g., litellm.example.com")
+    done
+
+    if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
+        if [ -z "${MONITORING_ENABLED}" ]; then
+            if gum confirm "Enable Prometheus monitoring?"; then
+                MONITORING_ENABLED="true"
+            fi
+        fi
+    fi
+
+    echo "Installing LiteLLM..."
+
+    just create-namespace
+
+    # Note: LiteLLM requires baseline due to Prisma needing /.cache write access
+    kubectl label namespace ${LITELLM_NAMESPACE} \
+        pod-security.kubernetes.io/enforce=baseline --overwrite
+
+    if [ "${MONITORING_ENABLED}" = "true" ]; then
+        kubectl label namespace ${LITELLM_NAMESPACE} \
+            buun.channel/enable-monitoring=true --overwrite
+    fi
+
+    just create-api-key-external-secret
+
+    echo "Setting up PostgreSQL database..."
+    just create-postgres-user-and-db
+    just create-postgres-secret
+
+    echo "Generating Helm values..."
+    gomplate -f litellm-values.gomplate.yaml -o litellm-values.yaml
+
+    echo "Installing LiteLLM Helm chart..."
+    helm upgrade --install litellm oci://ghcr.io/berriai/litellm-helm \
+        --version ${LITELLM_CHART_VERSION} -n ${LITELLM_NAMESPACE} --wait \
+        -f litellm-values.yaml
+
+    echo ""
+    echo "LiteLLM installed successfully!"
+    echo "Access LiteLLM at: https://${LITELLM_HOST}"
+
+# Upgrade LiteLLM
+upgrade:
+    #!/bin/bash
+    set -euo pipefail
+
+    just check-prerequisites
+    just verify-api-keys
+
+    while [ -z "${LITELLM_HOST}" ]; do
+        LITELLM_HOST=$(gum input --prompt="LiteLLM host (FQDN): " --width=80)
+    done
+
+    if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
+        if [ -z "${MONITORING_ENABLED}" ]; then
+            if gum confirm "Enable Prometheus monitoring?"; then
+                MONITORING_ENABLED="true"
+            fi
+        fi
+    fi
+
+    echo "Upgrading LiteLLM..."
+
+    if [ "${MONITORING_ENABLED}" = "true" ]; then
+        kubectl label namespace ${LITELLM_NAMESPACE} \
+            buun.channel/enable-monitoring=true --overwrite
+    fi
+
+    just create-api-key-external-secret
+
+    echo "Generating Helm values..."
+    gomplate -f litellm-values.gomplate.yaml -o litellm-values.yaml
+
+    echo "Upgrading LiteLLM Helm chart..."
+    helm upgrade litellm oci://ghcr.io/berriai/litellm-helm \
+        --version ${LITELLM_CHART_VERSION} -n ${LITELLM_NAMESPACE} --wait \
+        -f litellm-values.yaml
+
+    echo ""
+    echo "LiteLLM upgraded successfully!"
+    echo "Access LiteLLM at: https://${LITELLM_HOST}"
+
+# Uninstall LiteLLM
+uninstall:
+    #!/bin/bash
+    set -euo pipefail
+    if ! gum confirm "Uninstall LiteLLM?"; then
+        echo "Cancelled."
+        exit 0
+    fi
+    echo "Uninstalling LiteLLM..."
+    helm uninstall litellm -n ${LITELLM_NAMESPACE} --ignore-not-found --wait
+    just delete-api-key-external-secret
+    just delete-postgres-secret
+    just delete-namespace
+    echo "LiteLLM uninstalled."
+
+# Clean up all resources including database
+cleanup:
+    #!/bin/bash
+    set -euo pipefail
+    echo "This will delete:"
+    echo "  - LiteLLM deployment"
+    echo "  - LiteLLM database"
+    echo "  - All API keys from Vault"
+    echo ""
+    if ! gum confirm "Are you sure?"; then
+        echo "Cancelled."
+        exit 0
+    fi
+    just uninstall || true
+    just postgres::delete-db litellm || true
+    # Clean up API keys from Vault
+    providers=$(just get-required-providers 2>/dev/null || true)
+    for provider in $providers; do
+        just vault::delete "litellm/${provider}" || true
+    done
+    echo "Cleanup completed."
+
+# Generate virtual key
+generate-virtual-key user='' model='':
+    #!/bin/bash
+    set -euo pipefail
+    user="{{ user }}"
+    while [ -z "${user}" ]; do
+        user=$(gum input --prompt="User email: " --width=80)
+    done
+    model="{{ model }}"
+    if [ -z "${model}" ]; then
+        models=$(yq -r '.[].model_name' models.yaml 2>/dev/null || echo "")
+        if [ -n "$models" ]; then
+            model=$(echo "$models" | gum choose --header="Select model:")
+        else
+            model=$(gum input --prompt="Model: " --width=80)
+        fi
+    fi
+    master_key=$(kubectl get secret litellm-masterkey -n ${LITELLM_NAMESPACE} \
+        -o jsonpath="{.data.masterkey}" | base64 --decode)
+    curl "https://${LITELLM_HOST}/key/generate" \
+        --header "Authorization: Bearer ${master_key}" \
+        --header "Content-Type: application/json" \
+        --data-raw "{\"models\": [\"${model}\"], \"metadata\": {\"user\": \"${user}\"}}" | jq .
+
+# Get master key
+master-key:
+    @kubectl get secret litellm-masterkey -n ${LITELLM_NAMESPACE} \
+        -o jsonpath="{.data.masterkey}" | base64 --decode
--- a/litellm/litellm-values.gomplate.yaml
+++ b/litellm/litellm-values.gomplate.yaml
@@ -0,0 +1,65 @@
+# https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm
+# https://github.com/BerriAI/litellm/tree/main/litellm/proxy/example_config_yaml
+
+masterkeySecretName: ""
+masterkeySecretKey: ""
+
+# Note: LiteLLM image requires write access to /.cache for Prisma
+# Pod Security Standards must be set to "baseline" for this namespace
+podSecurityContext: {}
+
+securityContext: {}
+
+migrationJob:
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 1Gi
+
+environmentSecrets:
+  - apikey
+
+proxy_config:
+  model_list:
+{{ file.Read "models.yaml" | indent 4 }}
+
+db:
+  useExisting: true
+
+  endpoint: postgres-cluster-rw.postgres
+  database: litellm
+  secret:
+    name: postgres-auth
+    usernameKey: username
+    passwordKey: password
+
+  deployStandalone: false
+
+redis:
+  enabled: true
+
+ingress:
+  enabled: true
+  className: traefik
+  annotations:
+    kubernetes.io/ingress.class: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: websecure
+  hosts:
+    - host: {{ .Env.LITELLM_HOST }}
+      paths:
+        - path: /
+          pathType: ImplementationSpecific
+  tls:
+    - hosts:
+      - {{ .Env.LITELLM_HOST }}
+
+{{- if .Env.MONITORING_ENABLED }}
+serviceMonitor:
+  enabled: true
+  labels:
+    release: kube-prometheus-stack
+  interval: 30s
+  scrapeTimeout: 10s
+{{- end }}
--- a/litellm/models.example.yaml
+++ b/litellm/models.example.yaml
@@ -0,0 +1,106 @@
+# LiteLLM Model Configuration
+# Copy this file to models.yaml and customize for your environment.
+#
+# Usage:
+#   cp litellm/models.example.yaml litellm/models.yaml
+#   # Edit models.yaml to add/remove models
+#   just litellm::install
+#
+# API keys are stored in Vault and injected as environment variables.
+# Use: just litellm::set-api-key provider=<provider>
+#
+# Supported providers:
+#   - anthropic: Claude models (Opus, Sonnet, Haiku)
+#   - openai: GPT and o-series models
+#   - ollama: Local models (no API key required)
+#   - azure: Azure OpenAI
+#   - bedrock: AWS Bedrock
+#   - vertexai: Google Vertex AI
+#   - mistral: Mistral AI
+#   - groq: Groq (fast inference)
+#   - cohere: Cohere
+
+# Anthropic Claude (https://docs.anthropic.com/en/docs/about-claude/models/overview)
+- model_name: claude-sonnet
+  litellm_params:
+    model: anthropic/claude-sonnet-4-20250514
+    api_key: os.environ/ANTHROPIC_API_KEY
+
+- model_name: claude-haiku
+  litellm_params:
+    model: anthropic/claude-haiku-4-20251015
+    api_key: os.environ/ANTHROPIC_API_KEY
+
+# - model_name: claude-opus
+#   litellm_params:
+#     model: anthropic/claude-opus-4-20250514
+#     api_key: os.environ/ANTHROPIC_API_KEY
+
+# OpenAI (https://platform.openai.com/docs/models)
+# - model_name: gpt-4o
+#   litellm_params:
+#     model: openai/gpt-4o
+#     api_key: os.environ/OPENAI_API_KEY
+
+# - model_name: gpt-4o-mini
+#   litellm_params:
+#     model: openai/gpt-4o-mini
+#     api_key: os.environ/OPENAI_API_KEY
+
+# - model_name: o3
+#   litellm_params:
+#     model: openai/o3
+#     api_key: os.environ/OPENAI_API_KEY
+
+# - model_name: o4-mini
+#   litellm_params:
+#     model: openai/o4-mini
+#     api_key: os.environ/OPENAI_API_KEY
+
+# Ollama (local models - no API key required)
+# - model_name: llama4-scout
+#   litellm_params:
+#     model: ollama/llama4:scout
+#     api_base: http://ollama.ollama:11434
+
+# - model_name: qwen3
+#   litellm_params:
+#     model: ollama/qwen3:8b
+#     api_base: http://ollama.ollama:11434
+
+# - model_name: deepseek-r1
+#   litellm_params:
+#     model: ollama/deepseek-r1:8b
+#     api_base: http://ollama.ollama:11434
+
+# Mistral AI (https://docs.mistral.ai/getting-started/models/models_overview/)
+# - model_name: mistral-large
+#   litellm_params:
+#     model: mistral/mistral-large-latest
+#     api_key: os.environ/MISTRAL_API_KEY
+
+# - model_name: ministral-8b
+#   litellm_params:
+#     model: mistral/ministral-8b-latest
+#     api_key: os.environ/MISTRAL_API_KEY
+
+# - model_name: codestral
+#   litellm_params:
+#     model: mistral/codestral-latest
+#     api_key: os.environ/MISTRAL_API_KEY
+
+# Groq (fast inference - https://console.groq.com/docs/models)
+# - model_name: groq-llama4-scout
+#   litellm_params:
+#     model: groq/meta-llama/llama-4-scout-17b-16e-instruct
+#     api_key: os.environ/GROQ_API_KEY
+
+# - model_name: groq-llama3.3
+#   litellm_params:
+#     model: groq/llama-3.3-70b-versatile
+#     api_key: os.environ/GROQ_API_KEY
+
+# - model_name: groq-llama3.1
+#   litellm_params:
+#     model: groq/llama-3.1-8b-instant
+#     api_key: os.environ/GROQ_API_KEY