diff --git a/justfile b/justfile index 38509c5..5f8ff22 100644 --- a/justfile +++ b/justfile @@ -23,6 +23,7 @@ mod kserve mod langfuse mod lakekeeper mod librechat +mod litellm mod longhorn mod metabase mod mlflow diff --git a/litellm/.gitignore b/litellm/.gitignore new file mode 100644 index 0000000..7bb5d94 --- /dev/null +++ b/litellm/.gitignore @@ -0,0 +1,3 @@ +litellm-values.yaml +apikey-external-secret.yaml +models.yaml diff --git a/litellm/README.md b/litellm/README.md new file mode 100644 index 0000000..fd6c21f --- /dev/null +++ b/litellm/README.md @@ -0,0 +1,349 @@ +# LiteLLM + +Unified LLM gateway and proxy for accessing multiple LLM providers through a single OpenAI-compatible API: + +- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, Cohere, Azure, Bedrock, Vertex AI +- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK +- **Load Balancing & Fallback**: Automatic failover between providers +- **Virtual Keys**: Generate API keys for users with usage tracking +- **Cost Tracking**: Monitor spending across providers +- **Rate Limiting**: Control usage per key/user + +## Prerequisites + +- Kubernetes cluster (k3s) +- External Secrets Operator (required) +- PostgreSQL cluster (CloudNativePG) +- Vault for secrets management + +## Configuration Overview + +LiteLLM requires two types of configuration: + +1. **Environment variables** (`.env.local`): Host, namespace, chart version +2. **Model definitions** (`models.yaml`): LLM providers and models to expose + +This separation allows flexible model configuration without modifying environment files. + +## Installation + +### Step 1: Create Model Configuration + +Copy the example configuration and customize: + +```bash +cp litellm/models.example.yaml litellm/models.yaml +``` + +Edit `litellm/models.yaml` to configure your models: + +```yaml +# Anthropic Claude +- model_name: claude-sonnet + litellm_params: + model: anthropic/claude-3-7-sonnet-latest + api_key: os.environ/ANTHROPIC_API_KEY + +# OpenAI +- model_name: gpt-4o + litellm_params: + model: openai/gpt-4o + api_key: os.environ/OPENAI_API_KEY + +# Ollama (local models - no API key required) +- model_name: llama3 + litellm_params: + model: ollama/llama3.2 + api_base: http://ollama.ollama:11434 +``` + +### Step 2: Set API Keys + +For each provider that requires an API key: + +```bash +just litellm::set-api-key anthropic +just litellm::set-api-key openai +``` + +Or interactively select the provider: + +```bash +just litellm::set-api-key +``` + +API keys are stored in Vault and synced to Kubernetes via External Secrets Operator. + +### Step 3: Install LiteLLM + +```bash +just litellm::install +``` + +You will be prompted for: + +- **LiteLLM host (FQDN)**: e.g., `litellm.example.com` +- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed + +## Model Management + +### Add a Model Interactively + +```bash +just litellm::add-model +``` + +This guides you through: + +1. Selecting a provider +2. Choosing a model +3. Setting a model alias + +### Remove a Model + +```bash +just litellm::remove-model +``` + +### List Configured Models + +```bash +just litellm::list-models +``` + +### Example Output + +```text +Configured models: + - claude-sonnet: anthropic/claude-3-7-sonnet-latest + - claude-haiku: anthropic/claude-3-5-haiku-latest + - llama3: ollama/llama3.2 +``` + +## API Key Management + +### Set API Key for a Provider + +```bash +just litellm::set-api-key anthropic +``` + +### Get API Key (from Vault) + +```bash +just litellm::get-api-key anthropic +``` + +### Verify All Required Keys + +```bash +just litellm::verify-api-keys +``` + +## Environment Variables + +| Variable | Default | Description | +| -------- | ------- | ----------- | +| `LITELLM_NAMESPACE` | `litellm` | Kubernetes namespace | +| `LITELLM_CHART_VERSION` | `0.1.825` | Helm chart version | +| `LITELLM_HOST` | (prompt) | External hostname (FQDN) | +| `OLLAMA_NAMESPACE` | `ollama` | Ollama namespace for local models | +| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor | + +## API Usage + +LiteLLM exposes an OpenAI-compatible API at `https://your-litellm-host/`. + +### Get Master Key + +```bash +just litellm::master-key +``` + +### Generate Virtual Key for a User + +```bash +just litellm::generate-virtual-key user@example.com +``` + +### OpenAI SDK Example + +```python +from openai import OpenAI + +client = OpenAI( + base_url="https://litellm.example.com", + api_key="sk-..." # Virtual key or master key +) + +response = client.chat.completions.create( + model="claude-sonnet", # Use your model alias + messages=[{"role": "user", "content": "Hello!"}] +) +print(response.choices[0].message.content) +``` + +### curl Example + +```bash +curl https://litellm.example.com/v1/chat/completions \ + -H "Authorization: Bearer sk-..." \ + -H "Content-Type: application/json" \ + -d '{ + "model": "claude-sonnet", + "messages": [{"role": "user", "content": "Hello!"}] + }' +``` + +## Supported Providers + +| Provider | Model Prefix | API Key Required | +| -------- | ------------ | ---------------- | +| Anthropic | `anthropic/` | Yes | +| OpenAI | `openai/` | Yes | +| Ollama | `ollama/` | No (uses `api_base`) | +| Mistral | `mistral/` | Yes | +| Groq | `groq/` | Yes | +| Cohere | `cohere/` | Yes | +| Azure OpenAI | `azure/` | Yes | +| AWS Bedrock | `bedrock/` | Yes | +| Google Vertex AI | `vertexai/` | Yes | + +## Architecture + +```plain +External Users/Applications + | +Cloudflare Tunnel (HTTPS) + | +Traefik Ingress (HTTPS) + | +LiteLLM Proxy (HTTP inside cluster) + |-- PostgreSQL (usage tracking, virtual keys) + |-- Redis (caching, rate limiting) + |-- External Secrets (API keys from Vault) + | + +-- Anthropic API + +-- OpenAI API + +-- Ollama (local) + +-- Other providers... +``` + +## Upgrade + +After modifying `models.yaml` or updating API keys: + +```bash +just litellm::upgrade +``` + +## Uninstall + +```bash +just litellm::uninstall +``` + +This removes: + +- Helm release and all Kubernetes resources +- Namespace +- External Secrets + +**Note**: The following resources are NOT deleted: + +- PostgreSQL database (use `just postgres::delete-db litellm`) +- API keys in Vault + +### Full Cleanup + +To remove everything including database and Vault secrets: + +```bash +just litellm::cleanup +``` + +## Troubleshooting + +### Check Pod Status + +```bash +kubectl get pods -n litellm +``` + +Expected pods: + +- `litellm-*` - LiteLLM proxy +- `litellm-redis-master-0` - Redis instance + +### View Logs + +```bash +kubectl logs -n litellm deployment/litellm --tail=100 +``` + +### API Key Not Working + +Verify the ExternalSecret is synced: + +```bash +kubectl get externalsecret -n litellm +kubectl get secret apikey -n litellm -o yaml +``` + +### Model Not Found + +Ensure the model is configured in `models.yaml` and the deployment is updated: + +```bash +just litellm::list-models +just litellm::upgrade +``` + +### Provider API Errors + +Check if the API key is set correctly: + +```bash +just litellm::get-api-key anthropic +``` + +If empty, set the API key: + +```bash +just litellm::set-api-key anthropic +``` + +### Database Connection Issues + +Check PostgreSQL connectivity: + +```bash +kubectl exec -n litellm deployment/litellm -- \ + psql -h postgres-cluster-rw.postgres -U litellm -d litellm -c "SELECT 1" +``` + +## Configuration Files + +| File | Description | +| ---- | ----------- | +| `models.yaml` | Model definitions (user-created, gitignored) | +| `models.example.yaml` | Example model configuration | +| `litellm-values.gomplate.yaml` | Helm values template | +| `apikey-external-secret.gomplate.yaml` | ExternalSecret for API keys | + +## Security Considerations + +- **Pod Security Standards**: Namespace configured with **baseline** enforcement + (LiteLLM's Prisma requires write access to `/.cache`, which prevents `restricted` level) +- **Secrets Management**: API keys stored in Vault, synced via External Secrets Operator +- **Virtual Keys**: Generate scoped API keys for users instead of sharing master key +- **TLS/HTTPS**: All external traffic encrypted via Traefik Ingress +- **Database Credentials**: Unique PostgreSQL user with minimal privileges + +## References + +- [LiteLLM Documentation](https://docs.litellm.ai/) +- [LiteLLM GitHub](https://github.com/BerriAI/litellm) +- [LiteLLM Helm Chart](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm) +- [Supported Models](https://docs.litellm.ai/docs/providers) +- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys) diff --git a/litellm/apikey-external-secret.gomplate.yaml b/litellm/apikey-external-secret.gomplate.yaml new file mode 100644 index 0000000..24421b7 --- /dev/null +++ b/litellm/apikey-external-secret.gomplate.yaml @@ -0,0 +1,29 @@ +{{- $models := (datasource "models") -}} +{{- $providerMap := dict -}} +{{- range $models -}} +{{- if has .litellm_params "api_key" -}} +{{- $parts := strings.Split "/" .litellm_params.model -}} +{{- $provider := index $parts 0 -}} +{{- $providerMap = merge $providerMap (dict $provider true) -}} +{{- end -}} +{{- end -}} +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: apikey-external-secret + namespace: {{ .Env.LITELLM_NAMESPACE }} +spec: + refreshInterval: 1h + secretStoreRef: + name: vault-secret-store + kind: ClusterSecretStore + target: + name: apikey + creationPolicy: Owner + data: +{{- range $provider, $_ := $providerMap }} + - secretKey: {{ $provider | strings.ToUpper }}_API_KEY + remoteRef: + key: litellm/{{ $provider }} + property: apikey +{{- end }} diff --git a/litellm/justfile b/litellm/justfile new file mode 100644 index 0000000..fed91b8 --- /dev/null +++ b/litellm/justfile @@ -0,0 +1,440 @@ +set fallback := true + +export LITELLM_NAMESPACE := env("LITELLM_NAMESPACE", "litellm") +export LITELLM_CHART_VERSION := env("LITELLM_CHART_VERSION", "0.1.825") +export LITELLM_HOST := env("LITELLM_HOST", "") +export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets") +export OLLAMA_NAMESPACE := env("OLLAMA_NAMESPACE", "ollama") +export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring") +export MONITORING_ENABLED := env("MONITORING_ENABLED", "") + +[private] +default: + @just --list --unsorted --list-submodules + +# Create LiteLLM namespace +create-namespace: + kubectl get namespace ${LITELLM_NAMESPACE} &>/dev/null || \ + kubectl create namespace ${LITELLM_NAMESPACE} + +# Delete LiteLLM namespace +delete-namespace: + kubectl delete namespace ${LITELLM_NAMESPACE} --ignore-not-found + +# Check prerequisites +check-prerequisites: + #!/bin/bash + set -euo pipefail + if ! helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then + echo "Error: External Secrets Operator is required but not installed." + echo "Please install External Secrets Operator first:" + echo " just external-secrets::install" + exit 1 + fi + if [ ! -f models.yaml ]; then + echo "Error: models.yaml not found." + echo "Please create models.yaml from the example:" + echo " cp models.example.yaml models.yaml" + echo "Then edit models.yaml to configure your models." + exit 1 + fi + echo "Prerequisites check passed." + +# Extract required providers from models.yaml +[private] +get-required-providers: + #!/bin/bash + set -euo pipefail + # Extract providers that require API keys (exclude ollama) + yq -r '.[] | select(.litellm_params.api_key != null) | .litellm_params.model' models.yaml | \ + cut -d'/' -f1 | sort -u + +# Verify all required API keys are set in Vault +verify-api-keys: + #!/bin/bash + set -euo pipefail + providers=$(just get-required-providers) + missing=() + for provider in $providers; do + if ! just vault::get "litellm/${provider}" apikey &>/dev/null; then + missing+=("$provider") + fi + done + if [ ${#missing[@]} -gt 0 ]; then + echo "Error: Missing API keys for the following providers:" + for p in "${missing[@]}"; do + echo " - $p" + done + echo "" + echo "Please set the API keys:" + for p in "${missing[@]}"; do + echo " just litellm::set-api-key provider=$p" + done + exit 1 + fi + echo "All required API keys are configured." + +# Set API key for a provider +set-api-key provider='': + #!/bin/bash + set -euo pipefail + provider="{{ provider }}" + if [ -z "${provider}" ]; then + available=$(just get-required-providers 2>/dev/null || echo "anthropic openai mistral groq cohere") + provider=$(echo "$available" | tr ' ' '\n' | gum choose --header="Select provider:") + fi + apikey=$(gum input --prompt="${provider} API key: " --password --width=80) + if [ -z "${apikey}" ]; then + echo "Error: API key cannot be empty" + exit 1 + fi + just vault::put "litellm/${provider}" apikey="${apikey}" + echo "API key for ${provider} has been stored in Vault location 'litellm/${provider}'." + +# Get API key for a provider +get-api-key provider='': + #!/bin/bash + set -euo pipefail + provider="{{ provider }}" + if [ -z "${provider}" ]; then + echo "Usage: just litellm::get-api-key provider=" + exit 1 + fi + just vault::get "litellm/${provider}" apikey + +# Add a model interactively +add-model: + #!/bin/bash + set -euo pipefail + if [ ! -f models.yaml ]; then + echo "Creating models.yaml from example..." + cp models.example.yaml models.yaml + fi + + echo "Add a new model to LiteLLM" + echo "" + + provider=$(gum choose --header="Select provider:" \ + "anthropic" "openai" "ollama" "mistral" "groq" "cohere" "azure" "bedrock" "vertexai") + + model_name=$(gum input --prompt="Model alias (e.g., claude-sonnet): " --width=60) + if [ -z "${model_name}" ]; then + echo "Error: Model name is required" + exit 1 + fi + + case $provider in + anthropic) + model=$(gum choose --header="Select Anthropic model:" \ + "claude-sonnet-4-20250514" \ + "claude-haiku-4-20251015" \ + "claude-opus-4-20250514") + api_key_line=" api_key: os.environ/ANTHROPIC_API_KEY" + ;; + openai) + model=$(gum choose --header="Select OpenAI model:" \ + "gpt-4o" \ + "gpt-4o-mini" \ + "o3" \ + "o4-mini") + api_key_line=" api_key: os.environ/OPENAI_API_KEY" + ;; + ollama) + model=$(gum input --prompt="Ollama model name: " --width=60 --placeholder="qwen3:8b") + api_key_line=" api_base: http://ollama.${OLLAMA_NAMESPACE}:11434" + ;; + mistral) + model=$(gum choose --header="Select Mistral model:" \ + "mistral-large-latest" \ + "ministral-8b-latest" \ + "codestral-latest") + api_key_line=" api_key: os.environ/MISTRAL_API_KEY" + ;; + groq) + model=$(gum choose --header="Select Groq model:" \ + "meta-llama/llama-4-scout-17b-16e-instruct" \ + "llama-3.3-70b-versatile" \ + "llama-3.1-8b-instant") + api_key_line=" api_key: os.environ/GROQ_API_KEY" + ;; + cohere) + model=$(gum choose --header="Select Cohere model:" \ + "command-r-plus" \ + "command-r" \ + "command-light") + api_key_line=" api_key: os.environ/COHERE_API_KEY" + ;; + *) + model=$(gum input --prompt="Model identifier: " --width=60) + api_key_line=" api_key: os.environ/${provider^^}_API_KEY" + ;; + esac + + echo "" >> models.yaml + echo "- model_name: ${model_name}" >> models.yaml + echo " litellm_params:" >> models.yaml + echo " model: ${provider}/${model}" >> models.yaml + echo "${api_key_line}" >> models.yaml + + echo "" + echo "Model '${model_name}' added to models.yaml" + if [ "$provider" != "ollama" ]; then + echo "" + echo "Don't forget to set the API key if not already done:" + echo " just litellm::set-api-key provider=${provider}" + fi + echo "" + echo "Run 'just litellm::install' or 'just litellm::upgrade' to apply changes." + +# Remove a model interactively +remove-model: + #!/bin/bash + set -euo pipefail + if [ ! -f models.yaml ]; then + echo "Error: models.yaml not found" + exit 1 + fi + models=$(yq -r '.[].model_name' models.yaml) + if [ -z "$models" ]; then + echo "No models configured." + exit 0 + fi + model=$(echo "$models" | gum choose --header="Select model to remove:") + if gum confirm "Remove model '${model}'?"; then + yq -i "del(.[] | select(.model_name == \"${model}\"))" models.yaml + echo "Model '${model}' removed from models.yaml" + echo "Run 'just litellm::upgrade' to apply changes." + else + echo "Cancelled." + fi + +# List configured models +list-models: + #!/bin/bash + set -euo pipefail + if [ ! -f models.yaml ]; then + echo "No models.yaml found. Create one with:" + echo " cp models.example.yaml models.yaml" + exit 0 + fi + echo "Configured models:" + yq -r '.[] | " - \(.model_name): \(.litellm_params.model)"' models.yaml + +# Create API key external secret +create-api-key-external-secret: + #!/bin/bash + set -euo pipefail + gomplate -d models=models.yaml -f apikey-external-secret.gomplate.yaml -o apikey-external-secret.yaml + kubectl apply -f apikey-external-secret.yaml + echo "Waiting for API key secret to be ready..." + kubectl wait --for=condition=Ready externalsecret/apikey-external-secret \ + -n ${LITELLM_NAMESPACE} --timeout=60s + +# Delete API key external secret +delete-api-key-external-secret: + kubectl delete externalsecret apikey-external-secret -n ${LITELLM_NAMESPACE} --ignore-not-found + kubectl delete secret apikey -n ${LITELLM_NAMESPACE} --ignore-not-found + +# Create Postgres user and database +create-postgres-user-and-db: + #!/bin/bash + set -euo pipefail + if just postgres::user-exists litellm &>/dev/null; then + echo "PostgreSQL user 'litellm' already exists" + else + echo "Creating PostgreSQL user and database..." + PG_PASSWORD=$(just utils::random-password) + just postgres::create-user-and-db litellm litellm "${PG_PASSWORD}" + just vault::put litellm/db username=litellm password="${PG_PASSWORD}" + echo "PostgreSQL user and database created." + fi + +# Delete Postgres user and database +delete-postgres-user-and-db: + #!/bin/bash + set -euo pipefail + if gum confirm "Delete PostgreSQL user and database 'litellm'?"; then + just postgres::delete-user-and-db litellm litellm || true + just vault::delete litellm/db || true + echo "PostgreSQL user and database deleted." + else + echo "Cancelled." + fi + +# Create Postgres secret +create-postgres-secret: + #!/bin/bash + set -euo pipefail + if kubectl get secret postgres-auth -n ${LITELLM_NAMESPACE} &>/dev/null; then + echo "Postgres auth secret already exists" + exit 0 + fi + PG_USERNAME=$(just vault::get litellm/db username) + PG_PASSWORD=$(just vault::get litellm/db password) + kubectl create secret generic postgres-auth \ + --from-literal=username="${PG_USERNAME}" \ + --from-literal=password="${PG_PASSWORD}" \ + -n ${LITELLM_NAMESPACE} + +# Delete Postgres secret +delete-postgres-secret: + kubectl delete secret postgres-auth -n ${LITELLM_NAMESPACE} --ignore-not-found + +# Install LiteLLM +install: + #!/bin/bash + set -euo pipefail + + just check-prerequisites + just verify-api-keys + + while [ -z "${LITELLM_HOST}" ]; do + LITELLM_HOST=$(gum input --prompt="LiteLLM host (FQDN): " --width=80 \ + --placeholder="e.g., litellm.example.com") + done + + if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then + if [ -z "${MONITORING_ENABLED}" ]; then + if gum confirm "Enable Prometheus monitoring?"; then + MONITORING_ENABLED="true" + fi + fi + fi + + echo "Installing LiteLLM..." + + just create-namespace + + # Note: LiteLLM requires baseline due to Prisma needing /.cache write access + kubectl label namespace ${LITELLM_NAMESPACE} \ + pod-security.kubernetes.io/enforce=baseline --overwrite + + if [ "${MONITORING_ENABLED}" = "true" ]; then + kubectl label namespace ${LITELLM_NAMESPACE} \ + buun.channel/enable-monitoring=true --overwrite + fi + + just create-api-key-external-secret + + echo "Setting up PostgreSQL database..." + just create-postgres-user-and-db + just create-postgres-secret + + echo "Generating Helm values..." + gomplate -f litellm-values.gomplate.yaml -o litellm-values.yaml + + echo "Installing LiteLLM Helm chart..." + helm upgrade --install litellm oci://ghcr.io/berriai/litellm-helm \ + --version ${LITELLM_CHART_VERSION} -n ${LITELLM_NAMESPACE} --wait \ + -f litellm-values.yaml + + echo "" + echo "LiteLLM installed successfully!" + echo "Access LiteLLM at: https://${LITELLM_HOST}" + +# Upgrade LiteLLM +upgrade: + #!/bin/bash + set -euo pipefail + + just check-prerequisites + just verify-api-keys + + while [ -z "${LITELLM_HOST}" ]; do + LITELLM_HOST=$(gum input --prompt="LiteLLM host (FQDN): " --width=80) + done + + if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then + if [ -z "${MONITORING_ENABLED}" ]; then + if gum confirm "Enable Prometheus monitoring?"; then + MONITORING_ENABLED="true" + fi + fi + fi + + echo "Upgrading LiteLLM..." + + if [ "${MONITORING_ENABLED}" = "true" ]; then + kubectl label namespace ${LITELLM_NAMESPACE} \ + buun.channel/enable-monitoring=true --overwrite + fi + + just create-api-key-external-secret + + echo "Generating Helm values..." + gomplate -f litellm-values.gomplate.yaml -o litellm-values.yaml + + echo "Upgrading LiteLLM Helm chart..." + helm upgrade litellm oci://ghcr.io/berriai/litellm-helm \ + --version ${LITELLM_CHART_VERSION} -n ${LITELLM_NAMESPACE} --wait \ + -f litellm-values.yaml + + echo "" + echo "LiteLLM upgraded successfully!" + echo "Access LiteLLM at: https://${LITELLM_HOST}" + +# Uninstall LiteLLM +uninstall: + #!/bin/bash + set -euo pipefail + if ! gum confirm "Uninstall LiteLLM?"; then + echo "Cancelled." + exit 0 + fi + echo "Uninstalling LiteLLM..." + helm uninstall litellm -n ${LITELLM_NAMESPACE} --ignore-not-found --wait + just delete-api-key-external-secret + just delete-postgres-secret + just delete-namespace + echo "LiteLLM uninstalled." + +# Clean up all resources including database +cleanup: + #!/bin/bash + set -euo pipefail + echo "This will delete:" + echo " - LiteLLM deployment" + echo " - LiteLLM database" + echo " - All API keys from Vault" + echo "" + if ! gum confirm "Are you sure?"; then + echo "Cancelled." + exit 0 + fi + just uninstall || true + just postgres::delete-db litellm || true + # Clean up API keys from Vault + providers=$(just get-required-providers 2>/dev/null || true) + for provider in $providers; do + just vault::delete "litellm/${provider}" || true + done + echo "Cleanup completed." + +# Generate virtual key +generate-virtual-key user='' model='': + #!/bin/bash + set -euo pipefail + user="{{ user }}" + while [ -z "${user}" ]; do + user=$(gum input --prompt="User email: " --width=80) + done + model="{{ model }}" + if [ -z "${model}" ]; then + models=$(yq -r '.[].model_name' models.yaml 2>/dev/null || echo "") + if [ -n "$models" ]; then + model=$(echo "$models" | gum choose --header="Select model:") + else + model=$(gum input --prompt="Model: " --width=80) + fi + fi + master_key=$(kubectl get secret litellm-masterkey -n ${LITELLM_NAMESPACE} \ + -o jsonpath="{.data.masterkey}" | base64 --decode) + curl "https://${LITELLM_HOST}/key/generate" \ + --header "Authorization: Bearer ${master_key}" \ + --header "Content-Type: application/json" \ + --data-raw "{\"models\": [\"${model}\"], \"metadata\": {\"user\": \"${user}\"}}" | jq . + +# Get master key +master-key: + @kubectl get secret litellm-masterkey -n ${LITELLM_NAMESPACE} \ + -o jsonpath="{.data.masterkey}" | base64 --decode diff --git a/litellm/litellm-values.gomplate.yaml b/litellm/litellm-values.gomplate.yaml new file mode 100644 index 0000000..fb19d24 --- /dev/null +++ b/litellm/litellm-values.gomplate.yaml @@ -0,0 +1,65 @@ +# https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm +# https://github.com/BerriAI/litellm/tree/main/litellm/proxy/example_config_yaml + +masterkeySecretName: "" +masterkeySecretKey: "" + +# Note: LiteLLM image requires write access to /.cache for Prisma +# Pod Security Standards must be set to "baseline" for this namespace +podSecurityContext: {} + +securityContext: {} + +migrationJob: + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + memory: 1Gi + +environmentSecrets: + - apikey + +proxy_config: + model_list: +{{ file.Read "models.yaml" | indent 4 }} + +db: + useExisting: true + + endpoint: postgres-cluster-rw.postgres + database: litellm + secret: + name: postgres-auth + usernameKey: username + passwordKey: password + + deployStandalone: false + +redis: + enabled: true + +ingress: + enabled: true + className: traefik + annotations: + kubernetes.io/ingress.class: traefik + traefik.ingress.kubernetes.io/router.entrypoints: websecure + hosts: + - host: {{ .Env.LITELLM_HOST }} + paths: + - path: / + pathType: ImplementationSpecific + tls: + - hosts: + - {{ .Env.LITELLM_HOST }} + +{{- if .Env.MONITORING_ENABLED }} +serviceMonitor: + enabled: true + labels: + release: kube-prometheus-stack + interval: 30s + scrapeTimeout: 10s +{{- end }} diff --git a/litellm/models.example.yaml b/litellm/models.example.yaml new file mode 100644 index 0000000..d519c3c --- /dev/null +++ b/litellm/models.example.yaml @@ -0,0 +1,106 @@ +# LiteLLM Model Configuration +# Copy this file to models.yaml and customize for your environment. +# +# Usage: +# cp litellm/models.example.yaml litellm/models.yaml +# # Edit models.yaml to add/remove models +# just litellm::install +# +# API keys are stored in Vault and injected as environment variables. +# Use: just litellm::set-api-key provider= +# +# Supported providers: +# - anthropic: Claude models (Opus, Sonnet, Haiku) +# - openai: GPT and o-series models +# - ollama: Local models (no API key required) +# - azure: Azure OpenAI +# - bedrock: AWS Bedrock +# - vertexai: Google Vertex AI +# - mistral: Mistral AI +# - groq: Groq (fast inference) +# - cohere: Cohere + +# Anthropic Claude (https://docs.anthropic.com/en/docs/about-claude/models/overview) +- model_name: claude-sonnet + litellm_params: + model: anthropic/claude-sonnet-4-20250514 + api_key: os.environ/ANTHROPIC_API_KEY + +- model_name: claude-haiku + litellm_params: + model: anthropic/claude-haiku-4-20251015 + api_key: os.environ/ANTHROPIC_API_KEY + +# - model_name: claude-opus +# litellm_params: +# model: anthropic/claude-opus-4-20250514 +# api_key: os.environ/ANTHROPIC_API_KEY + +# OpenAI (https://platform.openai.com/docs/models) +# - model_name: gpt-4o +# litellm_params: +# model: openai/gpt-4o +# api_key: os.environ/OPENAI_API_KEY + +# - model_name: gpt-4o-mini +# litellm_params: +# model: openai/gpt-4o-mini +# api_key: os.environ/OPENAI_API_KEY + +# - model_name: o3 +# litellm_params: +# model: openai/o3 +# api_key: os.environ/OPENAI_API_KEY + +# - model_name: o4-mini +# litellm_params: +# model: openai/o4-mini +# api_key: os.environ/OPENAI_API_KEY + +# Ollama (local models - no API key required) +# - model_name: llama4-scout +# litellm_params: +# model: ollama/llama4:scout +# api_base: http://ollama.ollama:11434 + +# - model_name: qwen3 +# litellm_params: +# model: ollama/qwen3:8b +# api_base: http://ollama.ollama:11434 + +# - model_name: deepseek-r1 +# litellm_params: +# model: ollama/deepseek-r1:8b +# api_base: http://ollama.ollama:11434 + +# Mistral AI (https://docs.mistral.ai/getting-started/models/models_overview/) +# - model_name: mistral-large +# litellm_params: +# model: mistral/mistral-large-latest +# api_key: os.environ/MISTRAL_API_KEY + +# - model_name: ministral-8b +# litellm_params: +# model: mistral/ministral-8b-latest +# api_key: os.environ/MISTRAL_API_KEY + +# - model_name: codestral +# litellm_params: +# model: mistral/codestral-latest +# api_key: os.environ/MISTRAL_API_KEY + +# Groq (fast inference - https://console.groq.com/docs/models) +# - model_name: groq-llama4-scout +# litellm_params: +# model: groq/meta-llama/llama-4-scout-17b-16e-instruct +# api_key: os.environ/GROQ_API_KEY + +# - model_name: groq-llama3.3 +# litellm_params: +# model: groq/llama-3.3-70b-versatile +# api_key: os.environ/GROQ_API_KEY + +# - model_name: groq-llama3.1 +# litellm_params: +# model: groq/llama-3.1-8b-instant +# api_key: os.environ/GROQ_API_KEY