Compare commits

...

19 Commits

Author SHA1 Message Date
baschno
50b0094e86 bump b3sup 2025-12-13 14:25:43 +01:00
Masaki Yatsu
0d45433ea9 fix(litellm): fix langfuse integration 2025-12-11 11:43:37 +09:00
Masaki Yatsu
a8599b66f4 fix(minio): fix OIDC and add public access recipes 2025-12-10 13:26:41 +09:00
Masaki Yatsu
1924e56ad7 chore(ollama): set ollama resource by env-vars 2025-12-10 10:16:05 +09:00
Masaki Yatsu
dae4e9d7ac feat(temporal): enable worker insight 2025-12-08 20:47:19 +09:00
Masaki Yatsu
cfcb278c4d fix(miniflux): fix login error 2025-12-08 13:22:00 +09:00
Masaki Yatsu
022c85c0dc feat(temporal): upgrade Temporal 2025-12-08 09:23:17 +09:00
Masaki Yatsu
3ac8a72df6 docs: write about Temporal 2025-12-07 16:20:17 +09:00
Masaki Yatsu
ca0a8dacba feat(temporal): install Temporal 2025-12-07 16:18:50 +09:00
Masaki Yatsu
fb1e4c20fa chore(ollama): set longer timeout to deploy Ollama 2025-12-06 10:21:52 +09:00
Masaki Yatsu
593da33d64 chore(langfuse): adjust resources 2025-12-05 10:20:16 +09:00
Masaki Yatsu
f3bc41e9eb feat(litellm): litellm -> langfuse integration and some fixes 2025-12-05 10:11:46 +09:00
Masaki Yatsu
98b03704d7 fix(jupyterhub): set sticky sessions for websocket 2025-12-05 10:09:21 +09:00
Masaki Yatsu
6fa0d27f7d feat(nats): install NATS 2025-12-04 15:48:53 +09:00
Masaki Yatsu
d9ee90c32c feat(clickhouse): enable Prometheus monitoring 2025-12-04 11:34:22 +09:00
Masaki Yatsu
7dc732268e docs: write about LiteLLM 2025-12-04 00:21:13 +09:00
Masaki Yatsu
2955d7d783 feat(litellm): SSO and user management 2025-12-04 00:19:14 +09:00
Masaki Yatsu
5055a36d87 feat(litellm): install LiteLLM 2025-12-03 23:05:23 +09:00
Masaki Yatsu
46fdff720f chore(librechat): adjust kubernetes resouces 2025-12-03 21:01:28 +09:00
39 changed files with 3758 additions and 53 deletions

View File

@@ -429,6 +429,7 @@ When writing Markdown documentation:
``` ```
2. **Always validate with markdownlint-cli2**: 2. **Always validate with markdownlint-cli2**:
- Run `markdownlint-cli2 <file>` before committing any Markdown files - Run from the project root directory to use `.markdownlint.yaml` config:
`cd <top-dir> && markdownlint-cli2 <relative-path>`
- Fix all linting errors to ensure consistent formatting - Fix all linting errors to ensure consistent formatting
- Pay attention to code block language specifications (MD040) and list formatting (MD029) - Pay attention to code block language specifications (MD040) and list formatting (MD029)

View File

@@ -63,6 +63,7 @@ A remotely accessible Kubernetes home lab with OIDC authentication. Build a mode
### LLM & AI Applications (Optional) ### LLM & AI Applications (Optional)
- **[Ollama](https://ollama.com/)**: Local LLM inference server with GPU acceleration - **[Ollama](https://ollama.com/)**: Local LLM inference server with GPU acceleration
- **[LiteLLM](https://litellm.ai/)**: Unified LLM gateway for accessing multiple providers through OpenAI-compatible API
- **[LibreChat](https://www.librechat.ai/)**: Web-based chat interface with multi-model support and MCP integration - **[LibreChat](https://www.librechat.ai/)**: Web-based chat interface with multi-model support and MCP integration
- **[Langfuse](https://langfuse.com/)**: LLM observability and analytics platform for tracking and debugging AI applications - **[Langfuse](https://langfuse.com/)**: LLM observability and analytics platform for tracking and debugging AI applications
@@ -70,6 +71,7 @@ A remotely accessible Kubernetes home lab with OIDC authentication. Build a mode
- **[Dagster](https://dagster.io/)**: Modern data orchestration platform - **[Dagster](https://dagster.io/)**: Modern data orchestration platform
- **[Apache Airflow](https://airflow.apache.org/)**: Workflow orchestration and task scheduling - **[Apache Airflow](https://airflow.apache.org/)**: Workflow orchestration and task scheduling
- **[Temporal](https://temporal.io/)**: Durable workflow execution for distributed applications
### Security & Compliance (Optional) ### Security & Compliance (Optional)
@@ -346,6 +348,18 @@ LLM observability and analytics platform:
[📖 See Langfuse Documentation](./langfuse/README.md) [📖 See Langfuse Documentation](./langfuse/README.md)
### LiteLLM
Unified LLM gateway and proxy:
- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, and more through single API
- **OpenAI-Compatible**: Drop-in replacement for OpenAI SDK
- **Virtual Keys**: Generate scoped API keys for users with usage tracking
- **Cost Tracking**: Monitor spending across all LLM providers
- **Keycloak Authentication**: OAuth2 for Admin UI with role-based access
[📖 See LiteLLM Documentation](./litellm/README.md)
### Dagster ### Dagster
Modern data orchestration platform: Modern data orchestration platform:
@@ -366,6 +380,17 @@ Workflow orchestration platform:
[📖 See Airflow Documentation](./airflow/README.md) [📖 See Airflow Documentation](./airflow/README.md)
### Temporal
Durable workflow execution platform:
- **Durable Execution**: Workflows survive process and infrastructure failures
- **Saga Pattern**: Implement distributed transactions with compensating actions
- **Multi-Language SDKs**: Go, Python, TypeScript, Java, .NET, PHP
- **Keycloak Authentication**: OAuth2 for Web UI access
[📖 See Temporal Documentation](./temporal/README.md)
### Fairwinds Polaris ### Fairwinds Polaris
Kubernetes configuration validation and best practices auditing: Kubernetes configuration validation and best practices auditing:
@@ -485,6 +510,7 @@ kubectl --context yourpc-oidc get nodes
# JupyterHub: https://jupyter.yourdomain.com # JupyterHub: https://jupyter.yourdomain.com
# MLflow: https://mlflow.yourdomain.com # MLflow: https://mlflow.yourdomain.com
# Langfuse: https://langfuse.yourdomain.com # Langfuse: https://langfuse.yourdomain.com
# LiteLLM: https://litellm.yourdomain.com
# LibreChat: https://chat.yourdomain.com # LibreChat: https://chat.yourdomain.com
``` ```

View File

@@ -2,4 +2,5 @@ clickhouse-credentials-external-secret.yaml
clickhouse-ingress.yaml clickhouse-ingress.yaml
clickhouse-installation-template.yaml clickhouse-installation-template.yaml
clickhouse-operator-values.yaml clickhouse-operator-values.yaml
clickhouse-servicemonitor.yaml
clickhouse.yaml clickhouse.yaml

View File

@@ -44,3 +44,31 @@ ClickHouse can use the following Linux capabilities for enhanced performance, bu
| `SYS_NICE` | Thread priority control via `os_thread_priority` | Setting has no effect | | `SYS_NICE` | Thread priority control via `os_thread_priority` | Setting has no effect |
These capabilities are disabled by default to comply with baseline Pod Security Standards. To enable them, the namespace must allow privileged pods, and you need to uncomment the `add` line in `clickhouse-installation-template.yaml`. These capabilities are disabled by default to comply with baseline Pod Security Standards. To enable them, the namespace must allow privileged pods, and you need to uncomment the `add` line in `clickhouse-installation-template.yaml`.
## Monitoring
ClickHouse exposes Prometheus metrics on port 9363. When Prometheus (kube-prometheus-stack) is installed, monitoring can be enabled during installation or manually.
### Enable Monitoring
```bash
just clickhouse::setup-monitoring
```
This creates a ServiceMonitor and a metrics Service for Prometheus to scrape.
### Grafana Dashboard
Import the ClickHouse dashboard from Grafana.com:
1. Open Grafana → **Dashboards****New****Import**
2. Enter Dashboard ID: `14192`
3. Click **Load**, select **Prometheus** data source, then **Import**
The dashboard includes panels for memory, connections, queries, I/O, replication, merge operations, cache, and ZooKeeper metrics.
### Remove Monitoring
```bash
just clickhouse::remove-monitoring
```

View File

@@ -20,6 +20,10 @@ spec:
containers: containers:
- name: clickhouse - name: clickhouse
image: {{ .Env.CLICKHOUSE_IMAGE }} image: {{ .Env.CLICKHOUSE_IMAGE }}
ports:
- name: prometheus
containerPort: 9363
protocol: TCP
resources: resources:
requests: requests:
cpu: {{ .Env.CLICKHOUSE_CPU_REQUEST }} cpu: {{ .Env.CLICKHOUSE_CPU_REQUEST }}

View File

@@ -0,0 +1,67 @@
{{- if .Env.MONITORING_ENABLED }}
---
apiVersion: v1
kind: Service
metadata:
name: clickhouse-metrics
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
labels:
app: clickhouse
clickhouse.altinity.com/chi: clickhouse
spec:
type: ClusterIP
ports:
- name: prometheus
port: 9363
targetPort: 9363
protocol: TCP
selector:
clickhouse.altinity.com/chi: clickhouse
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: clickhouse
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
labels:
app: clickhouse
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: clickhouse
clickhouse.altinity.com/chi: clickhouse
namespaceSelector:
matchNames:
- {{ .Env.CLICKHOUSE_NAMESPACE }}
endpoints:
- port: prometheus
path: /metrics
interval: 30s
scrapeTimeout: 10s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: clickhouse-operator
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
labels:
app: clickhouse-operator
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: altinity-clickhouse-operator
namespaceSelector:
matchNames:
- {{ .Env.CLICKHOUSE_NAMESPACE }}
endpoints:
- port: ch-metrics
path: /metrics
interval: 30s
scrapeTimeout: 10s
- port: op-metrics
path: /metrics
interval: 30s
scrapeTimeout: 10s
{{- end }}

View File

@@ -29,6 +29,17 @@ spec:
<schema_type>transposed</schema_type> <schema_type>transposed</schema_type>
</asynchronous_metric_log> </asynchronous_metric_log>
</clickhouse> </clickhouse>
# Enable Prometheus metrics endpoint
prometheus.xml: |
<clickhouse>
<prometheus>
<endpoint>/metrics</endpoint>
<port>9363</port>
<metrics>true</metrics>
<events>true</events>
<asynchronous_metrics>true</asynchronous_metrics>
</prometheus>
</clickhouse>
users: users:
admin/k8s_secret_password: clickhouse-credentials/admin admin/k8s_secret_password: clickhouse-credentials/admin
admin/networks/ip: "::/0" admin/networks/ip: "::/0"

View File

@@ -5,6 +5,8 @@ export CLICKHOUSE_HOST := env("CLICKHOUSE_HOST", "")
export CLICKHOUSE_CHART_VERSION := env("CLICKHOUSE_CHART_VERSION", "0.25.5") export CLICKHOUSE_CHART_VERSION := env("CLICKHOUSE_CHART_VERSION", "0.25.5")
export CLICKHOUSE_IMAGE := env("CLICKHOUSE_IMAGE", "clickhouse/clickhouse-server:25.10") export CLICKHOUSE_IMAGE := env("CLICKHOUSE_IMAGE", "clickhouse/clickhouse-server:25.10")
export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets") export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring")
export MONITORING_ENABLED := env("MONITORING_ENABLED", "")
# ClickHouse resource settings # ClickHouse resource settings
export CLICKHOUSE_MEMORY_REQUEST := env("CLICKHOUSE_MEMORY_REQUEST", "1Gi") export CLICKHOUSE_MEMORY_REQUEST := env("CLICKHOUSE_MEMORY_REQUEST", "1Gi")
@@ -107,6 +109,16 @@ install:
--placeholder="e.g., clickhouse.example.com" --placeholder="e.g., clickhouse.example.com"
) )
done done
# Check if Prometheus is available and ask about monitoring
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
if [ -z "${MONITORING_ENABLED}" ]; then
if gum confirm "Enable Prometheus monitoring?"; then
MONITORING_ENABLED="true"
else
MONITORING_ENABLED="false"
fi
fi
fi
echo "Installing ClickHouse..." echo "Installing ClickHouse..."
just create-namespace just create-namespace
just install-zookeeper just install-zookeeper
@@ -124,6 +136,10 @@ install:
kubectl wait --for=jsonpath='{.status.status}'=Completed \ kubectl wait --for=jsonpath='{.status.status}'=Completed \
clickhouseinstallation/clickhouse -n ${CLICKHOUSE_NAMESPACE} --timeout=600s clickhouseinstallation/clickhouse -n ${CLICKHOUSE_NAMESPACE} --timeout=600s
just setup-ingress ${CLICKHOUSE_HOST} just setup-ingress ${CLICKHOUSE_HOST}
# Setup monitoring if enabled
if [ "${MONITORING_ENABLED}" = "true" ]; then
just setup-monitoring
fi
echo "ClickHouse installation completed successfully" echo "ClickHouse installation completed successfully"
echo "ClickHouse API at: https://${CLICKHOUSE_HOST}" echo "ClickHouse API at: https://${CLICKHOUSE_HOST}"
@@ -137,6 +153,27 @@ setup-ingress host:
kubectl apply -n ${CLICKHOUSE_NAMESPACE} -f clickhouse-ingress.yaml kubectl apply -n ${CLICKHOUSE_NAMESPACE} -f clickhouse-ingress.yaml
echo "ClickHouse Ingress configured successfully" echo "ClickHouse Ingress configured successfully"
# Setup Prometheus monitoring for ClickHouse
setup-monitoring:
#!/bin/bash
set -euo pipefail
echo "Setting up Prometheus monitoring for ClickHouse..."
kubectl label namespace ${CLICKHOUSE_NAMESPACE} buun.channel/enable-monitoring=true --overwrite
MONITORING_ENABLED="true" gomplate -f clickhouse-servicemonitor.gomplate.yaml \
-o clickhouse-servicemonitor.yaml
kubectl apply -f clickhouse-servicemonitor.yaml
echo "Prometheus monitoring configured successfully"
# Remove Prometheus monitoring for ClickHouse
remove-monitoring:
#!/bin/bash
set -euo pipefail
echo "Removing Prometheus monitoring for ClickHouse..."
kubectl delete servicemonitor clickhouse clickhouse-operator -n ${CLICKHOUSE_NAMESPACE} --ignore-not-found
kubectl delete service clickhouse-metrics -n ${CLICKHOUSE_NAMESPACE} --ignore-not-found
kubectl label namespace ${CLICKHOUSE_NAMESPACE} buun.channel/enable-monitoring- --ignore-not-found
echo "Prometheus monitoring removed"
# Uninstall ClickHouse (delete_volumes='false' to preserve PVCs and namespace) # Uninstall ClickHouse (delete_volumes='false' to preserve PVCs and namespace)
uninstall delete-volumes='true': uninstall delete-volumes='true':
#!/bin/bash #!/bin/bash

View File

@@ -33,33 +33,55 @@ install:
) )
done done
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then # Check if PostgreSQL user already exists
MINIFLUX_DB_PASSWORD=$( if just postgres::user-exists ${MINIFLUX_DB_USERNAME} &>/dev/null; then
gum input --prompt="Database password (empty to auto-generate): " \ echo "PostgreSQL user '${MINIFLUX_DB_USERNAME}' already exists."
--width=100 --password # Use existing password from Vault
) if existing_db_pw=$(just vault::get miniflux/db password 2>/dev/null); then
echo "Using existing database password from Vault."
export MINIFLUX_DB_PASSWORD="${existing_db_pw}"
else
echo "Error: User exists but password not found in Vault." >&2
echo "Please delete the user first: just postgres::delete-user ${MINIFLUX_DB_USERNAME}" >&2
exit 1
fi
else
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
MINIFLUX_DB_PASSWORD=$(just utils::random-password) MINIFLUX_DB_PASSWORD=$(
echo "Generated random password: ${MINIFLUX_DB_PASSWORD}" gum input --prompt="Database password (empty to auto-generate): " \
--width=100 --password
)
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
MINIFLUX_DB_PASSWORD=$(just utils::random-password)
echo "Generated random password for database."
fi
fi fi
export MINIFLUX_DB_PASSWORD
just postgres::create-user-and-db \
${MINIFLUX_DB_USERNAME} ${MINIFLUX_DB_NAME} ${MINIFLUX_DB_PASSWORD}
just vault::put miniflux/db username=${MINIFLUX_DB_USERNAME} \
password=${MINIFLUX_DB_PASSWORD} database=${MINIFLUX_DB_NAME}
fi fi
just postgres::create-user-and-db \
${MINIFLUX_DB_USERNAME} ${MINIFLUX_DB_NAME} ${MINIFLUX_DB_PASSWORD}
just vault::put miniflux/db username=${MINIFLUX_DB_USERNAME} \
password=${MINIFLUX_DB_PASSWORD} database=${MINIFLUX_DB_NAME}
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then # Check if admin password exists in Vault
MINIFLUX_ADMIN_PASSWORD=$( if existing_admin_pw=$(just vault::get miniflux/admin password 2>/dev/null); then
gum input --prompt="Admin password (empty to auto-generate): " \ echo "Using existing admin password from Vault."
--width=100 --password export MINIFLUX_ADMIN_PASSWORD="${existing_admin_pw}"
) else
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
MINIFLUX_ADMIN_PASSWORD=$(just utils::random-password) MINIFLUX_ADMIN_PASSWORD=$(
echo "Generated random password: ${MINIFLUX_ADMIN_PASSWORD}" gum input --prompt="Admin password (empty to auto-generate): " \
--width=100 --password
)
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
MINIFLUX_ADMIN_PASSWORD=$(just utils::random-password)
echo "Generated random password for admin."
fi
fi fi
export MINIFLUX_ADMIN_PASSWORD
just vault::put miniflux/admin username=${MINIFLUX_ADMIN_USERNAME} \
password=${MINIFLUX_ADMIN_PASSWORD}
fi fi
just vault::put miniflux/admin username=${MINIFLUX_ADMIN_USERNAME} \
password=${MINIFLUX_ADMIN_PASSWORD}
# https://github.com/gabe565/charts/tree/main/charts/miniflux # https://github.com/gabe565/charts/tree/main/charts/miniflux
MINIFLUX_NAMESPACE=${MINIFLUX_NAMESPACE} \ MINIFLUX_NAMESPACE=${MINIFLUX_NAMESPACE} \
@@ -80,3 +102,26 @@ admin-username:
# Print admin password # Print admin password
admin-password: admin-password:
@just vault::get miniflux/admin password @just vault::get miniflux/admin password
# Reset admin password (deletes admin user from DB so it gets recreated on pod restart)
reset-admin-password:
#!/bin/bash
set -euo pipefail
echo "This will reset the admin password to the value stored in Vault."
if ! gum confirm "Continue?"; then
echo "Cancelled."
exit 0
fi
echo "Deleting admin user from database..."
kubectl exec -n postgres postgres-cluster-1 -c postgres -- \
psql -U postgres -d ${MINIFLUX_DB_NAME} -c "DELETE FROM users WHERE username = '${MINIFLUX_ADMIN_USERNAME}';"
echo "Restarting Miniflux pod..."
kubectl rollout restart deployment/miniflux -n ${MINIFLUX_NAMESPACE}
kubectl rollout status deployment/miniflux -n ${MINIFLUX_NAMESPACE} --timeout=60s
echo ""
echo "Admin password has been reset."
echo "Username: $(just vault::get miniflux/admin username)"
echo "Password: $(just vault::get miniflux/admin password)"

View File

@@ -15,6 +15,8 @@ ingress:
env: env:
DATABASE_URL: "postgresql://{{ .Env.MINIFLUX_DB_USERNAME }}:{{ .Env.MINIFLUX_DB_PASSWORD }}@postgres-cluster-rw.postgres:5432/{{ .Env.MINIFLUX_DB_NAME }}" DATABASE_URL: "postgresql://{{ .Env.MINIFLUX_DB_USERNAME }}:{{ .Env.MINIFLUX_DB_PASSWORD }}@postgres-cluster-rw.postgres:5432/{{ .Env.MINIFLUX_DB_NAME }}"
BASE_URL: "https://{{ .Env.MINIFLUX_HOST }}"
HTTPS: "1"
ADMIN_USERNAME: {{ .Env.MINIFLUX_ADMIN_USERNAME }} ADMIN_USERNAME: {{ .Env.MINIFLUX_ADMIN_USERNAME }}
ADMIN_PASSWORD: {{ .Env.MINIFLUX_ADMIN_PASSWORD }} ADMIN_PASSWORD: {{ .Env.MINIFLUX_ADMIN_PASSWORD }}

View File

@@ -445,6 +445,10 @@ ingress:
annotations: annotations:
kubernetes.io/ingress.class: traefik kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure traefik.ingress.kubernetes.io/router.entrypoints: websecure
# Enable sticky sessions for WebSocket connections (required for Jupyter RTC/MCP)
traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
traefik.ingress.kubernetes.io/service.sticky.cookie.name: jupyter-session
traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
ingressClassName: traefik ingressClassName: traefik
hosts: hosts:
- {{ .Env.JUPYTERHUB_HOST }} - {{ .Env.JUPYTERHUB_HOST }}

View File

@@ -23,10 +23,12 @@ mod kserve
mod langfuse mod langfuse
mod lakekeeper mod lakekeeper
mod librechat mod librechat
mod litellm
mod longhorn mod longhorn
mod metabase mod metabase
mod mlflow mod mlflow
mod minio mod minio
mod nats
mod nvidia-device-plugin mod nvidia-device-plugin
mod fairwinds-polaris mod fairwinds-polaris
mod oauth2-proxy mod oauth2-proxy
@@ -37,6 +39,7 @@ mod qdrant
mod querybook mod querybook
mod security mod security
mod superset mod superset
mod temporal
mod trino mod trino
mod utils mod utils
mod vault mod vault

View File

@@ -98,8 +98,8 @@ install:
--version ${LANGFUSE_CHART_VERSION} -n ${LANGFUSE_NAMESPACE} --wait \ --version ${LANGFUSE_CHART_VERSION} -n ${LANGFUSE_NAMESPACE} --wait \
-f langfuse-values.yaml -f langfuse-values.yaml
# Uninstall Langfuse # Uninstall Langfuse (delete-data: true to delete database and storage)
uninstall: uninstall delete-data='false':
#!/bin/bash #!/bin/bash
set -euo pipefail set -euo pipefail
helm uninstall langfuse -n ${LANGFUSE_NAMESPACE} --wait --ignore-not-found helm uninstall langfuse -n ${LANGFUSE_NAMESPACE} --wait --ignore-not-found
@@ -108,19 +108,29 @@ uninstall:
# Clean up Keycloak client and Vault secrets to avoid stale credentials # Clean up Keycloak client and Vault secrets to avoid stale credentials
just delete-keycloak-client || true just delete-keycloak-client || true
echo "Langfuse uninstalled successfully" if [ "{{ delete-data }}" = "true" ]; then
echo "" echo "Deleting database and storage..."
echo "Note: The following resources were NOT deleted:" just delete-postgres-user-and-db || true
echo " - PostgreSQL user and database (langfuse)" just delete-clickhouse-user || true
echo " - ClickHouse user and database (langfuse)" just delete-minio-user || true
echo " - MinIO user and bucket (langfuse)" just delete-keycloak-user || true
echo " - Keycloak user (langfuse)" just delete-salt || true
echo "" just delete-nextauth-secret || true
echo "To delete these resources, run:" just delete-redis-password || true
echo " just langfuse::delete-postgres-user-and-db" echo "Langfuse uninstalled with all data deleted."
echo " just langfuse::delete-clickhouse-user" else
echo " just langfuse::delete-minio-user" echo "Langfuse uninstalled successfully"
echo " just langfuse::delete-keycloak-user" echo ""
echo "Note: The following resources were NOT deleted:"
echo " - PostgreSQL user and database (langfuse)"
echo " - ClickHouse user and database (langfuse)"
echo " - MinIO user and bucket (langfuse)"
echo " - Keycloak user (langfuse)"
echo " - Vault secrets (langfuse/*)"
echo ""
echo "To delete all data, run:"
echo " just langfuse::uninstall true"
fi
# Create all secrets (PostgreSQL, Keycloak, MinIO, Redis) # Create all secrets (PostgreSQL, Keycloak, MinIO, Redis)
create-secrets: create-secrets:

View File

@@ -69,13 +69,13 @@ langfuse:
tls: tls:
enabled: true enabled: true
# Resource configuration based on Goldilocks/VPA recommendations # Resource recommendations from Goldilocks VPA
# CPU limits increased to handle startup spikes # web target: cpu=15m, memory=717Mi
web: web:
resources: resources:
requests: requests:
cpu: 15m cpu: 25m
memory: 704Mi memory: 768Mi
limits: limits:
cpu: 100m cpu: 100m
memory: 1.5Gi memory: 1.5Gi
@@ -89,10 +89,12 @@ langfuse:
timeoutSeconds: 30 timeoutSeconds: 30
failureThreshold: 5 failureThreshold: 5
# Resource recommendations from Goldilocks VPA
# worker target: cpu=15m, memory=380Mi
worker: worker:
resources: resources:
requests: requests:
cpu: 15m cpu: 25m
memory: 512Mi memory: 512Mi
limits: limits:
cpu: 100m cpu: 100m
@@ -113,6 +115,16 @@ redis:
username: "default" username: "default"
existingSecret: redis-auth existingSecret: redis-auth
existingSecretPasswordKey: secret existingSecretPasswordKey: secret
# Resource recommendations from Goldilocks VPA
# valkey target: cpu=15m, memory=100Mi
master:
resources:
requests:
cpu: 25m
memory: 128Mi
limits:
cpu: 100m
memory: 256Mi
clickhouse: clickhouse:
deploy: false deploy: false

View File

@@ -94,10 +94,10 @@ ingress:
resources: resources:
requests: requests:
cpu: 100m cpu: 25m
memory: 512Mi memory: 512Mi
limits: limits:
cpu: 1000m cpu: 100m
memory: 1Gi memory: 1Gi
mongodb: mongodb:
@@ -110,6 +110,13 @@ mongodb:
tag: "latest" tag: "latest"
persistence: persistence:
size: 8Gi size: 8Gi
resources:
requests:
cpu: 75m
memory: 512Mi
limits:
cpu: 500m
memory: 2Gi
podSecurityContext: podSecurityContext:
fsGroup: 1001 fsGroup: 1001
seccompProfile: seccompProfile:
@@ -132,6 +139,13 @@ meilisearch:
tag: "v1.7.3" tag: "v1.7.3"
auth: auth:
existingMasterKeySecret: "librechat-credentials-env" existingMasterKeySecret: "librechat-credentials-env"
resources:
requests:
cpu: 25m
memory: 256Mi
limits:
cpu: 100m
memory: 1Gi
podSecurityContext: podSecurityContext:
fsGroup: 1000 fsGroup: 1000
seccompProfile: seccompProfile:

3
litellm/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
litellm-values.yaml
apikey-external-secret.yaml
models.yaml

547
litellm/README.md Normal file
View File

@@ -0,0 +1,547 @@
# LiteLLM
Unified LLM gateway and proxy for accessing multiple LLM providers through a single OpenAI-compatible API:
- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, Cohere, Azure, Bedrock, Vertex AI
- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK
- **Load Balancing & Fallback**: Automatic failover between providers
- **Virtual Keys**: Generate API keys for users with usage tracking
- **Cost Tracking**: Monitor spending across providers
- **Rate Limiting**: Control usage per key/user
## Prerequisites
- Kubernetes cluster (k3s)
- External Secrets Operator (required)
- PostgreSQL cluster (CloudNativePG)
- Vault for secrets management
## Configuration Overview
LiteLLM requires two types of configuration:
1. **Environment variables** (`.env.local`): Host, namespace, chart version
2. **Model definitions** (`models.yaml`): LLM providers and models to expose
This separation allows flexible model configuration without modifying environment files.
## Installation
### Step 1: Create Model Configuration
Copy the example configuration and customize:
```bash
cp litellm/models.example.yaml litellm/models.yaml
```
Edit `litellm/models.yaml` to configure your models:
```yaml
# Anthropic Claude
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-3-7-sonnet-latest
api_key: os.environ/ANTHROPIC_API_KEY
# OpenAI
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
# Ollama (local models - no API key required)
- model_name: llama3
litellm_params:
model: ollama/llama3.2
api_base: http://ollama.ollama:11434
```
### Step 2: Set API Keys
For each provider that requires an API key:
```bash
just litellm::set-api-key anthropic
just litellm::set-api-key openai
```
Or interactively select the provider:
```bash
just litellm::set-api-key
```
API keys are stored in Vault and synced to Kubernetes via External Secrets Operator.
### Step 3: Install LiteLLM
```bash
just litellm::install
```
You will be prompted for:
- **LiteLLM host (FQDN)**: e.g., `litellm.example.com`
- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed
## Model Management
### Add a Model Interactively
```bash
just litellm::add-model
```
This guides you through:
1. Selecting a provider
2. Choosing a model
3. Setting a model alias
### Remove a Model
```bash
just litellm::remove-model
```
### List Configured Models
```bash
just litellm::list-models
```
### Example Output
```text
Configured models:
- claude-sonnet: anthropic/claude-3-7-sonnet-latest
- claude-haiku: anthropic/claude-3-5-haiku-latest
- llama3: ollama/llama3.2
```
## API Key Management
### Set API Key for a Provider
```bash
just litellm::set-api-key anthropic
```
### Get API Key (from Vault)
```bash
just litellm::get-api-key anthropic
```
### Verify All Required Keys
```bash
just litellm::verify-api-keys
```
## Environment Variables
| Variable | Default | Description |
| -------- | ------- | ----------- |
| `LITELLM_NAMESPACE` | `litellm` | Kubernetes namespace |
| `LITELLM_CHART_VERSION` | `0.1.825` | Helm chart version |
| `LITELLM_HOST` | (prompt) | External hostname (FQDN) |
| `OLLAMA_NAMESPACE` | `ollama` | Ollama namespace for local models |
| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor |
## Authentication
LiteLLM has two types of authentication:
1. **API Access**: Uses Master Key or Virtual Keys for programmatic access
2. **Admin UI**: Uses Keycloak SSO for browser-based access
### Enable SSO for Admin UI
After installing LiteLLM, enable Keycloak authentication for the Admin UI:
```bash
just litellm::setup-oidc
```
This will:
- Create a Keycloak client for LiteLLM
- Store the client secret in Vault
- Configure LiteLLM with OIDC environment variables
- Upgrade the deployment with SSO enabled
### Disable SSO
To disable SSO and return to unauthenticated Admin UI access:
```bash
just litellm::disable-oidc
```
### SSO Configuration Details
| Setting | Value |
| ------- | ----- |
| Callback URL | `https://<litellm-host>/sso/callback` |
| Authorization Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth` |
| Token Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/token` |
| Userinfo Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/userinfo` |
| Scope | `openid email profile` |
## User Management
SSO users are automatically created in LiteLLM when they first log in. By default, new users are assigned the `internal_user_viewer` role (read-only access).
### List Users
```bash
just litellm::list-users
```
### Assign Role to User
Interactively select user and role:
```bash
just litellm::assign-role
```
Or specify directly:
```bash
just litellm::assign-role buun proxy_admin
```
### User Roles
| Role | Description |
| ---- | ----------- |
| `proxy_admin` | Full admin access (manage keys, users, models, settings) |
| `proxy_admin_viewer` | Admin read-only access |
| `internal_user` | Can create and manage own API keys |
| `internal_user_viewer` | Read-only access (default for SSO users) |
**Note**: To manage API keys in the Admin UI, users need at least `internal_user` or `proxy_admin` role.
## API Usage
LiteLLM exposes an OpenAI-compatible API at `https://your-litellm-host/`.
### Get Master Key
```bash
just litellm::master-key
```
### Generate Virtual Key for a User
```bash
just litellm::generate-virtual-key buun
```
This will prompt for a model selection and generate an API key for the specified user. Select `all` to grant access to all models.
### OpenAI SDK Example
```python
from openai import OpenAI
client = OpenAI(
base_url="https://litellm.example.com",
api_key="sk-..." # Virtual key or master key
)
response = client.chat.completions.create(
model="claude-sonnet", # Use your model alias
messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```
### curl Example
```bash
curl https://litellm.example.com/v1/chat/completions \
-H "Authorization: Bearer sk-..." \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
## Team Management
Teams allow you to group users and configure team-specific settings such as Langfuse projects for observability.
### Create a Team
```bash
just litellm::create-team
```
Or with a name directly:
```bash
just litellm::create-team name="project-alpha"
```
### List Teams
```bash
just litellm::list-teams
```
### Get Team Info
```bash
just litellm::get-team team_id=<team-id>
```
### Delete a Team
```bash
just litellm::delete-team team_id=<team-id>
```
### Generate Virtual Key for a Team
```bash
just litellm::generate-team-key
```
This will prompt for team selection and username. The generated key inherits the team's settings (including Langfuse project configuration).
## Langfuse Integration
[Langfuse](https://langfuse.com/) provides LLM observability with tracing, monitoring, and analytics. LiteLLM can send traces to Langfuse for every API call.
### Enable Langfuse Integration
During installation (`just litellm::install`) or upgrade (`just litellm::upgrade`), you will be prompted to enable Langfuse integration. Alternatively:
```bash
just litellm::setup-langfuse
```
You will need Langfuse API keys (Public Key and Secret Key) from the Langfuse UI: **Settings > API Keys**.
### Set Langfuse API Keys
```bash
just litellm::set-langfuse-keys
```
### Disable Langfuse Integration
```bash
just litellm::disable-langfuse
```
### Per-Team Langfuse Projects
Each team can have its own Langfuse project for isolated observability. This is useful when different projects or departments need separate trace data.
#### Setup Flow
1. Create a team:
```bash
just litellm::create-team name="project-alpha"
```
2. Create a Langfuse project for the team and get API keys from Langfuse UI
3. Configure the team's Langfuse project:
```bash
just litellm::set-team-langfuse-project
```
This will prompt for team selection and Langfuse API keys.
4. Generate a key for the team:
```bash
just litellm::generate-team-key
```
5. Use the team key for API calls - traces will be sent to the team's Langfuse project
#### Architecture
```plain
LiteLLM Proxy
|
+-- Default Langfuse Project (for keys without team)
|
+-- Team A --> Langfuse Project A
|
+-- Team B --> Langfuse Project B
```
### Environment Variables
| Variable | Default | Description |
| -------- | ------- | ----------- |
| `LITELLM_LANGFUSE_INTEGRATION_ENABLED` | (prompt) | Enable Langfuse integration |
| `LANGFUSE_HOST` | (prompt) | Langfuse instance hostname |
## Supported Providers
| Provider | Model Prefix | API Key Required |
| -------- | ------------ | ---------------- |
| Anthropic | `anthropic/` | Yes |
| OpenAI | `openai/` | Yes |
| Ollama | `ollama/` | No (uses `api_base`) |
| Mistral | `mistral/` | Yes |
| Groq | `groq/` | Yes |
| Cohere | `cohere/` | Yes |
| Azure OpenAI | `azure/` | Yes |
| AWS Bedrock | `bedrock/` | Yes |
| Google Vertex AI | `vertexai/` | Yes |
## Architecture
```plain
External Users/Applications
|
Cloudflare Tunnel (HTTPS)
|
Traefik Ingress (HTTPS)
|
LiteLLM Proxy (HTTP inside cluster)
|-- PostgreSQL (usage tracking, virtual keys)
|-- Redis (caching, rate limiting)
|-- External Secrets (API keys from Vault)
|
+-- Anthropic API
+-- OpenAI API
+-- Ollama (local)
+-- Other providers...
```
## Upgrade
After modifying `models.yaml` or updating API keys:
```bash
just litellm::upgrade
```
## Uninstall
```bash
just litellm::uninstall
```
This removes:
- Helm release and all Kubernetes resources
- Namespace
- External Secrets
**Note**: The following resources are NOT deleted:
- PostgreSQL database (use `just postgres::delete-db litellm`)
- API keys in Vault
### Full Cleanup
To remove everything including database and Vault secrets:
```bash
just litellm::cleanup
```
## Troubleshooting
### Check Pod Status
```bash
kubectl get pods -n litellm
```
Expected pods:
- `litellm-*` - LiteLLM proxy
- `litellm-redis-master-0` - Redis instance
### View Logs
```bash
kubectl logs -n litellm deployment/litellm --tail=100
```
### API Key Not Working
Verify the ExternalSecret is synced:
```bash
kubectl get externalsecret -n litellm
kubectl get secret apikey -n litellm -o yaml
```
### Model Not Found
Ensure the model is configured in `models.yaml` and the deployment is updated:
```bash
just litellm::list-models
just litellm::upgrade
```
### Provider API Errors
Check if the API key is set correctly:
```bash
just litellm::get-api-key anthropic
```
If empty, set the API key:
```bash
just litellm::set-api-key anthropic
```
### Database Connection Issues
Check PostgreSQL connectivity:
```bash
kubectl exec -n litellm deployment/litellm -- \
psql -h postgres-cluster-rw.postgres -U litellm -d litellm -c "SELECT 1"
```
## Configuration Files
| File | Description |
| ---- | ----------- |
| `models.yaml` | Model definitions (user-created, gitignored) |
| `models.example.yaml` | Example model configuration |
| `litellm-values.gomplate.yaml` | Helm values template |
| `apikey-external-secret.gomplate.yaml` | ExternalSecret for API keys |
| `keycloak-auth-external-secret.gomplate.yaml` | ExternalSecret for Keycloak OIDC |
| `langfuse-auth-external-secret.gomplate.yaml` | ExternalSecret for Langfuse API keys |
## Security Considerations
- **Pod Security Standards**: Namespace configured with **baseline** enforcement
(LiteLLM's Prisma requires write access to `/.cache`, which prevents `restricted` level)
- **Secrets Management**: API keys stored in Vault, synced via External Secrets Operator
- **Virtual Keys**: Generate scoped API keys for users instead of sharing master key
- **TLS/HTTPS**: All external traffic encrypted via Traefik Ingress
- **Database Credentials**: Unique PostgreSQL user with minimal privileges
## References
- [LiteLLM Documentation](https://docs.litellm.ai/)
- [LiteLLM GitHub](https://github.com/BerriAI/litellm)
- [LiteLLM Helm Chart](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm)
- [Supported Models](https://docs.litellm.ai/docs/providers)
- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys)
- [Langfuse Integration](https://docs.litellm.ai/docs/proxy/logging#langfuse)
- [Team-based Logging](https://docs.litellm.ai/docs/proxy/team_logging)

View File

@@ -0,0 +1,29 @@
{{- $models := (datasource "models") -}}
{{- $providerMap := dict -}}
{{- range $models -}}
{{- if has .litellm_params "api_key" -}}
{{- $parts := strings.Split "/" .litellm_params.model -}}
{{- $provider := index $parts 0 -}}
{{- $providerMap = merge $providerMap (dict $provider true) -}}
{{- end -}}
{{- end -}}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: apikey-external-secret
namespace: {{ .Env.LITELLM_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: apikey
creationPolicy: Owner
data:
{{- range $provider, $_ := $providerMap }}
- secretKey: {{ $provider | strings.ToUpper }}_API_KEY
remoteRef:
key: litellm/{{ $provider }}
property: apikey
{{- end }}

1074
litellm/justfile Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,22 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: keycloak-auth-external-secret
namespace: {{ .Env.LITELLM_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: keycloak-auth
creationPolicy: Owner
data:
- secretKey: GENERIC_CLIENT_ID
remoteRef:
key: keycloak/client/litellm
property: client_id
- secretKey: GENERIC_CLIENT_SECRET
remoteRef:
key: keycloak/client/litellm
property: client_secret

View File

@@ -0,0 +1,22 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: langfuse-auth-external-secret
namespace: {{ .Env.LITELLM_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: langfuse-auth
creationPolicy: Owner
data:
- secretKey: LANGFUSE_PUBLIC_KEY
remoteRef:
key: litellm/langfuse
property: public_key
- secretKey: LANGFUSE_SECRET_KEY
remoteRef:
key: litellm/langfuse
property: secret_key

View File

@@ -0,0 +1,114 @@
# https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm
# https://github.com/BerriAI/litellm/tree/main/litellm/proxy/example_config_yaml
masterkeySecretName: ""
masterkeySecretKey: ""
# Note: LiteLLM image requires write access to /.cache for Prisma
# Pod Security Standards must be set to "baseline" for this namespace
podSecurityContext: {}
securityContext: {}
# Resource recommendations from Goldilocks VPA
# litellm target: cpu=11m, memory=549Mi
resources:
requests:
cpu: 25m
memory: 512Mi
limits:
cpu: 100m
memory: 1Gi
migrationJob:
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
memory: 1Gi
environmentSecrets:
- apikey
{{- if .Env.LITELLM_OIDC_ENABLED }}
- keycloak-auth
{{- end }}
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
- langfuse-auth
{{- end }}
extraEnvVars:
{{- if .Env.LITELLM_OIDC_ENABLED }}
- name: PROXY_BASE_URL
value: "https://{{ .Env.LITELLM_HOST }}"
- name: GENERIC_AUTHORIZATION_ENDPOINT
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/auth"
- name: GENERIC_TOKEN_ENDPOINT
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/token"
- name: GENERIC_USERINFO_ENDPOINT
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/userinfo"
- name: GENERIC_SCOPE
value: "openid email profile"
{{- end }}
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
- name: LANGFUSE_HOST
value: "https://{{ .Env.LANGFUSE_HOST }}"
{{- end }}
proxy_config:
model_list:
{{ file.Read "models.yaml" | indent 4 }}
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
litellm_settings:
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
{{- end }}
db:
useExisting: true
endpoint: postgres-cluster-rw.postgres
database: litellm
secret:
name: postgres-auth
usernameKey: username
passwordKey: password
deployStandalone: false
redis:
enabled: true
# Resource recommendations from Goldilocks VPA
# redis target: cpu=15m, memory=100Mi
master:
resources:
requests:
cpu: 25m
memory: 128Mi
limits:
cpu: 100m
memory: 256Mi
ingress:
enabled: true
className: traefik
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
hosts:
- host: {{ .Env.LITELLM_HOST }}
paths:
- path: /
pathType: ImplementationSpecific
tls:
- hosts:
- {{ .Env.LITELLM_HOST }}
{{- if .Env.MONITORING_ENABLED }}
serviceMonitor:
enabled: true
labels:
release: kube-prometheus-stack
interval: 30s
scrapeTimeout: 10s
{{- end }}

106
litellm/models.example.yaml Normal file
View File

@@ -0,0 +1,106 @@
# LiteLLM Model Configuration
# Copy this file to models.yaml and customize for your environment.
#
# Usage:
# cp litellm/models.example.yaml litellm/models.yaml
# # Edit models.yaml to add/remove models
# just litellm::install
#
# API keys are stored in Vault and injected as environment variables.
# Use: just litellm::set-api-key provider=<provider>
#
# Supported providers:
# - anthropic: Claude models (Opus, Sonnet, Haiku)
# - openai: GPT and o-series models
# - ollama: Local models (no API key required)
# - azure: Azure OpenAI
# - bedrock: AWS Bedrock
# - vertexai: Google Vertex AI
# - mistral: Mistral AI
# - groq: Groq (fast inference)
# - cohere: Cohere
# Anthropic Claude (https://docs.anthropic.com/en/docs/about-claude/models/overview)
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-haiku
litellm_params:
model: anthropic/claude-haiku-4-20251015
api_key: os.environ/ANTHROPIC_API_KEY
# - model_name: claude-opus
# litellm_params:
# model: anthropic/claude-opus-4-20250514
# api_key: os.environ/ANTHROPIC_API_KEY
# OpenAI (https://platform.openai.com/docs/models)
# - model_name: gpt-4o
# litellm_params:
# model: openai/gpt-4o
# api_key: os.environ/OPENAI_API_KEY
# - model_name: gpt-4o-mini
# litellm_params:
# model: openai/gpt-4o-mini
# api_key: os.environ/OPENAI_API_KEY
# - model_name: o3
# litellm_params:
# model: openai/o3
# api_key: os.environ/OPENAI_API_KEY
# - model_name: o4-mini
# litellm_params:
# model: openai/o4-mini
# api_key: os.environ/OPENAI_API_KEY
# Ollama (local models - no API key required)
# - model_name: llama4-scout
# litellm_params:
# model: ollama/llama4:scout
# api_base: http://ollama.ollama:11434
# - model_name: qwen3
# litellm_params:
# model: ollama/qwen3:8b
# api_base: http://ollama.ollama:11434
# - model_name: deepseek-r1
# litellm_params:
# model: ollama/deepseek-r1:8b
# api_base: http://ollama.ollama:11434
# Mistral AI (https://docs.mistral.ai/getting-started/models/models_overview/)
# - model_name: mistral-large
# litellm_params:
# model: mistral/mistral-large-latest
# api_key: os.environ/MISTRAL_API_KEY
# - model_name: ministral-8b
# litellm_params:
# model: mistral/ministral-8b-latest
# api_key: os.environ/MISTRAL_API_KEY
# - model_name: codestral
# litellm_params:
# model: mistral/codestral-latest
# api_key: os.environ/MISTRAL_API_KEY
# Groq (fast inference - https://console.groq.com/docs/models)
# - model_name: groq-llama4-scout
# litellm_params:
# model: groq/meta-llama/llama-4-scout-17b-16e-instruct
# api_key: os.environ/GROQ_API_KEY
# - model_name: groq-llama3.3
# litellm_params:
# model: groq/llama-3.3-70b-versatile
# api_key: os.environ/GROQ_API_KEY
# - model_name: groq-llama3.1
# litellm_params:
# model: groq/llama-3.1-8b-instant
# api_key: os.environ/GROQ_API_KEY

View File

@@ -221,6 +221,83 @@ just minio::bucket-exists mybucket
This returns exit code 0 if the bucket exists, 1 otherwise. This returns exit code 0 if the bucket exists, 1 otherwise.
## Public Access
MinIO allows you to configure anonymous (public) access to buckets or specific prefixes for serving static content like images.
### Set Public Download Access
Enable public read access for a bucket or prefix:
```bash
# Set public access for entire bucket
just minio::set-public-download mybucket
# Set public access for specific prefix only
just minio::set-public-download mybucket/public
```
After setting public access, files can be accessed without authentication:
```text
https://your-minio-host/mybucket/public/image.png
```
### Check Public Access Status
View current anonymous access policy:
```bash
just minio::show-public-access mybucket
```
Possible values:
- `private`: No anonymous access (default)
- `download`: Public read access
- `upload`: Public write access
- `public`: Public read and write access
- `custom`: Custom policy applied
### Remove Public Access
Revoke anonymous access:
```bash
just minio::remove-public-access mybucket/public
```
### Using mc Commands
```bash
# Set public download (read-only)
mc anonymous set download myminio/mybucket/public
# Set public upload (write-only)
mc anonymous set upload myminio/mybucket/uploads
# Set full public access (read and write)
mc anonymous set public myminio/mybucket
# Remove public access
mc anonymous set none myminio/mybucket
# Check current policy
mc anonymous get myminio/mybucket
```
### Presigned URLs (Temporary Access)
For temporary access to private objects, use presigned URLs:
```bash
# Generate URL valid for 7 days
mc share download myminio/mybucket/private-file.pdf --expire=168h
# Generate upload URL
mc share upload myminio/mybucket/uploads/ --expire=1h
```
## User Management ## User Management
### Create MinIO User ### Create MinIO User

View File

@@ -47,6 +47,7 @@ create-root-credentials:
gomplate -f minio-root-external-secret.gomplate.yaml | kubectl apply -f - gomplate -f minio-root-external-secret.gomplate.yaml | kubectl apply -f -
echo "Waiting for ExternalSecret to sync..." echo "Waiting for ExternalSecret to sync..."
sleep 2
kubectl wait --for=condition=Ready externalsecret/minio \ kubectl wait --for=condition=Ready externalsecret/minio \
-n ${MINIO_NAMESPACE} --timeout=60s -n ${MINIO_NAMESPACE} --timeout=60s
else else
@@ -96,7 +97,12 @@ install:
--placeholder="e.g., minio-console.example.com" --placeholder="e.g., minio-console.example.com"
) )
fi fi
# Generate OIDC client secret for confidential client
OIDC_CLIENT_SECRET=$(just utils::random-password)
just keycloak::create-client realm=${KEYCLOAK_REALM} client_id=${MINIO_OIDC_CLIENT_ID} \ just keycloak::create-client realm=${KEYCLOAK_REALM} client_id=${MINIO_OIDC_CLIENT_ID} \
client_secret="${OIDC_CLIENT_SECRET}" \
redirect_url="https://${MINIO_HOST}/oauth_callback,https://${MINIO_CONSOLE_HOST}/oauth_callback" redirect_url="https://${MINIO_HOST}/oauth_callback,https://${MINIO_CONSOLE_HOST}/oauth_callback"
just add-keycloak-minio-policy just add-keycloak-minio-policy
just create-namespace just create-namespace
@@ -105,6 +111,28 @@ install:
pod-security.kubernetes.io/enforce=restricted --overwrite pod-security.kubernetes.io/enforce=restricted --overwrite
just create-root-credentials just create-root-credentials
# Store OIDC client secret
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
echo "Storing OIDC client secret in Vault..."
just vault::put minio/oidc client_id="${MINIO_OIDC_CLIENT_ID}" client_secret="${OIDC_CLIENT_SECRET}"
kubectl delete externalsecret minio-oidc -n ${MINIO_NAMESPACE} --ignore-not-found
gomplate -f minio-oidc-external-secret.gomplate.yaml | kubectl apply -f -
echo "Waiting for ExternalSecret to sync..."
sleep 2
kubectl wait --for=condition=Ready externalsecret/minio-oidc \
-n ${MINIO_NAMESPACE} --timeout=60s
else
echo "Creating OIDC client secret directly..."
kubectl delete secret minio-oidc -n ${MINIO_NAMESPACE} --ignore-not-found
kubectl create secret generic minio-oidc -n ${MINIO_NAMESPACE} \
--from-literal=clientId="${MINIO_OIDC_CLIENT_ID}" \
--from-literal=clientSecret="${OIDC_CLIENT_SECRET}"
if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then
just vault::put minio/oidc client_id="${MINIO_OIDC_CLIENT_ID}" client_secret="${OIDC_CLIENT_SECRET}"
fi
fi
just add-helm-repo just add-helm-repo
gomplate -f minio-values.gomplate.yaml -o minio-values.yaml gomplate -f minio-values.gomplate.yaml -o minio-values.yaml
helm upgrade --install minio minio/minio \ helm upgrade --install minio minio/minio \
@@ -260,3 +288,70 @@ grant-policy user='' policy='':
mc admin policy attach local ${POLICY} --user=${USER}" mc admin policy attach local ${POLICY} --user=${USER}"
echo "✅ Policy ${POLICY} granted to user ${USER}" echo "✅ Policy ${POLICY} granted to user ${USER}"
# Set public download access for a bucket or prefix
set-public-download path='':
#!/bin/bash
set -euo pipefail
PATH_ARG="{{ path }}"
while [ -z "${PATH_ARG}" ]; do
PATH_ARG=$(
gum input --prompt="Bucket/prefix path: " --width=100 \
--placeholder="e.g., my-bucket/public"
)
done
ROOT_USER=$(just root-username)
ROOT_PASSWORD=$(just root-password)
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc anonymous set download local/${PATH_ARG}
echo "✅ Public download access enabled for ${PATH_ARG}"
# Remove public access from a bucket or prefix
remove-public-access path='':
#!/bin/bash
set -euo pipefail
PATH_ARG="{{ path }}"
while [ -z "${PATH_ARG}" ]; do
PATH_ARG=$(
gum input --prompt="Bucket/prefix path: " --width=100 \
--placeholder="e.g., my-bucket/public"
)
done
ROOT_USER=$(just root-username)
ROOT_PASSWORD=$(just root-password)
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc anonymous set none local/${PATH_ARG}
echo "✅ Public access removed from ${PATH_ARG}"
# Show anonymous access policy for a bucket or prefix
show-public-access path='':
#!/bin/bash
set -euo pipefail
PATH_ARG="{{ path }}"
while [ -z "${PATH_ARG}" ]; do
PATH_ARG=$(
gum input --prompt="Bucket/prefix path: " --width=100 \
--placeholder="e.g., my-bucket"
)
done
ROOT_USER=$(just root-username)
ROOT_PASSWORD=$(just root-password)
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
mc anonymous get local/${PATH_ARG}

View File

@@ -0,0 +1,22 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: minio-oidc
namespace: {{ .Env.MINIO_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: minio-oidc
creationPolicy: Owner
data:
- secretKey: clientId
remoteRef:
key: minio/oidc
property: client_id
- secretKey: clientSecret
remoteRef:
key: minio/oidc
property: client_secret

View File

@@ -7,8 +7,9 @@ existingSecret: "minio"
oidc: oidc:
enabled: true enabled: true
configUrl: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/.well-known/openid-configuration" configUrl: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/.well-known/openid-configuration"
clientId: "{{ .Env.MINIO_OIDC_CLIENT_ID }}" existingClientSecretName: "minio-oidc"
clientSecret: "" existingClientIdKey: "clientId"
existingClientSecretKey: "clientSecret"
claimName: "minioPolicy" claimName: "minioPolicy"
scopes: "openid,profile,email" scopes: "openid,profile,email"
redirectUri: "https://{{ .Env.MINIO_CONSOLE_HOST }}/oauth_callback" redirectUri: "https://{{ .Env.MINIO_CONSOLE_HOST }}/oauth_callback"

View File

@@ -3,11 +3,11 @@ gomplate = "4.3.3"
gum = "0.16.2" gum = "0.16.2"
helm = "3.19.0" helm = "3.19.0"
just = "1.42.4" just = "1.42.4"
k3sup = "0.13.10" k3sup = "0.13.11"
kubelogin = "1.34.0" kubelogin = "1.34.0"
node = "22.18.0" node = "22.18.0"
python = "3.12.11" python = "3.12.11"
telepresence = "2.25.0" telepresence = "2.25.1"
trivy = "0.67.2" trivy = "0.67.2"
uv = "0.8.7" uv = "0.8.7"
vault = "1.20.2" vault = "1.20.2"

113
nats/justfile Normal file
View File

@@ -0,0 +1,113 @@
set fallback := true
export NATS_NAMESPACE := env("NATS_NAMESPACE", "nats")
export NATS_CHART_VERSION := env("NATS_CHART_VERSION", "2.12.2")
export NATS_REPLICAS := env("NATS_REPLICAS", "1")
export NATS_JETSTREAM_ENABLED := env("NATS_JETSTREAM_ENABLED", "true")
export NATS_JETSTREAM_STORAGE_SIZE := env("NATS_JETSTREAM_STORAGE_SIZE", "10Gi")
export NATS_JETSTREAM_MEMORY_SIZE := env("NATS_JETSTREAM_MEMORY_SIZE", "256Mi")
[private]
default:
@just --list --unsorted --list-submodules
# Add Helm repository
add-helm-repo:
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update nats
# Remove Helm repository
remove-helm-repo:
helm repo remove nats
# Create NATS namespace
create-namespace:
#!/bin/bash
set -euo pipefail
if ! kubectl get namespace ${NATS_NAMESPACE} &>/dev/null; then
kubectl create namespace ${NATS_NAMESPACE}
fi
kubectl label namespace ${NATS_NAMESPACE} \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=latest \
pod-security.kubernetes.io/warn=restricted \
pod-security.kubernetes.io/warn-version=latest \
--overwrite
# Delete NATS namespace
delete-namespace:
kubectl delete namespace ${NATS_NAMESPACE} --ignore-not-found
# Install NATS
install:
#!/bin/bash
set -euo pipefail
just create-namespace
just add-helm-repo
gomplate -f values.gomplate.yaml -o values.yaml
helm upgrade --install nats nats/nats \
--version ${NATS_CHART_VERSION} \
-n ${NATS_NAMESPACE} \
-f values.yaml \
--wait
echo ""
echo "NATS installed successfully"
echo "Namespace: ${NATS_NAMESPACE}"
echo "Replicas: ${NATS_REPLICAS}"
echo "JetStream enabled: ${NATS_JETSTREAM_ENABLED}"
echo ""
echo "Internal URL: nats://nats.${NATS_NAMESPACE}.svc:4222"
# Upgrade NATS
upgrade:
#!/bin/bash
set -euo pipefail
gomplate -f values.gomplate.yaml -o values.yaml
helm upgrade nats nats/nats \
--version ${NATS_CHART_VERSION} \
-n ${NATS_NAMESPACE} \
-f values.yaml \
--wait
echo "NATS upgraded successfully"
# Uninstall NATS
uninstall:
#!/bin/bash
set -euo pipefail
if ! gum confirm "Are you sure you want to uninstall NATS?"; then
echo "Aborted"
exit 0
fi
helm uninstall nats -n ${NATS_NAMESPACE} --wait --ignore-not-found
just delete-namespace
echo "NATS uninstalled"
# Show NATS status
status:
kubectl get pods -n ${NATS_NAMESPACE}
kubectl get svc -n ${NATS_NAMESPACE}
# Show NATS logs
logs:
kubectl logs -n ${NATS_NAMESPACE} -l app.kubernetes.io/name=nats -f
# Show server info via monitoring endpoint
server-info:
kubectl exec -n ${NATS_NAMESPACE} nats-0 -c nats -- \
wget -qO- http://localhost:8222/varz | head -50
# Show JetStream info via monitoring endpoint
js-info:
kubectl exec -n ${NATS_NAMESPACE} nats-0 -c nats -- \
wget -qO- http://localhost:8222/jsz
# Port forward for local testing
port-forward:
@echo "NATS available at localhost:4222"
@echo "Monitor available at http://localhost:8222"
kubectl port-forward -n ${NATS_NAMESPACE} svc/nats 4222:4222 8222:8222

64
nats/values.gomplate.yaml Normal file
View File

@@ -0,0 +1,64 @@
config:
cluster:
enabled: {{ if gt (conv.ToInt .Env.NATS_REPLICAS) 1 }}true{{ else }}false{{ end }}
replicas: {{ .Env.NATS_REPLICAS }}
jetstream:
enabled: {{ .Env.NATS_JETSTREAM_ENABLED }}
fileStore:
enabled: true
dir: /data
pvc:
enabled: true
size: {{ .Env.NATS_JETSTREAM_STORAGE_SIZE }}
memoryStore:
enabled: true
maxSize: {{ .Env.NATS_JETSTREAM_MEMORY_SIZE }}
monitor:
enabled: true
port: 8222
container:
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 512Mi
merge:
securityContext:
runAsUser: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
reloader:
enabled: true
merge:
securityContext:
runAsUser: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
podTemplate:
merge:
spec:
securityContext:
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
natsBox:
enabled: false

64
nats/values.yaml Normal file
View File

@@ -0,0 +1,64 @@
config:
cluster:
enabled: false
replicas: 1
jetstream:
enabled: true
fileStore:
enabled: true
dir: /data
pvc:
enabled: true
size: 10Gi
memoryStore:
enabled: true
maxSize: 256Mi
monitor:
enabled: true
port: 8222
container:
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 512Mi
merge:
securityContext:
runAsUser: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
reloader:
enabled: true
merge:
securityContext:
runAsUser: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
capabilities:
drop:
- ALL
podTemplate:
merge:
spec:
securityContext:
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
natsBox:
enabled: false

View File

@@ -8,6 +8,11 @@ export OLLAMA_GPU_TYPE := env("OLLAMA_GPU_TYPE", "nvidia")
export OLLAMA_GPU_COUNT := env("OLLAMA_GPU_COUNT", "1") export OLLAMA_GPU_COUNT := env("OLLAMA_GPU_COUNT", "1")
export OLLAMA_MODELS := env("OLLAMA_MODELS", "") export OLLAMA_MODELS := env("OLLAMA_MODELS", "")
export OLLAMA_STORAGE_SIZE := env("OLLAMA_STORAGE_SIZE", "30Gi") export OLLAMA_STORAGE_SIZE := env("OLLAMA_STORAGE_SIZE", "30Gi")
export OLLAMA_HELM_TIMEOUT := env("OLLAMA_HELM_TIMEOUT", "60m")
export OLLAMA_MEMORY_REQUEST := env("OLLAMA_MEMORY_REQUEST", "2Gi")
export OLLAMA_MEMORY_LIMIT := env("OLLAMA_MEMORY_LIMIT", "12Gi")
export OLLAMA_CPU_REQUEST := env("OLLAMA_CPU_REQUEST", "25m")
export OLLAMA_CPU_LIMIT := env("OLLAMA_CPU_LIMIT", "100m")
[private] [private]
default: default:
@@ -66,7 +71,8 @@ install:
gomplate -f values.gomplate.yaml -o values.yaml gomplate -f values.gomplate.yaml -o values.yaml
helm upgrade --install ollama ollama/ollama \ helm upgrade --install ollama ollama/ollama \
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} --wait \ --version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} \
--wait --timeout ${OLLAMA_HELM_TIMEOUT} \
-f values.yaml -f values.yaml
echo "" echo ""
@@ -97,7 +103,8 @@ upgrade:
gomplate -f values.gomplate.yaml -o values.yaml gomplate -f values.gomplate.yaml -o values.yaml
helm upgrade ollama ollama/ollama \ helm upgrade ollama ollama/ollama \
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} --wait \ --version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} \
--wait --timeout ${OLLAMA_HELM_TIMEOUT} \
-f values.yaml -f values.yaml
echo "Ollama upgraded successfully" echo "Ollama upgraded successfully"

View File

@@ -36,11 +36,11 @@ securityContext:
resources: resources:
requests: requests:
cpu: 25m cpu: {{ .Env.OLLAMA_CPU_REQUEST }}
memory: 2Gi memory: {{ .Env.OLLAMA_MEMORY_REQUEST }}
limits: limits:
cpu: 100m cpu: {{ .Env.OLLAMA_CPU_LIMIT }}
memory: 8Gi memory: {{ .Env.OLLAMA_MEMORY_LIMIT }}
persistentVolume: persistentVolume:
enabled: true enabled: true

1
temporal/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
temporal-values.yaml

401
temporal/README.md Normal file
View File

@@ -0,0 +1,401 @@
# Temporal
Durable workflow execution platform for building reliable distributed applications:
- **Durable Execution**: Workflows survive process and infrastructure failures
- **Language Support**: SDKs for Go, Java, Python, TypeScript, .NET, PHP
- **Visibility**: Query and observe workflow state via Web UI and APIs
- **Scalability**: Horizontally scalable architecture
- **Multi-tenancy**: Namespace-based isolation for workflows
## Prerequisites
- Kubernetes cluster (k3s)
- PostgreSQL cluster (CloudNativePG)
- Keycloak installed and configured
- Vault for secrets management
- External Secrets Operator (optional, for Vault integration)
## Installation
```bash
just temporal::install
```
You will be prompted for:
- **Temporal host (FQDN)**: e.g., `temporal.example.com`
- **Keycloak host (FQDN)**: e.g., `auth.example.com`
- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed
### What Gets Installed
- Temporal Server (frontend, history, matching, worker services)
- Temporal Web UI with Keycloak OIDC authentication
- Temporal Admin Tools for cluster management
- PostgreSQL databases (`temporal`, `temporal_visibility`)
- Keycloak OAuth client (confidential client)
- Vault secrets (if External Secrets Operator is available)
## Configuration
Environment variables (set in `.env.local` or override):
| Variable | Default | Description |
| -------- | ------- | ----------- |
| `TEMPORAL_NAMESPACE` | `temporal` | Kubernetes namespace |
| `TEMPORAL_CHART_VERSION` | `0.52.0` | Helm chart version |
| `TEMPORAL_HOST` | (prompt) | External hostname (FQDN) |
| `TEMPORAL_OIDC_CLIENT_ID` | `temporal` | Keycloak client ID |
| `KEYCLOAK_HOST` | (prompt) | Keycloak hostname (FQDN) |
| `KEYCLOAK_REALM` | `buunstack` | Keycloak realm |
| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor |
## Architecture
```plain
External Users
|
Cloudflare Tunnel (HTTPS)
|
Traefik Ingress (HTTPS)
|
Temporal Web UI (HTTP inside cluster)
|-- OAuth --> Keycloak (authentication)
|
Temporal Server
|-- Frontend Service (gRPC :7233)
| |-- Client connections
| |-- Workflow/Activity APIs
|
|-- History Service
| |-- Workflow state management
| |-- Event sourcing
|
|-- Matching Service
| |-- Task queue management
| |-- Worker polling
|
|-- Worker Service
| |-- System workflows
| |-- Archival
|
PostgreSQL (temporal, temporal_visibility)
```
**Key Components**:
- **Frontend**: Entry point for all client requests (gRPC API)
- **History**: Maintains workflow execution history and state
- **Matching**: Routes tasks to appropriate workers
- **Worker**: Executes internal system workflows
- **Web UI**: Browser-based workflow monitoring and management
- **Admin Tools**: CLI tools for cluster administration
## Usage
### Access Web UI
1. Navigate to `https://your-temporal-host/`
2. Authenticate via Keycloak SSO
3. Select a namespace to view workflows
### Temporal CLI Setup (Local Development)
The Temporal gRPC endpoint is only accessible within the cluster network. Use [Telepresence](https://www.telepresence.io/) to connect from your local machine.
#### Step 1: Connect to the Cluster
```bash
telepresence connect
```
#### Step 2: Configure Temporal CLI
Set environment variables (add to `.bashrc`, `.zshrc`, or use direnv):
```bash
export TEMPORAL_ADDRESS="temporal-frontend.temporal:7233"
export TEMPORAL_NAMESPACE="default"
```
Or create a named environment for multiple clusters:
```bash
# Configure named environment
temporal env set --env buun -k address -v temporal-frontend.temporal:7233
temporal env set --env buun -k namespace -v default
# Use with commands
temporal workflow list --env buun
```
#### Step 3: Verify Connection
```bash
# Check telepresence status
telepresence status
# Test Temporal connection
temporal operator namespace list
```
#### CLI Examples
```bash
# List workflows
temporal workflow list
# Describe a workflow
temporal workflow describe --workflow-id my-workflow-id
# Query workflow state
temporal workflow query --workflow-id my-workflow-id --type my-query
# Signal a workflow
temporal workflow signal --workflow-id my-workflow-id --name my-signal
# Terminate a workflow
temporal workflow terminate --workflow-id my-workflow-id --reason "manual termination"
```
### Create a Temporal Namespace
Before running workflows, create a namespace:
```bash
just temporal::create-temporal-namespace default
```
With custom retention period:
```bash
just temporal::create-temporal-namespace myapp 7d
```
### List Temporal Namespaces
```bash
just temporal::list-temporal-namespaces
```
### Cluster Health Check
```bash
just temporal::cluster-info
```
### Connect Workers
Workers connect to the Temporal Frontend service. From within the cluster:
```text
temporal-frontend.temporal:7233
```
Example Python worker:
```python
from temporalio.client import Client
from temporalio.worker import Worker
async def main():
client = await Client.connect("temporal-frontend.temporal:7233")
worker = Worker(
client,
task_queue="my-task-queue",
workflows=[MyWorkflow],
activities=[my_activity],
)
await worker.run()
```
Example Go worker:
```go
import (
"go.temporal.io/sdk/client"
"go.temporal.io/sdk/worker"
)
func main() {
c, _ := client.Dial(client.Options{
HostPort: "temporal-frontend.temporal:7233",
})
defer c.Close()
w := worker.New(c, "my-task-queue", worker.Options{})
w.RegisterWorkflow(MyWorkflow)
w.RegisterActivity(MyActivity)
w.Run(worker.InterruptCh())
}
```
## Authentication
### Web UI (OIDC)
- Users authenticate via Keycloak
- Standard OIDC flow with Authorization Code grant
- Configured via environment variables in the Web UI deployment
### gRPC API
- By default, no authentication is required for gRPC connections within the cluster
- For production, configure mTLS or JWT-based authorization
## Management
### Upgrade Temporal
```bash
just temporal::upgrade
```
### Uninstall
```bash
just temporal::uninstall
```
This removes:
- Helm release and all Kubernetes resources
- Namespace
- Keycloak client
**Note**: The following resources are NOT deleted:
- PostgreSQL databases (`temporal`, `temporal_visibility`)
- Vault secrets
### Full Cleanup
To remove everything including databases and Vault secrets:
```bash
just temporal::uninstall true
```
Or manually:
```bash
just temporal::delete-postgres-user-and-db
```
## Troubleshooting
### Check Pod Status
```bash
kubectl get pods -n temporal
```
Expected pods:
- `temporal-frontend-*` - Frontend service
- `temporal-history-*` - History service
- `temporal-matching-*` - Matching service
- `temporal-worker-*` - Worker service
- `temporal-web-*` - Web UI
- `temporal-admintools-*` - Admin tools
### View Logs
```bash
# Frontend logs
kubectl logs -n temporal deployment/temporal-frontend --tail=100
# History logs
kubectl logs -n temporal deployment/temporal-history --tail=100
# Web UI logs
kubectl logs -n temporal deployment/temporal-web --tail=100
```
### Database Connection Issues
Check PostgreSQL connectivity:
```bash
kubectl exec -n temporal deployment/temporal-admintools -- \
psql -h postgres-cluster-rw.postgres -U temporal -d temporal -c "SELECT 1"
```
### Schema Issues
If schema initialization fails, check the schema job:
```bash
kubectl logs -n temporal -l app.kubernetes.io/component=schema --all-containers
```
### Service Discovery Issues
Verify services are running:
```bash
kubectl get svc -n temporal
```
Test frontend connectivity from admin tools:
```bash
kubectl exec -n temporal deployment/temporal-admintools -- \
tctl cluster health
```
### Web UI Login Issues
Verify Keycloak client configuration:
```bash
just keycloak::get-client buunstack temporal
```
Check Web UI environment variables:
```bash
kubectl get deployment temporal-web -n temporal -o jsonpath='{.spec.template.spec.containers[0].env}' | jq
```
## Configuration Files
| File | Description |
| ---- | ----------- |
| `temporal-values.gomplate.yaml` | Helm values template |
| `postgres-external-secret.gomplate.yaml` | PostgreSQL credentials ExternalSecret |
| `keycloak-auth-external-secret.gomplate.yaml` | Keycloak OIDC credentials ExternalSecret |
## Security Considerations
- **Pod Security Standards**: Namespace configured with **baseline** enforcement
- **Server Security**: Temporal server components run with restricted-compliant security contexts
### Why Not Restricted?
The namespace cannot use `restricted` Pod Security Standards due to the Temporal Web UI image (`temporalio/ui`):
- The image writes configuration files to `./config/docker.yaml` at startup
- The container's filesystem is owned by root (UID 0)
- When running as non-root user (UID 1000), the container cannot write to these paths
- Error: `unable to create open ./config/docker.yaml: permission denied`
The Temporal server components (frontend, history, matching, worker) **do** meet `restricted` requirements and run with full security hardening. Only the Web UI component requires `baseline`.
### Server Security Context
Temporal server components (frontend, history, matching, worker) run with:
- `runAsNonRoot: true`
- `runAsUser: 1000`
- `allowPrivilegeEscalation: false`
- `seccompProfile.type: RuntimeDefault`
- `capabilities.drop: [ALL]`
## References
- [Temporal Documentation](https://docs.temporal.io/)
- [Temporal GitHub](https://github.com/temporalio/temporal)
- [Temporal Helm Charts](https://github.com/temporalio/helm-charts)

434
temporal/justfile Normal file
View File

@@ -0,0 +1,434 @@
set fallback := true
export TEMPORAL_NAMESPACE := env("TEMPORAL_NAMESPACE", "temporal")
export TEMPORAL_CHART_VERSION := env("TEMPORAL_CHART_VERSION", "0.72.0")
export TEMPORAL_HOST := env("TEMPORAL_HOST", "")
export TEMPORAL_OIDC_CLIENT_ID := env("TEMPORAL_OIDC_CLIENT_ID", "temporal")
export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring")
export MONITORING_ENABLED := env("MONITORING_ENABLED", "")
export KEYCLOAK_REALM := env("KEYCLOAK_REALM", "buunstack")
export KEYCLOAK_HOST := env("KEYCLOAK_HOST", "")
export K8S_VAULT_NAMESPACE := env("K8S_VAULT_NAMESPACE", "vault")
[private]
default:
@just --list --unsorted --list-submodules
# Add Helm repository
add-helm-repo:
helm repo add temporal https://go.temporal.io/helm-charts
helm repo update temporal
# Remove Helm repository
remove-helm-repo:
helm repo remove temporal
# Create Temporal namespace
create-namespace:
kubectl get namespace ${TEMPORAL_NAMESPACE} &>/dev/null || \
kubectl create namespace ${TEMPORAL_NAMESPACE}
# Delete Temporal namespace
delete-namespace:
kubectl delete namespace ${TEMPORAL_NAMESPACE} --ignore-not-found
# Create PostgreSQL user and databases for Temporal
create-postgres-user-and-db:
#!/bin/bash
set -euo pipefail
if just postgres::user-exists temporal &>/dev/null; then
echo "PostgreSQL user 'temporal' already exists"
else
echo "Creating PostgreSQL user and databases..."
PG_PASSWORD=$(just utils::random-password)
just postgres::create-user-and-db temporal temporal "${PG_PASSWORD}"
just postgres::create-db temporal_visibility
just postgres::grant temporal_visibility temporal
just vault::put temporal/db username=temporal password="${PG_PASSWORD}"
echo "PostgreSQL user and databases created."
fi
# Delete PostgreSQL user and databases
delete-postgres-user-and-db:
#!/bin/bash
set -euo pipefail
if gum confirm "Delete PostgreSQL user and databases for Temporal?"; then
just postgres::delete-db temporal || true
just postgres::delete-db temporal_visibility || true
just postgres::delete-user temporal || true
just vault::delete temporal/db || true
echo "PostgreSQL user and databases deleted."
else
echo "Cancelled."
fi
# Create Postgres secret
create-postgres-secret:
#!/bin/bash
set -euo pipefail
if kubectl get secret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} &>/dev/null; then
echo "Postgres auth secret already exists"
exit 0
fi
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
echo "External Secrets Operator detected. Creating ExternalSecret..."
kubectl delete externalsecret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
gomplate -f postgres-external-secret.gomplate.yaml | kubectl apply -f -
echo "Waiting for ExternalSecret to sync..."
kubectl wait --for=condition=Ready externalsecret/temporal-postgres-auth \
-n ${TEMPORAL_NAMESPACE} --timeout=60s
else
echo "Creating Kubernetes Secret directly..."
PG_USERNAME=$(just vault::get temporal/db username)
PG_PASSWORD=$(just vault::get temporal/db password)
kubectl create secret generic temporal-postgres-auth \
--from-literal=password="${PG_PASSWORD}" \
-n ${TEMPORAL_NAMESPACE}
fi
# Delete Postgres secret
delete-postgres-secret:
kubectl delete externalsecret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
kubectl delete secret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
# Create Keycloak client for Temporal Web UI
create-keycloak-client:
#!/bin/bash
set -euo pipefail
while [ -z "${TEMPORAL_HOST}" ]; do
TEMPORAL_HOST=$(
gum input --prompt="Temporal host (FQDN): " --width=100 \
--placeholder="e.g., temporal.example.com"
)
done
echo "Creating Keycloak client for Temporal..."
just keycloak::delete-client ${KEYCLOAK_REALM} ${TEMPORAL_OIDC_CLIENT_ID} || true
CLIENT_SECRET=$(just utils::random-password)
just keycloak::create-client \
realm=${KEYCLOAK_REALM} \
client_id=${TEMPORAL_OIDC_CLIENT_ID} \
redirect_url="https://${TEMPORAL_HOST}/*" \
client_secret="${CLIENT_SECRET}"
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
kubectl create secret generic temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
--from-literal=client_id="${TEMPORAL_OIDC_CLIENT_ID}" \
--from-literal=client_secret="${CLIENT_SECRET}"
echo "Keycloak client created successfully"
echo "Client ID: ${TEMPORAL_OIDC_CLIENT_ID}"
echo "Redirect URI: https://${TEMPORAL_HOST}/*"
# Delete Keycloak client
delete-keycloak-client:
#!/bin/bash
set -euo pipefail
echo "Deleting Keycloak client for Temporal..."
just keycloak::delete-client ${KEYCLOAK_REALM} ${TEMPORAL_OIDC_CLIENT_ID} || true
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
if just vault::exist keycloak/client/temporal &>/dev/null; then
just vault::delete keycloak/client/temporal
fi
# Create Keycloak auth secret
create-keycloak-auth-secret:
#!/bin/bash
set -euo pipefail
if kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} &>/dev/null; then
oauth_client_id=$(kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
-o jsonpath='{.data.client_id}' | base64 -d)
oauth_client_secret=$(kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
-o jsonpath='{.data.client_secret}' | base64 -d)
elif helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null && \
just vault::get keycloak/client/temporal client_secret &>/dev/null; then
oauth_client_id=$(just vault::get keycloak/client/temporal client_id)
oauth_client_secret=$(just vault::get keycloak/client/temporal client_secret)
else
echo "Error: Cannot retrieve OAuth client secret. Please run 'just temporal::create-keycloak-client' first."
exit 1
fi
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
echo "External Secrets Operator detected. Storing secrets in Vault..."
just vault::put keycloak/client/temporal \
client_id="${oauth_client_id}" \
client_secret="${oauth_client_secret}"
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
kubectl delete externalsecret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
gomplate -f keycloak-auth-external-secret.gomplate.yaml | kubectl apply -f -
echo "Waiting for ExternalSecret to sync..."
kubectl wait --for=condition=Ready externalsecret/temporal-web-auth \
-n ${TEMPORAL_NAMESPACE} --timeout=60s
echo "ExternalSecret created successfully"
else
echo "External Secrets Operator not found. Creating Kubernetes Secret directly..."
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
kubectl create secret generic temporal-web-auth -n ${TEMPORAL_NAMESPACE} \
--from-literal=TEMPORAL_AUTH_CLIENT_ID="${oauth_client_id}" \
--from-literal=TEMPORAL_AUTH_CLIENT_SECRET="${oauth_client_secret}"
if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then
just vault::put keycloak/client/temporal \
client_id="${oauth_client_id}" \
client_secret="${oauth_client_secret}"
fi
echo "Kubernetes Secret created successfully"
fi
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
# Delete Keycloak auth secret
delete-keycloak-auth-secret:
kubectl delete externalsecret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
# Initialize Temporal database schema
init-schema:
#!/bin/bash
set -euo pipefail
echo "Initializing Temporal database schema..."
PG_HOST="postgres-cluster-rw.postgres"
PG_PORT="5432"
PG_USER=$(just vault::get temporal/db username)
PG_PASSWORD=$(just vault::get temporal/db password)
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
if [ -z "${POD_NAME}" ]; then
echo "Admin tools pod not found. Running schema setup job..."
exit 0
fi
echo "Setting up main database schema..."
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
temporal-sql-tool --plugin postgres12 \
--endpoint ${PG_HOST} --port ${PG_PORT} \
--user ${PG_USER} --password ${PG_PASSWORD} \
--database temporal \
setup-schema -v 0.0
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
temporal-sql-tool --plugin postgres12 \
--endpoint ${PG_HOST} --port ${PG_PORT} \
--user ${PG_USER} --password ${PG_PASSWORD} \
--database temporal \
update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned
echo "Setting up visibility database schema..."
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
temporal-sql-tool --plugin postgres12 \
--endpoint ${PG_HOST} --port ${PG_PORT} \
--user ${PG_USER} --password ${PG_PASSWORD} \
--database temporal_visibility \
setup-schema -v 0.0
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
temporal-sql-tool --plugin postgres12 \
--endpoint ${PG_HOST} --port ${PG_PORT} \
--user ${PG_USER} --password ${PG_PASSWORD} \
--database temporal_visibility \
update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versioned
echo "Schema initialization complete."
# Install Temporal
install:
#!/bin/bash
set -euo pipefail
while [ -z "${TEMPORAL_HOST}" ]; do
TEMPORAL_HOST=$(gum input --prompt="Temporal host (FQDN): " --width=80 \
--placeholder="e.g., temporal.example.com")
done
while [ -z "${KEYCLOAK_HOST}" ]; do
KEYCLOAK_HOST=$(gum input --prompt="Keycloak host (FQDN): " --width=80 \
--placeholder="e.g., auth.example.com")
done
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
if [ -z "${MONITORING_ENABLED}" ]; then
if gum confirm "Enable Prometheus monitoring?"; then
MONITORING_ENABLED="true"
fi
fi
fi
echo "Installing Temporal..."
just create-namespace
kubectl label namespace ${TEMPORAL_NAMESPACE} \
pod-security.kubernetes.io/enforce=baseline --overwrite
if [ "${MONITORING_ENABLED}" = "true" ]; then
kubectl label namespace ${TEMPORAL_NAMESPACE} \
buun.channel/enable-monitoring=true --overwrite
fi
echo "Setting up PostgreSQL database..."
just create-postgres-user-and-db
just create-postgres-secret
echo "Setting up Keycloak OIDC authentication..."
just create-keycloak-client
just create-keycloak-auth-secret
echo "Generating Helm values..."
just add-helm-repo
gomplate -f temporal-values.gomplate.yaml -o temporal-values.yaml
echo "Installing Temporal Helm chart..."
helm upgrade --cleanup-on-fail --install temporal temporal/temporal \
--version ${TEMPORAL_CHART_VERSION} -n ${TEMPORAL_NAMESPACE} --wait \
-f temporal-values.yaml --timeout 15m
echo "Configuring dynamic config for Worker Insights..."
kubectl patch configmap temporal-dynamic-config -n ${TEMPORAL_NAMESPACE} --type merge -p '{
"data": {
"dynamic_config.yaml": "frontend.WorkerHeartbeatsEnabled:\n - value: true\nfrontend.ListWorkersEnabled:\n - value: true\n"
}
}'
echo "Restarting frontend to apply dynamic config..."
kubectl rollout restart deployment temporal-frontend -n ${TEMPORAL_NAMESPACE}
kubectl rollout status deployment temporal-frontend -n ${TEMPORAL_NAMESPACE} --timeout=120s
echo ""
echo "Temporal installed successfully!"
echo "Access Temporal Web UI at: https://${TEMPORAL_HOST}"
echo ""
echo "OIDC authentication is configured with Keycloak."
echo "Users can login with their Keycloak credentials."
# Upgrade Temporal
upgrade:
#!/bin/bash
set -euo pipefail
while [ -z "${TEMPORAL_HOST}" ]; do
TEMPORAL_HOST=$(gum input --prompt="Temporal host (FQDN): " --width=80)
done
while [ -z "${KEYCLOAK_HOST}" ]; do
KEYCLOAK_HOST=$(gum input --prompt="Keycloak host (FQDN): " --width=80)
done
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
if [ -z "${MONITORING_ENABLED}" ]; then
if gum confirm "Enable Prometheus monitoring?"; then
MONITORING_ENABLED="true"
fi
fi
fi
if [ "${MONITORING_ENABLED}" = "true" ]; then
kubectl label namespace ${TEMPORAL_NAMESPACE} \
buun.channel/enable-monitoring=true --overwrite
fi
echo "Upgrading Temporal..."
gomplate -f temporal-values.gomplate.yaml -o temporal-values.yaml
helm upgrade temporal temporal/temporal \
--version ${TEMPORAL_CHART_VERSION} -n ${TEMPORAL_NAMESPACE} --wait \
-f temporal-values.yaml --timeout 15m
echo "Configuring dynamic config for Worker Insights..."
kubectl patch configmap temporal-dynamic-config -n ${TEMPORAL_NAMESPACE} --type merge -p '{
"data": {
"dynamic_config.yaml": "frontend.WorkerHeartbeatsEnabled:\n - value: true\nfrontend.ListWorkersEnabled:\n - value: true\n"
}
}'
echo "Restarting frontend to apply dynamic config..."
kubectl rollout restart deployment temporal-frontend -n ${TEMPORAL_NAMESPACE}
kubectl rollout status deployment temporal-frontend -n ${TEMPORAL_NAMESPACE} --timeout=120s
echo ""
echo "Temporal upgraded successfully!"
echo "Access Temporal Web UI at: https://${TEMPORAL_HOST}"
# Uninstall Temporal (delete-data: true to delete database and Vault secrets)
uninstall delete-data='false':
#!/bin/bash
set -euo pipefail
if ! gum confirm "Uninstall Temporal?"; then
echo "Cancelled."
exit 0
fi
echo "Uninstalling Temporal..."
helm uninstall temporal -n ${TEMPORAL_NAMESPACE} --ignore-not-found --wait
just delete-keycloak-auth-secret || true
just delete-keycloak-client || true
just delete-postgres-secret
just delete-namespace
if [ "{{ delete-data }}" = "true" ]; then
echo "Deleting database and Vault secrets..."
just postgres::delete-db temporal || true
just postgres::delete-db temporal_visibility || true
just postgres::delete-user temporal || true
just vault::delete temporal/db || true
just vault::delete keycloak/client/temporal || true
echo "Temporal uninstalled with all data deleted."
else
echo "Temporal uninstalled."
echo ""
echo "Note: The following resources were NOT deleted:"
echo " - PostgreSQL user and databases (temporal, temporal_visibility)"
echo " - Vault secrets (temporal/db, keycloak/client/temporal)"
echo ""
echo "To delete all data, run:"
echo " just temporal::uninstall true"
fi
# Create a Temporal namespace (workflow namespace, not Kubernetes)
create-temporal-namespace name='' retention='3d':
#!/bin/bash
set -euo pipefail
name="{{ name }}"
retention="{{ retention }}"
while [ -z "${name}" ]; do
name=$(gum input --prompt="Namespace name: " --width=80 --placeholder="e.g., default")
done
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
tctl --namespace "${name}" namespace register --retention "${retention}"
echo "Namespace '${name}' created with retention ${retention}."
# List Temporal namespaces
list-temporal-namespaces:
#!/bin/bash
set -euo pipefail
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- tctl namespace list
# Get Temporal cluster info
cluster-info:
#!/bin/bash
set -euo pipefail
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
-o jsonpath='{.items[0].metadata.name}')
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- tctl cluster health

View File

@@ -0,0 +1,22 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: temporal-web-auth
namespace: {{ .Env.TEMPORAL_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: temporal-web-auth
creationPolicy: Owner
data:
- secretKey: TEMPORAL_AUTH_CLIENT_ID
remoteRef:
key: keycloak/client/temporal
property: client_id
- secretKey: TEMPORAL_AUTH_CLIENT_SECRET
remoteRef:
key: keycloak/client/temporal
property: client_secret

View File

@@ -0,0 +1,18 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: temporal-postgres-auth
namespace: {{ .Env.TEMPORAL_NAMESPACE }}
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-secret-store
kind: ClusterSecretStore
target:
name: temporal-postgres-auth
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: temporal/db
property: password

View File

@@ -0,0 +1,204 @@
server:
replicaCount: 1
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
config:
persistence:
default:
driver: "sql"
sql:
driver: "postgres12"
host: "postgres-cluster-rw.postgres"
port: 5432
database: temporal
user: temporal
existingSecret: temporal-postgres-auth
maxConns: 20
maxIdleConns: 20
maxConnLifetime: "1h"
visibility:
driver: "sql"
sql:
driver: "postgres12"
host: "postgres-cluster-rw.postgres"
port: 5432
database: temporal_visibility
user: temporal
existingSecret: temporal-postgres-auth
maxConns: 20
maxIdleConns: 20
maxConnLifetime: "1h"
{{- if .Env.MONITORING_ENABLED }}
metrics:
serviceMonitor:
enabled: true
additionalLabels:
release: kube-prometheus-stack
{{- end }}
frontend:
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
history:
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
matching:
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
worker:
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
admintools:
enabled: true
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
web:
enabled: true
replicaCount: 1
service:
type: ClusterIP
port: 8080
ingress:
enabled: true
className: traefik
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
hosts:
- {{ .Env.TEMPORAL_HOST }}
tls:
- secretName: temporal-web-tls
hosts:
- {{ .Env.TEMPORAL_HOST }}
additionalEnv:
- name: TEMPORAL_AUTH_ENABLED
value: "true"
- name: TEMPORAL_AUTH_PROVIDER_URL
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}"
- name: TEMPORAL_AUTH_SCOPES
value: "openid,profile,email"
- name: TEMPORAL_AUTH_CALLBACK_URL
value: "https://{{ .Env.TEMPORAL_HOST }}/auth/sso/callback"
additionalEnvSecretName: temporal-web-auth
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
cassandra:
enabled: false
mysql:
enabled: false
postgresql:
enabled: false
elasticsearch:
enabled: false
prometheus:
enabled: false
grafana:
enabled: false
schema:
createDatabase:
enabled: false
setup:
enabled: true
backoffLimit: 100
update:
enabled: true
backoffLimit: 100
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false