Compare commits
19 Commits
7234840eba
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
50b0094e86 | ||
|
|
0d45433ea9 | ||
|
|
a8599b66f4 | ||
|
|
1924e56ad7 | ||
|
|
dae4e9d7ac | ||
|
|
cfcb278c4d | ||
|
|
022c85c0dc | ||
|
|
3ac8a72df6 | ||
|
|
ca0a8dacba | ||
|
|
fb1e4c20fa | ||
|
|
593da33d64 | ||
|
|
f3bc41e9eb | ||
|
|
98b03704d7 | ||
|
|
6fa0d27f7d | ||
|
|
d9ee90c32c | ||
|
|
7dc732268e | ||
|
|
2955d7d783 | ||
|
|
5055a36d87 | ||
|
|
46fdff720f |
@@ -429,6 +429,7 @@ When writing Markdown documentation:
|
|||||||
```
|
```
|
||||||
|
|
||||||
2. **Always validate with markdownlint-cli2**:
|
2. **Always validate with markdownlint-cli2**:
|
||||||
- Run `markdownlint-cli2 <file>` before committing any Markdown files
|
- Run from the project root directory to use `.markdownlint.yaml` config:
|
||||||
|
`cd <top-dir> && markdownlint-cli2 <relative-path>`
|
||||||
- Fix all linting errors to ensure consistent formatting
|
- Fix all linting errors to ensure consistent formatting
|
||||||
- Pay attention to code block language specifications (MD040) and list formatting (MD029)
|
- Pay attention to code block language specifications (MD040) and list formatting (MD029)
|
||||||
|
|||||||
26
README.md
26
README.md
@@ -63,6 +63,7 @@ A remotely accessible Kubernetes home lab with OIDC authentication. Build a mode
|
|||||||
### LLM & AI Applications (Optional)
|
### LLM & AI Applications (Optional)
|
||||||
|
|
||||||
- **[Ollama](https://ollama.com/)**: Local LLM inference server with GPU acceleration
|
- **[Ollama](https://ollama.com/)**: Local LLM inference server with GPU acceleration
|
||||||
|
- **[LiteLLM](https://litellm.ai/)**: Unified LLM gateway for accessing multiple providers through OpenAI-compatible API
|
||||||
- **[LibreChat](https://www.librechat.ai/)**: Web-based chat interface with multi-model support and MCP integration
|
- **[LibreChat](https://www.librechat.ai/)**: Web-based chat interface with multi-model support and MCP integration
|
||||||
- **[Langfuse](https://langfuse.com/)**: LLM observability and analytics platform for tracking and debugging AI applications
|
- **[Langfuse](https://langfuse.com/)**: LLM observability and analytics platform for tracking and debugging AI applications
|
||||||
|
|
||||||
@@ -70,6 +71,7 @@ A remotely accessible Kubernetes home lab with OIDC authentication. Build a mode
|
|||||||
|
|
||||||
- **[Dagster](https://dagster.io/)**: Modern data orchestration platform
|
- **[Dagster](https://dagster.io/)**: Modern data orchestration platform
|
||||||
- **[Apache Airflow](https://airflow.apache.org/)**: Workflow orchestration and task scheduling
|
- **[Apache Airflow](https://airflow.apache.org/)**: Workflow orchestration and task scheduling
|
||||||
|
- **[Temporal](https://temporal.io/)**: Durable workflow execution for distributed applications
|
||||||
|
|
||||||
### Security & Compliance (Optional)
|
### Security & Compliance (Optional)
|
||||||
|
|
||||||
@@ -346,6 +348,18 @@ LLM observability and analytics platform:
|
|||||||
|
|
||||||
[📖 See Langfuse Documentation](./langfuse/README.md)
|
[📖 See Langfuse Documentation](./langfuse/README.md)
|
||||||
|
|
||||||
|
### LiteLLM
|
||||||
|
|
||||||
|
Unified LLM gateway and proxy:
|
||||||
|
|
||||||
|
- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, and more through single API
|
||||||
|
- **OpenAI-Compatible**: Drop-in replacement for OpenAI SDK
|
||||||
|
- **Virtual Keys**: Generate scoped API keys for users with usage tracking
|
||||||
|
- **Cost Tracking**: Monitor spending across all LLM providers
|
||||||
|
- **Keycloak Authentication**: OAuth2 for Admin UI with role-based access
|
||||||
|
|
||||||
|
[📖 See LiteLLM Documentation](./litellm/README.md)
|
||||||
|
|
||||||
### Dagster
|
### Dagster
|
||||||
|
|
||||||
Modern data orchestration platform:
|
Modern data orchestration platform:
|
||||||
@@ -366,6 +380,17 @@ Workflow orchestration platform:
|
|||||||
|
|
||||||
[📖 See Airflow Documentation](./airflow/README.md)
|
[📖 See Airflow Documentation](./airflow/README.md)
|
||||||
|
|
||||||
|
### Temporal
|
||||||
|
|
||||||
|
Durable workflow execution platform:
|
||||||
|
|
||||||
|
- **Durable Execution**: Workflows survive process and infrastructure failures
|
||||||
|
- **Saga Pattern**: Implement distributed transactions with compensating actions
|
||||||
|
- **Multi-Language SDKs**: Go, Python, TypeScript, Java, .NET, PHP
|
||||||
|
- **Keycloak Authentication**: OAuth2 for Web UI access
|
||||||
|
|
||||||
|
[📖 See Temporal Documentation](./temporal/README.md)
|
||||||
|
|
||||||
### Fairwinds Polaris
|
### Fairwinds Polaris
|
||||||
|
|
||||||
Kubernetes configuration validation and best practices auditing:
|
Kubernetes configuration validation and best practices auditing:
|
||||||
@@ -485,6 +510,7 @@ kubectl --context yourpc-oidc get nodes
|
|||||||
# JupyterHub: https://jupyter.yourdomain.com
|
# JupyterHub: https://jupyter.yourdomain.com
|
||||||
# MLflow: https://mlflow.yourdomain.com
|
# MLflow: https://mlflow.yourdomain.com
|
||||||
# Langfuse: https://langfuse.yourdomain.com
|
# Langfuse: https://langfuse.yourdomain.com
|
||||||
|
# LiteLLM: https://litellm.yourdomain.com
|
||||||
# LibreChat: https://chat.yourdomain.com
|
# LibreChat: https://chat.yourdomain.com
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
1
clickhouse/.gitignore
vendored
1
clickhouse/.gitignore
vendored
@@ -2,4 +2,5 @@ clickhouse-credentials-external-secret.yaml
|
|||||||
clickhouse-ingress.yaml
|
clickhouse-ingress.yaml
|
||||||
clickhouse-installation-template.yaml
|
clickhouse-installation-template.yaml
|
||||||
clickhouse-operator-values.yaml
|
clickhouse-operator-values.yaml
|
||||||
|
clickhouse-servicemonitor.yaml
|
||||||
clickhouse.yaml
|
clickhouse.yaml
|
||||||
|
|||||||
@@ -44,3 +44,31 @@ ClickHouse can use the following Linux capabilities for enhanced performance, bu
|
|||||||
| `SYS_NICE` | Thread priority control via `os_thread_priority` | Setting has no effect |
|
| `SYS_NICE` | Thread priority control via `os_thread_priority` | Setting has no effect |
|
||||||
|
|
||||||
These capabilities are disabled by default to comply with baseline Pod Security Standards. To enable them, the namespace must allow privileged pods, and you need to uncomment the `add` line in `clickhouse-installation-template.yaml`.
|
These capabilities are disabled by default to comply with baseline Pod Security Standards. To enable them, the namespace must allow privileged pods, and you need to uncomment the `add` line in `clickhouse-installation-template.yaml`.
|
||||||
|
|
||||||
|
## Monitoring
|
||||||
|
|
||||||
|
ClickHouse exposes Prometheus metrics on port 9363. When Prometheus (kube-prometheus-stack) is installed, monitoring can be enabled during installation or manually.
|
||||||
|
|
||||||
|
### Enable Monitoring
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just clickhouse::setup-monitoring
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates a ServiceMonitor and a metrics Service for Prometheus to scrape.
|
||||||
|
|
||||||
|
### Grafana Dashboard
|
||||||
|
|
||||||
|
Import the ClickHouse dashboard from Grafana.com:
|
||||||
|
|
||||||
|
1. Open Grafana → **Dashboards** → **New** → **Import**
|
||||||
|
2. Enter Dashboard ID: `14192`
|
||||||
|
3. Click **Load**, select **Prometheus** data source, then **Import**
|
||||||
|
|
||||||
|
The dashboard includes panels for memory, connections, queries, I/O, replication, merge operations, cache, and ZooKeeper metrics.
|
||||||
|
|
||||||
|
### Remove Monitoring
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just clickhouse::remove-monitoring
|
||||||
|
```
|
||||||
|
|||||||
@@ -20,6 +20,10 @@ spec:
|
|||||||
containers:
|
containers:
|
||||||
- name: clickhouse
|
- name: clickhouse
|
||||||
image: {{ .Env.CLICKHOUSE_IMAGE }}
|
image: {{ .Env.CLICKHOUSE_IMAGE }}
|
||||||
|
ports:
|
||||||
|
- name: prometheus
|
||||||
|
containerPort: 9363
|
||||||
|
protocol: TCP
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: {{ .Env.CLICKHOUSE_CPU_REQUEST }}
|
cpu: {{ .Env.CLICKHOUSE_CPU_REQUEST }}
|
||||||
|
|||||||
67
clickhouse/clickhouse-servicemonitor.gomplate.yaml
Normal file
67
clickhouse/clickhouse-servicemonitor.gomplate.yaml
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
{{- if .Env.MONITORING_ENABLED }}
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: clickhouse-metrics
|
||||||
|
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
|
||||||
|
labels:
|
||||||
|
app: clickhouse
|
||||||
|
clickhouse.altinity.com/chi: clickhouse
|
||||||
|
spec:
|
||||||
|
type: ClusterIP
|
||||||
|
ports:
|
||||||
|
- name: prometheus
|
||||||
|
port: 9363
|
||||||
|
targetPort: 9363
|
||||||
|
protocol: TCP
|
||||||
|
selector:
|
||||||
|
clickhouse.altinity.com/chi: clickhouse
|
||||||
|
---
|
||||||
|
apiVersion: monitoring.coreos.com/v1
|
||||||
|
kind: ServiceMonitor
|
||||||
|
metadata:
|
||||||
|
name: clickhouse
|
||||||
|
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
|
||||||
|
labels:
|
||||||
|
app: clickhouse
|
||||||
|
release: kube-prometheus-stack
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: clickhouse
|
||||||
|
clickhouse.altinity.com/chi: clickhouse
|
||||||
|
namespaceSelector:
|
||||||
|
matchNames:
|
||||||
|
- {{ .Env.CLICKHOUSE_NAMESPACE }}
|
||||||
|
endpoints:
|
||||||
|
- port: prometheus
|
||||||
|
path: /metrics
|
||||||
|
interval: 30s
|
||||||
|
scrapeTimeout: 10s
|
||||||
|
---
|
||||||
|
apiVersion: monitoring.coreos.com/v1
|
||||||
|
kind: ServiceMonitor
|
||||||
|
metadata:
|
||||||
|
name: clickhouse-operator
|
||||||
|
namespace: {{ .Env.CLICKHOUSE_NAMESPACE }}
|
||||||
|
labels:
|
||||||
|
app: clickhouse-operator
|
||||||
|
release: kube-prometheus-stack
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app.kubernetes.io/name: altinity-clickhouse-operator
|
||||||
|
namespaceSelector:
|
||||||
|
matchNames:
|
||||||
|
- {{ .Env.CLICKHOUSE_NAMESPACE }}
|
||||||
|
endpoints:
|
||||||
|
- port: ch-metrics
|
||||||
|
path: /metrics
|
||||||
|
interval: 30s
|
||||||
|
scrapeTimeout: 10s
|
||||||
|
- port: op-metrics
|
||||||
|
path: /metrics
|
||||||
|
interval: 30s
|
||||||
|
scrapeTimeout: 10s
|
||||||
|
{{- end }}
|
||||||
@@ -29,6 +29,17 @@ spec:
|
|||||||
<schema_type>transposed</schema_type>
|
<schema_type>transposed</schema_type>
|
||||||
</asynchronous_metric_log>
|
</asynchronous_metric_log>
|
||||||
</clickhouse>
|
</clickhouse>
|
||||||
|
# Enable Prometheus metrics endpoint
|
||||||
|
prometheus.xml: |
|
||||||
|
<clickhouse>
|
||||||
|
<prometheus>
|
||||||
|
<endpoint>/metrics</endpoint>
|
||||||
|
<port>9363</port>
|
||||||
|
<metrics>true</metrics>
|
||||||
|
<events>true</events>
|
||||||
|
<asynchronous_metrics>true</asynchronous_metrics>
|
||||||
|
</prometheus>
|
||||||
|
</clickhouse>
|
||||||
users:
|
users:
|
||||||
admin/k8s_secret_password: clickhouse-credentials/admin
|
admin/k8s_secret_password: clickhouse-credentials/admin
|
||||||
admin/networks/ip: "::/0"
|
admin/networks/ip: "::/0"
|
||||||
|
|||||||
@@ -5,6 +5,8 @@ export CLICKHOUSE_HOST := env("CLICKHOUSE_HOST", "")
|
|||||||
export CLICKHOUSE_CHART_VERSION := env("CLICKHOUSE_CHART_VERSION", "0.25.5")
|
export CLICKHOUSE_CHART_VERSION := env("CLICKHOUSE_CHART_VERSION", "0.25.5")
|
||||||
export CLICKHOUSE_IMAGE := env("CLICKHOUSE_IMAGE", "clickhouse/clickhouse-server:25.10")
|
export CLICKHOUSE_IMAGE := env("CLICKHOUSE_IMAGE", "clickhouse/clickhouse-server:25.10")
|
||||||
export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
|
export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
|
||||||
|
export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring")
|
||||||
|
export MONITORING_ENABLED := env("MONITORING_ENABLED", "")
|
||||||
|
|
||||||
# ClickHouse resource settings
|
# ClickHouse resource settings
|
||||||
export CLICKHOUSE_MEMORY_REQUEST := env("CLICKHOUSE_MEMORY_REQUEST", "1Gi")
|
export CLICKHOUSE_MEMORY_REQUEST := env("CLICKHOUSE_MEMORY_REQUEST", "1Gi")
|
||||||
@@ -107,6 +109,16 @@ install:
|
|||||||
--placeholder="e.g., clickhouse.example.com"
|
--placeholder="e.g., clickhouse.example.com"
|
||||||
)
|
)
|
||||||
done
|
done
|
||||||
|
# Check if Prometheus is available and ask about monitoring
|
||||||
|
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
|
||||||
|
if [ -z "${MONITORING_ENABLED}" ]; then
|
||||||
|
if gum confirm "Enable Prometheus monitoring?"; then
|
||||||
|
MONITORING_ENABLED="true"
|
||||||
|
else
|
||||||
|
MONITORING_ENABLED="false"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
echo "Installing ClickHouse..."
|
echo "Installing ClickHouse..."
|
||||||
just create-namespace
|
just create-namespace
|
||||||
just install-zookeeper
|
just install-zookeeper
|
||||||
@@ -124,6 +136,10 @@ install:
|
|||||||
kubectl wait --for=jsonpath='{.status.status}'=Completed \
|
kubectl wait --for=jsonpath='{.status.status}'=Completed \
|
||||||
clickhouseinstallation/clickhouse -n ${CLICKHOUSE_NAMESPACE} --timeout=600s
|
clickhouseinstallation/clickhouse -n ${CLICKHOUSE_NAMESPACE} --timeout=600s
|
||||||
just setup-ingress ${CLICKHOUSE_HOST}
|
just setup-ingress ${CLICKHOUSE_HOST}
|
||||||
|
# Setup monitoring if enabled
|
||||||
|
if [ "${MONITORING_ENABLED}" = "true" ]; then
|
||||||
|
just setup-monitoring
|
||||||
|
fi
|
||||||
echo "ClickHouse installation completed successfully"
|
echo "ClickHouse installation completed successfully"
|
||||||
echo "ClickHouse API at: https://${CLICKHOUSE_HOST}"
|
echo "ClickHouse API at: https://${CLICKHOUSE_HOST}"
|
||||||
|
|
||||||
@@ -137,6 +153,27 @@ setup-ingress host:
|
|||||||
kubectl apply -n ${CLICKHOUSE_NAMESPACE} -f clickhouse-ingress.yaml
|
kubectl apply -n ${CLICKHOUSE_NAMESPACE} -f clickhouse-ingress.yaml
|
||||||
echo "ClickHouse Ingress configured successfully"
|
echo "ClickHouse Ingress configured successfully"
|
||||||
|
|
||||||
|
# Setup Prometheus monitoring for ClickHouse
|
||||||
|
setup-monitoring:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Setting up Prometheus monitoring for ClickHouse..."
|
||||||
|
kubectl label namespace ${CLICKHOUSE_NAMESPACE} buun.channel/enable-monitoring=true --overwrite
|
||||||
|
MONITORING_ENABLED="true" gomplate -f clickhouse-servicemonitor.gomplate.yaml \
|
||||||
|
-o clickhouse-servicemonitor.yaml
|
||||||
|
kubectl apply -f clickhouse-servicemonitor.yaml
|
||||||
|
echo "Prometheus monitoring configured successfully"
|
||||||
|
|
||||||
|
# Remove Prometheus monitoring for ClickHouse
|
||||||
|
remove-monitoring:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Removing Prometheus monitoring for ClickHouse..."
|
||||||
|
kubectl delete servicemonitor clickhouse clickhouse-operator -n ${CLICKHOUSE_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl delete service clickhouse-metrics -n ${CLICKHOUSE_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl label namespace ${CLICKHOUSE_NAMESPACE} buun.channel/enable-monitoring- --ignore-not-found
|
||||||
|
echo "Prometheus monitoring removed"
|
||||||
|
|
||||||
# Uninstall ClickHouse (delete_volumes='false' to preserve PVCs and namespace)
|
# Uninstall ClickHouse (delete_volumes='false' to preserve PVCs and namespace)
|
||||||
uninstall delete-volumes='true':
|
uninstall delete-volumes='true':
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|||||||
@@ -33,6 +33,19 @@ install:
|
|||||||
)
|
)
|
||||||
done
|
done
|
||||||
|
|
||||||
|
# Check if PostgreSQL user already exists
|
||||||
|
if just postgres::user-exists ${MINIFLUX_DB_USERNAME} &>/dev/null; then
|
||||||
|
echo "PostgreSQL user '${MINIFLUX_DB_USERNAME}' already exists."
|
||||||
|
# Use existing password from Vault
|
||||||
|
if existing_db_pw=$(just vault::get miniflux/db password 2>/dev/null); then
|
||||||
|
echo "Using existing database password from Vault."
|
||||||
|
export MINIFLUX_DB_PASSWORD="${existing_db_pw}"
|
||||||
|
else
|
||||||
|
echo "Error: User exists but password not found in Vault." >&2
|
||||||
|
echo "Please delete the user first: just postgres::delete-user ${MINIFLUX_DB_USERNAME}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
|
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
|
||||||
MINIFLUX_DB_PASSWORD=$(
|
MINIFLUX_DB_PASSWORD=$(
|
||||||
gum input --prompt="Database password (empty to auto-generate): " \
|
gum input --prompt="Database password (empty to auto-generate): " \
|
||||||
@@ -40,14 +53,21 @@ install:
|
|||||||
)
|
)
|
||||||
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
|
if [ -z "${MINIFLUX_DB_PASSWORD}" ]; then
|
||||||
MINIFLUX_DB_PASSWORD=$(just utils::random-password)
|
MINIFLUX_DB_PASSWORD=$(just utils::random-password)
|
||||||
echo "Generated random password: ${MINIFLUX_DB_PASSWORD}"
|
echo "Generated random password for database."
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
export MINIFLUX_DB_PASSWORD
|
||||||
just postgres::create-user-and-db \
|
just postgres::create-user-and-db \
|
||||||
${MINIFLUX_DB_USERNAME} ${MINIFLUX_DB_NAME} ${MINIFLUX_DB_PASSWORD}
|
${MINIFLUX_DB_USERNAME} ${MINIFLUX_DB_NAME} ${MINIFLUX_DB_PASSWORD}
|
||||||
just vault::put miniflux/db username=${MINIFLUX_DB_USERNAME} \
|
just vault::put miniflux/db username=${MINIFLUX_DB_USERNAME} \
|
||||||
password=${MINIFLUX_DB_PASSWORD} database=${MINIFLUX_DB_NAME}
|
password=${MINIFLUX_DB_PASSWORD} database=${MINIFLUX_DB_NAME}
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if admin password exists in Vault
|
||||||
|
if existing_admin_pw=$(just vault::get miniflux/admin password 2>/dev/null); then
|
||||||
|
echo "Using existing admin password from Vault."
|
||||||
|
export MINIFLUX_ADMIN_PASSWORD="${existing_admin_pw}"
|
||||||
|
else
|
||||||
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
|
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
|
||||||
MINIFLUX_ADMIN_PASSWORD=$(
|
MINIFLUX_ADMIN_PASSWORD=$(
|
||||||
gum input --prompt="Admin password (empty to auto-generate): " \
|
gum input --prompt="Admin password (empty to auto-generate): " \
|
||||||
@@ -55,11 +75,13 @@ install:
|
|||||||
)
|
)
|
||||||
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
|
if [ -z "${MINIFLUX_ADMIN_PASSWORD}" ]; then
|
||||||
MINIFLUX_ADMIN_PASSWORD=$(just utils::random-password)
|
MINIFLUX_ADMIN_PASSWORD=$(just utils::random-password)
|
||||||
echo "Generated random password: ${MINIFLUX_ADMIN_PASSWORD}"
|
echo "Generated random password for admin."
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
export MINIFLUX_ADMIN_PASSWORD
|
||||||
just vault::put miniflux/admin username=${MINIFLUX_ADMIN_USERNAME} \
|
just vault::put miniflux/admin username=${MINIFLUX_ADMIN_USERNAME} \
|
||||||
password=${MINIFLUX_ADMIN_PASSWORD}
|
password=${MINIFLUX_ADMIN_PASSWORD}
|
||||||
|
fi
|
||||||
|
|
||||||
# https://github.com/gabe565/charts/tree/main/charts/miniflux
|
# https://github.com/gabe565/charts/tree/main/charts/miniflux
|
||||||
MINIFLUX_NAMESPACE=${MINIFLUX_NAMESPACE} \
|
MINIFLUX_NAMESPACE=${MINIFLUX_NAMESPACE} \
|
||||||
@@ -80,3 +102,26 @@ admin-username:
|
|||||||
# Print admin password
|
# Print admin password
|
||||||
admin-password:
|
admin-password:
|
||||||
@just vault::get miniflux/admin password
|
@just vault::get miniflux/admin password
|
||||||
|
|
||||||
|
# Reset admin password (deletes admin user from DB so it gets recreated on pod restart)
|
||||||
|
reset-admin-password:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "This will reset the admin password to the value stored in Vault."
|
||||||
|
if ! gum confirm "Continue?"; then
|
||||||
|
echo "Cancelled."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Deleting admin user from database..."
|
||||||
|
kubectl exec -n postgres postgres-cluster-1 -c postgres -- \
|
||||||
|
psql -U postgres -d ${MINIFLUX_DB_NAME} -c "DELETE FROM users WHERE username = '${MINIFLUX_ADMIN_USERNAME}';"
|
||||||
|
|
||||||
|
echo "Restarting Miniflux pod..."
|
||||||
|
kubectl rollout restart deployment/miniflux -n ${MINIFLUX_NAMESPACE}
|
||||||
|
kubectl rollout status deployment/miniflux -n ${MINIFLUX_NAMESPACE} --timeout=60s
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Admin password has been reset."
|
||||||
|
echo "Username: $(just vault::get miniflux/admin username)"
|
||||||
|
echo "Password: $(just vault::get miniflux/admin password)"
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ ingress:
|
|||||||
|
|
||||||
env:
|
env:
|
||||||
DATABASE_URL: "postgresql://{{ .Env.MINIFLUX_DB_USERNAME }}:{{ .Env.MINIFLUX_DB_PASSWORD }}@postgres-cluster-rw.postgres:5432/{{ .Env.MINIFLUX_DB_NAME }}"
|
DATABASE_URL: "postgresql://{{ .Env.MINIFLUX_DB_USERNAME }}:{{ .Env.MINIFLUX_DB_PASSWORD }}@postgres-cluster-rw.postgres:5432/{{ .Env.MINIFLUX_DB_NAME }}"
|
||||||
|
BASE_URL: "https://{{ .Env.MINIFLUX_HOST }}"
|
||||||
|
HTTPS: "1"
|
||||||
ADMIN_USERNAME: {{ .Env.MINIFLUX_ADMIN_USERNAME }}
|
ADMIN_USERNAME: {{ .Env.MINIFLUX_ADMIN_USERNAME }}
|
||||||
ADMIN_PASSWORD: {{ .Env.MINIFLUX_ADMIN_PASSWORD }}
|
ADMIN_PASSWORD: {{ .Env.MINIFLUX_ADMIN_PASSWORD }}
|
||||||
|
|
||||||
|
|||||||
@@ -445,6 +445,10 @@ ingress:
|
|||||||
annotations:
|
annotations:
|
||||||
kubernetes.io/ingress.class: traefik
|
kubernetes.io/ingress.class: traefik
|
||||||
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||||
|
# Enable sticky sessions for WebSocket connections (required for Jupyter RTC/MCP)
|
||||||
|
traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
|
||||||
|
traefik.ingress.kubernetes.io/service.sticky.cookie.name: jupyter-session
|
||||||
|
traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
|
||||||
ingressClassName: traefik
|
ingressClassName: traefik
|
||||||
hosts:
|
hosts:
|
||||||
- {{ .Env.JUPYTERHUB_HOST }}
|
- {{ .Env.JUPYTERHUB_HOST }}
|
||||||
|
|||||||
3
justfile
3
justfile
@@ -23,10 +23,12 @@ mod kserve
|
|||||||
mod langfuse
|
mod langfuse
|
||||||
mod lakekeeper
|
mod lakekeeper
|
||||||
mod librechat
|
mod librechat
|
||||||
|
mod litellm
|
||||||
mod longhorn
|
mod longhorn
|
||||||
mod metabase
|
mod metabase
|
||||||
mod mlflow
|
mod mlflow
|
||||||
mod minio
|
mod minio
|
||||||
|
mod nats
|
||||||
mod nvidia-device-plugin
|
mod nvidia-device-plugin
|
||||||
mod fairwinds-polaris
|
mod fairwinds-polaris
|
||||||
mod oauth2-proxy
|
mod oauth2-proxy
|
||||||
@@ -37,6 +39,7 @@ mod qdrant
|
|||||||
mod querybook
|
mod querybook
|
||||||
mod security
|
mod security
|
||||||
mod superset
|
mod superset
|
||||||
|
mod temporal
|
||||||
mod trino
|
mod trino
|
||||||
mod utils
|
mod utils
|
||||||
mod vault
|
mod vault
|
||||||
|
|||||||
@@ -98,8 +98,8 @@ install:
|
|||||||
--version ${LANGFUSE_CHART_VERSION} -n ${LANGFUSE_NAMESPACE} --wait \
|
--version ${LANGFUSE_CHART_VERSION} -n ${LANGFUSE_NAMESPACE} --wait \
|
||||||
-f langfuse-values.yaml
|
-f langfuse-values.yaml
|
||||||
|
|
||||||
# Uninstall Langfuse
|
# Uninstall Langfuse (delete-data: true to delete database and storage)
|
||||||
uninstall:
|
uninstall delete-data='false':
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
helm uninstall langfuse -n ${LANGFUSE_NAMESPACE} --wait --ignore-not-found
|
helm uninstall langfuse -n ${LANGFUSE_NAMESPACE} --wait --ignore-not-found
|
||||||
@@ -108,6 +108,17 @@ uninstall:
|
|||||||
# Clean up Keycloak client and Vault secrets to avoid stale credentials
|
# Clean up Keycloak client and Vault secrets to avoid stale credentials
|
||||||
just delete-keycloak-client || true
|
just delete-keycloak-client || true
|
||||||
|
|
||||||
|
if [ "{{ delete-data }}" = "true" ]; then
|
||||||
|
echo "Deleting database and storage..."
|
||||||
|
just delete-postgres-user-and-db || true
|
||||||
|
just delete-clickhouse-user || true
|
||||||
|
just delete-minio-user || true
|
||||||
|
just delete-keycloak-user || true
|
||||||
|
just delete-salt || true
|
||||||
|
just delete-nextauth-secret || true
|
||||||
|
just delete-redis-password || true
|
||||||
|
echo "Langfuse uninstalled with all data deleted."
|
||||||
|
else
|
||||||
echo "Langfuse uninstalled successfully"
|
echo "Langfuse uninstalled successfully"
|
||||||
echo ""
|
echo ""
|
||||||
echo "Note: The following resources were NOT deleted:"
|
echo "Note: The following resources were NOT deleted:"
|
||||||
@@ -115,12 +126,11 @@ uninstall:
|
|||||||
echo " - ClickHouse user and database (langfuse)"
|
echo " - ClickHouse user and database (langfuse)"
|
||||||
echo " - MinIO user and bucket (langfuse)"
|
echo " - MinIO user and bucket (langfuse)"
|
||||||
echo " - Keycloak user (langfuse)"
|
echo " - Keycloak user (langfuse)"
|
||||||
|
echo " - Vault secrets (langfuse/*)"
|
||||||
echo ""
|
echo ""
|
||||||
echo "To delete these resources, run:"
|
echo "To delete all data, run:"
|
||||||
echo " just langfuse::delete-postgres-user-and-db"
|
echo " just langfuse::uninstall true"
|
||||||
echo " just langfuse::delete-clickhouse-user"
|
fi
|
||||||
echo " just langfuse::delete-minio-user"
|
|
||||||
echo " just langfuse::delete-keycloak-user"
|
|
||||||
|
|
||||||
# Create all secrets (PostgreSQL, Keycloak, MinIO, Redis)
|
# Create all secrets (PostgreSQL, Keycloak, MinIO, Redis)
|
||||||
create-secrets:
|
create-secrets:
|
||||||
|
|||||||
@@ -69,13 +69,13 @@ langfuse:
|
|||||||
tls:
|
tls:
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|
||||||
# Resource configuration based on Goldilocks/VPA recommendations
|
# Resource recommendations from Goldilocks VPA
|
||||||
# CPU limits increased to handle startup spikes
|
# web target: cpu=15m, memory=717Mi
|
||||||
web:
|
web:
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 15m
|
cpu: 25m
|
||||||
memory: 704Mi
|
memory: 768Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 100m
|
cpu: 100m
|
||||||
memory: 1.5Gi
|
memory: 1.5Gi
|
||||||
@@ -89,10 +89,12 @@ langfuse:
|
|||||||
timeoutSeconds: 30
|
timeoutSeconds: 30
|
||||||
failureThreshold: 5
|
failureThreshold: 5
|
||||||
|
|
||||||
|
# Resource recommendations from Goldilocks VPA
|
||||||
|
# worker target: cpu=15m, memory=380Mi
|
||||||
worker:
|
worker:
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 15m
|
cpu: 25m
|
||||||
memory: 512Mi
|
memory: 512Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 100m
|
cpu: 100m
|
||||||
@@ -113,6 +115,16 @@ redis:
|
|||||||
username: "default"
|
username: "default"
|
||||||
existingSecret: redis-auth
|
existingSecret: redis-auth
|
||||||
existingSecretPasswordKey: secret
|
existingSecretPasswordKey: secret
|
||||||
|
# Resource recommendations from Goldilocks VPA
|
||||||
|
# valkey target: cpu=15m, memory=100Mi
|
||||||
|
master:
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
clickhouse:
|
clickhouse:
|
||||||
deploy: false
|
deploy: false
|
||||||
|
|||||||
@@ -94,10 +94,10 @@ ingress:
|
|||||||
|
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 100m
|
cpu: 25m
|
||||||
memory: 512Mi
|
memory: 512Mi
|
||||||
limits:
|
limits:
|
||||||
cpu: 1000m
|
cpu: 100m
|
||||||
memory: 1Gi
|
memory: 1Gi
|
||||||
|
|
||||||
mongodb:
|
mongodb:
|
||||||
@@ -110,6 +110,13 @@ mongodb:
|
|||||||
tag: "latest"
|
tag: "latest"
|
||||||
persistence:
|
persistence:
|
||||||
size: 8Gi
|
size: 8Gi
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 75m
|
||||||
|
memory: 512Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 2Gi
|
||||||
podSecurityContext:
|
podSecurityContext:
|
||||||
fsGroup: 1001
|
fsGroup: 1001
|
||||||
seccompProfile:
|
seccompProfile:
|
||||||
@@ -132,6 +139,13 @@ meilisearch:
|
|||||||
tag: "v1.7.3"
|
tag: "v1.7.3"
|
||||||
auth:
|
auth:
|
||||||
existingMasterKeySecret: "librechat-credentials-env"
|
existingMasterKeySecret: "librechat-credentials-env"
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 256Mi
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 1Gi
|
||||||
podSecurityContext:
|
podSecurityContext:
|
||||||
fsGroup: 1000
|
fsGroup: 1000
|
||||||
seccompProfile:
|
seccompProfile:
|
||||||
|
|||||||
3
litellm/.gitignore
vendored
Normal file
3
litellm/.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
litellm-values.yaml
|
||||||
|
apikey-external-secret.yaml
|
||||||
|
models.yaml
|
||||||
547
litellm/README.md
Normal file
547
litellm/README.md
Normal file
@@ -0,0 +1,547 @@
|
|||||||
|
# LiteLLM
|
||||||
|
|
||||||
|
Unified LLM gateway and proxy for accessing multiple LLM providers through a single OpenAI-compatible API:
|
||||||
|
|
||||||
|
- **Multi-Provider Support**: Anthropic, OpenAI, Ollama, Mistral, Groq, Cohere, Azure, Bedrock, Vertex AI
|
||||||
|
- **OpenAI-Compatible API**: Drop-in replacement for OpenAI SDK
|
||||||
|
- **Load Balancing & Fallback**: Automatic failover between providers
|
||||||
|
- **Virtual Keys**: Generate API keys for users with usage tracking
|
||||||
|
- **Cost Tracking**: Monitor spending across providers
|
||||||
|
- **Rate Limiting**: Control usage per key/user
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Kubernetes cluster (k3s)
|
||||||
|
- External Secrets Operator (required)
|
||||||
|
- PostgreSQL cluster (CloudNativePG)
|
||||||
|
- Vault for secrets management
|
||||||
|
|
||||||
|
## Configuration Overview
|
||||||
|
|
||||||
|
LiteLLM requires two types of configuration:
|
||||||
|
|
||||||
|
1. **Environment variables** (`.env.local`): Host, namespace, chart version
|
||||||
|
2. **Model definitions** (`models.yaml`): LLM providers and models to expose
|
||||||
|
|
||||||
|
This separation allows flexible model configuration without modifying environment files.
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
### Step 1: Create Model Configuration
|
||||||
|
|
||||||
|
Copy the example configuration and customize:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp litellm/models.example.yaml litellm/models.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
Edit `litellm/models.yaml` to configure your models:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Anthropic Claude
|
||||||
|
- model_name: claude-sonnet
|
||||||
|
litellm_params:
|
||||||
|
model: anthropic/claude-3-7-sonnet-latest
|
||||||
|
api_key: os.environ/ANTHROPIC_API_KEY
|
||||||
|
|
||||||
|
# OpenAI
|
||||||
|
- model_name: gpt-4o
|
||||||
|
litellm_params:
|
||||||
|
model: openai/gpt-4o
|
||||||
|
api_key: os.environ/OPENAI_API_KEY
|
||||||
|
|
||||||
|
# Ollama (local models - no API key required)
|
||||||
|
- model_name: llama3
|
||||||
|
litellm_params:
|
||||||
|
model: ollama/llama3.2
|
||||||
|
api_base: http://ollama.ollama:11434
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Set API Keys
|
||||||
|
|
||||||
|
For each provider that requires an API key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-api-key anthropic
|
||||||
|
just litellm::set-api-key openai
|
||||||
|
```
|
||||||
|
|
||||||
|
Or interactively select the provider:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-api-key
|
||||||
|
```
|
||||||
|
|
||||||
|
API keys are stored in Vault and synced to Kubernetes via External Secrets Operator.
|
||||||
|
|
||||||
|
### Step 3: Install LiteLLM
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::install
|
||||||
|
```
|
||||||
|
|
||||||
|
You will be prompted for:
|
||||||
|
|
||||||
|
- **LiteLLM host (FQDN)**: e.g., `litellm.example.com`
|
||||||
|
- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed
|
||||||
|
|
||||||
|
## Model Management
|
||||||
|
|
||||||
|
### Add a Model Interactively
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::add-model
|
||||||
|
```
|
||||||
|
|
||||||
|
This guides you through:
|
||||||
|
|
||||||
|
1. Selecting a provider
|
||||||
|
2. Choosing a model
|
||||||
|
3. Setting a model alias
|
||||||
|
|
||||||
|
### Remove a Model
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::remove-model
|
||||||
|
```
|
||||||
|
|
||||||
|
### List Configured Models
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::list-models
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example Output
|
||||||
|
|
||||||
|
```text
|
||||||
|
Configured models:
|
||||||
|
- claude-sonnet: anthropic/claude-3-7-sonnet-latest
|
||||||
|
- claude-haiku: anthropic/claude-3-5-haiku-latest
|
||||||
|
- llama3: ollama/llama3.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Key Management
|
||||||
|
|
||||||
|
### Set API Key for a Provider
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-api-key anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get API Key (from Vault)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::get-api-key anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify All Required Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::verify-api-keys
|
||||||
|
```
|
||||||
|
|
||||||
|
## Environment Variables
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
| -------- | ------- | ----------- |
|
||||||
|
| `LITELLM_NAMESPACE` | `litellm` | Kubernetes namespace |
|
||||||
|
| `LITELLM_CHART_VERSION` | `0.1.825` | Helm chart version |
|
||||||
|
| `LITELLM_HOST` | (prompt) | External hostname (FQDN) |
|
||||||
|
| `OLLAMA_NAMESPACE` | `ollama` | Ollama namespace for local models |
|
||||||
|
| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor |
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
LiteLLM has two types of authentication:
|
||||||
|
|
||||||
|
1. **API Access**: Uses Master Key or Virtual Keys for programmatic access
|
||||||
|
2. **Admin UI**: Uses Keycloak SSO for browser-based access
|
||||||
|
|
||||||
|
### Enable SSO for Admin UI
|
||||||
|
|
||||||
|
After installing LiteLLM, enable Keycloak authentication for the Admin UI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::setup-oidc
|
||||||
|
```
|
||||||
|
|
||||||
|
This will:
|
||||||
|
|
||||||
|
- Create a Keycloak client for LiteLLM
|
||||||
|
- Store the client secret in Vault
|
||||||
|
- Configure LiteLLM with OIDC environment variables
|
||||||
|
- Upgrade the deployment with SSO enabled
|
||||||
|
|
||||||
|
### Disable SSO
|
||||||
|
|
||||||
|
To disable SSO and return to unauthenticated Admin UI access:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::disable-oidc
|
||||||
|
```
|
||||||
|
|
||||||
|
### SSO Configuration Details
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
| ------- | ----- |
|
||||||
|
| Callback URL | `https://<litellm-host>/sso/callback` |
|
||||||
|
| Authorization Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth` |
|
||||||
|
| Token Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/token` |
|
||||||
|
| Userinfo Endpoint | `https://<keycloak-host>/realms/<realm>/protocol/openid-connect/userinfo` |
|
||||||
|
| Scope | `openid email profile` |
|
||||||
|
|
||||||
|
## User Management
|
||||||
|
|
||||||
|
SSO users are automatically created in LiteLLM when they first log in. By default, new users are assigned the `internal_user_viewer` role (read-only access).
|
||||||
|
|
||||||
|
### List Users
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::list-users
|
||||||
|
```
|
||||||
|
|
||||||
|
### Assign Role to User
|
||||||
|
|
||||||
|
Interactively select user and role:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::assign-role
|
||||||
|
```
|
||||||
|
|
||||||
|
Or specify directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::assign-role buun proxy_admin
|
||||||
|
```
|
||||||
|
|
||||||
|
### User Roles
|
||||||
|
|
||||||
|
| Role | Description |
|
||||||
|
| ---- | ----------- |
|
||||||
|
| `proxy_admin` | Full admin access (manage keys, users, models, settings) |
|
||||||
|
| `proxy_admin_viewer` | Admin read-only access |
|
||||||
|
| `internal_user` | Can create and manage own API keys |
|
||||||
|
| `internal_user_viewer` | Read-only access (default for SSO users) |
|
||||||
|
|
||||||
|
**Note**: To manage API keys in the Admin UI, users need at least `internal_user` or `proxy_admin` role.
|
||||||
|
|
||||||
|
## API Usage
|
||||||
|
|
||||||
|
LiteLLM exposes an OpenAI-compatible API at `https://your-litellm-host/`.
|
||||||
|
|
||||||
|
### Get Master Key
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::master-key
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generate Virtual Key for a User
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::generate-virtual-key buun
|
||||||
|
```
|
||||||
|
|
||||||
|
This will prompt for a model selection and generate an API key for the specified user. Select `all` to grant access to all models.
|
||||||
|
|
||||||
|
### OpenAI SDK Example
|
||||||
|
|
||||||
|
```python
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(
|
||||||
|
base_url="https://litellm.example.com",
|
||||||
|
api_key="sk-..." # Virtual key or master key
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="claude-sonnet", # Use your model alias
|
||||||
|
messages=[{"role": "user", "content": "Hello!"}]
|
||||||
|
)
|
||||||
|
print(response.choices[0].message.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
### curl Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl https://litellm.example.com/v1/chat/completions \
|
||||||
|
-H "Authorization: Bearer sk-..." \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "claude-sonnet",
|
||||||
|
"messages": [{"role": "user", "content": "Hello!"}]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Team Management
|
||||||
|
|
||||||
|
Teams allow you to group users and configure team-specific settings such as Langfuse projects for observability.
|
||||||
|
|
||||||
|
### Create a Team
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::create-team
|
||||||
|
```
|
||||||
|
|
||||||
|
Or with a name directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::create-team name="project-alpha"
|
||||||
|
```
|
||||||
|
|
||||||
|
### List Teams
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::list-teams
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Team Info
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::get-team team_id=<team-id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Delete a Team
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::delete-team team_id=<team-id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Generate Virtual Key for a Team
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::generate-team-key
|
||||||
|
```
|
||||||
|
|
||||||
|
This will prompt for team selection and username. The generated key inherits the team's settings (including Langfuse project configuration).
|
||||||
|
|
||||||
|
## Langfuse Integration
|
||||||
|
|
||||||
|
[Langfuse](https://langfuse.com/) provides LLM observability with tracing, monitoring, and analytics. LiteLLM can send traces to Langfuse for every API call.
|
||||||
|
|
||||||
|
### Enable Langfuse Integration
|
||||||
|
|
||||||
|
During installation (`just litellm::install`) or upgrade (`just litellm::upgrade`), you will be prompted to enable Langfuse integration. Alternatively:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::setup-langfuse
|
||||||
|
```
|
||||||
|
|
||||||
|
You will need Langfuse API keys (Public Key and Secret Key) from the Langfuse UI: **Settings > API Keys**.
|
||||||
|
|
||||||
|
### Set Langfuse API Keys
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-langfuse-keys
|
||||||
|
```
|
||||||
|
|
||||||
|
### Disable Langfuse Integration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::disable-langfuse
|
||||||
|
```
|
||||||
|
|
||||||
|
### Per-Team Langfuse Projects
|
||||||
|
|
||||||
|
Each team can have its own Langfuse project for isolated observability. This is useful when different projects or departments need separate trace data.
|
||||||
|
|
||||||
|
#### Setup Flow
|
||||||
|
|
||||||
|
1. Create a team:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::create-team name="project-alpha"
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create a Langfuse project for the team and get API keys from Langfuse UI
|
||||||
|
|
||||||
|
3. Configure the team's Langfuse project:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-team-langfuse-project
|
||||||
|
```
|
||||||
|
|
||||||
|
This will prompt for team selection and Langfuse API keys.
|
||||||
|
|
||||||
|
4. Generate a key for the team:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::generate-team-key
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Use the team key for API calls - traces will be sent to the team's Langfuse project
|
||||||
|
|
||||||
|
#### Architecture
|
||||||
|
|
||||||
|
```plain
|
||||||
|
LiteLLM Proxy
|
||||||
|
|
|
||||||
|
+-- Default Langfuse Project (for keys without team)
|
||||||
|
|
|
||||||
|
+-- Team A --> Langfuse Project A
|
||||||
|
|
|
||||||
|
+-- Team B --> Langfuse Project B
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
| -------- | ------- | ----------- |
|
||||||
|
| `LITELLM_LANGFUSE_INTEGRATION_ENABLED` | (prompt) | Enable Langfuse integration |
|
||||||
|
| `LANGFUSE_HOST` | (prompt) | Langfuse instance hostname |
|
||||||
|
|
||||||
|
## Supported Providers
|
||||||
|
|
||||||
|
| Provider | Model Prefix | API Key Required |
|
||||||
|
| -------- | ------------ | ---------------- |
|
||||||
|
| Anthropic | `anthropic/` | Yes |
|
||||||
|
| OpenAI | `openai/` | Yes |
|
||||||
|
| Ollama | `ollama/` | No (uses `api_base`) |
|
||||||
|
| Mistral | `mistral/` | Yes |
|
||||||
|
| Groq | `groq/` | Yes |
|
||||||
|
| Cohere | `cohere/` | Yes |
|
||||||
|
| Azure OpenAI | `azure/` | Yes |
|
||||||
|
| AWS Bedrock | `bedrock/` | Yes |
|
||||||
|
| Google Vertex AI | `vertexai/` | Yes |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```plain
|
||||||
|
External Users/Applications
|
||||||
|
|
|
||||||
|
Cloudflare Tunnel (HTTPS)
|
||||||
|
|
|
||||||
|
Traefik Ingress (HTTPS)
|
||||||
|
|
|
||||||
|
LiteLLM Proxy (HTTP inside cluster)
|
||||||
|
|-- PostgreSQL (usage tracking, virtual keys)
|
||||||
|
|-- Redis (caching, rate limiting)
|
||||||
|
|-- External Secrets (API keys from Vault)
|
||||||
|
|
|
||||||
|
+-- Anthropic API
|
||||||
|
+-- OpenAI API
|
||||||
|
+-- Ollama (local)
|
||||||
|
+-- Other providers...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
After modifying `models.yaml` or updating API keys:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
## Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
This removes:
|
||||||
|
|
||||||
|
- Helm release and all Kubernetes resources
|
||||||
|
- Namespace
|
||||||
|
- External Secrets
|
||||||
|
|
||||||
|
**Note**: The following resources are NOT deleted:
|
||||||
|
|
||||||
|
- PostgreSQL database (use `just postgres::delete-db litellm`)
|
||||||
|
- API keys in Vault
|
||||||
|
|
||||||
|
### Full Cleanup
|
||||||
|
|
||||||
|
To remove everything including database and Vault secrets:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::cleanup
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Check Pod Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get pods -n litellm
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected pods:
|
||||||
|
|
||||||
|
- `litellm-*` - LiteLLM proxy
|
||||||
|
- `litellm-redis-master-0` - Redis instance
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl logs -n litellm deployment/litellm --tail=100
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Key Not Working
|
||||||
|
|
||||||
|
Verify the ExternalSecret is synced:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get externalsecret -n litellm
|
||||||
|
kubectl get secret apikey -n litellm -o yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Model Not Found
|
||||||
|
|
||||||
|
Ensure the model is configured in `models.yaml` and the deployment is updated:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::list-models
|
||||||
|
just litellm::upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
### Provider API Errors
|
||||||
|
|
||||||
|
Check if the API key is set correctly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::get-api-key anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
If empty, set the API key:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just litellm::set-api-key anthropic
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Connection Issues
|
||||||
|
|
||||||
|
Check PostgreSQL connectivity:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl exec -n litellm deployment/litellm -- \
|
||||||
|
psql -h postgres-cluster-rw.postgres -U litellm -d litellm -c "SELECT 1"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration Files
|
||||||
|
|
||||||
|
| File | Description |
|
||||||
|
| ---- | ----------- |
|
||||||
|
| `models.yaml` | Model definitions (user-created, gitignored) |
|
||||||
|
| `models.example.yaml` | Example model configuration |
|
||||||
|
| `litellm-values.gomplate.yaml` | Helm values template |
|
||||||
|
| `apikey-external-secret.gomplate.yaml` | ExternalSecret for API keys |
|
||||||
|
| `keycloak-auth-external-secret.gomplate.yaml` | ExternalSecret for Keycloak OIDC |
|
||||||
|
| `langfuse-auth-external-secret.gomplate.yaml` | ExternalSecret for Langfuse API keys |
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
- **Pod Security Standards**: Namespace configured with **baseline** enforcement
|
||||||
|
(LiteLLM's Prisma requires write access to `/.cache`, which prevents `restricted` level)
|
||||||
|
- **Secrets Management**: API keys stored in Vault, synced via External Secrets Operator
|
||||||
|
- **Virtual Keys**: Generate scoped API keys for users instead of sharing master key
|
||||||
|
- **TLS/HTTPS**: All external traffic encrypted via Traefik Ingress
|
||||||
|
- **Database Credentials**: Unique PostgreSQL user with minimal privileges
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [LiteLLM Documentation](https://docs.litellm.ai/)
|
||||||
|
- [LiteLLM GitHub](https://github.com/BerriAI/litellm)
|
||||||
|
- [LiteLLM Helm Chart](https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm)
|
||||||
|
- [Supported Models](https://docs.litellm.ai/docs/providers)
|
||||||
|
- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys)
|
||||||
|
- [Langfuse Integration](https://docs.litellm.ai/docs/proxy/logging#langfuse)
|
||||||
|
- [Team-based Logging](https://docs.litellm.ai/docs/proxy/team_logging)
|
||||||
29
litellm/apikey-external-secret.gomplate.yaml
Normal file
29
litellm/apikey-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{{- $models := (datasource "models") -}}
|
||||||
|
{{- $providerMap := dict -}}
|
||||||
|
{{- range $models -}}
|
||||||
|
{{- if has .litellm_params "api_key" -}}
|
||||||
|
{{- $parts := strings.Split "/" .litellm_params.model -}}
|
||||||
|
{{- $provider := index $parts 0 -}}
|
||||||
|
{{- $providerMap = merge $providerMap (dict $provider true) -}}
|
||||||
|
{{- end -}}
|
||||||
|
{{- end -}}
|
||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: apikey-external-secret
|
||||||
|
namespace: {{ .Env.LITELLM_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: apikey
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
{{- range $provider, $_ := $providerMap }}
|
||||||
|
- secretKey: {{ $provider | strings.ToUpper }}_API_KEY
|
||||||
|
remoteRef:
|
||||||
|
key: litellm/{{ $provider }}
|
||||||
|
property: apikey
|
||||||
|
{{- end }}
|
||||||
1074
litellm/justfile
Normal file
1074
litellm/justfile
Normal file
File diff suppressed because it is too large
Load Diff
22
litellm/keycloak-auth-external-secret.gomplate.yaml
Normal file
22
litellm/keycloak-auth-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: keycloak-auth-external-secret
|
||||||
|
namespace: {{ .Env.LITELLM_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: keycloak-auth
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
- secretKey: GENERIC_CLIENT_ID
|
||||||
|
remoteRef:
|
||||||
|
key: keycloak/client/litellm
|
||||||
|
property: client_id
|
||||||
|
- secretKey: GENERIC_CLIENT_SECRET
|
||||||
|
remoteRef:
|
||||||
|
key: keycloak/client/litellm
|
||||||
|
property: client_secret
|
||||||
22
litellm/langfuse-auth-external-secret.gomplate.yaml
Normal file
22
litellm/langfuse-auth-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: langfuse-auth-external-secret
|
||||||
|
namespace: {{ .Env.LITELLM_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: langfuse-auth
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
- secretKey: LANGFUSE_PUBLIC_KEY
|
||||||
|
remoteRef:
|
||||||
|
key: litellm/langfuse
|
||||||
|
property: public_key
|
||||||
|
- secretKey: LANGFUSE_SECRET_KEY
|
||||||
|
remoteRef:
|
||||||
|
key: litellm/langfuse
|
||||||
|
property: secret_key
|
||||||
114
litellm/litellm-values.gomplate.yaml
Normal file
114
litellm/litellm-values.gomplate.yaml
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
# https://github.com/BerriAI/litellm/tree/main/deploy/charts/litellm-helm
|
||||||
|
# https://github.com/BerriAI/litellm/tree/main/litellm/proxy/example_config_yaml
|
||||||
|
|
||||||
|
masterkeySecretName: ""
|
||||||
|
masterkeySecretKey: ""
|
||||||
|
|
||||||
|
# Note: LiteLLM image requires write access to /.cache for Prisma
|
||||||
|
# Pod Security Standards must be set to "baseline" for this namespace
|
||||||
|
podSecurityContext: {}
|
||||||
|
|
||||||
|
securityContext: {}
|
||||||
|
|
||||||
|
# Resource recommendations from Goldilocks VPA
|
||||||
|
# litellm target: cpu=11m, memory=549Mi
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 512Mi
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 1Gi
|
||||||
|
|
||||||
|
migrationJob:
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 512Mi
|
||||||
|
limits:
|
||||||
|
memory: 1Gi
|
||||||
|
|
||||||
|
environmentSecrets:
|
||||||
|
- apikey
|
||||||
|
{{- if .Env.LITELLM_OIDC_ENABLED }}
|
||||||
|
- keycloak-auth
|
||||||
|
{{- end }}
|
||||||
|
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
|
||||||
|
- langfuse-auth
|
||||||
|
{{- end }}
|
||||||
|
|
||||||
|
extraEnvVars:
|
||||||
|
{{- if .Env.LITELLM_OIDC_ENABLED }}
|
||||||
|
- name: PROXY_BASE_URL
|
||||||
|
value: "https://{{ .Env.LITELLM_HOST }}"
|
||||||
|
- name: GENERIC_AUTHORIZATION_ENDPOINT
|
||||||
|
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/auth"
|
||||||
|
- name: GENERIC_TOKEN_ENDPOINT
|
||||||
|
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/token"
|
||||||
|
- name: GENERIC_USERINFO_ENDPOINT
|
||||||
|
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/protocol/openid-connect/userinfo"
|
||||||
|
- name: GENERIC_SCOPE
|
||||||
|
value: "openid email profile"
|
||||||
|
{{- end }}
|
||||||
|
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
|
||||||
|
- name: LANGFUSE_HOST
|
||||||
|
value: "https://{{ .Env.LANGFUSE_HOST }}"
|
||||||
|
{{- end }}
|
||||||
|
|
||||||
|
proxy_config:
|
||||||
|
model_list:
|
||||||
|
{{ file.Read "models.yaml" | indent 4 }}
|
||||||
|
{{- if .Env.LITELLM_LANGFUSE_INTEGRATION_ENABLED }}
|
||||||
|
litellm_settings:
|
||||||
|
success_callback: ["langfuse"]
|
||||||
|
failure_callback: ["langfuse"]
|
||||||
|
{{- end }}
|
||||||
|
|
||||||
|
db:
|
||||||
|
useExisting: true
|
||||||
|
|
||||||
|
endpoint: postgres-cluster-rw.postgres
|
||||||
|
database: litellm
|
||||||
|
secret:
|
||||||
|
name: postgres-auth
|
||||||
|
usernameKey: username
|
||||||
|
passwordKey: password
|
||||||
|
|
||||||
|
deployStandalone: false
|
||||||
|
|
||||||
|
redis:
|
||||||
|
enabled: true
|
||||||
|
# Resource recommendations from Goldilocks VPA
|
||||||
|
# redis target: cpu=15m, memory=100Mi
|
||||||
|
master:
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 25m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
|
ingress:
|
||||||
|
enabled: true
|
||||||
|
className: traefik
|
||||||
|
annotations:
|
||||||
|
kubernetes.io/ingress.class: traefik
|
||||||
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||||
|
hosts:
|
||||||
|
- host: {{ .Env.LITELLM_HOST }}
|
||||||
|
paths:
|
||||||
|
- path: /
|
||||||
|
pathType: ImplementationSpecific
|
||||||
|
tls:
|
||||||
|
- hosts:
|
||||||
|
- {{ .Env.LITELLM_HOST }}
|
||||||
|
|
||||||
|
{{- if .Env.MONITORING_ENABLED }}
|
||||||
|
serviceMonitor:
|
||||||
|
enabled: true
|
||||||
|
labels:
|
||||||
|
release: kube-prometheus-stack
|
||||||
|
interval: 30s
|
||||||
|
scrapeTimeout: 10s
|
||||||
|
{{- end }}
|
||||||
106
litellm/models.example.yaml
Normal file
106
litellm/models.example.yaml
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# LiteLLM Model Configuration
|
||||||
|
# Copy this file to models.yaml and customize for your environment.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# cp litellm/models.example.yaml litellm/models.yaml
|
||||||
|
# # Edit models.yaml to add/remove models
|
||||||
|
# just litellm::install
|
||||||
|
#
|
||||||
|
# API keys are stored in Vault and injected as environment variables.
|
||||||
|
# Use: just litellm::set-api-key provider=<provider>
|
||||||
|
#
|
||||||
|
# Supported providers:
|
||||||
|
# - anthropic: Claude models (Opus, Sonnet, Haiku)
|
||||||
|
# - openai: GPT and o-series models
|
||||||
|
# - ollama: Local models (no API key required)
|
||||||
|
# - azure: Azure OpenAI
|
||||||
|
# - bedrock: AWS Bedrock
|
||||||
|
# - vertexai: Google Vertex AI
|
||||||
|
# - mistral: Mistral AI
|
||||||
|
# - groq: Groq (fast inference)
|
||||||
|
# - cohere: Cohere
|
||||||
|
|
||||||
|
# Anthropic Claude (https://docs.anthropic.com/en/docs/about-claude/models/overview)
|
||||||
|
- model_name: claude-sonnet
|
||||||
|
litellm_params:
|
||||||
|
model: anthropic/claude-sonnet-4-20250514
|
||||||
|
api_key: os.environ/ANTHROPIC_API_KEY
|
||||||
|
|
||||||
|
- model_name: claude-haiku
|
||||||
|
litellm_params:
|
||||||
|
model: anthropic/claude-haiku-4-20251015
|
||||||
|
api_key: os.environ/ANTHROPIC_API_KEY
|
||||||
|
|
||||||
|
# - model_name: claude-opus
|
||||||
|
# litellm_params:
|
||||||
|
# model: anthropic/claude-opus-4-20250514
|
||||||
|
# api_key: os.environ/ANTHROPIC_API_KEY
|
||||||
|
|
||||||
|
# OpenAI (https://platform.openai.com/docs/models)
|
||||||
|
# - model_name: gpt-4o
|
||||||
|
# litellm_params:
|
||||||
|
# model: openai/gpt-4o
|
||||||
|
# api_key: os.environ/OPENAI_API_KEY
|
||||||
|
|
||||||
|
# - model_name: gpt-4o-mini
|
||||||
|
# litellm_params:
|
||||||
|
# model: openai/gpt-4o-mini
|
||||||
|
# api_key: os.environ/OPENAI_API_KEY
|
||||||
|
|
||||||
|
# - model_name: o3
|
||||||
|
# litellm_params:
|
||||||
|
# model: openai/o3
|
||||||
|
# api_key: os.environ/OPENAI_API_KEY
|
||||||
|
|
||||||
|
# - model_name: o4-mini
|
||||||
|
# litellm_params:
|
||||||
|
# model: openai/o4-mini
|
||||||
|
# api_key: os.environ/OPENAI_API_KEY
|
||||||
|
|
||||||
|
# Ollama (local models - no API key required)
|
||||||
|
# - model_name: llama4-scout
|
||||||
|
# litellm_params:
|
||||||
|
# model: ollama/llama4:scout
|
||||||
|
# api_base: http://ollama.ollama:11434
|
||||||
|
|
||||||
|
# - model_name: qwen3
|
||||||
|
# litellm_params:
|
||||||
|
# model: ollama/qwen3:8b
|
||||||
|
# api_base: http://ollama.ollama:11434
|
||||||
|
|
||||||
|
# - model_name: deepseek-r1
|
||||||
|
# litellm_params:
|
||||||
|
# model: ollama/deepseek-r1:8b
|
||||||
|
# api_base: http://ollama.ollama:11434
|
||||||
|
|
||||||
|
# Mistral AI (https://docs.mistral.ai/getting-started/models/models_overview/)
|
||||||
|
# - model_name: mistral-large
|
||||||
|
# litellm_params:
|
||||||
|
# model: mistral/mistral-large-latest
|
||||||
|
# api_key: os.environ/MISTRAL_API_KEY
|
||||||
|
|
||||||
|
# - model_name: ministral-8b
|
||||||
|
# litellm_params:
|
||||||
|
# model: mistral/ministral-8b-latest
|
||||||
|
# api_key: os.environ/MISTRAL_API_KEY
|
||||||
|
|
||||||
|
# - model_name: codestral
|
||||||
|
# litellm_params:
|
||||||
|
# model: mistral/codestral-latest
|
||||||
|
# api_key: os.environ/MISTRAL_API_KEY
|
||||||
|
|
||||||
|
# Groq (fast inference - https://console.groq.com/docs/models)
|
||||||
|
# - model_name: groq-llama4-scout
|
||||||
|
# litellm_params:
|
||||||
|
# model: groq/meta-llama/llama-4-scout-17b-16e-instruct
|
||||||
|
# api_key: os.environ/GROQ_API_KEY
|
||||||
|
|
||||||
|
# - model_name: groq-llama3.3
|
||||||
|
# litellm_params:
|
||||||
|
# model: groq/llama-3.3-70b-versatile
|
||||||
|
# api_key: os.environ/GROQ_API_KEY
|
||||||
|
|
||||||
|
# - model_name: groq-llama3.1
|
||||||
|
# litellm_params:
|
||||||
|
# model: groq/llama-3.1-8b-instant
|
||||||
|
# api_key: os.environ/GROQ_API_KEY
|
||||||
@@ -221,6 +221,83 @@ just minio::bucket-exists mybucket
|
|||||||
|
|
||||||
This returns exit code 0 if the bucket exists, 1 otherwise.
|
This returns exit code 0 if the bucket exists, 1 otherwise.
|
||||||
|
|
||||||
|
## Public Access
|
||||||
|
|
||||||
|
MinIO allows you to configure anonymous (public) access to buckets or specific prefixes for serving static content like images.
|
||||||
|
|
||||||
|
### Set Public Download Access
|
||||||
|
|
||||||
|
Enable public read access for a bucket or prefix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set public access for entire bucket
|
||||||
|
just minio::set-public-download mybucket
|
||||||
|
|
||||||
|
# Set public access for specific prefix only
|
||||||
|
just minio::set-public-download mybucket/public
|
||||||
|
```
|
||||||
|
|
||||||
|
After setting public access, files can be accessed without authentication:
|
||||||
|
|
||||||
|
```text
|
||||||
|
https://your-minio-host/mybucket/public/image.png
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Public Access Status
|
||||||
|
|
||||||
|
View current anonymous access policy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just minio::show-public-access mybucket
|
||||||
|
```
|
||||||
|
|
||||||
|
Possible values:
|
||||||
|
|
||||||
|
- `private`: No anonymous access (default)
|
||||||
|
- `download`: Public read access
|
||||||
|
- `upload`: Public write access
|
||||||
|
- `public`: Public read and write access
|
||||||
|
- `custom`: Custom policy applied
|
||||||
|
|
||||||
|
### Remove Public Access
|
||||||
|
|
||||||
|
Revoke anonymous access:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just minio::remove-public-access mybucket/public
|
||||||
|
```
|
||||||
|
|
||||||
|
### Using mc Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Set public download (read-only)
|
||||||
|
mc anonymous set download myminio/mybucket/public
|
||||||
|
|
||||||
|
# Set public upload (write-only)
|
||||||
|
mc anonymous set upload myminio/mybucket/uploads
|
||||||
|
|
||||||
|
# Set full public access (read and write)
|
||||||
|
mc anonymous set public myminio/mybucket
|
||||||
|
|
||||||
|
# Remove public access
|
||||||
|
mc anonymous set none myminio/mybucket
|
||||||
|
|
||||||
|
# Check current policy
|
||||||
|
mc anonymous get myminio/mybucket
|
||||||
|
```
|
||||||
|
|
||||||
|
### Presigned URLs (Temporary Access)
|
||||||
|
|
||||||
|
For temporary access to private objects, use presigned URLs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Generate URL valid for 7 days
|
||||||
|
mc share download myminio/mybucket/private-file.pdf --expire=168h
|
||||||
|
|
||||||
|
# Generate upload URL
|
||||||
|
mc share upload myminio/mybucket/uploads/ --expire=1h
|
||||||
|
```
|
||||||
|
|
||||||
## User Management
|
## User Management
|
||||||
|
|
||||||
### Create MinIO User
|
### Create MinIO User
|
||||||
|
|||||||
@@ -47,6 +47,7 @@ create-root-credentials:
|
|||||||
gomplate -f minio-root-external-secret.gomplate.yaml | kubectl apply -f -
|
gomplate -f minio-root-external-secret.gomplate.yaml | kubectl apply -f -
|
||||||
|
|
||||||
echo "Waiting for ExternalSecret to sync..."
|
echo "Waiting for ExternalSecret to sync..."
|
||||||
|
sleep 2
|
||||||
kubectl wait --for=condition=Ready externalsecret/minio \
|
kubectl wait --for=condition=Ready externalsecret/minio \
|
||||||
-n ${MINIO_NAMESPACE} --timeout=60s
|
-n ${MINIO_NAMESPACE} --timeout=60s
|
||||||
else
|
else
|
||||||
@@ -96,7 +97,12 @@ install:
|
|||||||
--placeholder="e.g., minio-console.example.com"
|
--placeholder="e.g., minio-console.example.com"
|
||||||
)
|
)
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Generate OIDC client secret for confidential client
|
||||||
|
OIDC_CLIENT_SECRET=$(just utils::random-password)
|
||||||
|
|
||||||
just keycloak::create-client realm=${KEYCLOAK_REALM} client_id=${MINIO_OIDC_CLIENT_ID} \
|
just keycloak::create-client realm=${KEYCLOAK_REALM} client_id=${MINIO_OIDC_CLIENT_ID} \
|
||||||
|
client_secret="${OIDC_CLIENT_SECRET}" \
|
||||||
redirect_url="https://${MINIO_HOST}/oauth_callback,https://${MINIO_CONSOLE_HOST}/oauth_callback"
|
redirect_url="https://${MINIO_HOST}/oauth_callback,https://${MINIO_CONSOLE_HOST}/oauth_callback"
|
||||||
just add-keycloak-minio-policy
|
just add-keycloak-minio-policy
|
||||||
just create-namespace
|
just create-namespace
|
||||||
@@ -105,6 +111,28 @@ install:
|
|||||||
pod-security.kubernetes.io/enforce=restricted --overwrite
|
pod-security.kubernetes.io/enforce=restricted --overwrite
|
||||||
|
|
||||||
just create-root-credentials
|
just create-root-credentials
|
||||||
|
|
||||||
|
# Store OIDC client secret
|
||||||
|
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
|
||||||
|
echo "Storing OIDC client secret in Vault..."
|
||||||
|
just vault::put minio/oidc client_id="${MINIO_OIDC_CLIENT_ID}" client_secret="${OIDC_CLIENT_SECRET}"
|
||||||
|
kubectl delete externalsecret minio-oidc -n ${MINIO_NAMESPACE} --ignore-not-found
|
||||||
|
gomplate -f minio-oidc-external-secret.gomplate.yaml | kubectl apply -f -
|
||||||
|
echo "Waiting for ExternalSecret to sync..."
|
||||||
|
sleep 2
|
||||||
|
kubectl wait --for=condition=Ready externalsecret/minio-oidc \
|
||||||
|
-n ${MINIO_NAMESPACE} --timeout=60s
|
||||||
|
else
|
||||||
|
echo "Creating OIDC client secret directly..."
|
||||||
|
kubectl delete secret minio-oidc -n ${MINIO_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl create secret generic minio-oidc -n ${MINIO_NAMESPACE} \
|
||||||
|
--from-literal=clientId="${MINIO_OIDC_CLIENT_ID}" \
|
||||||
|
--from-literal=clientSecret="${OIDC_CLIENT_SECRET}"
|
||||||
|
if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then
|
||||||
|
just vault::put minio/oidc client_id="${MINIO_OIDC_CLIENT_ID}" client_secret="${OIDC_CLIENT_SECRET}"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
just add-helm-repo
|
just add-helm-repo
|
||||||
gomplate -f minio-values.gomplate.yaml -o minio-values.yaml
|
gomplate -f minio-values.gomplate.yaml -o minio-values.yaml
|
||||||
helm upgrade --install minio minio/minio \
|
helm upgrade --install minio minio/minio \
|
||||||
@@ -260,3 +288,70 @@ grant-policy user='' policy='':
|
|||||||
mc admin policy attach local ${POLICY} --user=${USER}"
|
mc admin policy attach local ${POLICY} --user=${USER}"
|
||||||
|
|
||||||
echo "✅ Policy ${POLICY} granted to user ${USER}"
|
echo "✅ Policy ${POLICY} granted to user ${USER}"
|
||||||
|
|
||||||
|
# Set public download access for a bucket or prefix
|
||||||
|
set-public-download path='':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
PATH_ARG="{{ path }}"
|
||||||
|
while [ -z "${PATH_ARG}" ]; do
|
||||||
|
PATH_ARG=$(
|
||||||
|
gum input --prompt="Bucket/prefix path: " --width=100 \
|
||||||
|
--placeholder="e.g., my-bucket/public"
|
||||||
|
)
|
||||||
|
done
|
||||||
|
|
||||||
|
ROOT_USER=$(just root-username)
|
||||||
|
ROOT_PASSWORD=$(just root-password)
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc anonymous set download local/${PATH_ARG}
|
||||||
|
|
||||||
|
echo "✅ Public download access enabled for ${PATH_ARG}"
|
||||||
|
|
||||||
|
# Remove public access from a bucket or prefix
|
||||||
|
remove-public-access path='':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
PATH_ARG="{{ path }}"
|
||||||
|
while [ -z "${PATH_ARG}" ]; do
|
||||||
|
PATH_ARG=$(
|
||||||
|
gum input --prompt="Bucket/prefix path: " --width=100 \
|
||||||
|
--placeholder="e.g., my-bucket/public"
|
||||||
|
)
|
||||||
|
done
|
||||||
|
|
||||||
|
ROOT_USER=$(just root-username)
|
||||||
|
ROOT_PASSWORD=$(just root-password)
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc anonymous set none local/${PATH_ARG}
|
||||||
|
|
||||||
|
echo "✅ Public access removed from ${PATH_ARG}"
|
||||||
|
|
||||||
|
# Show anonymous access policy for a bucket or prefix
|
||||||
|
show-public-access path='':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
PATH_ARG="{{ path }}"
|
||||||
|
while [ -z "${PATH_ARG}" ]; do
|
||||||
|
PATH_ARG=$(
|
||||||
|
gum input --prompt="Bucket/prefix path: " --width=100 \
|
||||||
|
--placeholder="e.g., my-bucket"
|
||||||
|
)
|
||||||
|
done
|
||||||
|
|
||||||
|
ROOT_USER=$(just root-username)
|
||||||
|
ROOT_PASSWORD=$(just root-password)
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc alias set local http://localhost:9000 ${ROOT_USER} ${ROOT_PASSWORD}
|
||||||
|
|
||||||
|
kubectl -n ${MINIO_NAMESPACE} exec deploy/minio -- \
|
||||||
|
mc anonymous get local/${PATH_ARG}
|
||||||
|
|||||||
22
minio/minio-oidc-external-secret.gomplate.yaml
Normal file
22
minio/minio-oidc-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: minio-oidc
|
||||||
|
namespace: {{ .Env.MINIO_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: minio-oidc
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
- secretKey: clientId
|
||||||
|
remoteRef:
|
||||||
|
key: minio/oidc
|
||||||
|
property: client_id
|
||||||
|
- secretKey: clientSecret
|
||||||
|
remoteRef:
|
||||||
|
key: minio/oidc
|
||||||
|
property: client_secret
|
||||||
@@ -7,8 +7,9 @@ existingSecret: "minio"
|
|||||||
oidc:
|
oidc:
|
||||||
enabled: true
|
enabled: true
|
||||||
configUrl: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/.well-known/openid-configuration"
|
configUrl: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}/.well-known/openid-configuration"
|
||||||
clientId: "{{ .Env.MINIO_OIDC_CLIENT_ID }}"
|
existingClientSecretName: "minio-oidc"
|
||||||
clientSecret: ""
|
existingClientIdKey: "clientId"
|
||||||
|
existingClientSecretKey: "clientSecret"
|
||||||
claimName: "minioPolicy"
|
claimName: "minioPolicy"
|
||||||
scopes: "openid,profile,email"
|
scopes: "openid,profile,email"
|
||||||
redirectUri: "https://{{ .Env.MINIO_CONSOLE_HOST }}/oauth_callback"
|
redirectUri: "https://{{ .Env.MINIO_CONSOLE_HOST }}/oauth_callback"
|
||||||
|
|||||||
@@ -3,11 +3,11 @@ gomplate = "4.3.3"
|
|||||||
gum = "0.16.2"
|
gum = "0.16.2"
|
||||||
helm = "3.19.0"
|
helm = "3.19.0"
|
||||||
just = "1.42.4"
|
just = "1.42.4"
|
||||||
k3sup = "0.13.10"
|
k3sup = "0.13.11"
|
||||||
kubelogin = "1.34.0"
|
kubelogin = "1.34.0"
|
||||||
node = "22.18.0"
|
node = "22.18.0"
|
||||||
python = "3.12.11"
|
python = "3.12.11"
|
||||||
telepresence = "2.25.0"
|
telepresence = "2.25.1"
|
||||||
trivy = "0.67.2"
|
trivy = "0.67.2"
|
||||||
uv = "0.8.7"
|
uv = "0.8.7"
|
||||||
vault = "1.20.2"
|
vault = "1.20.2"
|
||||||
|
|||||||
113
nats/justfile
Normal file
113
nats/justfile
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
set fallback := true
|
||||||
|
|
||||||
|
export NATS_NAMESPACE := env("NATS_NAMESPACE", "nats")
|
||||||
|
export NATS_CHART_VERSION := env("NATS_CHART_VERSION", "2.12.2")
|
||||||
|
export NATS_REPLICAS := env("NATS_REPLICAS", "1")
|
||||||
|
export NATS_JETSTREAM_ENABLED := env("NATS_JETSTREAM_ENABLED", "true")
|
||||||
|
export NATS_JETSTREAM_STORAGE_SIZE := env("NATS_JETSTREAM_STORAGE_SIZE", "10Gi")
|
||||||
|
export NATS_JETSTREAM_MEMORY_SIZE := env("NATS_JETSTREAM_MEMORY_SIZE", "256Mi")
|
||||||
|
|
||||||
|
[private]
|
||||||
|
default:
|
||||||
|
@just --list --unsorted --list-submodules
|
||||||
|
|
||||||
|
# Add Helm repository
|
||||||
|
add-helm-repo:
|
||||||
|
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
|
||||||
|
helm repo update nats
|
||||||
|
|
||||||
|
# Remove Helm repository
|
||||||
|
remove-helm-repo:
|
||||||
|
helm repo remove nats
|
||||||
|
|
||||||
|
# Create NATS namespace
|
||||||
|
create-namespace:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if ! kubectl get namespace ${NATS_NAMESPACE} &>/dev/null; then
|
||||||
|
kubectl create namespace ${NATS_NAMESPACE}
|
||||||
|
fi
|
||||||
|
kubectl label namespace ${NATS_NAMESPACE} \
|
||||||
|
pod-security.kubernetes.io/enforce=restricted \
|
||||||
|
pod-security.kubernetes.io/enforce-version=latest \
|
||||||
|
pod-security.kubernetes.io/warn=restricted \
|
||||||
|
pod-security.kubernetes.io/warn-version=latest \
|
||||||
|
--overwrite
|
||||||
|
|
||||||
|
# Delete NATS namespace
|
||||||
|
delete-namespace:
|
||||||
|
kubectl delete namespace ${NATS_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
# Install NATS
|
||||||
|
install:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
just create-namespace
|
||||||
|
just add-helm-repo
|
||||||
|
|
||||||
|
gomplate -f values.gomplate.yaml -o values.yaml
|
||||||
|
helm upgrade --install nats nats/nats \
|
||||||
|
--version ${NATS_CHART_VERSION} \
|
||||||
|
-n ${NATS_NAMESPACE} \
|
||||||
|
-f values.yaml \
|
||||||
|
--wait
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "NATS installed successfully"
|
||||||
|
echo "Namespace: ${NATS_NAMESPACE}"
|
||||||
|
echo "Replicas: ${NATS_REPLICAS}"
|
||||||
|
echo "JetStream enabled: ${NATS_JETSTREAM_ENABLED}"
|
||||||
|
echo ""
|
||||||
|
echo "Internal URL: nats://nats.${NATS_NAMESPACE}.svc:4222"
|
||||||
|
|
||||||
|
# Upgrade NATS
|
||||||
|
upgrade:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
gomplate -f values.gomplate.yaml -o values.yaml
|
||||||
|
helm upgrade nats nats/nats \
|
||||||
|
--version ${NATS_CHART_VERSION} \
|
||||||
|
-n ${NATS_NAMESPACE} \
|
||||||
|
-f values.yaml \
|
||||||
|
--wait
|
||||||
|
|
||||||
|
echo "NATS upgraded successfully"
|
||||||
|
|
||||||
|
# Uninstall NATS
|
||||||
|
uninstall:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if ! gum confirm "Are you sure you want to uninstall NATS?"; then
|
||||||
|
echo "Aborted"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
helm uninstall nats -n ${NATS_NAMESPACE} --wait --ignore-not-found
|
||||||
|
just delete-namespace
|
||||||
|
echo "NATS uninstalled"
|
||||||
|
|
||||||
|
# Show NATS status
|
||||||
|
status:
|
||||||
|
kubectl get pods -n ${NATS_NAMESPACE}
|
||||||
|
kubectl get svc -n ${NATS_NAMESPACE}
|
||||||
|
|
||||||
|
# Show NATS logs
|
||||||
|
logs:
|
||||||
|
kubectl logs -n ${NATS_NAMESPACE} -l app.kubernetes.io/name=nats -f
|
||||||
|
|
||||||
|
# Show server info via monitoring endpoint
|
||||||
|
server-info:
|
||||||
|
kubectl exec -n ${NATS_NAMESPACE} nats-0 -c nats -- \
|
||||||
|
wget -qO- http://localhost:8222/varz | head -50
|
||||||
|
|
||||||
|
# Show JetStream info via monitoring endpoint
|
||||||
|
js-info:
|
||||||
|
kubectl exec -n ${NATS_NAMESPACE} nats-0 -c nats -- \
|
||||||
|
wget -qO- http://localhost:8222/jsz
|
||||||
|
|
||||||
|
# Port forward for local testing
|
||||||
|
port-forward:
|
||||||
|
@echo "NATS available at localhost:4222"
|
||||||
|
@echo "Monitor available at http://localhost:8222"
|
||||||
|
kubectl port-forward -n ${NATS_NAMESPACE} svc/nats 4222:4222 8222:8222
|
||||||
64
nats/values.gomplate.yaml
Normal file
64
nats/values.gomplate.yaml
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
config:
|
||||||
|
cluster:
|
||||||
|
enabled: {{ if gt (conv.ToInt .Env.NATS_REPLICAS) 1 }}true{{ else }}false{{ end }}
|
||||||
|
replicas: {{ .Env.NATS_REPLICAS }}
|
||||||
|
|
||||||
|
jetstream:
|
||||||
|
enabled: {{ .Env.NATS_JETSTREAM_ENABLED }}
|
||||||
|
fileStore:
|
||||||
|
enabled: true
|
||||||
|
dir: /data
|
||||||
|
pvc:
|
||||||
|
enabled: true
|
||||||
|
size: {{ .Env.NATS_JETSTREAM_STORAGE_SIZE }}
|
||||||
|
memoryStore:
|
||||||
|
enabled: true
|
||||||
|
maxSize: {{ .Env.NATS_JETSTREAM_MEMORY_SIZE }}
|
||||||
|
|
||||||
|
monitor:
|
||||||
|
enabled: true
|
||||||
|
port: 8222
|
||||||
|
|
||||||
|
container:
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
merge:
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
|
||||||
|
reloader:
|
||||||
|
enabled: true
|
||||||
|
merge:
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
|
||||||
|
podTemplate:
|
||||||
|
merge:
|
||||||
|
spec:
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1000
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
|
||||||
|
natsBox:
|
||||||
|
enabled: false
|
||||||
64
nats/values.yaml
Normal file
64
nats/values.yaml
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
config:
|
||||||
|
cluster:
|
||||||
|
enabled: false
|
||||||
|
replicas: 1
|
||||||
|
|
||||||
|
jetstream:
|
||||||
|
enabled: true
|
||||||
|
fileStore:
|
||||||
|
enabled: true
|
||||||
|
dir: /data
|
||||||
|
pvc:
|
||||||
|
enabled: true
|
||||||
|
size: 10Gi
|
||||||
|
memoryStore:
|
||||||
|
enabled: true
|
||||||
|
maxSize: 256Mi
|
||||||
|
|
||||||
|
monitor:
|
||||||
|
enabled: true
|
||||||
|
port: 8222
|
||||||
|
|
||||||
|
container:
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
merge:
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
|
||||||
|
reloader:
|
||||||
|
enabled: true
|
||||||
|
merge:
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
|
||||||
|
podTemplate:
|
||||||
|
merge:
|
||||||
|
spec:
|
||||||
|
securityContext:
|
||||||
|
fsGroup: 1000
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
|
||||||
|
natsBox:
|
||||||
|
enabled: false
|
||||||
@@ -8,6 +8,11 @@ export OLLAMA_GPU_TYPE := env("OLLAMA_GPU_TYPE", "nvidia")
|
|||||||
export OLLAMA_GPU_COUNT := env("OLLAMA_GPU_COUNT", "1")
|
export OLLAMA_GPU_COUNT := env("OLLAMA_GPU_COUNT", "1")
|
||||||
export OLLAMA_MODELS := env("OLLAMA_MODELS", "")
|
export OLLAMA_MODELS := env("OLLAMA_MODELS", "")
|
||||||
export OLLAMA_STORAGE_SIZE := env("OLLAMA_STORAGE_SIZE", "30Gi")
|
export OLLAMA_STORAGE_SIZE := env("OLLAMA_STORAGE_SIZE", "30Gi")
|
||||||
|
export OLLAMA_HELM_TIMEOUT := env("OLLAMA_HELM_TIMEOUT", "60m")
|
||||||
|
export OLLAMA_MEMORY_REQUEST := env("OLLAMA_MEMORY_REQUEST", "2Gi")
|
||||||
|
export OLLAMA_MEMORY_LIMIT := env("OLLAMA_MEMORY_LIMIT", "12Gi")
|
||||||
|
export OLLAMA_CPU_REQUEST := env("OLLAMA_CPU_REQUEST", "25m")
|
||||||
|
export OLLAMA_CPU_LIMIT := env("OLLAMA_CPU_LIMIT", "100m")
|
||||||
|
|
||||||
[private]
|
[private]
|
||||||
default:
|
default:
|
||||||
@@ -66,7 +71,8 @@ install:
|
|||||||
|
|
||||||
gomplate -f values.gomplate.yaml -o values.yaml
|
gomplate -f values.gomplate.yaml -o values.yaml
|
||||||
helm upgrade --install ollama ollama/ollama \
|
helm upgrade --install ollama ollama/ollama \
|
||||||
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} --wait \
|
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} \
|
||||||
|
--wait --timeout ${OLLAMA_HELM_TIMEOUT} \
|
||||||
-f values.yaml
|
-f values.yaml
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
@@ -97,7 +103,8 @@ upgrade:
|
|||||||
|
|
||||||
gomplate -f values.gomplate.yaml -o values.yaml
|
gomplate -f values.gomplate.yaml -o values.yaml
|
||||||
helm upgrade ollama ollama/ollama \
|
helm upgrade ollama ollama/ollama \
|
||||||
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} --wait \
|
--version ${OLLAMA_CHART_VERSION} -n ${OLLAMA_NAMESPACE} \
|
||||||
|
--wait --timeout ${OLLAMA_HELM_TIMEOUT} \
|
||||||
-f values.yaml
|
-f values.yaml
|
||||||
|
|
||||||
echo "Ollama upgraded successfully"
|
echo "Ollama upgraded successfully"
|
||||||
|
|||||||
@@ -36,11 +36,11 @@ securityContext:
|
|||||||
|
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: 25m
|
cpu: {{ .Env.OLLAMA_CPU_REQUEST }}
|
||||||
memory: 2Gi
|
memory: {{ .Env.OLLAMA_MEMORY_REQUEST }}
|
||||||
limits:
|
limits:
|
||||||
cpu: 100m
|
cpu: {{ .Env.OLLAMA_CPU_LIMIT }}
|
||||||
memory: 8Gi
|
memory: {{ .Env.OLLAMA_MEMORY_LIMIT }}
|
||||||
|
|
||||||
persistentVolume:
|
persistentVolume:
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|||||||
1
temporal/.gitignore
vendored
Normal file
1
temporal/.gitignore
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
temporal-values.yaml
|
||||||
401
temporal/README.md
Normal file
401
temporal/README.md
Normal file
@@ -0,0 +1,401 @@
|
|||||||
|
# Temporal
|
||||||
|
|
||||||
|
Durable workflow execution platform for building reliable distributed applications:
|
||||||
|
|
||||||
|
- **Durable Execution**: Workflows survive process and infrastructure failures
|
||||||
|
- **Language Support**: SDKs for Go, Java, Python, TypeScript, .NET, PHP
|
||||||
|
- **Visibility**: Query and observe workflow state via Web UI and APIs
|
||||||
|
- **Scalability**: Horizontally scalable architecture
|
||||||
|
- **Multi-tenancy**: Namespace-based isolation for workflows
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Kubernetes cluster (k3s)
|
||||||
|
- PostgreSQL cluster (CloudNativePG)
|
||||||
|
- Keycloak installed and configured
|
||||||
|
- Vault for secrets management
|
||||||
|
- External Secrets Operator (optional, for Vault integration)
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::install
|
||||||
|
```
|
||||||
|
|
||||||
|
You will be prompted for:
|
||||||
|
|
||||||
|
- **Temporal host (FQDN)**: e.g., `temporal.example.com`
|
||||||
|
- **Keycloak host (FQDN)**: e.g., `auth.example.com`
|
||||||
|
- **Enable Prometheus monitoring**: If kube-prometheus-stack is installed
|
||||||
|
|
||||||
|
### What Gets Installed
|
||||||
|
|
||||||
|
- Temporal Server (frontend, history, matching, worker services)
|
||||||
|
- Temporal Web UI with Keycloak OIDC authentication
|
||||||
|
- Temporal Admin Tools for cluster management
|
||||||
|
- PostgreSQL databases (`temporal`, `temporal_visibility`)
|
||||||
|
- Keycloak OAuth client (confidential client)
|
||||||
|
- Vault secrets (if External Secrets Operator is available)
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Environment variables (set in `.env.local` or override):
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
| -------- | ------- | ----------- |
|
||||||
|
| `TEMPORAL_NAMESPACE` | `temporal` | Kubernetes namespace |
|
||||||
|
| `TEMPORAL_CHART_VERSION` | `0.52.0` | Helm chart version |
|
||||||
|
| `TEMPORAL_HOST` | (prompt) | External hostname (FQDN) |
|
||||||
|
| `TEMPORAL_OIDC_CLIENT_ID` | `temporal` | Keycloak client ID |
|
||||||
|
| `KEYCLOAK_HOST` | (prompt) | Keycloak hostname (FQDN) |
|
||||||
|
| `KEYCLOAK_REALM` | `buunstack` | Keycloak realm |
|
||||||
|
| `MONITORING_ENABLED` | (prompt) | Enable Prometheus ServiceMonitor |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```plain
|
||||||
|
External Users
|
||||||
|
|
|
||||||
|
Cloudflare Tunnel (HTTPS)
|
||||||
|
|
|
||||||
|
Traefik Ingress (HTTPS)
|
||||||
|
|
|
||||||
|
Temporal Web UI (HTTP inside cluster)
|
||||||
|
|-- OAuth --> Keycloak (authentication)
|
||||||
|
|
|
||||||
|
Temporal Server
|
||||||
|
|-- Frontend Service (gRPC :7233)
|
||||||
|
| |-- Client connections
|
||||||
|
| |-- Workflow/Activity APIs
|
||||||
|
|
|
||||||
|
|-- History Service
|
||||||
|
| |-- Workflow state management
|
||||||
|
| |-- Event sourcing
|
||||||
|
|
|
||||||
|
|-- Matching Service
|
||||||
|
| |-- Task queue management
|
||||||
|
| |-- Worker polling
|
||||||
|
|
|
||||||
|
|-- Worker Service
|
||||||
|
| |-- System workflows
|
||||||
|
| |-- Archival
|
||||||
|
|
|
||||||
|
PostgreSQL (temporal, temporal_visibility)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Components**:
|
||||||
|
|
||||||
|
- **Frontend**: Entry point for all client requests (gRPC API)
|
||||||
|
- **History**: Maintains workflow execution history and state
|
||||||
|
- **Matching**: Routes tasks to appropriate workers
|
||||||
|
- **Worker**: Executes internal system workflows
|
||||||
|
- **Web UI**: Browser-based workflow monitoring and management
|
||||||
|
- **Admin Tools**: CLI tools for cluster administration
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Access Web UI
|
||||||
|
|
||||||
|
1. Navigate to `https://your-temporal-host/`
|
||||||
|
2. Authenticate via Keycloak SSO
|
||||||
|
3. Select a namespace to view workflows
|
||||||
|
|
||||||
|
### Temporal CLI Setup (Local Development)
|
||||||
|
|
||||||
|
The Temporal gRPC endpoint is only accessible within the cluster network. Use [Telepresence](https://www.telepresence.io/) to connect from your local machine.
|
||||||
|
|
||||||
|
#### Step 1: Connect to the Cluster
|
||||||
|
|
||||||
|
```bash
|
||||||
|
telepresence connect
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 2: Configure Temporal CLI
|
||||||
|
|
||||||
|
Set environment variables (add to `.bashrc`, `.zshrc`, or use direnv):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export TEMPORAL_ADDRESS="temporal-frontend.temporal:7233"
|
||||||
|
export TEMPORAL_NAMESPACE="default"
|
||||||
|
```
|
||||||
|
|
||||||
|
Or create a named environment for multiple clusters:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Configure named environment
|
||||||
|
temporal env set --env buun -k address -v temporal-frontend.temporal:7233
|
||||||
|
temporal env set --env buun -k namespace -v default
|
||||||
|
|
||||||
|
# Use with commands
|
||||||
|
temporal workflow list --env buun
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 3: Verify Connection
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check telepresence status
|
||||||
|
telepresence status
|
||||||
|
|
||||||
|
# Test Temporal connection
|
||||||
|
temporal operator namespace list
|
||||||
|
```
|
||||||
|
|
||||||
|
#### CLI Examples
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List workflows
|
||||||
|
temporal workflow list
|
||||||
|
|
||||||
|
# Describe a workflow
|
||||||
|
temporal workflow describe --workflow-id my-workflow-id
|
||||||
|
|
||||||
|
# Query workflow state
|
||||||
|
temporal workflow query --workflow-id my-workflow-id --type my-query
|
||||||
|
|
||||||
|
# Signal a workflow
|
||||||
|
temporal workflow signal --workflow-id my-workflow-id --name my-signal
|
||||||
|
|
||||||
|
# Terminate a workflow
|
||||||
|
temporal workflow terminate --workflow-id my-workflow-id --reason "manual termination"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create a Temporal Namespace
|
||||||
|
|
||||||
|
Before running workflows, create a namespace:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::create-temporal-namespace default
|
||||||
|
```
|
||||||
|
|
||||||
|
With custom retention period:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::create-temporal-namespace myapp 7d
|
||||||
|
```
|
||||||
|
|
||||||
|
### List Temporal Namespaces
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::list-temporal-namespaces
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cluster Health Check
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::cluster-info
|
||||||
|
```
|
||||||
|
|
||||||
|
### Connect Workers
|
||||||
|
|
||||||
|
Workers connect to the Temporal Frontend service. From within the cluster:
|
||||||
|
|
||||||
|
```text
|
||||||
|
temporal-frontend.temporal:7233
|
||||||
|
```
|
||||||
|
|
||||||
|
Example Python worker:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from temporalio.client import Client
|
||||||
|
from temporalio.worker import Worker
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
client = await Client.connect("temporal-frontend.temporal:7233")
|
||||||
|
|
||||||
|
worker = Worker(
|
||||||
|
client,
|
||||||
|
task_queue="my-task-queue",
|
||||||
|
workflows=[MyWorkflow],
|
||||||
|
activities=[my_activity],
|
||||||
|
)
|
||||||
|
await worker.run()
|
||||||
|
```
|
||||||
|
|
||||||
|
Example Go worker:
|
||||||
|
|
||||||
|
```go
|
||||||
|
import (
|
||||||
|
"go.temporal.io/sdk/client"
|
||||||
|
"go.temporal.io/sdk/worker"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
c, _ := client.Dial(client.Options{
|
||||||
|
HostPort: "temporal-frontend.temporal:7233",
|
||||||
|
})
|
||||||
|
defer c.Close()
|
||||||
|
|
||||||
|
w := worker.New(c, "my-task-queue", worker.Options{})
|
||||||
|
w.RegisterWorkflow(MyWorkflow)
|
||||||
|
w.RegisterActivity(MyActivity)
|
||||||
|
w.Run(worker.InterruptCh())
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Authentication
|
||||||
|
|
||||||
|
### Web UI (OIDC)
|
||||||
|
|
||||||
|
- Users authenticate via Keycloak
|
||||||
|
- Standard OIDC flow with Authorization Code grant
|
||||||
|
- Configured via environment variables in the Web UI deployment
|
||||||
|
|
||||||
|
### gRPC API
|
||||||
|
|
||||||
|
- By default, no authentication is required for gRPC connections within the cluster
|
||||||
|
- For production, configure mTLS or JWT-based authorization
|
||||||
|
|
||||||
|
## Management
|
||||||
|
|
||||||
|
### Upgrade Temporal
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
### Uninstall
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::uninstall
|
||||||
|
```
|
||||||
|
|
||||||
|
This removes:
|
||||||
|
|
||||||
|
- Helm release and all Kubernetes resources
|
||||||
|
- Namespace
|
||||||
|
- Keycloak client
|
||||||
|
|
||||||
|
**Note**: The following resources are NOT deleted:
|
||||||
|
|
||||||
|
- PostgreSQL databases (`temporal`, `temporal_visibility`)
|
||||||
|
- Vault secrets
|
||||||
|
|
||||||
|
### Full Cleanup
|
||||||
|
|
||||||
|
To remove everything including databases and Vault secrets:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::uninstall true
|
||||||
|
```
|
||||||
|
|
||||||
|
Or manually:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just temporal::delete-postgres-user-and-db
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Check Pod Status
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get pods -n temporal
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected pods:
|
||||||
|
|
||||||
|
- `temporal-frontend-*` - Frontend service
|
||||||
|
- `temporal-history-*` - History service
|
||||||
|
- `temporal-matching-*` - Matching service
|
||||||
|
- `temporal-worker-*` - Worker service
|
||||||
|
- `temporal-web-*` - Web UI
|
||||||
|
- `temporal-admintools-*` - Admin tools
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Frontend logs
|
||||||
|
kubectl logs -n temporal deployment/temporal-frontend --tail=100
|
||||||
|
|
||||||
|
# History logs
|
||||||
|
kubectl logs -n temporal deployment/temporal-history --tail=100
|
||||||
|
|
||||||
|
# Web UI logs
|
||||||
|
kubectl logs -n temporal deployment/temporal-web --tail=100
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Connection Issues
|
||||||
|
|
||||||
|
Check PostgreSQL connectivity:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl exec -n temporal deployment/temporal-admintools -- \
|
||||||
|
psql -h postgres-cluster-rw.postgres -U temporal -d temporal -c "SELECT 1"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Schema Issues
|
||||||
|
|
||||||
|
If schema initialization fails, check the schema job:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl logs -n temporal -l app.kubernetes.io/component=schema --all-containers
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service Discovery Issues
|
||||||
|
|
||||||
|
Verify services are running:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get svc -n temporal
|
||||||
|
```
|
||||||
|
|
||||||
|
Test frontend connectivity from admin tools:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl exec -n temporal deployment/temporal-admintools -- \
|
||||||
|
tctl cluster health
|
||||||
|
```
|
||||||
|
|
||||||
|
### Web UI Login Issues
|
||||||
|
|
||||||
|
Verify Keycloak client configuration:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just keycloak::get-client buunstack temporal
|
||||||
|
```
|
||||||
|
|
||||||
|
Check Web UI environment variables:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl get deployment temporal-web -n temporal -o jsonpath='{.spec.template.spec.containers[0].env}' | jq
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration Files
|
||||||
|
|
||||||
|
| File | Description |
|
||||||
|
| ---- | ----------- |
|
||||||
|
| `temporal-values.gomplate.yaml` | Helm values template |
|
||||||
|
| `postgres-external-secret.gomplate.yaml` | PostgreSQL credentials ExternalSecret |
|
||||||
|
| `keycloak-auth-external-secret.gomplate.yaml` | Keycloak OIDC credentials ExternalSecret |
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
- **Pod Security Standards**: Namespace configured with **baseline** enforcement
|
||||||
|
- **Server Security**: Temporal server components run with restricted-compliant security contexts
|
||||||
|
|
||||||
|
### Why Not Restricted?
|
||||||
|
|
||||||
|
The namespace cannot use `restricted` Pod Security Standards due to the Temporal Web UI image (`temporalio/ui`):
|
||||||
|
|
||||||
|
- The image writes configuration files to `./config/docker.yaml` at startup
|
||||||
|
- The container's filesystem is owned by root (UID 0)
|
||||||
|
- When running as non-root user (UID 1000), the container cannot write to these paths
|
||||||
|
- Error: `unable to create open ./config/docker.yaml: permission denied`
|
||||||
|
|
||||||
|
The Temporal server components (frontend, history, matching, worker) **do** meet `restricted` requirements and run with full security hardening. Only the Web UI component requires `baseline`.
|
||||||
|
|
||||||
|
### Server Security Context
|
||||||
|
|
||||||
|
Temporal server components (frontend, history, matching, worker) run with:
|
||||||
|
|
||||||
|
- `runAsNonRoot: true`
|
||||||
|
- `runAsUser: 1000`
|
||||||
|
- `allowPrivilegeEscalation: false`
|
||||||
|
- `seccompProfile.type: RuntimeDefault`
|
||||||
|
- `capabilities.drop: [ALL]`
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [Temporal Documentation](https://docs.temporal.io/)
|
||||||
|
- [Temporal GitHub](https://github.com/temporalio/temporal)
|
||||||
|
- [Temporal Helm Charts](https://github.com/temporalio/helm-charts)
|
||||||
434
temporal/justfile
Normal file
434
temporal/justfile
Normal file
@@ -0,0 +1,434 @@
|
|||||||
|
set fallback := true
|
||||||
|
|
||||||
|
export TEMPORAL_NAMESPACE := env("TEMPORAL_NAMESPACE", "temporal")
|
||||||
|
export TEMPORAL_CHART_VERSION := env("TEMPORAL_CHART_VERSION", "0.72.0")
|
||||||
|
export TEMPORAL_HOST := env("TEMPORAL_HOST", "")
|
||||||
|
export TEMPORAL_OIDC_CLIENT_ID := env("TEMPORAL_OIDC_CLIENT_ID", "temporal")
|
||||||
|
export EXTERNAL_SECRETS_NAMESPACE := env("EXTERNAL_SECRETS_NAMESPACE", "external-secrets")
|
||||||
|
export PROMETHEUS_NAMESPACE := env("PROMETHEUS_NAMESPACE", "monitoring")
|
||||||
|
export MONITORING_ENABLED := env("MONITORING_ENABLED", "")
|
||||||
|
export KEYCLOAK_REALM := env("KEYCLOAK_REALM", "buunstack")
|
||||||
|
export KEYCLOAK_HOST := env("KEYCLOAK_HOST", "")
|
||||||
|
export K8S_VAULT_NAMESPACE := env("K8S_VAULT_NAMESPACE", "vault")
|
||||||
|
|
||||||
|
[private]
|
||||||
|
default:
|
||||||
|
@just --list --unsorted --list-submodules
|
||||||
|
|
||||||
|
# Add Helm repository
|
||||||
|
add-helm-repo:
|
||||||
|
helm repo add temporal https://go.temporal.io/helm-charts
|
||||||
|
helm repo update temporal
|
||||||
|
|
||||||
|
# Remove Helm repository
|
||||||
|
remove-helm-repo:
|
||||||
|
helm repo remove temporal
|
||||||
|
|
||||||
|
# Create Temporal namespace
|
||||||
|
create-namespace:
|
||||||
|
kubectl get namespace ${TEMPORAL_NAMESPACE} &>/dev/null || \
|
||||||
|
kubectl create namespace ${TEMPORAL_NAMESPACE}
|
||||||
|
|
||||||
|
# Delete Temporal namespace
|
||||||
|
delete-namespace:
|
||||||
|
kubectl delete namespace ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
# Create PostgreSQL user and databases for Temporal
|
||||||
|
create-postgres-user-and-db:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if just postgres::user-exists temporal &>/dev/null; then
|
||||||
|
echo "PostgreSQL user 'temporal' already exists"
|
||||||
|
else
|
||||||
|
echo "Creating PostgreSQL user and databases..."
|
||||||
|
PG_PASSWORD=$(just utils::random-password)
|
||||||
|
just postgres::create-user-and-db temporal temporal "${PG_PASSWORD}"
|
||||||
|
just postgres::create-db temporal_visibility
|
||||||
|
just postgres::grant temporal_visibility temporal
|
||||||
|
just vault::put temporal/db username=temporal password="${PG_PASSWORD}"
|
||||||
|
echo "PostgreSQL user and databases created."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Delete PostgreSQL user and databases
|
||||||
|
delete-postgres-user-and-db:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if gum confirm "Delete PostgreSQL user and databases for Temporal?"; then
|
||||||
|
just postgres::delete-db temporal || true
|
||||||
|
just postgres::delete-db temporal_visibility || true
|
||||||
|
just postgres::delete-user temporal || true
|
||||||
|
just vault::delete temporal/db || true
|
||||||
|
echo "PostgreSQL user and databases deleted."
|
||||||
|
else
|
||||||
|
echo "Cancelled."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create Postgres secret
|
||||||
|
create-postgres-secret:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if kubectl get secret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} &>/dev/null; then
|
||||||
|
echo "Postgres auth secret already exists"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
|
||||||
|
echo "External Secrets Operator detected. Creating ExternalSecret..."
|
||||||
|
kubectl delete externalsecret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
gomplate -f postgres-external-secret.gomplate.yaml | kubectl apply -f -
|
||||||
|
echo "Waiting for ExternalSecret to sync..."
|
||||||
|
kubectl wait --for=condition=Ready externalsecret/temporal-postgres-auth \
|
||||||
|
-n ${TEMPORAL_NAMESPACE} --timeout=60s
|
||||||
|
else
|
||||||
|
echo "Creating Kubernetes Secret directly..."
|
||||||
|
PG_USERNAME=$(just vault::get temporal/db username)
|
||||||
|
PG_PASSWORD=$(just vault::get temporal/db password)
|
||||||
|
kubectl create secret generic temporal-postgres-auth \
|
||||||
|
--from-literal=password="${PG_PASSWORD}" \
|
||||||
|
-n ${TEMPORAL_NAMESPACE}
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Delete Postgres secret
|
||||||
|
delete-postgres-secret:
|
||||||
|
kubectl delete externalsecret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl delete secret temporal-postgres-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
# Create Keycloak client for Temporal Web UI
|
||||||
|
create-keycloak-client:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
while [ -z "${TEMPORAL_HOST}" ]; do
|
||||||
|
TEMPORAL_HOST=$(
|
||||||
|
gum input --prompt="Temporal host (FQDN): " --width=100 \
|
||||||
|
--placeholder="e.g., temporal.example.com"
|
||||||
|
)
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Creating Keycloak client for Temporal..."
|
||||||
|
|
||||||
|
just keycloak::delete-client ${KEYCLOAK_REALM} ${TEMPORAL_OIDC_CLIENT_ID} || true
|
||||||
|
|
||||||
|
CLIENT_SECRET=$(just utils::random-password)
|
||||||
|
|
||||||
|
just keycloak::create-client \
|
||||||
|
realm=${KEYCLOAK_REALM} \
|
||||||
|
client_id=${TEMPORAL_OIDC_CLIENT_ID} \
|
||||||
|
redirect_url="https://${TEMPORAL_HOST}/*" \
|
||||||
|
client_secret="${CLIENT_SECRET}"
|
||||||
|
|
||||||
|
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl create secret generic temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
|
||||||
|
--from-literal=client_id="${TEMPORAL_OIDC_CLIENT_ID}" \
|
||||||
|
--from-literal=client_secret="${CLIENT_SECRET}"
|
||||||
|
|
||||||
|
echo "Keycloak client created successfully"
|
||||||
|
echo "Client ID: ${TEMPORAL_OIDC_CLIENT_ID}"
|
||||||
|
echo "Redirect URI: https://${TEMPORAL_HOST}/*"
|
||||||
|
|
||||||
|
# Delete Keycloak client
|
||||||
|
delete-keycloak-client:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Deleting Keycloak client for Temporal..."
|
||||||
|
just keycloak::delete-client ${KEYCLOAK_REALM} ${TEMPORAL_OIDC_CLIENT_ID} || true
|
||||||
|
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
if just vault::exist keycloak/client/temporal &>/dev/null; then
|
||||||
|
just vault::delete keycloak/client/temporal
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create Keycloak auth secret
|
||||||
|
create-keycloak-auth-secret:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} &>/dev/null; then
|
||||||
|
oauth_client_id=$(kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
|
||||||
|
-o jsonpath='{.data.client_id}' | base64 -d)
|
||||||
|
oauth_client_secret=$(kubectl get secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} \
|
||||||
|
-o jsonpath='{.data.client_secret}' | base64 -d)
|
||||||
|
elif helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null && \
|
||||||
|
just vault::get keycloak/client/temporal client_secret &>/dev/null; then
|
||||||
|
oauth_client_id=$(just vault::get keycloak/client/temporal client_id)
|
||||||
|
oauth_client_secret=$(just vault::get keycloak/client/temporal client_secret)
|
||||||
|
else
|
||||||
|
echo "Error: Cannot retrieve OAuth client secret. Please run 'just temporal::create-keycloak-client' first."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then
|
||||||
|
echo "External Secrets Operator detected. Storing secrets in Vault..."
|
||||||
|
|
||||||
|
just vault::put keycloak/client/temporal \
|
||||||
|
client_id="${oauth_client_id}" \
|
||||||
|
client_secret="${oauth_client_secret}"
|
||||||
|
|
||||||
|
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl delete externalsecret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
gomplate -f keycloak-auth-external-secret.gomplate.yaml | kubectl apply -f -
|
||||||
|
|
||||||
|
echo "Waiting for ExternalSecret to sync..."
|
||||||
|
kubectl wait --for=condition=Ready externalsecret/temporal-web-auth \
|
||||||
|
-n ${TEMPORAL_NAMESPACE} --timeout=60s
|
||||||
|
|
||||||
|
echo "ExternalSecret created successfully"
|
||||||
|
else
|
||||||
|
echo "External Secrets Operator not found. Creating Kubernetes Secret directly..."
|
||||||
|
|
||||||
|
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl create secret generic temporal-web-auth -n ${TEMPORAL_NAMESPACE} \
|
||||||
|
--from-literal=TEMPORAL_AUTH_CLIENT_ID="${oauth_client_id}" \
|
||||||
|
--from-literal=TEMPORAL_AUTH_CLIENT_SECRET="${oauth_client_secret}"
|
||||||
|
|
||||||
|
if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then
|
||||||
|
just vault::put keycloak/client/temporal \
|
||||||
|
client_id="${oauth_client_id}" \
|
||||||
|
client_secret="${oauth_client_secret}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Kubernetes Secret created successfully"
|
||||||
|
fi
|
||||||
|
|
||||||
|
kubectl delete secret temporal-oauth-temp -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
# Delete Keycloak auth secret
|
||||||
|
delete-keycloak-auth-secret:
|
||||||
|
kubectl delete externalsecret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
kubectl delete secret temporal-web-auth -n ${TEMPORAL_NAMESPACE} --ignore-not-found
|
||||||
|
|
||||||
|
# Initialize Temporal database schema
|
||||||
|
init-schema:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Initializing Temporal database schema..."
|
||||||
|
|
||||||
|
PG_HOST="postgres-cluster-rw.postgres"
|
||||||
|
PG_PORT="5432"
|
||||||
|
PG_USER=$(just vault::get temporal/db username)
|
||||||
|
PG_PASSWORD=$(just vault::get temporal/db password)
|
||||||
|
|
||||||
|
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "")
|
||||||
|
|
||||||
|
if [ -z "${POD_NAME}" ]; then
|
||||||
|
echo "Admin tools pod not found. Running schema setup job..."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Setting up main database schema..."
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
|
||||||
|
temporal-sql-tool --plugin postgres12 \
|
||||||
|
--endpoint ${PG_HOST} --port ${PG_PORT} \
|
||||||
|
--user ${PG_USER} --password ${PG_PASSWORD} \
|
||||||
|
--database temporal \
|
||||||
|
setup-schema -v 0.0
|
||||||
|
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
|
||||||
|
temporal-sql-tool --plugin postgres12 \
|
||||||
|
--endpoint ${PG_HOST} --port ${PG_PORT} \
|
||||||
|
--user ${PG_USER} --password ${PG_PASSWORD} \
|
||||||
|
--database temporal \
|
||||||
|
update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned
|
||||||
|
|
||||||
|
echo "Setting up visibility database schema..."
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
|
||||||
|
temporal-sql-tool --plugin postgres12 \
|
||||||
|
--endpoint ${PG_HOST} --port ${PG_PORT} \
|
||||||
|
--user ${PG_USER} --password ${PG_PASSWORD} \
|
||||||
|
--database temporal_visibility \
|
||||||
|
setup-schema -v 0.0
|
||||||
|
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
|
||||||
|
temporal-sql-tool --plugin postgres12 \
|
||||||
|
--endpoint ${PG_HOST} --port ${PG_PORT} \
|
||||||
|
--user ${PG_USER} --password ${PG_PASSWORD} \
|
||||||
|
--database temporal_visibility \
|
||||||
|
update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versioned
|
||||||
|
|
||||||
|
echo "Schema initialization complete."
|
||||||
|
|
||||||
|
# Install Temporal
|
||||||
|
install:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
while [ -z "${TEMPORAL_HOST}" ]; do
|
||||||
|
TEMPORAL_HOST=$(gum input --prompt="Temporal host (FQDN): " --width=80 \
|
||||||
|
--placeholder="e.g., temporal.example.com")
|
||||||
|
done
|
||||||
|
|
||||||
|
while [ -z "${KEYCLOAK_HOST}" ]; do
|
||||||
|
KEYCLOAK_HOST=$(gum input --prompt="Keycloak host (FQDN): " --width=80 \
|
||||||
|
--placeholder="e.g., auth.example.com")
|
||||||
|
done
|
||||||
|
|
||||||
|
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
|
||||||
|
if [ -z "${MONITORING_ENABLED}" ]; then
|
||||||
|
if gum confirm "Enable Prometheus monitoring?"; then
|
||||||
|
MONITORING_ENABLED="true"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Installing Temporal..."
|
||||||
|
|
||||||
|
just create-namespace
|
||||||
|
|
||||||
|
kubectl label namespace ${TEMPORAL_NAMESPACE} \
|
||||||
|
pod-security.kubernetes.io/enforce=baseline --overwrite
|
||||||
|
|
||||||
|
if [ "${MONITORING_ENABLED}" = "true" ]; then
|
||||||
|
kubectl label namespace ${TEMPORAL_NAMESPACE} \
|
||||||
|
buun.channel/enable-monitoring=true --overwrite
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Setting up PostgreSQL database..."
|
||||||
|
just create-postgres-user-and-db
|
||||||
|
just create-postgres-secret
|
||||||
|
|
||||||
|
echo "Setting up Keycloak OIDC authentication..."
|
||||||
|
just create-keycloak-client
|
||||||
|
just create-keycloak-auth-secret
|
||||||
|
|
||||||
|
echo "Generating Helm values..."
|
||||||
|
just add-helm-repo
|
||||||
|
gomplate -f temporal-values.gomplate.yaml -o temporal-values.yaml
|
||||||
|
|
||||||
|
echo "Installing Temporal Helm chart..."
|
||||||
|
helm upgrade --cleanup-on-fail --install temporal temporal/temporal \
|
||||||
|
--version ${TEMPORAL_CHART_VERSION} -n ${TEMPORAL_NAMESPACE} --wait \
|
||||||
|
-f temporal-values.yaml --timeout 15m
|
||||||
|
|
||||||
|
echo "Configuring dynamic config for Worker Insights..."
|
||||||
|
kubectl patch configmap temporal-dynamic-config -n ${TEMPORAL_NAMESPACE} --type merge -p '{
|
||||||
|
"data": {
|
||||||
|
"dynamic_config.yaml": "frontend.WorkerHeartbeatsEnabled:\n - value: true\nfrontend.ListWorkersEnabled:\n - value: true\n"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
|
||||||
|
echo "Restarting frontend to apply dynamic config..."
|
||||||
|
kubectl rollout restart deployment temporal-frontend -n ${TEMPORAL_NAMESPACE}
|
||||||
|
kubectl rollout status deployment temporal-frontend -n ${TEMPORAL_NAMESPACE} --timeout=120s
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Temporal installed successfully!"
|
||||||
|
echo "Access Temporal Web UI at: https://${TEMPORAL_HOST}"
|
||||||
|
echo ""
|
||||||
|
echo "OIDC authentication is configured with Keycloak."
|
||||||
|
echo "Users can login with their Keycloak credentials."
|
||||||
|
|
||||||
|
# Upgrade Temporal
|
||||||
|
upgrade:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
while [ -z "${TEMPORAL_HOST}" ]; do
|
||||||
|
TEMPORAL_HOST=$(gum input --prompt="Temporal host (FQDN): " --width=80)
|
||||||
|
done
|
||||||
|
|
||||||
|
while [ -z "${KEYCLOAK_HOST}" ]; do
|
||||||
|
KEYCLOAK_HOST=$(gum input --prompt="Keycloak host (FQDN): " --width=80)
|
||||||
|
done
|
||||||
|
|
||||||
|
if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then
|
||||||
|
if [ -z "${MONITORING_ENABLED}" ]; then
|
||||||
|
if gum confirm "Enable Prometheus monitoring?"; then
|
||||||
|
MONITORING_ENABLED="true"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "${MONITORING_ENABLED}" = "true" ]; then
|
||||||
|
kubectl label namespace ${TEMPORAL_NAMESPACE} \
|
||||||
|
buun.channel/enable-monitoring=true --overwrite
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Upgrading Temporal..."
|
||||||
|
|
||||||
|
gomplate -f temporal-values.gomplate.yaml -o temporal-values.yaml
|
||||||
|
|
||||||
|
helm upgrade temporal temporal/temporal \
|
||||||
|
--version ${TEMPORAL_CHART_VERSION} -n ${TEMPORAL_NAMESPACE} --wait \
|
||||||
|
-f temporal-values.yaml --timeout 15m
|
||||||
|
|
||||||
|
echo "Configuring dynamic config for Worker Insights..."
|
||||||
|
kubectl patch configmap temporal-dynamic-config -n ${TEMPORAL_NAMESPACE} --type merge -p '{
|
||||||
|
"data": {
|
||||||
|
"dynamic_config.yaml": "frontend.WorkerHeartbeatsEnabled:\n - value: true\nfrontend.ListWorkersEnabled:\n - value: true\n"
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
|
||||||
|
echo "Restarting frontend to apply dynamic config..."
|
||||||
|
kubectl rollout restart deployment temporal-frontend -n ${TEMPORAL_NAMESPACE}
|
||||||
|
kubectl rollout status deployment temporal-frontend -n ${TEMPORAL_NAMESPACE} --timeout=120s
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Temporal upgraded successfully!"
|
||||||
|
echo "Access Temporal Web UI at: https://${TEMPORAL_HOST}"
|
||||||
|
|
||||||
|
# Uninstall Temporal (delete-data: true to delete database and Vault secrets)
|
||||||
|
uninstall delete-data='false':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
if ! gum confirm "Uninstall Temporal?"; then
|
||||||
|
echo "Cancelled."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Uninstalling Temporal..."
|
||||||
|
helm uninstall temporal -n ${TEMPORAL_NAMESPACE} --ignore-not-found --wait
|
||||||
|
|
||||||
|
just delete-keycloak-auth-secret || true
|
||||||
|
just delete-keycloak-client || true
|
||||||
|
just delete-postgres-secret
|
||||||
|
just delete-namespace
|
||||||
|
|
||||||
|
if [ "{{ delete-data }}" = "true" ]; then
|
||||||
|
echo "Deleting database and Vault secrets..."
|
||||||
|
just postgres::delete-db temporal || true
|
||||||
|
just postgres::delete-db temporal_visibility || true
|
||||||
|
just postgres::delete-user temporal || true
|
||||||
|
just vault::delete temporal/db || true
|
||||||
|
just vault::delete keycloak/client/temporal || true
|
||||||
|
echo "Temporal uninstalled with all data deleted."
|
||||||
|
else
|
||||||
|
echo "Temporal uninstalled."
|
||||||
|
echo ""
|
||||||
|
echo "Note: The following resources were NOT deleted:"
|
||||||
|
echo " - PostgreSQL user and databases (temporal, temporal_visibility)"
|
||||||
|
echo " - Vault secrets (temporal/db, keycloak/client/temporal)"
|
||||||
|
echo ""
|
||||||
|
echo "To delete all data, run:"
|
||||||
|
echo " just temporal::uninstall true"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create a Temporal namespace (workflow namespace, not Kubernetes)
|
||||||
|
create-temporal-namespace name='' retention='3d':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
name="{{ name }}"
|
||||||
|
retention="{{ retention }}"
|
||||||
|
while [ -z "${name}" ]; do
|
||||||
|
name=$(gum input --prompt="Namespace name: " --width=80 --placeholder="e.g., default")
|
||||||
|
done
|
||||||
|
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}')
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- \
|
||||||
|
tctl --namespace "${name}" namespace register --retention "${retention}"
|
||||||
|
echo "Namespace '${name}' created with retention ${retention}."
|
||||||
|
|
||||||
|
# List Temporal namespaces
|
||||||
|
list-temporal-namespaces:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}')
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- tctl namespace list
|
||||||
|
|
||||||
|
# Get Temporal cluster info
|
||||||
|
cluster-info:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
POD_NAME=$(kubectl get pods -n ${TEMPORAL_NAMESPACE} -l app.kubernetes.io/name=temporal-admintools \
|
||||||
|
-o jsonpath='{.items[0].metadata.name}')
|
||||||
|
kubectl exec -n ${TEMPORAL_NAMESPACE} ${POD_NAME} -- tctl cluster health
|
||||||
22
temporal/keycloak-auth-external-secret.gomplate.yaml
Normal file
22
temporal/keycloak-auth-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: temporal-web-auth
|
||||||
|
namespace: {{ .Env.TEMPORAL_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: temporal-web-auth
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
- secretKey: TEMPORAL_AUTH_CLIENT_ID
|
||||||
|
remoteRef:
|
||||||
|
key: keycloak/client/temporal
|
||||||
|
property: client_id
|
||||||
|
- secretKey: TEMPORAL_AUTH_CLIENT_SECRET
|
||||||
|
remoteRef:
|
||||||
|
key: keycloak/client/temporal
|
||||||
|
property: client_secret
|
||||||
18
temporal/postgres-external-secret.gomplate.yaml
Normal file
18
temporal/postgres-external-secret.gomplate.yaml
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
apiVersion: external-secrets.io/v1
|
||||||
|
kind: ExternalSecret
|
||||||
|
metadata:
|
||||||
|
name: temporal-postgres-auth
|
||||||
|
namespace: {{ .Env.TEMPORAL_NAMESPACE }}
|
||||||
|
spec:
|
||||||
|
refreshInterval: 1h
|
||||||
|
secretStoreRef:
|
||||||
|
name: vault-secret-store
|
||||||
|
kind: ClusterSecretStore
|
||||||
|
target:
|
||||||
|
name: temporal-postgres-auth
|
||||||
|
creationPolicy: Owner
|
||||||
|
data:
|
||||||
|
- secretKey: password
|
||||||
|
remoteRef:
|
||||||
|
key: temporal/db
|
||||||
|
property: password
|
||||||
204
temporal/temporal-values.gomplate.yaml
Normal file
204
temporal/temporal-values.gomplate.yaml
Normal file
@@ -0,0 +1,204 @@
|
|||||||
|
server:
|
||||||
|
replicaCount: 1
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsGroup: 1000
|
||||||
|
fsGroup: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
config:
|
||||||
|
persistence:
|
||||||
|
default:
|
||||||
|
driver: "sql"
|
||||||
|
sql:
|
||||||
|
driver: "postgres12"
|
||||||
|
host: "postgres-cluster-rw.postgres"
|
||||||
|
port: 5432
|
||||||
|
database: temporal
|
||||||
|
user: temporal
|
||||||
|
existingSecret: temporal-postgres-auth
|
||||||
|
maxConns: 20
|
||||||
|
maxIdleConns: 20
|
||||||
|
maxConnLifetime: "1h"
|
||||||
|
|
||||||
|
visibility:
|
||||||
|
driver: "sql"
|
||||||
|
sql:
|
||||||
|
driver: "postgres12"
|
||||||
|
host: "postgres-cluster-rw.postgres"
|
||||||
|
port: 5432
|
||||||
|
database: temporal_visibility
|
||||||
|
user: temporal
|
||||||
|
existingSecret: temporal-postgres-auth
|
||||||
|
maxConns: 20
|
||||||
|
maxIdleConns: 20
|
||||||
|
maxConnLifetime: "1h"
|
||||||
|
|
||||||
|
{{- if .Env.MONITORING_ENABLED }}
|
||||||
|
metrics:
|
||||||
|
serviceMonitor:
|
||||||
|
enabled: true
|
||||||
|
additionalLabels:
|
||||||
|
release: kube-prometheus-stack
|
||||||
|
{{- end }}
|
||||||
|
|
||||||
|
frontend:
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
history:
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
matching:
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
worker:
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
|
||||||
|
admintools:
|
||||||
|
enabled: true
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsGroup: 1000
|
||||||
|
fsGroup: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 200m
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
|
web:
|
||||||
|
enabled: true
|
||||||
|
replicaCount: 1
|
||||||
|
service:
|
||||||
|
type: ClusterIP
|
||||||
|
port: 8080
|
||||||
|
ingress:
|
||||||
|
enabled: true
|
||||||
|
className: traefik
|
||||||
|
annotations:
|
||||||
|
traefik.ingress.kubernetes.io/router.entrypoints: websecure
|
||||||
|
traefik.ingress.kubernetes.io/router.tls: "true"
|
||||||
|
hosts:
|
||||||
|
- {{ .Env.TEMPORAL_HOST }}
|
||||||
|
tls:
|
||||||
|
- secretName: temporal-web-tls
|
||||||
|
hosts:
|
||||||
|
- {{ .Env.TEMPORAL_HOST }}
|
||||||
|
additionalEnv:
|
||||||
|
- name: TEMPORAL_AUTH_ENABLED
|
||||||
|
value: "true"
|
||||||
|
- name: TEMPORAL_AUTH_PROVIDER_URL
|
||||||
|
value: "https://{{ .Env.KEYCLOAK_HOST }}/realms/{{ .Env.KEYCLOAK_REALM }}"
|
||||||
|
- name: TEMPORAL_AUTH_SCOPES
|
||||||
|
value: "openid,profile,email"
|
||||||
|
- name: TEMPORAL_AUTH_CALLBACK_URL
|
||||||
|
value: "https://{{ .Env.TEMPORAL_HOST }}/auth/sso/callback"
|
||||||
|
additionalEnvSecretName: temporal-web-auth
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 200m
|
||||||
|
memory: 256Mi
|
||||||
|
|
||||||
|
cassandra:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
mysql:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
postgresql:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
elasticsearch:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
prometheus:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
grafana:
|
||||||
|
enabled: false
|
||||||
|
|
||||||
|
schema:
|
||||||
|
createDatabase:
|
||||||
|
enabled: false
|
||||||
|
setup:
|
||||||
|
enabled: true
|
||||||
|
backoffLimit: 100
|
||||||
|
update:
|
||||||
|
enabled: true
|
||||||
|
backoffLimit: 100
|
||||||
|
securityContext:
|
||||||
|
runAsUser: 1000
|
||||||
|
runAsGroup: 1000
|
||||||
|
fsGroup: 1000
|
||||||
|
runAsNonRoot: true
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
containerSecurityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: false
|
||||||
Reference in New Issue
Block a user