baschno/buun-stack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(jupyterhub): admin vault token renewal

Browse Source
This commit is contained in:
Masaki Yatsu
2025-09-08 18:43:08 +09:00
parent c82c6aa22b
commit 4df776c181
6 changed files with 322 additions and 127 deletions
Download Patch File Download Diff File Expand all files Collapse all files

130
docs/jupyterhub.md
View File

@@ -126,7 +126,7 @@ Vault integration enables secure secrets management directly from Jupyter notebo
- **ExternalSecret** to fetch the admin token from Vault - **ExternalSecret** to fetch the admin token from Vault
- **Renewable tokens** with unlimited Max TTL to avoid 30-day system limitations - **Renewable tokens** with unlimited Max TTL to avoid 30-day system limitations
- **Token renewal script** that automatically renews tokens every 12 hours - **Token renewal script** that automatically renews tokens at TTL/2 intervals (minimum 30 seconds)
- **User-specific tokens** created during notebook spawn with isolated access - **User-specific tokens** created during notebook spawn with isolated access
### Architecture ### Architecture
@@ -214,7 +214,7 @@ secrets.delete('api-keys', field='github') # Delete only github field
### Security Features ### Security Features
- **User isolation**: Each user receives an orphan token with access only to their namespace - **User isolation**: Each user receives an orphan token with access only to their namespace
- **Automatic renewal**: Token renewal script renews admin token every 12 hours - **Automatic renewal**: Token renewal script renews admin token at TTL/2 intervals (minimum 30 seconds)
- **ExternalSecret integration**: Admin token fetched securely from Vault - **ExternalSecret integration**: Admin token fetched securely from Vault
- **Orphan tokens**: User tokens are orphan tokens, not limited by parent policy restrictions - **Orphan tokens**: User tokens are orphan tokens, not limited by parent policy restrictions
- **Audit trail**: All secret access is logged in Vault - **Audit trail**: All secret access is logged in Vault
@@ -228,7 +228,7 @@ The admin token is managed through:
1. **Creation**: `just jupyterhub::create-jupyterhub-vault-token` creates renewable token 1. **Creation**: `just jupyterhub::create-jupyterhub-vault-token` creates renewable token
2. **Storage**: Stored in Vault at `secret/jupyterhub/vault-token` 2. **Storage**: Stored in Vault at `secret/jupyterhub/vault-token`
3. **Retrieval**: ExternalSecret fetches and mounts as Kubernetes Secret 3. **Retrieval**: ExternalSecret fetches and mounts as Kubernetes Secret
4. **Renewal**: `vault-token-renewer.sh` script renews every 12 hours 4. **Renewal**: `vault-token-renewer.sh` script renews at TTL/2 intervals
#### User Tokens #### User Tokens
@@ -239,6 +239,112 @@ User tokens are created dynamically:
3. **Creates orphan token** with user policy (requires `sudo` permission) 3. **Creates orphan token** with user policy (requires `sudo` permission)
4. **Sets environment variable** `NOTEBOOK_VAULT_TOKEN` in notebook container 4. **Sets environment variable** `NOTEBOOK_VAULT_TOKEN` in notebook container
## Token Renewal Implementation
### Admin Token Renewal
The admin token renewal is handled by a sidecar container (`vault-agent`) running alongside the JupyterHub hub:
**Implementation Details:**
1. **Renewal Script**: `/vault/config/vault-token-renewer.sh`
- Runs in the `vault-agent` sidecar container
- Uses Vault 1.17.5 image with HashiCorp Vault CLI
2. **Environment-Based TTL Configuration**:
```bash
# Reads TTL from environment variable (set in .env.local)
TTL_RAW="${JUPYTERHUB_VAULT_TOKEN_TTL}" # e.g., "5m", "24h"
# Converts to seconds and calculates renewal interval
RENEWAL_INTERVAL=$((TTL_SECONDS / 2)) # TTL/2 with minimum 30s
```
3. **Token Source**: ExternalSecret → Kubernetes Secret → mounted file
```bash
# Token retrieved from ExternalSecret-managed mount
ADMIN_TOKEN=$(cat /vault/admin-token/token)
```
4. **Renewal Loop**:
```bash
while true; do
vault token renew >/dev/null 2>&1
sleep $RENEWAL_INTERVAL
done
```
5. **Error Handling**: If renewal fails, re-retrieves token from ExternalSecret mount
**Key Files:**
- `vault-token-renewer.sh`: Main renewal script
- `jupyterhub-vault-token-external-secret.gomplate.yaml`: ExternalSecret configuration
- `vault-agent-config` ConfigMap: Contains the renewal script
### User Token Renewal
User token renewal is handled within the notebook environment by the `buunstack` Python package:
**Implementation Details:**
1. **Token Source**: Environment variable set by pre-spawn hook
```python
# In pre_spawn_hook.gomplate.py
spawner.environment["NOTEBOOK_VAULT_TOKEN"] = user_vault_token
```
2. **Automatic Renewal**: Built into `SecretStore` class operations
```python
# In buunstack/secrets.py
def _ensure_authenticated(self):
token_info = self.client.auth.token.lookup_self()
ttl = token_info.get("data", {}).get("ttl", 0)
renewable = token_info.get("data", {}).get("renewable", False)
# Renew if TTL < 10 minutes and renewable
if renewable and ttl > 0 and ttl < 600:
self.client.auth.token.renew_self()
```
3. **Renewal Trigger**: Every `SecretStore` operation (get, put, delete, list)
- Checks token validity before operation
- Automatically renews if TTL < 10 minutes
- Transparent to user code
4. **Token Configuration** (set during creation):
- **TTL**: `NOTEBOOK_VAULT_TOKEN_TTL` (default: 24h)
- **Max TTL**: `NOTEBOOK_VAULT_TOKEN_MAX_TTL` (default: 168h = 7 days)
- **Policy**: User-specific `jupyter-user-{username}`
- **Type**: Orphan token (independent of parent token lifecycle)
5. **Expiry Handling**: When token reaches Max TTL:
- Cannot be renewed further
- User must restart notebook server (triggers new token creation)
- Prevented by `JUPYTERHUB_CULL_MAX_AGE` setting (6 days < 7 day Max TTL)
**Key Files:**
- `pre_spawn_hook.gomplate.py`: User token creation logic
- `buunstack/secrets.py`: Token renewal implementation
- `user_policy.hcl`: User token permissions template
### Token Lifecycle Summary
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Admin Token │ │ User Token │ │ Pod Lifecycle │
│ │ │ │ │ │
│ Created: Manual │ │ Created: Spawn │ │ Max Age: 6 days │
│ TTL: 5m-24h │ │ TTL: 24h │ │ Auto-restart │
│ Max TTL: ∞ │ │ Max TTL: 7 days │ │ before expiry │
│ Renewal: Auto │ │ Renewal: Auto │ │ │
│ Interval: TTL/2 │ │ Trigger: Usage │ │ │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│ │ │
▼ ▼ ▼
vault-agent buunstack.py cull.maxAge
sidecar SecretStore pod restart
```
## Storage Options ## Storage Options
### Default Storage ### Default Storage
@@ -288,10 +394,14 @@ JUPYTER_PYTHON_KERNEL_TAG=python-3.12-28
IMAGE_REGISTRY=localhost:30500 IMAGE_REGISTRY=localhost:30500
# Vault token TTL settings # Vault token TTL settings
JUPYTERHUB_VAULT_TOKEN_TTL=24h # Admin token: renewed every 12h JUPYTERHUB_VAULT_TOKEN_TTL=24h # Admin token: renewed at TTL/2 intervals
NOTEBOOK_VAULT_TOKEN_TTL=24h # User token: 1 day NOTEBOOK_VAULT_TOKEN_TTL=24h # User token: 1 day
NOTEBOOK_VAULT_TOKEN_MAX_TTL=168h # User token: 7 days max NOTEBOOK_VAULT_TOKEN_MAX_TTL=168h # User token: 7 days max
# Server pod lifecycle settings
JUPYTERHUB_CULL_MAX_AGE=518400 # Max pod age in seconds (6 days = 518400s)
# MUST be < NOTEBOOK_VAULT_TOKEN_MAX_TTL to prevent token expiry
# Logging # Logging
JUPYTER_BUUNSTACK_LOG_LEVEL=warning # Options: debug, info, warning, error JUPYTER_BUUNSTACK_LOG_LEVEL=warning # Options: debug, info, warning, error
``` ```
@@ -413,7 +523,7 @@ The system uses a three-tier token approach:
1. **Renewable Admin Token**: 1. **Renewable Admin Token**:
- Created with `explicit-max-ttl=0` (unlimited Max TTL) - Created with `explicit-max-ttl=0` (unlimited Max TTL)
- Renewed automatically every 12 hours - Renewed automatically at TTL/2 intervals (minimum 30 seconds)
- Stored in Vault and fetched via ExternalSecret - Stored in Vault and fetched via ExternalSecret
2. **Orphan User Tokens**: 2. **Orphan User Tokens**:
@@ -438,7 +548,7 @@ The system uses a three-tier token approach:
- **Image Size**: Buun-stack images are ~13GB, plan storage accordingly - **Image Size**: Buun-stack images are ~13GB, plan storage accordingly
- **Pull Time**: Initial pulls take 5-15 minutes depending on network - **Pull Time**: Initial pulls take 5-15 minutes depending on network
- **Resource Usage**: Data science workloads require adequate CPU/memory - **Resource Usage**: Data science workloads require adequate CPU/memory
- **Token Renewal**: Minimal overhead (renewal every 12 hours) - **Token Renewal**: Minimal overhead (renewal at TTL/2 intervals)
For production deployments, consider: For production deployments, consider:
@@ -451,8 +561,10 @@ For production deployments, consider:
1. **Annual Token Recreation**: While tokens have unlimited Max TTL, best practice suggests recreating them annually 1. **Annual Token Recreation**: While tokens have unlimited Max TTL, best practice suggests recreating them annually
2. **Cull Settings**: Server idle timeout is set to 2 hours by default. Adjust `cull.timeout` and `cull.every` in the Helm values for different requirements 2. **Token Expiry and Pod Lifecycle**: User tokens have a maximum TTL of 7 days (`NOTEBOOK_VAULT_TOKEN_MAX_TTL=168h`). To prevent token expiry in long-running server pods, `JUPYTERHUB_CULL_MAX_AGE` is set to 6 days (518400s) by default. This ensures pods are restarted with fresh tokens before expiry.
3. **NFS Storage**: When using NFS storage, ensure proper permissions are set on the NFS server. The default `JUPYTER_FSGID` is 100 3. **Cull Settings**: Server idle timeout is set to 2 hours by default. Adjust `cull.timeout` and `cull.every` in the Helm values for different requirements
4. **ExternalSecret Dependency**: Requires External Secrets Operator to be installed and configured 4. **NFS Storage**: When using NFS storage, ensure proper permissions are set on the NFS server. The default `JUPYTER_FSGID` is 100
5. **ExternalSecret Dependency**: Requires External Secrets Operator to be installed and configured

1
jupyterhub/.gitignore vendored
View File

@@ -1,3 +1,4 @@
jupyterhub-values.yaml jupyterhub-values.yaml
pre_spawn_hook.py
vault-agent-config.hcl vault-agent-config.hcl
/notebooks/ /notebooks/

126
jupyterhub/jupyterhub-values.gomplate.yaml
View File

@@ -23,6 +23,11 @@ hub:
mode: 0644 mode: 0644
stringData: | stringData: |
{{ .Env.USER_POLICY_HCL | strings.Indent 8 }} {{ .Env.USER_POLICY_HCL | strings.Indent 8 }}
pre_spawn_hook.py:
mountPath: /srv/jupyterhub/pre_spawn_hook.py
mode: 0644
stringData: |
{{ file.Read "pre_spawn_hook.py" | strings.Indent 8 }}
# Override the default command to run our startup script first # Override the default command to run our startup script first
command: command:
@@ -56,112 +61,10 @@ hub:
- email - email
extraConfig: extraConfig:
pre-spawn-hook: | load-pre-spawn-hook: |
# Set environment variables for spawned containers # Load pre_spawn_hook from external file
import hvac with open('/srv/jupyterhub/pre_spawn_hook.py', 'r') as f:
exec(f.read())
{{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }}
def get_vault_token():
"""Read Vault token from file"""
import os
token_file = '/vault/secrets/vault-token'
try:
with open(token_file, 'r') as f:
token = f.read().strip()
if token:
return token
except FileNotFoundError:
print(f"Token file not found: {token_file}")
except Exception as e:
print(f"Error reading token file {token_file}: {e}")
return None
{{- end }}
async def pre_spawn_hook(spawner):
"""Set essential environment variables for spawned containers"""
# PostgreSQL configuration
spawner.environment["POSTGRES_HOST"] = "postgres-cluster-rw.postgres"
spawner.environment["POSTGRES_PORT"] = "5432"
# JupyterHub API configuration
spawner.environment["JUPYTERHUB_API_URL"] = "http://hub:8081/hub/api"
# Logging configuration
spawner.environment["BUUNSTACK_LOG_LEVEL"] = "{{ .Env.JUPYTER_BUUNSTACK_LOG_LEVEL }}"
{{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }}
# Create user-specific Vault token directly
try:
username = spawner.user.name
# Step 1: Initialize admin Vault client with file-based token
import os
vault_addr = os.environ.get("VAULT_ADDR", "{{ .Env.VAULT_ADDR }}")
vault_token = get_vault_token()
spawner.log.info(f"pre_spawn_hook starting for {username}")
spawner.log.info(f"Vault address: {vault_addr}")
spawner.log.info(f"Vault token source: {'file' if os.path.exists('/vault/secrets/vault-token') else 'env'}")
spawner.log.info(f"Vault token present: {bool(vault_token)}, length: {len(vault_token) if vault_token else 0}")
if not vault_token:
raise Exception("No Vault token available from file or environment")
vault_client = hvac.Client(url=vault_addr, verify=False)
vault_client.token = vault_token
if not vault_client.is_authenticated():
raise Exception("Admin token is not authenticated")
# Step 2: Create user-specific policy
user_policy_name = "jupyter-user-{}".format(username)
# Read policy template from file
import os
policy_template_path = "/srv/jupyterhub/user_policy.hcl"
with open(policy_template_path, 'r') as f:
policy_template = f.read()
# Replace {username} placeholder with actual username
user_policy = policy_template.replace("{username}", username)
# Write user-specific policy
try:
vault_client.sys.create_or_update_policy(user_policy_name, user_policy)
spawner.log.info("✅ Created policy: {}".format(user_policy_name))
except Exception as policy_e:
spawner.log.warning("Policy creation failed (may already exist): {}".format(policy_e))
# Step 3: Create user-specific token
# Get TTL settings from environment variables
user_token_ttl = os.environ.get("NOTEBOOK_VAULT_TOKEN_TTL", "24h")
user_token_max_ttl = os.environ.get("NOTEBOOK_VAULT_TOKEN_MAX_TTL", "168h")
token_response = vault_client.auth.token.create_orphan(
policies=[user_policy_name],
ttl=user_token_ttl,
renewable=True,
display_name="notebook-{}".format(username),
explicit_max_ttl=user_token_max_ttl
)
user_vault_token = token_response["auth"]["client_token"]
lease_duration = token_response["auth"].get("lease_duration", 3600)
# Set user-specific Vault token as environment variable
spawner.environment["NOTEBOOK_VAULT_TOKEN"] = user_vault_token
spawner.log.info("✅ User-specific Vault token created for {} (TTL: {}s, renewable, max TTL: {})".format(username, lease_duration, user_token_max_ttl))
except Exception as e:
spawner.log.error("Failed to create user-specific Vault token for {}: {}".format(spawner.user.name, e))
import traceback
spawner.log.error("Full traceback: {}".format(traceback.format_exc()))
{{- end }}
c.KubeSpawner.pre_spawn_hook = pre_spawn_hook
{{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }} {{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }}
# Vault Agent sidecar configuration # Vault Agent sidecar configuration
@@ -186,7 +89,7 @@ hub:
extraContainers: extraContainers:
- name: vault-agent - name: vault-agent
image: hashicorp/vault:1.15.2 image: hashicorp/vault:1.17.5
securityContext: securityContext:
runAsUser: 100 runAsUser: 100
runAsGroup: 101 runAsGroup: 101
@@ -205,6 +108,8 @@ hub:
env: env:
- name: VAULT_ADDR - name: VAULT_ADDR
value: {{ .Env.VAULT_ADDR | quote }} value: {{ .Env.VAULT_ADDR | quote }}
- name: JUPYTERHUB_VAULT_TOKEN_TTL
value: {{ .Env.JUPYTERHUB_VAULT_TOKEN_TTL | quote }}
volumeMounts: volumeMounts:
- name: vault-secrets - name: vault-secrets
mountPath: /vault/secrets mountPath: /vault/secrets
@@ -378,7 +283,12 @@ cull:
# timeout: 300 # 5 minutes idle timeout (for testing) │ │ # timeout: 300 # 5 minutes idle timeout (for testing) │ │
# every: 60 # Check every 1 minute (for testing) │ │ # every: 60 # Check every 1 minute (for testing) │ │
# maxAge: 86400 # Maximum age of a server pod (1 day) # Maximum age of a server pod before forced restart
# IMPORTANT: This must be less than NOTEBOOK_VAULT_TOKEN_MAX_TTL to prevent token expiry
# - NOTEBOOK_VAULT_TOKEN_MAX_TTL: {{ .Env.NOTEBOOK_VAULT_TOKEN_MAX_TTL }} (7 days = 604800s)
# - JUPYTERHUB_CULL_MAX_AGE: {{ .Env.JUPYTERHUB_CULL_MAX_AGE }}s (6 days = 518400s)
# Pod restart creates new user token, preventing 7-day token expiry
maxAge: {{ .Env.JUPYTERHUB_CULL_MAX_AGE }}
adminUsers: true # Also cull admin users' server pods adminUsers: true # Also cull admin users' server pods
users: false # Don't delete user accounts, only stop server pods users: false # Don't delete user accounts, only stop server pods

15
jupyterhub/justfile
View File

@@ -22,6 +22,7 @@ export JUPYTER_PROFILE_BUUN_STACK_CUDA_ENABLED := env("JUPYTER_PROFILE_BUUN_STAC
export JUPYTERHUB_VAULT_TOKEN_TTL := env("JUPYTERHUB_VAULT_TOKEN_TTL", "24h") export JUPYTERHUB_VAULT_TOKEN_TTL := env("JUPYTERHUB_VAULT_TOKEN_TTL", "24h")
export NOTEBOOK_VAULT_TOKEN_TTL := env("NOTEBOOK_VAULT_TOKEN_TTL", "24h") export NOTEBOOK_VAULT_TOKEN_TTL := env("NOTEBOOK_VAULT_TOKEN_TTL", "24h")
export NOTEBOOK_VAULT_TOKEN_MAX_TTL := env("NOTEBOOK_VAULT_TOKEN_MAX_TTL", "168h") export NOTEBOOK_VAULT_TOKEN_MAX_TTL := env("NOTEBOOK_VAULT_TOKEN_MAX_TTL", "168h")
export JUPYTERHUB_CULL_MAX_AGE := env("JUPYTERHUB_CULL_MAX_AGE", "518400")
export VAULT_AGENT_LOG_LEVEL := env("VAULT_AGENT_LOG_LEVEL", "info") export VAULT_AGENT_LOG_LEVEL := env("VAULT_AGENT_LOG_LEVEL", "info")
export JUPYTER_BUUNSTACK_LOG_LEVEL := env("JUPYTER_BUUNSTACK_LOG_LEVEL", "warning") export JUPYTER_BUUNSTACK_LOG_LEVEL := env("JUPYTER_BUUNSTACK_LOG_LEVEL", "warning")
export IMAGE_REGISTRY := env("IMAGE_REGISTRY", "localhost:30500") export IMAGE_REGISTRY := env("IMAGE_REGISTRY", "localhost:30500")
@@ -146,6 +147,10 @@ install root_token='':
export USER_POLICY_HCL="" export USER_POLICY_HCL=""
fi fi
# Generate pre_spawn_hook.py
echo "Generating pre_spawn_hook.py..."
gomplate -f pre_spawn_hook.gomplate.py -o pre_spawn_hook.py
# https://z2jh.jupyter.org/en/stable/ # https://z2jh.jupyter.org/en/stable/
gomplate -f jupyterhub-values.gomplate.yaml -o jupyterhub-values.yaml gomplate -f jupyterhub-values.gomplate.yaml -o jupyterhub-values.yaml
@@ -261,7 +266,7 @@ setup-vault-integration root_token='':
echo " User Token TTL: ${NOTEBOOK_VAULT_TOKEN_TTL}" echo " User Token TTL: ${NOTEBOOK_VAULT_TOKEN_TTL}"
echo " User Token Max TTL: ${NOTEBOOK_VAULT_TOKEN_MAX_TTL}" echo " User Token Max TTL: ${NOTEBOOK_VAULT_TOKEN_MAX_TTL}"
echo " Vault Agent Log Level: ${VAULT_AGENT_LOG_LEVEL}" echo " Vault Agent Log Level: ${VAULT_AGENT_LOG_LEVEL}"
echo " Auto-renewal: Every $(( $(echo ${JUPYTERHUB_VAULT_TOKEN_TTL} | sed 's/m/*60/g; s/h/*3600/g; s/s//g' | bc) / 2 ))s (TTL/2)" echo " Auto-renewal: Every TTL/2 (minimum 30s) based on actual token TTL"
echo "" echo ""
echo "Users can now access Vault from notebooks using:" echo "Users can now access Vault from notebooks using:"
echo " from buunstack import SecretStore" echo " from buunstack import SecretStore"
@@ -295,10 +300,10 @@ create-jupyterhub-vault-token root_token='':
# Create admin vault token with unlimited max TTL # Create admin vault token with unlimited max TTL
echo "" echo ""
echo "Creating admin token (TTL: 24h, Max TTL: unlimited)..." echo "Creating admin token (TTL: ${JUPYTERHUB_VAULT_TOKEN_TTL}, Max TTL: unlimited)..."
TOKEN_RESPONSE=$(vault token create \ TOKEN_RESPONSE=$(vault token create \
-policy=jupyterhub-admin \ -policy=jupyterhub-admin \
-ttl=24h \ -ttl=${JUPYTERHUB_VAULT_TOKEN_TTL} \
-explicit-max-ttl=0 \ -explicit-max-ttl=0 \
-display-name="jupyterhub-admin" \ -display-name="jupyterhub-admin" \
-renewable=true \ -renewable=true \
@@ -320,9 +325,9 @@ create-jupyterhub-vault-token root_token='':
echo "✅ Admin token created and stored successfully!" echo "✅ Admin token created and stored successfully!"
echo "" echo ""
echo "Token behavior:" echo "Token behavior:"
echo " - TTL: 24 hours (will expire in 24h without renewal)" echo " - TTL: ${JUPYTERHUB_VAULT_TOKEN_TTL} (will expire without renewal)"
echo " - Max TTL: Unlimited (can be renewed forever)" echo " - Max TTL: Unlimited (can be renewed forever)"
echo " - Vault Agent will renew every 12 hours" echo " - Vault Agent will renew at TTL/2 intervals (minimum 30s)"
echo " - No more 30-day limitation!" echo " - No more 30-day limitation!"
echo "" echo ""
echo "Token stored at: secret/jupyterhub/vault-token" echo "Token stored at: secret/jupyterhub/vault-token"

105
jupyterhub/pre_spawn_hook.gomplate.py Normal file
View File

@@ -0,0 +1,105 @@
# JupyterHub pre_spawn_hook
# Sets up user environment and creates user-specific Vault tokens
import hvac
import os
{{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }}
def get_vault_token():
"""Read Vault token from file"""
token_file = '/vault/secrets/vault-token'
try:
with open(token_file, 'r') as f:
token = f.read().strip()
if token:
return token
except FileNotFoundError:
print(f"Token file not found: {token_file}")
except Exception as e:
print(f"Error reading token file {token_file}: {e}")
return None
{{- end }}
async def pre_spawn_hook(spawner):
"""Set essential environment variables for spawned containers"""
# PostgreSQL configuration
spawner.environment["POSTGRES_HOST"] = "postgres-cluster-rw.postgres"
spawner.environment["POSTGRES_PORT"] = "5432"
# JupyterHub API configuration
spawner.environment["JUPYTERHUB_API_URL"] = "http://hub:8081/hub/api"
# Logging configuration
spawner.environment["BUUNSTACK_LOG_LEVEL"] = "{{ .Env.JUPYTER_BUUNSTACK_LOG_LEVEL }}"
{{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }}
# Create user-specific Vault token directly
try:
username = spawner.user.name
# Step 1: Initialize admin Vault client with file-based token
vault_addr = os.environ.get("VAULT_ADDR", "{{ .Env.VAULT_ADDR }}")
vault_token = get_vault_token()
spawner.log.info(f"pre_spawn_hook starting for {username}")
spawner.log.info(f"Vault address: {vault_addr}")
spawner.log.info(f"Vault token source: {'file' if os.path.exists('/vault/secrets/vault-token') else 'env'}")
spawner.log.info(f"Vault token present: {bool(vault_token)}, length: {len(vault_token) if vault_token else 0}")
if not vault_token:
raise Exception("No Vault token available from file or environment")
vault_client = hvac.Client(url=vault_addr, verify=False)
vault_client.token = vault_token
if not vault_client.is_authenticated():
raise Exception("Admin token is not authenticated")
# Step 2: Create user-specific policy
user_policy_name = "jupyter-user-{}".format(username)
# Read policy template from file
policy_template_path = "/srv/jupyterhub/user_policy.hcl"
with open(policy_template_path, 'r') as f:
policy_template = f.read()
# Replace {username} placeholder with actual username
user_policy = policy_template.replace("{username}", username)
# Write user-specific policy
try:
vault_client.sys.create_or_update_policy(user_policy_name, user_policy)
spawner.log.info("✅ Created policy: {}".format(user_policy_name))
except Exception as policy_e:
spawner.log.warning("Policy creation failed (may already exist): {}".format(policy_e))
# Step 3: Create user-specific token
# Get TTL settings from environment variables
user_token_ttl = os.environ.get("NOTEBOOK_VAULT_TOKEN_TTL", "24h")
user_token_max_ttl = os.environ.get("NOTEBOOK_VAULT_TOKEN_MAX_TTL", "168h")
token_response = vault_client.auth.token.create_orphan(
policies=[user_policy_name],
ttl=user_token_ttl,
renewable=True,
display_name="notebook-{}".format(username),
explicit_max_ttl=user_token_max_ttl
)
user_vault_token = token_response["auth"]["client_token"]
lease_duration = token_response["auth"].get("lease_duration", 3600)
# Set user-specific Vault token as environment variable
spawner.environment["NOTEBOOK_VAULT_TOKEN"] = user_vault_token
spawner.log.info("✅ User-specific Vault token created for {} (TTL: {}s, renewable, max TTL: {})".format(username, lease_duration, user_token_max_ttl))
except Exception as e:
spawner.log.error("Failed to create user-specific Vault token for {}: {}".format(spawner.user.name, e))
import traceback
spawner.log.error("Full traceback: {}".format(traceback.format_exc()))
{{- end }}
# Set the hook
c.KubeSpawner.pre_spawn_hook = pre_spawn_hook

70
jupyterhub/vault-token-renewer.sh
View File

@@ -1,4 +1,4 @@
#!/bin/sh #!/bin/bash
# Script to handle admin token retrieval and renewal # Script to handle admin token retrieval and renewal
set -e set -e
@@ -23,7 +23,69 @@ if [ -z "$ADMIN_TOKEN" ]; then
fi fi
echo "Admin token retrieved from ExternalSecret" echo "Admin token retrieved from ExternalSecret"
echo "$ADMIN_TOKEN" > /vault/secrets/vault-token echo "$ADMIN_TOKEN" >/vault/secrets/vault-token
# Calculate renewal interval (TTL/2, minimum 30 seconds)
# Use JUPYTERHUB_VAULT_TOKEN_TTL environment variable if available
if [ -n "${JUPYTERHUB_VAULT_TOKEN_TTL}" ]; then
echo "Using TTL from environment variable: ${JUPYTERHUB_VAULT_TOKEN_TTL}"
TTL_RAW="${JUPYTERHUB_VAULT_TOKEN_TTL}"
else
echo "Looking up token TTL..."
if vault token lookup >/dev/null 2>&1; then
echo "Token is valid, using default 5m interval for now"
TTL_RAW="300" # 5 minutes for testing
else
echo "Token lookup failed, using default TTL"
TTL_RAW="86400"
fi
fi
echo "Raw TTL: $TTL_RAW"
# Convert TTL format (e.g., "4m9s", "3600", "0") to seconds
convert_ttl_to_seconds() {
local ttl="$1"
# If already a number (seconds), return as-is
if echo "$ttl" | grep -E '^[0-9]+$' >/dev/null; then
echo "$ttl"
return
fi
# If contains time units (e.g., "4m9s")
local hours=0
local minutes=0
local seconds=0
if echo "$ttl" | grep -E '[0-9]+h' >/dev/null; then
hours=$(echo "$ttl" | sed -n 's/.*\([0-9]\+\)h.*/\1/p')
seconds=$((seconds + hours * 3600))
fi
if echo "$ttl" | grep -E '[0-9]+m' >/dev/null; then
minutes=$(echo "$ttl" | sed -n 's/.*\([0-9]\+\)m.*/\1/p')
seconds=$((seconds + minutes * 60))
fi
if echo "$ttl" | grep -E '[0-9]+s' >/dev/null; then
secs=$(echo "$ttl" | sed -n 's/.*\([0-9]\+\)s.*/\1/p')
seconds=$((seconds + secs))
fi
echo "$seconds"
}
TTL_SECONDS=$(convert_ttl_to_seconds "$TTL_RAW")
if [ "$TTL_SECONDS" = "0" ]; then
# If TTL is 0 (never expires), use default 12h interval
RENEWAL_INTERVAL=43200
else
# Renew at TTL/2, with minimum of 30 seconds
RENEWAL_INTERVAL=$((TTL_SECONDS / 2))
if [ "$RENEWAL_INTERVAL" -lt 30 ]; then
RENEWAL_INTERVAL=30
fi
fi
echo "Token TTL: ${TTL_SECONDS}s, renewal interval: ${RENEWAL_INTERVAL}s"
# Start token renewal loop # Start token renewal loop
export VAULT_TOKEN="$ADMIN_TOKEN" export VAULT_TOKEN="$ADMIN_TOKEN"
@@ -36,12 +98,12 @@ while true; do
# Re-read token from mounted secret # Re-read token from mounted secret
ADMIN_TOKEN=$(cat /vault/admin-token/token 2>/dev/null || echo "") ADMIN_TOKEN=$(cat /vault/admin-token/token 2>/dev/null || echo "")
if [ -n "$ADMIN_TOKEN" ]; then if [ -n "$ADMIN_TOKEN" ]; then
echo "$ADMIN_TOKEN" > /vault/secrets/vault-token echo "$ADMIN_TOKEN" >/vault/secrets/vault-token
export VAULT_TOKEN="$ADMIN_TOKEN" export VAULT_TOKEN="$ADMIN_TOKEN"
echo "$(date): Token re-retrieved successfully from ExternalSecret" echo "$(date): Token re-retrieved successfully from ExternalSecret"
else else
echo "$(date): Failed to re-retrieve token from ExternalSecret" echo "$(date): Failed to re-retrieve token from ExternalSecret"
fi fi
fi fi
sleep 43200 # 12 hours sleep $RENEWAL_INTERVAL
done done