diff --git a/jupyterhub/.gitignore b/jupyterhub/.gitignore index def5bb4..037e846 100644 --- a/jupyterhub/.gitignore +++ b/jupyterhub/.gitignore @@ -1,4 +1,5 @@ jupyterhub-values.yaml +jupyterhub-admin-service-token-external-secret.yaml jupyterhub-crypt-key-external-secret.yaml pre_spawn_hook.py vault-agent-config.hcl diff --git a/jupyterhub/README.md b/jupyterhub/README.md index 34eb92c..8904984 100644 --- a/jupyterhub/README.md +++ b/jupyterhub/README.md @@ -7,6 +7,8 @@ JupyterHub provides a multi-user Jupyter notebook environment with Keycloak OIDC - [Installation](#installation) - [Prerequisites](#prerequisites) - [Access](#access) +- [MCP Server Integration](#mcp-server-integration) +- [Programmatic API Access](#programmatic-api-access) - [Kernel Images](#kernel-images) - [Profile Configuration](#profile-configuration) - [GPU Support](#gpu-support) @@ -49,6 +51,237 @@ This will prompt for: Access JupyterHub at your configured host (e.g., `https://jupyter.example.com`) and authenticate via Keycloak. +## MCP Server Integration + +JupyterHub includes [jupyter-mcp-server](https://jupyter-mcp-server.datalayer.tech/) as a Jupyter Server Extension, enabling MCP (Model Context Protocol) clients to interact with Jupyter notebooks programmatically. + +### Overview + +The MCP server provides a standardized interface for AI assistants and other MCP clients to: + +- List and manage files on the Jupyter server +- Create, read, and edit notebook cells +- Execute code in notebook kernels +- Manage kernel sessions + +### Enabling MCP Server + +MCP server support is controlled by the `JUPYTER_MCP_SERVER_ENABLED` environment variable. During installation, you will be prompted: + +```bash +just jupyterhub::install +# "Enable jupyter-mcp-server for Claude Code integration? (y/N)" +``` + +Or set the environment variable before installation: + +```bash +JUPYTER_MCP_SERVER_ENABLED=true just jupyterhub::install +``` + +### Kernel Image Requirements + +The MCP server requires jupyter-mcp-server to be installed and enabled in the kernel image. + +**Buun-Stack profiles** (`buun-stack`, `buun-stack-cuda`) include jupyter-mcp-server pre-installed and enabled. No additional setup is required. + +**Other profiles** (minimal, base, datascience, pyspark, pytorch, tensorflow) do not include jupyter-mcp-server. To use MCP with these images, install the required packages in your notebook: + +```bash +pip install 'jupyter-mcp-server==0.21.0' 'jupyter-mcp-tools>=0.1.4' +pip uninstall -y pycrdt datalayer_pycrdt +pip install 'datalayer_pycrdt==0.12.17' +jupyter server extension enable jupyter_mcp_server +``` + +After installation, restart your Jupyter server for the extension to take effect. + +### MCP Endpoint + +When enabled, each user's Jupyter server exposes an MCP endpoint at: + +```text +https:///user//mcp +``` + +### Authentication + +MCP clients must authenticate using a JupyterHub API token. Obtain a token using: + +```bash +# Get token for a user (creates user if not exists) +just jupyterhub::get-token +``` + +The token should be passed in the `Authorization` header: + +```text +Authorization: token +``` + +### Client Configuration + +#### Generic MCP Client Configuration + +For any MCP client that supports HTTP transport: + +```bash +just jupyterhub::setup-mcp-server +``` + +This displays the MCP server URL, authentication details, and available tools. + +#### Claude Code Configuration + +For Claude Code specifically: + +```bash +just jupyterhub::setup-claude-mcp-server +``` + +This provides a ready-to-use `.mcp.json` configuration: + +```json +{ + "mcpServers": { + "jupyter-": { + "type": "http", + "url": "https:///user//mcp", + "headers": { + "Authorization": "token ${JUPYTERHUB_TOKEN}" + } + } + } +} +``` + +Set the environment variable: + +```bash +export JUPYTERHUB_TOKEN= +``` + +### Checking MCP Status + +Verify MCP server status for a user: + +```bash +just jupyterhub::mcp-status +``` + +This checks: + +- User pod is running +- jupyter-mcp-server extension is enabled +- MCP endpoint is responding + +### Technical Details + +- **Transport**: HTTP (streamable-http) +- **Extension**: jupyter-mcp-server (installed in kernel images) +- **Environment Variable**: `JUPYTERHUB_ALLOW_TOKEN_IN_URL=1` enables WebSocket token authentication + +## Programmatic API Access + +buun-stack configures JupyterHub to allow programmatic API access, enabling token generation and user management without requiring users to log in first. This is achieved by registering a **Service** in JupyterHub. + +### What is a JupyterHub Service? + +In JupyterHub, a **Service** is a registered entity (external program or script) that can access the JupyterHub API using a pre-configured token. While regular users obtain tokens by logging in, services use tokens registered in the JupyterHub configuration. + +```python +# Register a service with its API token +c.JupyterHub.services = [ + { + 'name': 'admin-service', + 'api_token': '', + } +] + +# Grant permissions to the service +c.JupyterHub.load_roles = [ + { + 'name': 'admin-service-role', + 'scopes': ['admin:users', 'tokens', 'admin:servers'], + 'services': ['admin-service'], + } +] +``` + +When an API request includes `Authorization: token `, JupyterHub identifies the token owner (in this case, `admin-service`) and applies the corresponding permissions. + +### How It Works + +```text +┌─────────────────────────────────────────────────────────────────┐ +│ just jupyterhub::get-token │ +│ │ +│ 1. Retrieve service token from Kubernetes Secret │ +│ 2. Call JupyterHub API with the token │ +└──────────────────────────────┬──────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ JupyterHub API │ +│ │ +│ 1. Receive: Authorization: token │ +│ 2. Identify: This token belongs to "admin-service" │ +│ 3. Check permissions: admin-service has admin:users, tokens │ +│ 4. Execute: Create user token and return it │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Service Configuration + +The service is automatically configured during JupyterHub installation: + +1. **Token Generation**: A random token is generated using `just utils::random-password` +2. **Secret Storage**: Token is stored in Vault (if External Secrets Operator is available) or as a Kubernetes Secret +3. **Service Registration**: JupyterHub is configured with the service and appropriate RBAC roles + +### RBAC Permissions + +The registered service has the following scopes: + +- `admin:users` - Create, read, update, delete users +- `tokens` - Create and manage API tokens +- `admin:servers` - Start and stop user servers + +### Usage + +#### Get Token for a User + +```bash +# Creates user if not exists, returns API token +just jupyterhub::get-token +``` + +This command: + +1. Checks if the user exists in JupyterHub +2. Creates the user if not found +3. Generates an API token with appropriate scopes +4. Returns the token for use with MCP or other API clients + +#### Manual Token Management + +The service token is stored in: + +- **Vault path**: `secret/jupyterhub/admin-service` (key: `token`) +- **Kubernetes Secret**: `jupyterhub-admin-service-token` in the JupyterHub namespace + +To recreate the service token: + +```bash +just jupyterhub::create-admin-service-token-secret +``` + +### Security Considerations + +- The service token has elevated privileges; protect it accordingly +- Tokens are stored encrypted in Vault when External Secrets Operator is available +- User tokens generated via the service have limited scopes (`access:servers!user=`, `self`) + ## Kernel Images ### Important Note @@ -391,7 +624,7 @@ Vault integration enables secure secrets management directly from Jupyter notebo └──────────────────────┘ ``` -### Prerequisites +### Vault Integration Prerequisites Vault integration requires: diff --git a/jupyterhub/images/datastack-cuda-notebook/Dockerfile b/jupyterhub/images/datastack-cuda-notebook/Dockerfile index 8f982ad..9b8a0ce 100644 --- a/jupyterhub/images/datastack-cuda-notebook/Dockerfile +++ b/jupyterhub/images/datastack-cuda-notebook/Dockerfile @@ -167,11 +167,16 @@ RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip pip install -i "${pip_ RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip pip install -i "${pip_repository_url}" \ 'dlt[clickhouse,databricks,deltalake,dremio,duckdb,filesystem,parquet,postgres,pyiceberg,qdrant,redshift,s3,snowflake,sql-database,sqlalchemy,workspace]' +# jupyter-mcp-server as Jupyter Server Extension # https://jupyter-mcp-server.datalayer.tech/setup/jupyter/local_mcp/ -# RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip \ -# pip install -i "${pip_repository_url}" 'jupyterlab==4.4.1' 'jupyter-collaboration==4.0.2' \ -# && pip uninstall -y pycrdt datalayer_pycrdt \ -# && pip install -i "${pip_repository_url}" 'datalayer_pycrdt==0.12.17' +# Provides /mcp/v1 endpoint for Claude Code integration +RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip \ + pip install -i "${pip_repository_url}" \ + 'jupyter-mcp-server==0.21.0' \ + 'jupyter-mcp-tools>=0.1.4' \ + && pip uninstall -y pycrdt datalayer_pycrdt \ + && pip install -i "${pip_repository_url}" 'datalayer_pycrdt==0.12.17' \ + && jupyter server extension enable jupyter_mcp_server # Install PyTorch with CUDA 12.x support (https://pytorch.org/get-started/locally/) # langchain-openai must be updated to avoid pydantic v2 error diff --git a/jupyterhub/images/datastack-notebook/Dockerfile b/jupyterhub/images/datastack-notebook/Dockerfile index 29accd0..a1ef305 100644 --- a/jupyterhub/images/datastack-notebook/Dockerfile +++ b/jupyterhub/images/datastack-notebook/Dockerfile @@ -166,11 +166,16 @@ RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip pip install -i "${pip_ RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip pip install -i "${pip_repository_url}" \ 'dlt[clickhouse,databricks,deltalake,dremio,duckdb,filesystem,parquet,postgres,pyiceberg,qdrant,redshift,s3,snowflake,sql-database,sqlalchemy,workspace]' +# jupyter-mcp-server as Jupyter Server Extension # https://jupyter-mcp-server.datalayer.tech/setup/jupyter/local_mcp/ -# RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip \ -# pip install -i "${pip_repository_url}" 'jupyterlab==4.4.1' 'jupyter-collaboration==4.0.2' \ -# && pip uninstall -y pycrdt datalayer_pycrdt \ -# && pip install -i "${pip_repository_url}" 'datalayer_pycrdt==0.12.17' +# Provides /mcp/v1 endpoint for Claude Code integration +RUN --mount=type=cache,target=/home/${NB_USER}/.cache/pip \ + pip install -i "${pip_repository_url}" \ + 'jupyter-mcp-server==0.21.0' \ + 'jupyter-mcp-tools>=0.1.4' \ + && pip uninstall -y pycrdt datalayer_pycrdt \ + && pip install -i "${pip_repository_url}" 'datalayer_pycrdt==0.12.17' \ + && jupyter server extension enable jupyter_mcp_server # Install PyTorch with pip (https://pytorch.org/get-started/locally/) # langchain-openai must be updated to avoid pydantic v2 error diff --git a/jupyterhub/jupyterhub-admin-service-token-external-secret.gomplate.yaml b/jupyterhub/jupyterhub-admin-service-token-external-secret.gomplate.yaml new file mode 100644 index 0000000..ad9b5c3 --- /dev/null +++ b/jupyterhub/jupyterhub-admin-service-token-external-secret.gomplate.yaml @@ -0,0 +1,18 @@ +apiVersion: external-secrets.io/v1 +kind: ExternalSecret +metadata: + name: jupyterhub-admin-service-token + namespace: {{ .Env.JUPYTERHUB_NAMESPACE }} +spec: + refreshInterval: 1h + secretStoreRef: + name: vault-secret-store + kind: ClusterSecretStore + target: + name: jupyterhub-admin-service-token + creationPolicy: Owner + data: + - secretKey: token + remoteRef: + key: jupyterhub/admin-service + property: token diff --git a/jupyterhub/jupyterhub-values.gomplate.yaml b/jupyterhub/jupyterhub-values.gomplate.yaml index 8ca1f4a..2ce98e2 100644 --- a/jupyterhub/jupyterhub-values.gomplate.yaml +++ b/jupyterhub/jupyterhub-values.gomplate.yaml @@ -5,6 +5,11 @@ hub: secretKeyRef: name: jupyterhub-crypt-key key: crypt-key + JUPYTERHUB_ADMIN_SERVICE_TOKEN: + valueFrom: + secretKeyRef: + name: jupyterhub-admin-service-token + key: token VAULT_ADDR: {{ .Env.VAULT_ADDR | quote }} NOTEBOOK_VAULT_TOKEN_TTL: {{ .Env.NOTEBOOK_VAULT_TOKEN_TTL | quote }} NOTEBOOK_VAULT_TOKEN_MAX_TTL: {{ .Env.NOTEBOOK_VAULT_TOKEN_MAX_TTL | quote }} @@ -81,6 +86,25 @@ hub: } } + admin-service: | + # Admin service for token management via API (used by get-token recipe) + import os + admin_token = os.environ.get('JUPYTERHUB_ADMIN_SERVICE_TOKEN', '') + if admin_token: + c.JupyterHub.services = [ + { + 'name': 'admin-service', + 'api_token': admin_token, + } + ] + c.JupyterHub.load_roles = [ + { + 'name': 'admin-service-role', + 'scopes': ['admin:users', 'tokens', 'admin:servers'], + 'services': ['admin-service'], + } + ] + {{- if eq .Env.JUPYTERHUB_VAULT_INTEGRATION_ENABLED "true" }} # Vault token renewal sidecar configuration extraVolumes: @@ -172,34 +196,6 @@ singleuser: allowPrivilegeEscalation: false # Additional security context via extraPodConfig - extraPodConfig: - securityContext: - runAsNonRoot: true - seccompProfile: - type: RuntimeDefault - - storage: - {{ if env.Getenv "PVC_NAME" -}} - type: static - static: - pvcName: {{ .Env.PVC_NAME }} - {{ else -}} - type: dynamic - dynamic: - {{ if env.Getenv "JUPYTERHUB_STORAGE_CLASS" -}} - storageClass: {{ .Env.JUPYTERHUB_STORAGE_CLASS }} - {{ end -}} - storageAccessModes: - - ReadWriteOnce - {{ end -}} - capacity: 10Gi - - extraEnv: - VAULT_ADDR: "{{ .Env.VAULT_ADDR }}" - NOTEBOOK_VAULT_TOKEN_TTL: "{{ .Env.NOTEBOOK_VAULT_TOKEN_TTL }}" - NOTEBOOK_VAULT_TOKEN_MAX_TTL: "{{ .Env.NOTEBOOK_VAULT_TOKEN_MAX_TTL }}" - # JUPYTERHUB_SINGLEUSER_EXTENSION: "0" - {{- if eq .Env.JUPYTERHUB_GPU_ENABLED "true" }} extraPodConfig: runtimeClassName: nvidia @@ -210,8 +206,24 @@ singleuser: extraResource: limits: nvidia.com/gpu: "{{ .Env.JUPYTERHUB_GPU_LIMIT }}" + {{- else }} + extraPodConfig: + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault {{- end }} + extraEnv: + VAULT_ADDR: "{{ .Env.VAULT_ADDR }}" + NOTEBOOK_VAULT_TOKEN_TTL: "{{ .Env.NOTEBOOK_VAULT_TOKEN_TTL }}" + NOTEBOOK_VAULT_TOKEN_MAX_TTL: "{{ .Env.NOTEBOOK_VAULT_TOKEN_MAX_TTL }}" + {{- if eq .Env.JUPYTER_MCP_SERVER_ENABLED "true" }} + # Enable WebSocket token authentication for jupyter-mcp-server extension + # https://github.com/datalayer/jupyter-mcp-server/issues/61 + JUPYTERHUB_ALLOW_TOKEN_IN_URL: "1" + {{- end }} + storage: {{ if env.Getenv "PVC_NAME" -}} type: static @@ -228,7 +240,6 @@ singleuser: {{ end -}} capacity: 10Gi {{- if eq .Env.JUPYTERHUB_AIRFLOW_DAGS_PERSISTENCE_ENABLED "true" }} - # Mount Airflow DAGs when both are in the same namespace (jupyter) extraVolumes: - name: airflow-dags persistentVolumeClaim: diff --git a/jupyterhub/justfile b/jupyterhub/justfile index f346d8c..feee092 100644 --- a/jupyterhub/justfile +++ b/jupyterhub/justfile @@ -9,7 +9,7 @@ export JUPYTERHUB_NFS_PV_ENABLED := env("JUPYTERHUB_NFS_PV_ENABLED", "") export JUPYTERHUB_STORAGE_CLASS := env("JUPYTERHUB_STORAGE_CLASS", "") export JUPYTERHUB_VAULT_INTEGRATION_ENABLED := env("JUPYTERHUB_VAULT_INTEGRATION_ENABLED", "") export JUPYTERHUB_AIRFLOW_DAGS_PERSISTENCE_ENABLED := env("JUPYTERHUB_AIRFLOW_DAGS_PERSISTENCE_ENABLED", "") -export JUPYTER_PYTHON_KERNEL_TAG := env("JUPYTER_PYTHON_KERNEL_TAG", "python-3.12-52") +export JUPYTER_PYTHON_KERNEL_TAG := env("JUPYTER_PYTHON_KERNEL_TAG", "python-3.12-53") export KERNEL_IMAGE_BUUN_STACK_REPOSITORY := env("KERNEL_IMAGE_BUUN_STACK_REPOSITORY", "buun-stack-notebook") export KERNEL_IMAGE_BUUN_STACK_CUDA_REPOSITORY := env("KERNEL_IMAGE_BUUN_STACK_CUDA_REPOSITORY", "buun-stack-cuda-notebook") export JUPYTER_PROFILE_MINIMAL_ENABLED := env("JUPYTER_PROFILE_MINIMAL_ENABLED", "false") @@ -22,6 +22,7 @@ export JUPYTER_PROFILE_BUUN_STACK_ENABLED := env("JUPYTER_PROFILE_BUUN_STACK_ENA export JUPYTER_PROFILE_BUUN_STACK_CUDA_ENABLED := env("JUPYTER_PROFILE_BUUN_STACK_CUDA_ENABLED", "false") export JUPYTERHUB_GPU_ENABLED := env("JUPYTERHUB_GPU_ENABLED", "") export JUPYTERHUB_GPU_LIMIT := env("JUPYTERHUB_GPU_LIMIT", "1") +export JUPYTER_MCP_SERVER_ENABLED := env("JUPYTER_MCP_SERVER_ENABLED", "") export JUPYTERHUB_VAULT_TOKEN_TTL := env("JUPYTERHUB_VAULT_TOKEN_TTL", "24h") export NOTEBOOK_VAULT_TOKEN_TTL := env("NOTEBOOK_VAULT_TOKEN_TTL", "24h") export NOTEBOOK_VAULT_TOKEN_MAX_TTL := env("NOTEBOOK_VAULT_TOKEN_MAX_TTL", "168h") @@ -65,6 +66,37 @@ create-namespace: delete-namespace: kubectl delete namespace ${JUPYTERHUB_NAMESPACE} --ignore-not-found +# Create JupyterHub admin service token secret +create-admin-service-token-secret: + #!/bin/bash + set -euo pipefail + admin_token=$(just utils::random-password) + + if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then + echo "External Secrets Operator detected. Storing admin service token in Vault..." + just vault::put jupyterhub/admin-service token="${admin_token}" + + kubectl delete secret jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} --ignore-not-found + kubectl delete externalsecret jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} --ignore-not-found + + gomplate -f jupyterhub-admin-service-token-external-secret.gomplate.yaml \ + -o jupyterhub-admin-service-token-external-secret.yaml + kubectl apply -f jupyterhub-admin-service-token-external-secret.yaml + + echo "Waiting for ExternalSecret to sync..." + kubectl wait --for=condition=Ready externalsecret/jupyterhub-admin-service-token \ + -n ${JUPYTERHUB_NAMESPACE} --timeout=60s + else + echo "External Secrets Operator not found. Creating secret directly..." + kubectl delete secret jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} --ignore-not-found + kubectl create secret generic jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} \ + --from-literal=token="${admin_token}" + + if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then + just vault::put jupyterhub/admin-service token="${admin_token}" + fi + fi + # Create JupyterHub crypt key secret create-crypt-key-secret: #!/bin/bash @@ -113,11 +145,14 @@ install root_token='': kubectl label namespace ${JUPYTERHUB_NAMESPACE} \ pod-security.kubernetes.io/enforce=restricted --overwrite - # Create crypt key secret if it doesn't exist if ! kubectl get secret jupyterhub-crypt-key -n ${JUPYTERHUB_NAMESPACE} &>/dev/null; then just create-crypt-key-secret fi + if ! kubectl get secret jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} &>/dev/null; then + just create-admin-service-token-secret + fi + if helm status kube-prometheus-stack -n ${PROMETHEUS_NAMESPACE} &>/dev/null; then if [ -z "${MONITORING_ENABLED}" ]; then if gum confirm "Enable Prometheus monitoring (ServiceMonitor)?"; then @@ -130,7 +165,6 @@ install root_token='': MONITORING_ENABLED="false" fi - # Check if nvidia-device-plugin is installed if helm status nvidia-device-plugin -n ${NVIDIA_DEVICE_PLUGIN_NAMESPACE:-nvidia-device-plugin} &>/dev/null; then if [ -z "${JUPYTERHUB_GPU_ENABLED}" ]; then if gum confirm "Enable GPU support for JupyterHub notebooks?"; then @@ -197,7 +231,6 @@ install root_token='': kubectl apply -n ${JUPYTERHUB_NAMESPACE} -f nfs-pvc.yaml fi - # Setup Airflow DAG storage sharing (same namespace) if [ -z "${JUPYTERHUB_AIRFLOW_DAGS_PERSISTENCE_ENABLED}" ]; then if gum confirm "Enable Airflow DAG storage mounting (requires Airflow in same namespace)?"; then JUPYTERHUB_AIRFLOW_DAGS_PERSISTENCE_ENABLED="true" @@ -214,7 +247,19 @@ install root_token='': echo " kubectl delete pods -n jupyter -l app.kubernetes.io/component=singleuser-server" fi - # Setup Vault Agent for automatic token management + if [ -z "${JUPYTER_MCP_SERVER_ENABLED}" ]; then + if gum confirm "Enable jupyter-mcp-server?"; then + JUPYTER_MCP_SERVER_ENABLED="true" + else + JUPYTER_MCP_SERVER_ENABLED="false" + fi + fi + if [ "${JUPYTER_MCP_SERVER_ENABLED}" = "true" ]; then + echo "✅ jupyter-mcp-server enabled" + echo " MCP endpoint: https://${JUPYTERHUB_HOST}/user/{username}/mcp" + echo " Use 'just jupyterhub::get-token ' to get API token" + fi + if [ -z "${JUPYTERHUB_VAULT_INTEGRATION_ENABLED}" ]; then if gum confirm "Are you going to enable Vault integration?"; then JUPYTERHUB_VAULT_INTEGRATION_ENABLED=true @@ -243,7 +288,6 @@ install root_token='': export USER_POLICY_HCL="" fi - # Generate pre_spawn_hook.py echo "Generating pre_spawn_hook.py..." gomplate -f pre_spawn_hook.gomplate.py -o pre_spawn_hook.py @@ -253,7 +297,8 @@ install root_token='': helm upgrade --cleanup-on-fail --install jupyterhub jupyterhub/jupyterhub \ --version ${JUPYTERHUB_CHART_VERSION} -n ${JUPYTERHUB_NAMESPACE} \ --timeout=20m -f jupyterhub-values.yaml - # wait deployments manually because `helm upgrade --wait` does not work for JupyterHub + + # Wait deployments manually because `helm upgrade --wait` does not work for JupyterHub just k8s::wait-deployments-ready ${JUPYTERHUB_NAMESPACE} hub proxy if [ "${MONITORING_ENABLED}" = "true" ]; then @@ -272,7 +317,9 @@ uninstall: kubectl delete pods -n ${JUPYTERHUB_NAMESPACE} -l app.kubernetes.io/component=singleuser-server kubectl delete -n ${JUPYTERHUB_NAMESPACE} pvc jupyter-nfs-pvc --ignore-not-found kubectl delete -n ${JUPYTERHUB_NAMESPACE} secret jupyterhub-crypt-key --ignore-not-found + kubectl delete -n ${JUPYTERHUB_NAMESPACE} secret jupyterhub-admin-service-token --ignore-not-found kubectl delete -n ${JUPYTERHUB_NAMESPACE} externalsecret jupyterhub-crypt-key --ignore-not-found + kubectl delete -n ${JUPYTERHUB_NAMESPACE} externalsecret jupyterhub-admin-service-token --ignore-not-found kubectl delete -n ${JUPYTERHUB_NAMESPACE} externalsecret jupyterhub-vault-token --ignore-not-found if kubectl get pv jupyter-nfs-pv &>/dev/null; then kubectl patch pv jupyter-nfs-pv -p '{"spec":{"claimRef":null}}' @@ -281,6 +328,7 @@ uninstall: # Clean up Vault entries if present if helm status vault -n ${K8S_VAULT_NAMESPACE} &>/dev/null; then just vault::delete jupyterhub/config || true + just vault::delete jupyterhub/admin-service || true fi # Delete JupyterHub PV and StorageClass @@ -364,6 +412,7 @@ setup-vault-integration root_token='': bound_service_account_names=hub \ bound_service_account_namespaces=jupyter \ policies=jupyterhub-admin \ + audience=vault \ ttl=${JUPYTERHUB_VAULT_TOKEN_TTL} \ max_ttl=720h @@ -430,7 +479,71 @@ create-jupyterhub-vault-token root_token='': echo "" echo "Token stored at: secret/jupyterhub/vault-token" -# Get JupyterHub API token for a user +# Get or create JupyterHub API token for a user +get-token username: + #!/bin/bash + set -euo pipefail + USERNAME="{{ username }}" + + if [ -z "${USERNAME}" ]; then + echo "Error: Username is required" >&2 + echo "Usage: just jupyterhub::get-token " >&2 + exit 1 + fi + + # Get admin service token from Secret + ADMIN_TOKEN=$(kubectl get secret jupyterhub-admin-service-token -n ${JUPYTERHUB_NAMESPACE} \ + -o jsonpath='{.data.token}' 2>/dev/null | base64 -d || true) + + if [ -z "${ADMIN_TOKEN}" ]; then + echo "Error: Could not retrieve admin service token" >&2 + echo "Make sure jupyterhub-admin-service-token secret exists" >&2 + exit 1 + fi + + # Check if user exists, if not create it + USER_EXISTS=$(kubectl exec -n ${JUPYTERHUB_NAMESPACE} deployment/hub -- \ + curl -s -o /dev/null -w "%{http_code}" \ + -H "Authorization: token ${ADMIN_TOKEN}" \ + "http://localhost:8081/hub/api/users/${USERNAME}" 2>/dev/null || echo "000") + + if [ "${USER_EXISTS}" = "404" ]; then + echo "User '${USERNAME}' not found, creating..." >&2 + CREATE_RESPONSE=$(kubectl exec -n ${JUPYTERHUB_NAMESPACE} deployment/hub -- \ + curl -s -X POST \ + -H "Authorization: token ${ADMIN_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"usernames\": [\"${USERNAME}\"]}" \ + "http://localhost:8081/hub/api/users" 2>/dev/null) + echo "User '${USERNAME}' created" >&2 + fi + + # Create token via JupyterHub API + # POST /hub/api/users/{name}/tokens + # Use access:servers!user={username} scope to allow access to the user's own servers + RESPONSE=$(kubectl exec -n ${JUPYTERHUB_NAMESPACE} deployment/hub -- \ + curl -s -X POST \ + -H "Authorization: token ${ADMIN_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"note\": \"MCP server token\", \"scopes\": [\"access:servers!user=${USERNAME}\", \"self\"]}" \ + "http://localhost:8081/hub/api/users/${USERNAME}/tokens" 2>/dev/null) + + TOKEN=$(echo "${RESPONSE}" | jq -r '.token // empty' 2>/dev/null) + + if [ -z "${TOKEN}" ]; then + ERROR=$(echo "${RESPONSE}" | jq -r '.message // .error // empty' 2>/dev/null) + if [ -n "${ERROR}" ]; then + echo "Error: ${ERROR}" >&2 + else + echo "Error: Failed to create token for user '${USERNAME}'" >&2 + echo "Response: ${RESPONSE}" >&2 + fi + exit 1 + fi + + echo "${TOKEN}" + +# Get JupyterHub API token for a user (from running pod) get-api-token username: #!/bin/bash set -euo pipefail @@ -474,8 +587,8 @@ get-api-token username: echo "${API_TOKEN}" -# Setup MCP server configuration for Claude Code (has auth problems) -setup-mcp-server username='' notebook='': +# Show MCP server configuration for a user +setup-mcp-server username='': #!/bin/bash set -euo pipefail @@ -499,78 +612,136 @@ setup-mcp-server username='' notebook='': exit 1 fi - # Get notebook path - NOTEBOOK="{{ notebook }}" - if [ -z "${NOTEBOOK}" ]; then - echo "" - echo "Available notebooks for user '${USERNAME}':" - kubectl exec -n ${JUPYTERHUB_NAMESPACE} jupyter-${USERNAME} -- \ - curl -s -H "Authorization: token ${API_TOKEN}" \ - "http://localhost:8888/user/${USERNAME}/api/contents" 2>/dev/null | \ - jq -r '.content[]? | select(.type=="notebook") | .path' | head -20 || true - echo "" - NOTEBOOK=$(gum input --prompt="Notebook path (required): " --width=100 --placeholder="e.g., Untitled.ipynb or path/to/notebook.ipynb") + # MCP endpoint URL (Jupyter Server Extension provides /mcp) + MCP_URL="https://${JUPYTERHUB_HOST}/user/${USERNAME}/mcp" - if [ -z "${NOTEBOOK}" ]; then - echo "Error: Notebook path is required for MCP server to function" >&2 - exit 1 - fi + echo "" + echo "✅ MCP Server is available at: ${MCP_URL}" + echo "" + echo "API Token: ${API_TOKEN}" + echo "" + echo "MCP Server Configuration:" + echo "" + echo " URL: ${MCP_URL}" + echo " Transport: HTTP (streamable-http)" + echo " Authentication: Bearer token via Authorization header" + echo "" + echo "Environment variable:" + echo " export JUPYTERHUB_TOKEN=${API_TOKEN}" + echo "" + echo "Available MCP tools:" + echo " - list_files: List files in the Jupyter server" + echo " - list_kernels: List available kernels" + echo " - use_notebook: Activate a notebook for operations" + echo " - list_notebooks: Show activated notebooks" + echo " - insert_cell, execute_cell, read_cell, delete_cell: Cell operations" + echo " - execute_code: Execute arbitrary code" + echo "" + echo "Note: Use 'list_files' first to find notebooks, then 'use_notebook' to activate one." + +# Show MCP server configuration for Claude Code (with .mcp.json example) +setup-claude-mcp-server username='': + #!/bin/bash + set -euo pipefail + + USERNAME="{{ username }}" + if [ -z "${USERNAME}" ]; then + USERNAME=$(gum input --prompt="JupyterHub username: " --width=100 --placeholder="e.g., buun") fi - # Create .mcp.json configuration - MCP_CONFIG_FILE="../.mcp.json" + if [ -z "${USERNAME}" ]; then + echo "Error: Username is required" >&2 + exit 1 + fi - echo "Creating MCP server configuration..." - cat > "${MCP_CONFIG_FILE}" </dev/null || true) + + if [ -z "${API_TOKEN}" ]; then + echo "Error: Could not get API token for user '${USERNAME}'" >&2 + echo "Make sure the user has an active Jupyter session" >&2 + exit 1 + fi + + # MCP endpoint URL (Jupyter Server Extension provides /mcp) + MCP_URL="https://${JUPYTERHUB_HOST}/user/${USERNAME}/mcp" + + echo "" + echo "✅ MCP Server is available at: ${MCP_URL}" + echo "" + echo "API Token: ${API_TOKEN}" + echo "" + echo "To configure Claude Code, add to your .mcp.json:" + echo "" + cat <" +# Show MCP server status for a user +mcp-status username='': + #!/bin/bash + set -euo pipefail + + USERNAME="{{ username }}" + if [ -z "${USERNAME}" ]; then + USERNAME=$(gum input --prompt="JupyterHub username: " --width=100 --placeholder="e.g., buun") + fi + + if [ -z "${USERNAME}" ]; then + echo "Error: Username is required" >&2 + exit 1 + fi + + # Check if user pod is running + POD_NAME=$(kubectl get pods -n ${JUPYTERHUB_NAMESPACE} \ + -l "app=jupyterhub,component=singleuser-server,hub.jupyter.org/username=${USERNAME}" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true) + + if [ -z "${POD_NAME}" ]; then + echo "❌ No running pod found for user '${USERNAME}'" + echo " Start a Jupyter session first at: https://${JUPYTERHUB_HOST}" + exit 1 + fi + + echo "✅ User pod running: ${POD_NAME}" + + # Check if jupyter-mcp-server extension is enabled + echo "" + echo "Checking jupyter-mcp-server extension status..." + kubectl exec -n ${JUPYTERHUB_NAMESPACE} ${POD_NAME} -- \ + jupyter server extension list 2>/dev/null | grep -E "(jupyter_mcp_server|enabled)" || \ + echo "⚠️ jupyter-mcp-server extension not found in listing" + + # Try to access MCP endpoint + echo "" + echo "Testing MCP endpoint..." + API_TOKEN=$(just jupyterhub::get-api-token ${USERNAME} 2>/dev/null || true) + if [ -n "${API_TOKEN}" ]; then + RESPONSE=$(kubectl exec -n ${JUPYTERHUB_NAMESPACE} ${POD_NAME} -- \ + curl -s -o /dev/null -w "%{http_code}" \ + -H "Authorization: token ${API_TOKEN}" \ + "http://localhost:8888/mcp" 2>/dev/null || echo "000") + if [ "${RESPONSE}" = "200" ] || [ "${RESPONSE}" = "405" ]; then + echo "✅ MCP endpoint responding (HTTP ${RESPONSE})" + echo " URL: https://${JUPYTERHUB_HOST}/user/${USERNAME}/mcp" + else + echo "⚠️ MCP endpoint returned HTTP ${RESPONSE}" + fi fi