diff --git a/lakekeeper/README.md b/lakekeeper/README.md index 1449b69..f8d78b0 100644 --- a/lakekeeper/README.md +++ b/lakekeeper/README.md @@ -35,6 +35,88 @@ The installation automatically: Access Lakekeeper at `https://lakekeeper.yourdomain.com` and authenticate via Keycloak. +## Warehouse Management + +### Creating Warehouses with Vended Credentials + +Create warehouses with STS (Security Token Service) enabled for automatic temporary credential management: + +```bash +# Create warehouse with default name and bucket +just lakekeeper::create-warehouse + +# Example: Create 'production' warehouse using 'warehouse' bucket +just lakekeeper::create-warehouse production warehouse +``` + +This creates a warehouse with: + +- **STS enabled** for vended credentials (temporary S3 tokens) +- **S3-compatible storage** (MinIO) with path-style access +- **Automatic credential rotation** via MinIO STS + +**Prerequisites**: + +- MinIO bucket must exist (create with `just minio::create-bucket `) +- API client credentials must be available in Vault + +**Benefits of Vended Credentials**: + +- No need to distribute static S3 credentials to clients +- Automatic credential expiration and rotation +- Better security through temporary tokens +- Centralized credential management + +### Creating Namespaces + +Namespaces organize tables within a warehouse (similar to databases in traditional systems): + +```bash +# Create Iceberg namespace in a warehouse +just lakekeeper::create-warehouse-namespace + +# Example: Create 'ecommerce' namespace in 'test' warehouse +just lakekeeper::create-warehouse-namespace test ecommerce +``` + +### Managing Warehouses + +List, view, and delete warehouses: + +```bash +# List all warehouses +just lakekeeper::list-warehouses + +# List all namespaces in a warehouse +just lakekeeper::list-warehouse-namespaces + +# Example: List namespaces in 'test' warehouse +just lakekeeper::list-warehouse-namespaces test + +# Delete a namespace from a warehouse (recursively deletes all tables) +just lakekeeper::delete-warehouse-namespace + +# Example: Delete 'ecommerce' namespace from 'test' warehouse (including all tables) +just lakekeeper::delete-warehouse-namespace test ecommerce + +# Delete a warehouse (must be empty) +just lakekeeper::delete-warehouse + +# Force delete a warehouse (automatically deletes all namespaces first) +just lakekeeper::delete-warehouse true + +# Example: Force delete 'test' warehouse with all its namespaces +just lakekeeper::delete-warehouse test true +``` + +**Important Notes**: + +- Namespace deletion is **recursive** - it will delete all tables and data within the namespace +- Warehouses must be empty before deletion. If a warehouse contains namespaces, you must either: + 1. Delete each namespace individually using `delete-warehouse-namespace`, then delete the warehouse + 2. Use force deletion (`delete-warehouse true`) to automatically delete all namespaces and their tables first +- All deletion operations require confirmation prompts to prevent accidental data loss + ## Programmatic Access ### API Client Credentials @@ -70,13 +152,47 @@ Configure dlt to use the API client credentials: export OIDC_CLIENT_ID=lakekeeper-api export OIDC_CLIENT_SECRET= export ICEBERG_CATALOG_URL=http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog -export ICEBERG_WAREHOUSE=default +export ICEBERG_WAREHOUSE=test # Use warehouse with vended credentials enabled +export KEYCLOAK_TOKEN_URL=https://auth.example.com/realms/buunstack/protocol/openid-connect/token +export OAUTH2_SCOPE=lakekeeper # Optional, defaults to "lakekeeper" ``` -The dlt Iceberg REST destination automatically uses these credentials for OAuth2 authentication. +The dlt Iceberg REST destination automatically uses these credentials for OAuth2 authentication and receives temporary S3 credentials via STS (vended credentials). + +**Notes**: + +- `KEYCLOAK_TOKEN_URL` is required because Lakekeeper v0.9.x uses external OAuth2 provider (Keycloak) instead of the deprecated `/v1/oauth/tokens` endpoint. +- `OAUTH2_SCOPE` must be set to `lakekeeper` (default) to include the audience claim in JWT tokens. PyIceberg defaults to `catalog` scope, which is not valid for Keycloak. +- **No S3 credentials needed** when using warehouses with vended credentials enabled (STS). Lakekeeper provides temporary S3 credentials automatically. + +#### Legacy Mode: Static S3 Credentials + +If using a warehouse with `vended-credentials-enabled=false`, you need to provide static S3 credentials: + +```bash +# Additional environment variables for static credentials mode +export S3_ENDPOINT_URL=http://minio.minio.svc.cluster.local:9000 +export S3_ACCESS_KEY_ID= +export S3_SECRET_ACCESS_KEY= +``` + +To get MinIO credentials: + +```bash +just vault::get minio/dlt access_key +just vault::get minio/dlt secret_key +``` + +Or create a dedicated MinIO user: + +```bash +just minio::create-user dlt "dlt-data" +``` #### PyIceberg +With vended credentials (recommended): + ```python from pyiceberg.catalog import load_catalog @@ -84,8 +200,30 @@ catalog = load_catalog( "rest_catalog", **{ "uri": "http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog", - "warehouse": "default", + "warehouse": "test", # Use warehouse with vended credentials enabled "credential": f"{client_id}:{client_secret}", # OAuth2 format + "oauth2-server-uri": "https://auth.example.com/realms/buunstack/protocol/openid-connect/token", + "scope": "lakekeeper", # Required for Keycloak (PyIceberg defaults to "catalog") + } +) +``` + +With static S3 credentials (legacy mode): + +```python +catalog = load_catalog( + "rest_catalog", + **{ + "uri": "http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog", + "warehouse": "default", + "credential": f"{client_id}:{client_secret}", + "oauth2-server-uri": "https://auth.example.com/realms/buunstack/protocol/openid-connect/token", + "scope": "lakekeeper", + # Static S3 credentials (only needed when vended credentials disabled) + "s3.endpoint": "http://minio.minio.svc.cluster.local:9000", + "s3.access-key-id": "", + "s3.secret-access-key": "", + "s3.path-style-access": "true", } ) ``` diff --git a/lakekeeper/justfile b/lakekeeper/justfile index 8bef729..b139af0 100644 --- a/lakekeeper/justfile +++ b/lakekeeper/justfile @@ -292,3 +292,481 @@ cleanup: else echo "Cleanup cancelled" fi + +# Create warehouse with vended credentials enabled (STS) +create-warehouse warehouse_name='default' bucket='warehouse': + #!/bin/bash + set -euo pipefail + echo "Creating warehouse '{{ warehouse_name }}' with vended credentials (STS) enabled..." + + # Get MinIO credentials + MINIO_ACCESS_KEY=$(kubectl get secret -n minio minio -o jsonpath='{.data.rootUser}' | base64 -d) + MINIO_SECRET_KEY=$(kubectl get secret -n minio minio -o jsonpath='{.data.rootPassword}' | base64 -d) + + # Create warehouse JSON configuration + WAREHOUSE_CONFIG=$(cat </dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + echo "Authenticating with Keycloak..." + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # Create warehouse + echo "Creating warehouse..." + RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$WAREHOUSE_CONFIG") + + HTTP_CODE=$(echo "$RESPONSE" | tail -n1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "Warehouse '{{ warehouse_name }}' created successfully with vended credentials enabled" + echo "Response: $BODY" + else + echo "Error: Failed to create warehouse (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi + +# Create Iceberg namespace in a warehouse +create-warehouse-namespace warehouse_name namespace: + #!/bin/bash + set -euo pipefail + echo "Creating namespace '{{ namespace }}' in warehouse '{{ warehouse_name }}'..." + + # Get API client credentials for authentication + CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + echo "Authenticating with Keycloak..." + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # Get warehouse ID from warehouse name + echo "Getting warehouse ID for '{{ warehouse_name }}'..." + WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1) + LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d') + + if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then + WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id') + if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then + echo "Error: Warehouse '{{ warehouse_name }}' not found" + echo "Available warehouses:" + echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names" + exit 1 + fi + echo "Warehouse ID: $WAREHOUSE_ID" + else + echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)" + echo "Response: $LIST_BODY" + exit 1 + fi + + # Create namespace + echo "Creating namespace..." + NAMESPACE_CONFIG=$(cat </dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # List warehouses + RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + HTTP_CODE=$(echo "$RESPONSE" | tail -n1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "Warehouses:" + echo "$BODY" | jq -r '.warehouses[] | " - \(.name) (ID: \(.id))"' + else + echo "Error: Failed to list warehouses (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi + +# Delete namespace from a warehouse +delete-warehouse-namespace warehouse_name namespace: + #!/bin/bash + set -euo pipefail + echo "This will delete namespace '{{ namespace }}' from warehouse '{{ warehouse_name }}'." + if ! gum confirm "Are you sure you want to proceed?"; then + echo "Deletion cancelled" + exit 0 + fi + + echo "Deleting namespace '{{ namespace }}' from warehouse '{{ warehouse_name }}'..." + + # Get API client credentials for authentication + CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # Get warehouse ID from warehouse name + WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1) + LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d') + + if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then + WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id') + if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then + echo "Error: Warehouse '{{ warehouse_name }}' not found" + echo "Available warehouses:" + echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names" + exit 1 + fi + else + echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)" + echo "Response: $LIST_BODY" + exit 1 + fi + + # Delete namespace with recursive flag to delete all tables + RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces/{{ namespace }}?recursive=true" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + HTTP_CODE=$(echo "$RESPONSE" | tail -n1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "Namespace '{{ namespace }}' deleted successfully from warehouse '{{ warehouse_name }}'" + elif [ "$HTTP_CODE" = "404" ]; then + echo "Namespace '{{ namespace }}' not found in warehouse '{{ warehouse_name }}'" + else + echo "Error: Failed to delete namespace (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi + +# List all namespaces in a warehouse +list-warehouse-namespaces warehouse_name: + #!/bin/bash + set -euo pipefail + echo "Listing namespaces in warehouse '{{ warehouse_name }}'..." + + # Get API client credentials for authentication + CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # Get warehouse ID from warehouse name + WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1) + LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d') + + if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then + WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id') + if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then + echo "Error: Warehouse '{{ warehouse_name }}' not found" + echo "Available warehouses:" + echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names" + exit 1 + fi + else + echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)" + echo "Response: $LIST_BODY" + exit 1 + fi + + # List namespaces + RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + HTTP_CODE=$(echo "$RESPONSE" | tail -n1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "Namespaces in warehouse '{{ warehouse_name }}':" + echo "$BODY" | jq -r '.namespaces[] | " - \(.[0])"' + else + echo "Error: Failed to list namespaces (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi + +# Delete warehouse +delete-warehouse warehouse_name force='false': + #!/bin/bash + set -euo pipefail + + # Get API client credentials for authentication + CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "") + if [ -z "$CLIENT_SECRET" ]; then + echo "Error: Could not retrieve API client credentials" + echo "Please ensure 'lakekeeper-api' client exists" + exit 1 + fi + + # Get OAuth2 token + TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials" \ + -d "client_id=lakekeeper-api" \ + -d "client_secret=$CLIENT_SECRET" \ + -d "scope=lakekeeper") + + ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token') + if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then + echo "Error: Failed to obtain access token" + echo "Response: $TOKEN_RESPONSE" + exit 1 + fi + + # Get warehouse ID from warehouse name + WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1) + LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d') + + if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then + WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id') + if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then + echo "Error: Warehouse '{{ warehouse_name }}' not found" + echo "Available warehouses:" + echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names" + exit 1 + fi + else + echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)" + echo "Response: $LIST_BODY" + exit 1 + fi + + # If force option is enabled, delete all namespaces first + if [ "{{ force }}" = "true" ]; then + echo "Force deletion enabled. Deleting all namespaces in warehouse '{{ warehouse_name }}'..." + + # List namespaces + NAMESPACE_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + NS_HTTP_CODE=$(echo "$NAMESPACE_RESPONSE" | tail -n1) + NS_BODY=$(echo "$NAMESPACE_RESPONSE" | sed '$d') + + if [ "$NS_HTTP_CODE" -ge 200 ] && [ "$NS_HTTP_CODE" -lt 300 ]; then + # Extract namespace names and delete each one + NAMESPACES=$(echo "$NS_BODY" | jq -r '.namespaces[] | .[0]') + if [ -n "$NAMESPACES" ]; then + echo "Found namespaces to delete:" + echo "$NAMESPACES" | while read -r ns; do + echo " - $ns" + done + + echo "$NAMESPACES" | while read -r ns; do + echo "Deleting namespace '$ns' (including all tables)..." + DEL_RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces/${ns}?recursive=true" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + DEL_HTTP_CODE=$(echo "$DEL_RESPONSE" | tail -n1) + if [ "$DEL_HTTP_CODE" -ge 200 ] && [ "$DEL_HTTP_CODE" -lt 300 ]; then + echo " Namespace '$ns' deleted" + else + DEL_BODY=$(echo "$DEL_RESPONSE" | sed '$d') + echo " Warning: Failed to delete namespace '$ns' (HTTP $DEL_HTTP_CODE)" + echo " Response: $DEL_BODY" + fi + done + else + echo "No namespaces found in warehouse '{{ warehouse_name }}'" + fi + fi + fi + + echo "This will delete the warehouse '{{ warehouse_name }}' and all its data." + if ! gum confirm "Are you sure you want to proceed?"; then + echo "Deletion cancelled" + exit 0 + fi + + echo "Deleting warehouse '{{ warehouse_name }}'..." + + # Delete warehouse + RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \ + "http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse/${WAREHOUSE_ID}" \ + -H "Authorization: Bearer $ACCESS_TOKEN") + + HTTP_CODE=$(echo "$RESPONSE" | tail -n1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "Warehouse '{{ warehouse_name }}' deleted successfully" + elif [ "$HTTP_CODE" = "409" ]; then + echo "Error: Warehouse is not empty (HTTP 409)" + echo "Response: $BODY" + echo "" + echo "The warehouse still contains namespaces or data." + echo "To delete all namespaces automatically, use:" + echo " just lakekeeper::delete-warehouse {{ warehouse_name }} true" + exit 1 + else + echo "Error: Failed to delete warehouse (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi