feat(lakekeeper): create warehouses with STS
This commit is contained in:
@@ -35,6 +35,88 @@ The installation automatically:
|
|||||||
|
|
||||||
Access Lakekeeper at `https://lakekeeper.yourdomain.com` and authenticate via Keycloak.
|
Access Lakekeeper at `https://lakekeeper.yourdomain.com` and authenticate via Keycloak.
|
||||||
|
|
||||||
|
## Warehouse Management
|
||||||
|
|
||||||
|
### Creating Warehouses with Vended Credentials
|
||||||
|
|
||||||
|
Create warehouses with STS (Security Token Service) enabled for automatic temporary credential management:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create warehouse with default name and bucket
|
||||||
|
just lakekeeper::create-warehouse <warehouse-name> <bucket-name>
|
||||||
|
|
||||||
|
# Example: Create 'production' warehouse using 'warehouse' bucket
|
||||||
|
just lakekeeper::create-warehouse production warehouse
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates a warehouse with:
|
||||||
|
|
||||||
|
- **STS enabled** for vended credentials (temporary S3 tokens)
|
||||||
|
- **S3-compatible storage** (MinIO) with path-style access
|
||||||
|
- **Automatic credential rotation** via MinIO STS
|
||||||
|
|
||||||
|
**Prerequisites**:
|
||||||
|
|
||||||
|
- MinIO bucket must exist (create with `just minio::create-bucket <bucket-name>`)
|
||||||
|
- API client credentials must be available in Vault
|
||||||
|
|
||||||
|
**Benefits of Vended Credentials**:
|
||||||
|
|
||||||
|
- No need to distribute static S3 credentials to clients
|
||||||
|
- Automatic credential expiration and rotation
|
||||||
|
- Better security through temporary tokens
|
||||||
|
- Centralized credential management
|
||||||
|
|
||||||
|
### Creating Namespaces
|
||||||
|
|
||||||
|
Namespaces organize tables within a warehouse (similar to databases in traditional systems):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Create Iceberg namespace in a warehouse
|
||||||
|
just lakekeeper::create-warehouse-namespace <warehouse-name> <namespace>
|
||||||
|
|
||||||
|
# Example: Create 'ecommerce' namespace in 'test' warehouse
|
||||||
|
just lakekeeper::create-warehouse-namespace test ecommerce
|
||||||
|
```
|
||||||
|
|
||||||
|
### Managing Warehouses
|
||||||
|
|
||||||
|
List, view, and delete warehouses:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all warehouses
|
||||||
|
just lakekeeper::list-warehouses
|
||||||
|
|
||||||
|
# List all namespaces in a warehouse
|
||||||
|
just lakekeeper::list-warehouse-namespaces <warehouse-name>
|
||||||
|
|
||||||
|
# Example: List namespaces in 'test' warehouse
|
||||||
|
just lakekeeper::list-warehouse-namespaces test
|
||||||
|
|
||||||
|
# Delete a namespace from a warehouse (recursively deletes all tables)
|
||||||
|
just lakekeeper::delete-warehouse-namespace <warehouse-name> <namespace>
|
||||||
|
|
||||||
|
# Example: Delete 'ecommerce' namespace from 'test' warehouse (including all tables)
|
||||||
|
just lakekeeper::delete-warehouse-namespace test ecommerce
|
||||||
|
|
||||||
|
# Delete a warehouse (must be empty)
|
||||||
|
just lakekeeper::delete-warehouse <warehouse-name>
|
||||||
|
|
||||||
|
# Force delete a warehouse (automatically deletes all namespaces first)
|
||||||
|
just lakekeeper::delete-warehouse <warehouse-name> true
|
||||||
|
|
||||||
|
# Example: Force delete 'test' warehouse with all its namespaces
|
||||||
|
just lakekeeper::delete-warehouse test true
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important Notes**:
|
||||||
|
|
||||||
|
- Namespace deletion is **recursive** - it will delete all tables and data within the namespace
|
||||||
|
- Warehouses must be empty before deletion. If a warehouse contains namespaces, you must either:
|
||||||
|
1. Delete each namespace individually using `delete-warehouse-namespace`, then delete the warehouse
|
||||||
|
2. Use force deletion (`delete-warehouse <name> true`) to automatically delete all namespaces and their tables first
|
||||||
|
- All deletion operations require confirmation prompts to prevent accidental data loss
|
||||||
|
|
||||||
## Programmatic Access
|
## Programmatic Access
|
||||||
|
|
||||||
### API Client Credentials
|
### API Client Credentials
|
||||||
@@ -70,13 +152,47 @@ Configure dlt to use the API client credentials:
|
|||||||
export OIDC_CLIENT_ID=lakekeeper-api
|
export OIDC_CLIENT_ID=lakekeeper-api
|
||||||
export OIDC_CLIENT_SECRET=<secret-from-creation>
|
export OIDC_CLIENT_SECRET=<secret-from-creation>
|
||||||
export ICEBERG_CATALOG_URL=http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog
|
export ICEBERG_CATALOG_URL=http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog
|
||||||
export ICEBERG_WAREHOUSE=default
|
export ICEBERG_WAREHOUSE=test # Use warehouse with vended credentials enabled
|
||||||
|
export KEYCLOAK_TOKEN_URL=https://auth.example.com/realms/buunstack/protocol/openid-connect/token
|
||||||
|
export OAUTH2_SCOPE=lakekeeper # Optional, defaults to "lakekeeper"
|
||||||
```
|
```
|
||||||
|
|
||||||
The dlt Iceberg REST destination automatically uses these credentials for OAuth2 authentication.
|
The dlt Iceberg REST destination automatically uses these credentials for OAuth2 authentication and receives temporary S3 credentials via STS (vended credentials).
|
||||||
|
|
||||||
|
**Notes**:
|
||||||
|
|
||||||
|
- `KEYCLOAK_TOKEN_URL` is required because Lakekeeper v0.9.x uses external OAuth2 provider (Keycloak) instead of the deprecated `/v1/oauth/tokens` endpoint.
|
||||||
|
- `OAUTH2_SCOPE` must be set to `lakekeeper` (default) to include the audience claim in JWT tokens. PyIceberg defaults to `catalog` scope, which is not valid for Keycloak.
|
||||||
|
- **No S3 credentials needed** when using warehouses with vended credentials enabled (STS). Lakekeeper provides temporary S3 credentials automatically.
|
||||||
|
|
||||||
|
#### Legacy Mode: Static S3 Credentials
|
||||||
|
|
||||||
|
If using a warehouse with `vended-credentials-enabled=false`, you need to provide static S3 credentials:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Additional environment variables for static credentials mode
|
||||||
|
export S3_ENDPOINT_URL=http://minio.minio.svc.cluster.local:9000
|
||||||
|
export S3_ACCESS_KEY_ID=<minio-access-key>
|
||||||
|
export S3_SECRET_ACCESS_KEY=<minio-secret-key>
|
||||||
|
```
|
||||||
|
|
||||||
|
To get MinIO credentials:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just vault::get minio/dlt access_key
|
||||||
|
just vault::get minio/dlt secret_key
|
||||||
|
```
|
||||||
|
|
||||||
|
Or create a dedicated MinIO user:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just minio::create-user dlt "dlt-data"
|
||||||
|
```
|
||||||
|
|
||||||
#### PyIceberg
|
#### PyIceberg
|
||||||
|
|
||||||
|
With vended credentials (recommended):
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from pyiceberg.catalog import load_catalog
|
from pyiceberg.catalog import load_catalog
|
||||||
|
|
||||||
@@ -84,8 +200,30 @@ catalog = load_catalog(
|
|||||||
"rest_catalog",
|
"rest_catalog",
|
||||||
**{
|
**{
|
||||||
"uri": "http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog",
|
"uri": "http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog",
|
||||||
"warehouse": "default",
|
"warehouse": "test", # Use warehouse with vended credentials enabled
|
||||||
"credential": f"{client_id}:{client_secret}", # OAuth2 format
|
"credential": f"{client_id}:{client_secret}", # OAuth2 format
|
||||||
|
"oauth2-server-uri": "https://auth.example.com/realms/buunstack/protocol/openid-connect/token",
|
||||||
|
"scope": "lakekeeper", # Required for Keycloak (PyIceberg defaults to "catalog")
|
||||||
|
}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
With static S3 credentials (legacy mode):
|
||||||
|
|
||||||
|
```python
|
||||||
|
catalog = load_catalog(
|
||||||
|
"rest_catalog",
|
||||||
|
**{
|
||||||
|
"uri": "http://lakekeeper.lakekeeper.svc.cluster.local:8181/catalog",
|
||||||
|
"warehouse": "default",
|
||||||
|
"credential": f"{client_id}:{client_secret}",
|
||||||
|
"oauth2-server-uri": "https://auth.example.com/realms/buunstack/protocol/openid-connect/token",
|
||||||
|
"scope": "lakekeeper",
|
||||||
|
# Static S3 credentials (only needed when vended credentials disabled)
|
||||||
|
"s3.endpoint": "http://minio.minio.svc.cluster.local:9000",
|
||||||
|
"s3.access-key-id": "<minio-access-key>",
|
||||||
|
"s3.secret-access-key": "<minio-secret-key>",
|
||||||
|
"s3.path-style-access": "true",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -292,3 +292,481 @@ cleanup:
|
|||||||
else
|
else
|
||||||
echo "Cleanup cancelled"
|
echo "Cleanup cancelled"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Create warehouse with vended credentials enabled (STS)
|
||||||
|
create-warehouse warehouse_name='default' bucket='warehouse':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Creating warehouse '{{ warehouse_name }}' with vended credentials (STS) enabled..."
|
||||||
|
|
||||||
|
# Get MinIO credentials
|
||||||
|
MINIO_ACCESS_KEY=$(kubectl get secret -n minio minio -o jsonpath='{.data.rootUser}' | base64 -d)
|
||||||
|
MINIO_SECRET_KEY=$(kubectl get secret -n minio minio -o jsonpath='{.data.rootPassword}' | base64 -d)
|
||||||
|
|
||||||
|
# Create warehouse JSON configuration
|
||||||
|
WAREHOUSE_CONFIG=$(cat <<EOF
|
||||||
|
{
|
||||||
|
"warehouse-name": "{{ warehouse_name }}",
|
||||||
|
"storage-credential": {
|
||||||
|
"type": "s3",
|
||||||
|
"aws-access-key-id": "$MINIO_ACCESS_KEY",
|
||||||
|
"aws-secret-access-key": "$MINIO_SECRET_KEY",
|
||||||
|
"credential-type": "access-key"
|
||||||
|
},
|
||||||
|
"storage-profile": {
|
||||||
|
"type": "s3",
|
||||||
|
"bucket": "{{ bucket }}",
|
||||||
|
"region": "us-east-1",
|
||||||
|
"sts-enabled": true,
|
||||||
|
"flavor": "s3-compat",
|
||||||
|
"endpoint": "http://minio.minio:9000",
|
||||||
|
"path-style-access": true,
|
||||||
|
"key-prefix": "{{ warehouse_name }}"
|
||||||
|
},
|
||||||
|
"delete-profile": {
|
||||||
|
"type": "hard"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
echo "Authenticating with Keycloak..."
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create warehouse
|
||||||
|
echo "Creating warehouse..."
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "$WAREHOUSE_CONFIG")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Warehouse '{{ warehouse_name }}' created successfully with vended credentials enabled"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
else
|
||||||
|
echo "Error: Failed to create warehouse (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create Iceberg namespace in a warehouse
|
||||||
|
create-warehouse-namespace warehouse_name namespace:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Creating namespace '{{ namespace }}' in warehouse '{{ warehouse_name }}'..."
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
echo "Authenticating with Keycloak..."
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get warehouse ID from warehouse name
|
||||||
|
echo "Getting warehouse ID for '{{ warehouse_name }}'..."
|
||||||
|
WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1)
|
||||||
|
LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then
|
||||||
|
WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id')
|
||||||
|
if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then
|
||||||
|
echo "Error: Warehouse '{{ warehouse_name }}' not found"
|
||||||
|
echo "Available warehouses:"
|
||||||
|
echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Warehouse ID: $WAREHOUSE_ID"
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)"
|
||||||
|
echo "Response: $LIST_BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Create namespace
|
||||||
|
echo "Creating namespace..."
|
||||||
|
NAMESPACE_CONFIG=$(cat <<EOF
|
||||||
|
{
|
||||||
|
"namespace": ["{{ namespace }}"],
|
||||||
|
"properties": {}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "$NAMESPACE_CONFIG")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Namespace '{{ namespace }}' created successfully in warehouse '{{ warehouse_name }}'"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
elif [ "$HTTP_CODE" = "409" ]; then
|
||||||
|
echo "Namespace '{{ namespace }}' already exists in warehouse '{{ warehouse_name }}'"
|
||||||
|
else
|
||||||
|
echo "Error: Failed to create namespace (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# List all warehouses
|
||||||
|
list-warehouses:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Listing all warehouses..."
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# List warehouses
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Warehouses:"
|
||||||
|
echo "$BODY" | jq -r '.warehouses[] | " - \(.name) (ID: \(.id))"'
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list warehouses (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Delete namespace from a warehouse
|
||||||
|
delete-warehouse-namespace warehouse_name namespace:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "This will delete namespace '{{ namespace }}' from warehouse '{{ warehouse_name }}'."
|
||||||
|
if ! gum confirm "Are you sure you want to proceed?"; then
|
||||||
|
echo "Deletion cancelled"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Deleting namespace '{{ namespace }}' from warehouse '{{ warehouse_name }}'..."
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get warehouse ID from warehouse name
|
||||||
|
WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1)
|
||||||
|
LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then
|
||||||
|
WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id')
|
||||||
|
if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then
|
||||||
|
echo "Error: Warehouse '{{ warehouse_name }}' not found"
|
||||||
|
echo "Available warehouses:"
|
||||||
|
echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)"
|
||||||
|
echo "Response: $LIST_BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Delete namespace with recursive flag to delete all tables
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces/{{ namespace }}?recursive=true" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Namespace '{{ namespace }}' deleted successfully from warehouse '{{ warehouse_name }}'"
|
||||||
|
elif [ "$HTTP_CODE" = "404" ]; then
|
||||||
|
echo "Namespace '{{ namespace }}' not found in warehouse '{{ warehouse_name }}'"
|
||||||
|
else
|
||||||
|
echo "Error: Failed to delete namespace (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# List all namespaces in a warehouse
|
||||||
|
list-warehouse-namespaces warehouse_name:
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
echo "Listing namespaces in warehouse '{{ warehouse_name }}'..."
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get warehouse ID from warehouse name
|
||||||
|
WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1)
|
||||||
|
LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then
|
||||||
|
WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id')
|
||||||
|
if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then
|
||||||
|
echo "Error: Warehouse '{{ warehouse_name }}' not found"
|
||||||
|
echo "Available warehouses:"
|
||||||
|
echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)"
|
||||||
|
echo "Response: $LIST_BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# List namespaces
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Namespaces in warehouse '{{ warehouse_name }}':"
|
||||||
|
echo "$BODY" | jq -r '.namespaces[] | " - \(.[0])"'
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list namespaces (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Delete warehouse
|
||||||
|
delete-warehouse warehouse_name force='false':
|
||||||
|
#!/bin/bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
# Get API client credentials for authentication
|
||||||
|
CLIENT_SECRET=$(just vault::get lakekeeper/api-client/lakekeeper-api client_secret 2>/dev/null || echo "")
|
||||||
|
if [ -z "$CLIENT_SECRET" ]; then
|
||||||
|
echo "Error: Could not retrieve API client credentials"
|
||||||
|
echo "Please ensure 'lakekeeper-api' client exists"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get OAuth2 token
|
||||||
|
TOKEN_RESPONSE=$(curl -s -X POST "https://${KEYCLOAK_HOST}/realms/${KEYCLOAK_REALM}/protocol/openid-connect/token" \
|
||||||
|
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||||
|
-d "grant_type=client_credentials" \
|
||||||
|
-d "client_id=lakekeeper-api" \
|
||||||
|
-d "client_secret=$CLIENT_SECRET" \
|
||||||
|
-d "scope=lakekeeper")
|
||||||
|
|
||||||
|
ACCESS_TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
|
||||||
|
if [ "$ACCESS_TOKEN" = "null" ] || [ -z "$ACCESS_TOKEN" ]; then
|
||||||
|
echo "Error: Failed to obtain access token"
|
||||||
|
echo "Response: $TOKEN_RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Get warehouse ID from warehouse name
|
||||||
|
WAREHOUSE_LIST_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
LIST_HTTP_CODE=$(echo "$WAREHOUSE_LIST_RESPONSE" | tail -n1)
|
||||||
|
LIST_BODY=$(echo "$WAREHOUSE_LIST_RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$LIST_HTTP_CODE" -ge 200 ] && [ "$LIST_HTTP_CODE" -lt 300 ]; then
|
||||||
|
WAREHOUSE_ID=$(echo "$LIST_BODY" | jq -r '.warehouses[] | select(.name == "{{ warehouse_name }}") | .id')
|
||||||
|
if [ -z "$WAREHOUSE_ID" ] || [ "$WAREHOUSE_ID" = "null" ]; then
|
||||||
|
echo "Error: Warehouse '{{ warehouse_name }}' not found"
|
||||||
|
echo "Available warehouses:"
|
||||||
|
echo "$LIST_BODY" | jq -r '.warehouses[] | .name' 2>/dev/null || echo "Could not parse warehouse names"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Error: Failed to list warehouses (HTTP $LIST_HTTP_CODE)"
|
||||||
|
echo "Response: $LIST_BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# If force option is enabled, delete all namespaces first
|
||||||
|
if [ "{{ force }}" = "true" ]; then
|
||||||
|
echo "Force deletion enabled. Deleting all namespaces in warehouse '{{ warehouse_name }}'..."
|
||||||
|
|
||||||
|
# List namespaces
|
||||||
|
NAMESPACE_RESPONSE=$(curl -s -w "\n%{http_code}" -X GET \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
NS_HTTP_CODE=$(echo "$NAMESPACE_RESPONSE" | tail -n1)
|
||||||
|
NS_BODY=$(echo "$NAMESPACE_RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$NS_HTTP_CODE" -ge 200 ] && [ "$NS_HTTP_CODE" -lt 300 ]; then
|
||||||
|
# Extract namespace names and delete each one
|
||||||
|
NAMESPACES=$(echo "$NS_BODY" | jq -r '.namespaces[] | .[0]')
|
||||||
|
if [ -n "$NAMESPACES" ]; then
|
||||||
|
echo "Found namespaces to delete:"
|
||||||
|
echo "$NAMESPACES" | while read -r ns; do
|
||||||
|
echo " - $ns"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "$NAMESPACES" | while read -r ns; do
|
||||||
|
echo "Deleting namespace '$ns' (including all tables)..."
|
||||||
|
DEL_RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/catalog/v1/${WAREHOUSE_ID}/namespaces/${ns}?recursive=true" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
DEL_HTTP_CODE=$(echo "$DEL_RESPONSE" | tail -n1)
|
||||||
|
if [ "$DEL_HTTP_CODE" -ge 200 ] && [ "$DEL_HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo " Namespace '$ns' deleted"
|
||||||
|
else
|
||||||
|
DEL_BODY=$(echo "$DEL_RESPONSE" | sed '$d')
|
||||||
|
echo " Warning: Failed to delete namespace '$ns' (HTTP $DEL_HTTP_CODE)"
|
||||||
|
echo " Response: $DEL_BODY"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
else
|
||||||
|
echo "No namespaces found in warehouse '{{ warehouse_name }}'"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "This will delete the warehouse '{{ warehouse_name }}' and all its data."
|
||||||
|
if ! gum confirm "Are you sure you want to proceed?"; then
|
||||||
|
echo "Deletion cancelled"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Deleting warehouse '{{ warehouse_name }}'..."
|
||||||
|
|
||||||
|
# Delete warehouse
|
||||||
|
RESPONSE=$(curl -s -w "\n%{http_code}" -X DELETE \
|
||||||
|
"http://lakekeeper.${LAKEKEEPER_NAMESPACE}.svc.cluster.local:8181/management/v1/warehouse/${WAREHOUSE_ID}" \
|
||||||
|
-H "Authorization: Bearer $ACCESS_TOKEN")
|
||||||
|
|
||||||
|
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)
|
||||||
|
BODY=$(echo "$RESPONSE" | sed '$d')
|
||||||
|
|
||||||
|
if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then
|
||||||
|
echo "Warehouse '{{ warehouse_name }}' deleted successfully"
|
||||||
|
elif [ "$HTTP_CODE" = "409" ]; then
|
||||||
|
echo "Error: Warehouse is not empty (HTTP 409)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
echo ""
|
||||||
|
echo "The warehouse still contains namespaces or data."
|
||||||
|
echo "To delete all namespaces automatically, use:"
|
||||||
|
echo " just lakekeeper::delete-warehouse {{ warehouse_name }} true"
|
||||||
|
exit 1
|
||||||
|
else
|
||||||
|
echo "Error: Failed to delete warehouse (HTTP $HTTP_CODE)"
|
||||||
|
echo "Response: $BODY"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|||||||
Reference in New Issue
Block a user