diff --git a/minio/README.md b/minio/README.md new file mode 100644 index 0000000..dda1a09 --- /dev/null +++ b/minio/README.md @@ -0,0 +1,608 @@ +# MinIO + +High-performance S3-compatible object storage for Kubernetes: + +- **MinIO**: S3-compatible object storage server +- **Keycloak OIDC Integration**: Centralized authentication via OpenID Connect +- **Policy-based Access Control**: Fine-grained permissions using MinIO policies +- **Bucket Management**: Create and manage storage buckets +- **User Management**: Create service accounts with dedicated buckets + +## Prerequisites + +- Kubernetes cluster (k3s) +- Keycloak (for OIDC authentication) +- External Secrets Operator (optional, for Vault integration) +- Vault (optional, for credential storage) +- Storage provisioner (e.g., Longhorn) + +## Installation + +```bash +just minio::install +``` + +You will be prompted for: + +1. **MinIO host (FQDN)**: e.g., `minio.example.com` (API endpoint) +2. **MinIO Console host (FQDN)**: e.g., `minio-console.example.com` (Web UI) + +### What Gets Installed + +1. MinIO server in standalone mode +2. Keycloak OIDC client with `minioPolicy` attribute mapper +3. Root credentials stored in Kubernetes Secret (and optionally Vault) +4. Ingress for both API and Console endpoints +5. Persistent volume for data storage + +The stack uses the official [MinIO Helm Chart](https://github.com/minio/minio/tree/master/helm/minio). + +## Pod Security Standards + +The minio namespace uses **restricted** Pod Security Standard enforcement. + +```bash +pod-security.kubernetes.io/enforce=restricted +``` + +### Security Context + +**Pod Security Context**: + +- `runAsUser: 1000` +- `runAsGroup: 1000` +- `fsGroup: 1000` +- `fsGroupChangePolicy: OnRootMismatch` +- `seccompProfile.type: RuntimeDefault` + +**Container Security Context**: + +- `allowPrivilegeEscalation: false` +- `capabilities.drop: [ALL]` +- `runAsNonRoot: true` +- `seccompProfile.type: RuntimeDefault` + +**Note**: `readOnlyRootFilesystem: false` is required because MinIO needs to write data and temporary files. + +### Init Jobs Security + +The `makeUserJob` and `makePolicyJob` init jobs also apply the same restricted security context to comply with Pod Security Standards. + +## Access + +### Web Console + +Access MinIO Console at `https://your-minio-console-host/` + +**Authentication**: Login with Keycloak (OIDC) + +### Root Credentials + +For administrative access: + +- Username: Retrieved via `just minio::root-username` +- Password: Retrieved via `just minio::root-password` + +### API Endpoint + +S3-compatible API: `https://your-minio-host/` + +## Configuration + +Environment variables (set in `.env.local` or override): + +```bash +MINIO_NAMESPACE=minio # Kubernetes namespace +MINIO_CHART_VERSION=5.4.0 # MinIO Helm chart version +MINIO_OIDC_CLIENT_ID=minio # Keycloak client ID +MINIO_STORAGE_SIZE=50Gi # Persistent volume size +MINIO_HOST= # MinIO API FQDN +MINIO_CONSOLE_HOST= # MinIO Console FQDN +KEYCLOAK_REALM=buunstack # Keycloak realm +``` + +## MinIO Client (mc) Setup + +The MinIO Client (`mc`) provides a command-line interface for managing MinIO. + +### Install mc + +```bash +# macOS +brew install minio-mc + +# Linux +curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o $HOME/bin/mc +chmod +x $HOME/bin/mc +``` + +### Configure mc Alias + +#### For Regular Users (Recommended) + +The recommended way to create credentials is through the MinIO Console web UI: + +1. **Login to MinIO Console**: Navigate to `https://your-minio-console-host/` +2. **Authenticate with Keycloak**: Click "Login with Keycloak" +3. **Create Access Key**: + - Go to **User** → **Access Keys** + - Click **Create access key** + - Copy the **Access Key** and **Secret Key** (shown only once!) + - Optionally set expiration and policy restrictions + - Save these credentials securely + +4. **Configure mc alias**: + +```bash +mc alias set myminio https://your-minio-host +``` + +**Note**: Access Keys created via Console are not stored anywhere by the system. Save them immediately when created. + +#### For Programmatic Service Accounts + +If you created a service account using `just minio::create-user` (for application use), the credentials are automatically stored in Vault: + +```bash +# Set up Vault access +export VAULT_ADDR="https://your-vault-host" +just vault::setup-token + +# Extract credentials from Vault +ACCESS_KEY=$(vault kv get -mount=secret -field=access_key myuser/minio) +SECRET_KEY=$(vault kv get -mount=secret -field=secret_key myuser/minio) +BUCKET=$(vault kv get -mount=secret -field=bucket myuser/minio) + +# Configure mc alias +mc alias set myuser-minio https://your-minio-host ${ACCESS_KEY} ${SECRET_KEY} +``` + +### Common mc Commands + +```bash +# List buckets +mc ls myminio + +# Create bucket +mc mb myminio/mybucket + +# Upload file +mc cp myfile.txt myminio/mybucket/ + +# Download file +mc cp myminio/mybucket/myfile.txt ./ + +# Remove file +mc rm myminio/mybucket/myfile.txt + +# List files in bucket +mc ls myminio/mybucket + +# Copy directory recursively +mc cp --recursive mydir/ myminio/mybucket/mydir/ + +# Set bucket policy (public read) +mc anonymous set download myminio/mybucket + +# Set bucket policy (private) +mc anonymous set none myminio/mybucket + +# Mirror local directory to bucket +mc mirror localdir/ myminio/mybucket/ + +# Get bucket versioning status +mc version info myminio/mybucket + +# Enable bucket versioning +mc version enable myminio/mybucket +``` + +## Bucket Management + +### Create Bucket + +Using Just recipes: + +```bash +just minio::create-bucket mybucket +``` + +Using mc: + +```bash +mc mb myminio/mybucket +``` + +### Check if Bucket Exists + +```bash +just minio::bucket-exists mybucket +``` + +This returns exit code 0 if the bucket exists, 1 otherwise. + +## User Management + +### Create MinIO User + +Create a MinIO user with dedicated bucket: + +```bash +just minio::create-user user=myuser bucket=mybucket +``` + +This will: + +1. Generate access key and secret key +2. Create the bucket +3. Create MinIO user with readwrite policy +4. Store credentials in Vault (if External Secrets is available) + +**Interactive mode** (prompts for username): + +```bash +just minio::create-user +``` + +The bucket defaults to `{username}-storage` if not specified. + +### Get Service Account Credentials + +For programmatically created service accounts (via `just minio::create-user`), retrieve credentials from Vault: + +```bash +# Set up Vault access +export VAULT_ADDR="https://your-vault-host" +just vault::setup-token + +# Get all stored information +vault kv get -mount=secret myuser/minio + +# Or get specific fields +vault kv get -mount=secret -field=access_key myuser/minio +vault kv get -mount=secret -field=secret_key myuser/minio +vault kv get -mount=secret -field=bucket myuser/minio +vault kv get -mount=secret -field=endpoint myuser/minio +``` + +**Note**: Regular user Access Keys created via MinIO Console are not stored in Vault. + +### Grant Policy to User + +Change user permissions: + +```bash +just minio::grant-policy user=myuser policy=readonly +``` + +Available policies: + +- `readwrite`: Full read and write access +- `readonly`: Read-only access +- `writeonly`: Write-only access + +## OIDC Authentication + +MinIO uses Keycloak for web console authentication via OIDC. + +### MinIO Policy Claim + +Users authenticate via Keycloak and receive MinIO policies based on the `minioPolicy` user attribute. + +**Default Policy**: `readwrite` + +**Available Policies**: + +- `readwrite`: Full access +- `readonly`: Read-only access +- `writeonly`: Write-only access + +### Set User Policy via Keycloak + +The `minioPolicy` attribute is automatically added to users upon client creation. To modify: + +1. Login to Keycloak Admin Console +2. Navigate to Users → Select User → Attributes +3. Set `minioPolicy` to desired policy (e.g., `readonly`) + +## Admin Operations + +### List Policies and Users + +Debug information about MinIO internal state: + +```bash +just minio::debug-info +``` + +This shows: + +- All MinIO policies +- All MinIO users + +### Using mc Admin Commands + +```bash +# List users +mc admin user list myminio + +# List policies +mc admin policy list myminio + +# Create policy from file +mc admin policy create myminio mypolicy /path/to/policy.json + +# Attach policy to user +mc admin policy attach myminio mypolicy --user=myuser + +# Server info +mc admin info myminio + +# Server stats +mc admin top locks myminio +``` + +## S3 API Usage + +### AWS CLI + +Configure AWS CLI to use MinIO: + +```bash +# Configure profile +aws configure --profile minio +# Enter: +# - AWS Access Key ID: (MinIO access key) +# - AWS Secret Access Key: (MinIO secret key) +# - Region: us-east-1 (default) +# - Output format: json + +# Use with custom endpoint +aws --profile minio --endpoint-url https://your-minio-host s3 ls + +# List buckets +aws --profile minio --endpoint-url https://your-minio-host s3 ls + +# Upload file +aws --profile minio --endpoint-url https://your-minio-host \ + s3 cp myfile.txt s3://mybucket/ + +# Download file +aws --profile minio --endpoint-url https://your-minio-host \ + s3 cp s3://mybucket/myfile.txt ./ +``` + +### Python (boto3) + +```python +import boto3 + +s3 = boto3.client( + 's3', + endpoint_url='https://your-minio-host', + aws_access_key_id='your-access-key', + aws_secret_access_key='your-secret-key', + region_name='us-east-1' +) + +# List buckets +response = s3.list_buckets() +for bucket in response['Buckets']: + print(bucket['Name']) + +# Upload file +s3.upload_file('myfile.txt', 'mybucket', 'myfile.txt') + +# Download file +s3.download_file('mybucket', 'myfile.txt', 'downloaded.txt') +``` + +### Environment Variables for Applications + +```bash +# S3-compatible endpoint +AWS_ENDPOINT_URL=https://your-minio-host + +# Or service-internal endpoint (from pods) +AWS_ENDPOINT_URL=http://minio.minio.svc.cluster.local:9000 + +# Credentials +AWS_ACCESS_KEY_ID=your-access-key +AWS_SECRET_ACCESS_KEY=your-secret-key +AWS_REGION=us-east-1 + +# MinIO-specific (some libraries) +MINIO_ENDPOINT=your-minio-host +MINIO_ACCESS_KEY=your-access-key +MINIO_SECRET_KEY=your-secret-key +MINIO_BUCKET=mybucket +``` + +## Integration Patterns + +### Application with Dedicated Bucket + +1. **Create MinIO service account and bucket**: + + ```bash + just minio::create-user user=myapp + ``` + + This will output the credentials and store them in Vault (if External Secrets is available). + +2. **Get credentials from Vault** (if needed later): + + ```bash + # Set up Vault access + export VAULT_ADDR="https://your-vault-host" + just vault::setup-token + + # Get credentials + ACCESS_KEY=$(vault kv get -mount=secret -field=access_key myapp/minio) + SECRET_KEY=$(vault kv get -mount=secret -field=secret_key myapp/minio) + BUCKET=$(vault kv get -mount=secret -field=bucket myapp/minio) + ``` + +3. **Create Kubernetes Secret** (if not using External Secrets): + + ```bash + kubectl create secret generic myapp-minio -n myapp \ + --from-literal=access-key=myapp \ + --from-literal=secret-key= \ + --from-literal=bucket=myapp-storage \ + --from-literal=endpoint=http://minio.minio.svc.cluster.local:9000 + ``` + +4. **Mount in application**: + + ```yaml + env: + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: myapp-minio + key: access-key + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: myapp-minio + key: secret-key + - name: AWS_ENDPOINT_URL + valueFrom: + secretKeyRef: + name: myapp-minio + key: endpoint + - name: S3_BUCKET + valueFrom: + secretKeyRef: + name: myapp-minio + key: bucket + ``` + +### External Secrets Integration + +If using External Secrets Operator: + +```yaml +apiVersion: external-secrets.io/v1beta1 +kind: ExternalSecret +metadata: + name: myapp-minio + namespace: myapp +spec: + refreshInterval: 1h + secretStoreRef: + name: vault-backend + kind: ClusterSecretStore + target: + name: myapp-minio + creationPolicy: Owner + data: + - secretKey: access-key + remoteRef: + key: myapp/minio + property: access_key + - secretKey: secret-key + remoteRef: + key: myapp/minio + property: secret_key + - secretKey: bucket + remoteRef: + key: myapp/minio + property: bucket + - secretKey: endpoint + remoteRef: + key: myapp/minio + property: endpoint +``` + +## Troubleshooting + +### Check MinIO Pod Status + +```bash +kubectl get pods -n minio +``` + +### View MinIO Logs + +```bash +kubectl logs -n minio deploy/minio +``` + +### Test S3 Connectivity + +```bash +# From inside cluster +kubectl run -it --rm debug --image=minio/mc --restart=Never -- \ + mc alias set test http://minio.minio.svc.cluster.local:9000 \ + $(just minio::root-username) \ + $(just minio::root-password) + +kubectl run -it --rm debug --image=minio/mc --restart=Never -- \ + mc ls test +``` + +### Check OIDC Configuration + +```bash +# Verify Keycloak client +just keycloak::get-client buunstack minio + +# Check MinIO environment +kubectl get deployment minio -n minio -o jsonpath='{.spec.template.spec.containers[0].env}' | jq +``` + +### Reset Root Credentials + +```bash +# Delete existing secret +kubectl delete secret minio -n minio + +# Recreate credentials +just minio::create-root-credentials +``` + +### Verify Bucket Permissions + +```bash +# List bucket policy +mc anonymous list myminio/mybucket + +# Check user policy +mc admin policy info myminio readwrite +``` + +## Management + +### Uninstall MinIO + +```bash +just minio::uninstall +``` + +This removes: + +- MinIO deployment +- MinIO namespace +- Keycloak client +- **Note**: PersistentVolumeClaim is also deleted, losing all data + +### Backup Before Uninstall + +```bash +# Mirror all buckets to local directory +mc mirror myminio/ ./minio-backup/ + +# Or use specific bucket +mc mirror myminio/important-bucket/ ./backup/ +``` + +## References + +- [MinIO Documentation](https://min.io/docs/minio/linux/index.html) +- [MinIO Client (mc) Guide](https://min.io/docs/minio/linux/reference/minio-mc.html) +- [MinIO Helm Chart](https://github.com/minio/minio/tree/master/helm/minio) +- [MinIO OIDC Identity Management](https://min.io/docs/minio/linux/operations/external-iam/configure-openid-external-identity-management.html) +- [S3 API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) +- [Resource Management Best Practices](../docs/resource-management.md) diff --git a/minio/justfile b/minio/justfile index eb53323..5473047 100644 --- a/minio/justfile +++ b/minio/justfile @@ -100,6 +100,10 @@ install: redirect_url="https://${MINIO_HOST}/oauth_callback,https://${MINIO_CONSOLE_HOST}/oauth_callback" just add-keycloak-minio-policy just create-namespace + + kubectl label namespace ${MINIO_NAMESPACE} \ + pod-security.kubernetes.io/enforce=restricted --overwrite + just create-root-credentials just add-helm-repo gomplate -f minio-values.gomplate.yaml -o minio-values.yaml @@ -229,24 +233,6 @@ create-user user='' bucket='': echo "✅ MinIO user and bucket created for ${USER}" -# Get MinIO credentials from Vault -get-user-credentials user='': - #!/bin/bash - set -euo pipefail - USER="{{ user }}" - - while [ -z "${USER}" ]; do - USER=$(gum input --prompt="Username: " --width=100 --placeholder="e.g., airbyte") - done - - if helm status external-secrets -n ${EXTERNAL_SECRETS_NAMESPACE} &>/dev/null; then - echo "Getting MinIO credentials for ${USER} from Vault..." - just vault::get ${USER}/minio - else - echo "External Secrets not available. Please check the credentials manually." - exit 1 - fi - # Grant policy to user for specific operations grant-policy user='' policy='': #!/bin/bash diff --git a/minio/minio-values.gomplate.yaml b/minio/minio-values.gomplate.yaml index c2cb778..445f986 100644 --- a/minio/minio-values.gomplate.yaml +++ b/minio/minio-values.gomplate.yaml @@ -49,3 +49,68 @@ resources: limits: cpu: 100m memory: 1Gi + +# Security context for Pod Security Standards (restricted) +securityContext: + enabled: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + fsGroupChangePolicy: "OnRootMismatch" + seccompProfile: + type: RuntimeDefault + +containerSecurityContext: + readOnlyRootFilesystem: false + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + capabilities: + drop: + - ALL + +# Security context for init jobs +makeUserJob: + securityContext: + enabled: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + fsGroupChangePolicy: "OnRootMismatch" + seccompProfile: + type: RuntimeDefault + containerSecurityContext: + readOnlyRootFilesystem: false + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + capabilities: + drop: + - ALL + +makePolicyJob: + securityContext: + enabled: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + fsGroupChangePolicy: "OnRootMismatch" + seccompProfile: + type: RuntimeDefault + containerSecurityContext: + readOnlyRootFilesystem: false + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + capabilities: + drop: + - ALL