# Pi Cluster - Kubernetes on Talos Linux

## Prerequisites
- `talosctl` installed
- `kubectl` installed
- `helm` installed
- DNS record for `controlplane.cdb-online.co.uk` pointing to VIP `10.0.1.11`

---

## 1. Generate Configuration

### Generate secrets only
```bash
talosctl gen secrets -o secrets.yaml
```

### Generate full config using secrets
```bash
talosctl gen config pi-cluster https://controlplane.cdb-online.co.uk:6443 \
  --with-secrets secrets.yaml \
  --config-patch @patches/t1-vip-patch.yaml \
  --output-dir ./
```

> `secrets.yaml` contains your cluster CA, tokens, and keys. Keep this safe and never commit it to git.

---

## 2. Validate Config
```bash
talosctl validate --config controlplane.yaml --mode metal
talosctl validate --config worker.yaml --mode metal
```

---

## 3. Apply Configuration

### Controlplane nodes (one at a time)
```bash
talosctl apply-config --nodes 10.0.1.12 --endpoints 10.0.1.12 --file controlplane.yaml
talosctl apply-config --nodes 10.0.1.13 --endpoints 10.0.1.13 --file controlplane.yaml
talosctl apply-config --nodes 10.0.1.14 --endpoints 10.0.1.14 --file controlplane.yaml
```

### Worker nodes
```bash
talosctl apply-config --nodes 10.0.1.21 --endpoints 10.0.1.21 --file worker.yaml
talosctl apply-config --nodes 10.0.1.22 --endpoints 10.0.1.22 --file worker.yaml
talosctl apply-config --nodes 10.0.1.23 --endpoints 10.0.1.23 --file worker.yaml
```

---

## 4. Bootstrap etcd
Only run this ONCE on the first controlplane node:
```bash
talosctl bootstrap --nodes 10.0.1.12 --endpoints 10.0.1.12
```

---

## 5. Watch Cluster Come Up
```bash
talosctl health --nodes 10.0.1.12 --endpoints 10.0.1.12
```

Wait for VIP `10.0.1.11` to appear before proceeding:
```bash
ping 10.0.1.11
```

---

## 6. Export Kubeconfig

### One time export
```bash
talosctl --nodes controlplane.cdb-online.co.uk \
  --endpoints controlplane.cdb-online.co.uk \
  kubeconfig ~/.kube/config
```

### Make it permanent - add to ~/.bashrc or ~/.zshrc
```bash
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc
```

### Verify kubectl is working
```bash
kubectl cluster-info
kubectl get nodes
```

---

## 7. Install MetalLB
```bash
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.3/config/manifests/metallb-native.yaml
```

Wait for MetalLB to be ready:
```bash
kubectl wait --namespace metallb-system \
  --for=condition=ready pod \
  --selector=app=metallb \
  --timeout=90s
```

Apply IP pool:
```bash
kubectl apply -f metallb/metallb.yaml
```

---

## 8. Deploy Applications
```bash
# Create namespaces and storage
kubectl apply -f deployments/grav-www-paulcdb-com.yaml
kubectl apply -f deployments/grav-www-cdb-online-co-uk.yaml
kubectl apply -f deployments/grav-www-holyislandarc-club.yaml
kubectl apply -f deployments/vaultwarden.yaml
kubectl apply -f deployments/uptimekuma.yaml
kubectl apply -f deployments/pairdrop.yaml
kubectl apply -f deployments/it-tools.yaml
kubectl apply -f deployments/forgejo.yaml
kubectl apply -f deployments/changedetection.yaml
```

Watch pods come up:
```bash
kubectl get pods -A -w
```

Check services have IPs:
```bash
kubectl get svc -A | grep LoadBalancer
```

---

## 9. Install Monitoring (kube-prometheus-stack)
```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values monitoring/kube-prometheus-values.yaml
```

Watch monitoring come up:
```bash
kubectl get pods -n monitoring -w
```

---

## 10. Updating Talos

### Bump version in controlplane.yaml and worker.yaml
```yaml
machine:
  install:
    image: ghcr.io/siderolabs/installer:v1.14.0
```

### Apply to controlplane nodes one at a time
```bash
talosctl apply-config --nodes 10.0.1.12 --endpoints 10.0.1.12 --file controlplane.yaml
# Wait for node to come back before doing the next one
talosctl apply-config --nodes 10.0.1.13 --endpoints 10.0.1.13 --file controlplane.yaml
talosctl apply-config --nodes 10.0.1.14 --endpoints 10.0.1.14 --file controlplane.yaml
```

### Apply to worker nodes
```bash
talosctl apply-config --nodes 10.0.1.21 --endpoints 10.0.1.21 --file worker.yaml
talosctl apply-config --nodes 10.0.1.22 --endpoints 10.0.1.22 --file worker.yaml
talosctl apply-config --nodes 10.0.1.23 --endpoints 10.0.1.23 --file worker.yaml
```

---

## 11. Updating Containers

### Restart a deployment to pull latest image
```bash
kubectl rollout restart deployment <name> -n <namespace>
```

### Apply updated yaml with new version
```bash
kubectl apply -f deployments/<name>.yaml
```

### Check rollout status
```bash
kubectl rollout status deployment <name> -n <namespace>
```

---

## 12. Using Secrets in Deployments

### Create a secret
```bash
kubectl create secret generic <name> \
  --from-literal=KEY=value \
  --namespace <namespace>
```

### Reference in deployment yaml
```yaml
env:
  - name: MY_PASSWORD
    valueFrom:
      secretKeyRef:
        name: <secret-name>
        key: KEY
```

### View existing secrets
```bash
kubectl get secrets -n <namespace>
kubectl describe secret <name> -n <namespace>
```

---

## 13. Useful Commands

### Talos
```bash
# Dashboard
talosctl dashboard --nodes 10.0.1.12 --endpoints 10.0.1.12

# Logs
talosctl logs etcd --nodes 10.0.1.12 --endpoints 10.0.1.12

# Service status
talosctl service etcd --nodes 10.0.1.12 --endpoints 10.0.1.12

# Node health
talosctl health --nodes 10.0.1.12 --endpoints 10.0.1.12
```

### Kubernetes
```bash
# Get all pods
kubectl get pods -A

# Get all services
kubectl get svc -A

# Describe a pod
kubectl describe pod <pod-name> -n <namespace>

# Pod logs
kubectl logs -n <namespace> -l app=<name>
kubectl logs -n <namespace> -l app=<name> --previous

# Restart deployment
kubectl rollout restart deployment <name> -n <namespace>

# Watch pods
kubectl get pods -A -w

# Get events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
```

### MetalLB
```bash
# Check IP pools
kubectl get ipaddresspools -n metallb-system

# Check advertisements
kubectl get l2advertisements -n metallb-system

# Annotate service with shared IP
kubectl annotate svc <name> -n <namespace> metallb.io/address-pool=metallb-ips --overwrite
kubectl annotate svc <name> -n <namespace> metallb.io/loadBalancerIPs=10.0.1.100 --overwrite
kubectl annotate svc <name> -n <namespace> metallb.io/allow-shared-ip=shared-web --overwrite
```

---

## Caddy Reverse Proxy Example
```
www.paulcdb.com {
reverse_proxy 10.0.1.100:8100
}
vault.paulcdb.com {
reverse_proxy 10.0.1.100:8600
}
status.paulcdb.com {
reverse_proxy 10.0.1.100:3001
}
git.paulcdb.com {
reverse_proxy 10.0.1.100:3000
}
drop.paulcdb.com {
reverse_proxy 10.0.1.100:3000
}
```
---

## File Structure
```
pi-cluster/
├── README.md
├── secrets.yaml          # NEVER COMMIT - add to .gitignore
├── controlplane.yaml
├── worker.yaml
├── patches/
│   └── t1-vip-patch.yaml
├── metallb/
│   └── metallb.yaml
├── monitoring/
│   └── kube-prometheus-values.yaml
└── deployments/
├── grav-www-paulcdb-com.yaml
├── grav-www-cdb-online-co-uk.yaml
├── grav-www-holyislandarc-club.yaml
├── vaultwarden.yaml
├── uptimekuma.yaml
├── pairdrop.yaml
├── it-tools.yaml
├── forgejo.yaml
├── changedetection.yaml
└── spoolman.yaml
```
---

## Important Notes
- Never commit `secrets.yaml` to git — add it to `.gitignore`
- Always apply controlplane updates one node at a time to maintain etcd quorum
- Only run `bootstrap` once on initial cluster setup
- All services sharing `10.0.1.100` must have the `metallb.io/allow-shared-ip: shared-web` annotation
- MetalLB pool name is `metallb-ips`
- NFS server is at `10.0.1.5`

## Add secrets.yaml to .gitignore before pushing to Forgejo:
echo "secrets.yaml" >> .gitignore