kind and Persistent Storage
11 July, 2019NB: As of kind 0.7.0, the below is no longer necessary! 🎉
I’ve been using kind instead of Minikube recently for any Kubernetes-related hacking about - specifically for KUDO. Why? Well, on macOS at least, it’s quicker to launch and it’s also more efficient from a resource usage perspective - if you happen to have Docker running anyway that is. Remember that Docker for Mac actually runs Linux in a VM via Hyperkit on your behalf; Minikube - if you configure it correctly - also creates a VM to deploy Kubernetes. Having two VMs with a committed amount of CPU and memory running behind the scenes doesn’t make sense. You might as well run it all in containers under the one VM (the Docker-provisioned one).
Anyway, I hit a problem early on trying to deploy either the ZooKeeper or Kafka operators, as they both have a StatefulSet defined with persistent storage requirements via volumeClaimTemplates. By default kind uses hostpath-provisioner which is somewhat limited, and so if you look at the logs for your service - for example one of the ZooKeeper pods in my case - you’ll see errors along the lines of:
Creating ZooKeeper log4j configuration
mkdir: cannot create directory '/var/lib/zookeeper/data': Permission denied
chown: cannot access '/var/lib/zookeeper/data': No such file or directory
mkdir: cannot create directory '/var/lib/zookeeper/data': Permission denied
chown: invalid group: 'zookeeper:USER'
/usr/bin/start-zookeeper: line 176: /var/lib/zookeeper/data/myid: No such file or directory
Fortunately, it’s possible to fix this by switching to the Local Path Provisioner by doing the following after launching your kind cluster (and before attempting to run any of the aforementioned services, obvs):
kubectl delete storageclass standard
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl annotate storageclass --overwrite local-path storageclass.kubernetes.io/is-default-class=true
I actually spotted the fix for this problem via the Makefile in the KUDO operators repository, so kudos (🤦‍♂️) to the original author for that one.