I used microk8s to set up a kubernetes cluster with three virtualbox machines in it. Everything seems to be fine, but the microk8s registry does not seem to work anymore. It used to work on a single machine, but I am not sure if it is meant to work for a multi-host cluster. Anyways, below is what happened (under Ubuntu 22.04 LTS, with microk8s installed by sudo snap install microk8s --classic --channel=1.27
):
Initially microk8s status
shows that the registry is disabled:
$ microk8s status microk8s is runninghigh-availability: yes datastore master nodes: 192.168.10.30:19001 192.168.10.31:19001 192.168.10.32:19001 datastore standby nodes: noneaddons: enabled: dns # (core) CoreDNS ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes hostpath-storage # (core) Storage class; allocates storage from host directory storage # (core) Alias to hostpath-storage add-on, deprecated disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard gpu # (core) Automatic enablement of Nvidia CUDA host-access # (core) Allow Pods connecting to Host services smoothly ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000
But when I use microk8s
to enable it, there is an error:
$ microk8s enable registryInfer repository core for addon registry Infer repository core for addon hostpath-storage Addon core/hostpath-storage is already enabled The registry will be created with the size of 20Gi. Default storage class will be used. Warning: Detected changes to resource container-registry which is currently being deleted.namespace/container-registry unchangedWarning: Detected changes to resource registry-claim which is currently being deleted.persistentvolumeclaim/registry-claim unchangedconfigmap/local-registry-hosting unchangedError from server (Forbidden): error when creating "STDIN": deployments.apps "registry" is forbidden: unable to create new content in namespace container-registry because it is being terminatedError from server (Forbidden): error when creating "STDIN": services "registry" is forbidden: unable to create new content in namespace container-registry because it is being terminated
microk8s disable registry
seems to work fine, but re-enabling registry gives the same error:
$ microk8s disable registryInfer repository core for addon registryAddon core/registry is already disabled
My questions are:
Is the built-in microk8s registry supposed to work in a multi-host cluster?
If not, how to set up an image registry in a multi-host microk8s cluster?