This proposal attempts to improve the existing local cluster experience for kubernetes. The current local cluster experience is sub-par and often not functional. There are several options to setup a local cluster (docker, vagrant, linux processes, etc) and we do not test any of them continuously. Here are some highlighted issues: - Docker based solution breaks with docker upgrades, does not support DNS, and many kubelet features are not functional yet inside a container. - Vagrant based solution are too heavy and have mostly failed on macOS. - Local linux cluster is poorly documented and is undiscoverable. From an end user perspective, they want to run a kubernetes cluster. They care less about how a cluster is setup locally and more about what they can do with a functional cluster.
From a high level the goal is to make it easy for a new user to run a Kubernetes cluster and play with curated examples that require least amount of knowledge about Kubernetes. These examples will only use kubectl and only a subset of Kubernetes features that are available will be exposed.
Following are some of the existing solutions that attempt to simplify local cluster deployments.
Spread’s UX is great! It is adapted from monokube and includes DNS as well. It satisfies almost all the requirements, excepting that of requiring docker to be pre-installed. It has a loose dependency on docker. New releases of docker might break this setup.
Kmachine is adapted from docker-machine. It exposes the entire docker-machine CLI. It is possible to repurpose Kmachine to meet all our requirements.
Single binary that runs all kube master components. Does not include DNS. This is only a part of the overall local cluster solution.
The kube-up.sh script included in Kubernetes release supports a few Vagrant based local cluster deployments. kube-up.sh is not user friendly. It typically takes a long time for the cluster to be set up using vagrant and often times is unsuccessful on macOS. The Core OS single machine guide uses Vagrant as well and it just works. Since we are targeting a single command install/teardown experience, vagrant needs to be an implementation detail and not be exposed to our users.
To avoid exposing users to third party software and external dependencies, we will build a toolbox that will be shipped with all the dependencies including all kubernetes components, hypervisor, base image, kubectl, etc. Note: Docker provides a similar toolbox. This “Localkube” tool will be referred to as “Minikube” in this proposal to avoid ambiguity against Spread’s existing “localkube”. The final name of this tool is TBD. Suggestions are welcome!
Minikube will provide a unified CLI to interact with the local cluster. The CLI will support only a few operations: - Start - creates & starts a local cluster along with setting up kubectl & networking (if necessary) - Stop - suspends the local cluster & preserves cluster state - Delete - deletes the local cluster completely - Upgrade - upgrades internal components to the latest available version (upgrades are not guaranteed to preserve cluster state)
For running and managing the kubernetes components themselves, we can re-use Spread’s localkube. Localkube is a self-contained go binary that includes all the master components including DNS and runs them using multiple go threads. Each Kubernetes release will include a localkube binary that has been tested exhaustively.
To support Windows and macOS, minikube will use libmachine internally to create and destroy virtual machines. Minikube will be shipped with an hypervisor (virtualbox) in the case of macOS. Minikube will include a base image that will be well tested.
In the case of Linux, since the cluster can be run locally, we ideally want to avoid setting up a VM. Since docker is the only fully supported runtime as of Kubernetes v1.2, we can initially use docker to run and manage localkube. There is risk of being incompatible with the existing version of docker. By using a VM, we can avoid such incompatibility issues though. Feedback from the community will be helpful here.
If the goal is to run outside of a VM, we can have minikube prompt the user if docker is unavailable or version is incompatible. Alternatives to docker for running the localkube core includes using rkt, setting up systemd services, or a System V Init script depending on the distro.
To summarize the pipeline is as follows:
minikube -> libmachine -> virtualbox/hyper V -> linux VM -> localkube
minikube -> docker -> localkube