Kubeadm Config Draft

Kubeadm Config to Beta

Table of Contents

A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template.

Table of Contents

Summary

Kubeadm uses MasterConfiguraton for two distinct but similar operations: Initialising a new cluster and upgrading an existing cluster. The former is typically created by hand by an administrator. It is stored on disk and passed to kubeadm init via command line flag. The latter is produced by kubeadm using supplied configuration files, command line options, and internal defaults. It will be stored in a ConfigMap so upgrade operations can find.

Right now the configuration format is unversioned. This means configuration file formats can change between kubeadm versions and there’s no safe way to update the configuration format.

We propose a stable versioning of this configuration, v1alpha2 and eventually v1beta1. Version information will be mandatory going forward, both for user-generated configuration files and machine-generated configuration maps.

There as an existing document describing current Kubernetes best practices around component configuration.

Motivation

After 1.10.0, we discovered a bug in the upgrade process. The MasterConfiguraton embedded a struct that had changed, which caused a backwards-incompatible change to the configuration format. This caused kubeadm upgrade to fail, because a newer version of kubeadm was attempting to deserialise an older version of the struct.

Because the configuration is often written and read by different versions of kubeadm compiled by different versions of kubernetes, it’s very important for this configuration file to be well-versioned.

Goals

  • kubeadm init fails if a configuration file isn’t versioned
  • the config map written out contains a version
  • the configuration struct does not embed any other structs
  • existing configuration files are converted on upgrade to a known, stable version
  • structs should be sparsely populated
  • all structs should have reasonable defaults so an empty config is still sensible

Non-Goals

  • kubeadm is able to read and write configuration files for older and newer versions of kubernetes than it was compiled with
  • substantially changing the schema of the MasterConfiguration

Proposal

The concrete proposal is as follows.

  1. Immediately start writing Kind and Version information into the MasterConfiguraton struct.
  2. Define the previous (1.9) version of the struct as v1alpha1.
  3. Duplicate the KubeProxyConfig struct that caused the schema change, adding the old version to the v1alpha1 struct.
  4. Create a new v1alpha2 directory mirroring the existing v1alpha1, which matches the 1.10 schema. This version need not duplicate the file as well.
  5. Warn users if their configuration files do not have a version and kind
  6. Use apimachinery’s conversion library to design migrations from the old (v1alpha1) versions to the new (v1alpha2) versions
  7. Determine the changes for v1beta1
  8. With v1beta1, enforce presence of version numbers in config files and ConfigMaps, erroring if not present.

User Stories [optional]

As a user upgrading with Kubeadm, I want the upgrade process to not fail with unfamiliar configuration.

In the past, the haphazard nature of the versioning system has meant it was hard to provide strong guarantees between versions. Implementing strong version guarantees mean any given configuration generated in the past by kubeadm will work with a future version of kubeadm. Deprecations can happen in the future in well-regulated ways.

As a infrastructure system using kubeadm, I want to be able to write configuration files that always work.

Having a configuration file that changes without notice makes it very difficult to write software that integrates with kubeadm. By providing strong version guarantees, we can guarantee that the files these tools produce will work with a given version of kubeadm.

Implementation Details/Notes/Constraints

The incident that caused the breakage in alpha wasn’t a field changed it Kubeadm, it was a struct referenced inside the MasterConfiguration struct. By completely owning our own configuration, changes in the rest of the project can’t unknowingly affect us. When we do need to interface with the rest of the project, we will do so explicitly in code and be protected by the compiler.

Risks and Mitigations

Moving to a strongly versioned configuration from a weakly versioned one must be done carefully so as not break kubeadm for existing users. We can start requiring versions of the existing v1alpha1 format, issuing warnings to users when Version and Kind aren’t present. These fields can be used today, they’re simply ignored. In the future, we could require them, and transition to using v1alpha1.

Graduation Criteria

This KEP can be considered complete once all currently supported versions of Kubeadm write out v1beta1-version structs.

Implementation History

Alternatives

Rather than creating our own copies of all structs in the MasterConfiguration struct, we could instead continue embedding the structs. To provide our guarantees, we would have to invest a lot more in automated testing for upgrades.