grow-flexvolume-size

Proposal for Growing FlexVolume Size

Authors: xingzhou

Goals

Since PVC resizing is introduced in Kubernetes v1.8, several volume plugins have already supported this feature, e.g. GlusterFS, AWS EBS. In this proposal, we are proposing to support FlexVolume expansion. So when user uses FlexVolume and corresponding volume driver to connect to his/her backend storage system, he/she can expand the PV size by updating PVC in Kubernetes.

Non Goals

  • We only consider expanding FlexVolume size in this proposal. Decreasing size of FlexVolume will be designed in the future.
  • In this proposal, user can only expand the FlexVolume size manually by updating PVC. Auto-expansion of FlexVolume based on specific meterings is not considered.
  • The proposal only contains the changes made in FlexVolume, volume driver changes which should be made by user are not included.

Implementation Designs

Prerequisites

  • Kubernetes should be at least v1.8.
  • Enable resizing by setting feature gate ExpandPersistentVolumeGate to true.
  • Enable PersistentVolumeClaimResize admission plugin(optional).
  • Follow the UI of PV resizing, including:
    • Only dynamic provisioning supports volume resizing
    • Set StorageClass attribute allowVolumeExpansion to true

Admission Control Changes

Whether or not a specific volume plugin supports volume expansion is validated and checked in PV resize admission plugin. In general, we can list FlexVolume as the ones that support volume expansion and leave the actual expansion capability check to the underneath volume driver when PV resize controller calls the ExpandVolumeDevice method of FlexVolume.

In PV resize admission plugin, add the following check to checkVolumePlugin method:

// checkVolumePlugin checks whether the volume plugin supports resize
func (pvcr *persistentVolumeClaimResize) checkVolumePlugin(pv *api.PersistentVolume) bool {
  ...
  if pv.Spec.FlexVolume != nil {
    return true
  }
  ...
}

FlexVolume Plugin Changes

FlexVolume relies on underneath volume driver to implement various volume functions, e.g. attach/detach. As a result, volume driver will decide whether volume can be expanded or not.

By default, we assume all kinds of flex volume drivers support resizing. If they do not, flex volume plugin can detect this during resizing call to flex volume driver and always throw out error to stop the resizing process. So as a result, to implement resizing feature in flex volume plugin, the plugin itself must implement the following ExpandableVolumePlugin interfaces:

ExpandVolumeDevice

Volume resizing controller invokes this method while receiving a valid PVC resizing request. FlexVolume plugin calls the underneath volume driver’s corresponding expandvolume method with three parameters, including new size of volume(number in bytes), old size of volume(number in bytes) and volume spec, to expand PV. Once the expansion is done, volume driver should return the new size(number in bytes) of the volume to FlexVolume.

A sample implementation of ExpandVolumeDevice method is like:

func (plugin *flexVolumePlugin) ExpandVolumeDevice(spec *volume.Spec, newSize resource.Quantity, oldSize resource.Quantity) (resource.Quantity, error) {
  const timeout = 10*time.Minute
 
  call := plugin.NewDriverCallWithTimeout(expandVolumeCmd, timeout)
  call.Append(newSize.Value())
  call.Append(oldSize.Value())
  call.AppendSpec(spec, plugin.host, nil)

  // If the volume driver does not support resizing, Flex Volume Plugin can throw out error here
  // to stop expand controller's resizing process. 
  ds, err := call.Run()
  if err != nil {
      return resource.NewQuantity(0, resource.BinarySI), err
  }
 
  return  resource.NewQuantity(ds.ActualVolumeSize, resource.BinarySI), nil
}

Add a new field in type DriverStatus named ActualVolumeSize to identify the new expanded size of the volume returned by underneath volume driver:

// DriverStatus represents the return value of the driver callout.
type DriverStatus struct {
  ...
  ActualVolumeSize int64 `json:"volumeNewSize,omitempty"`
}

RequiresFSResize

RequiresFSResize is a method to implement ExpandableVolumePlugin interface. The return value of this method identifies whether or not a file system resize is required once physical volume get expanded. If the return value is true, PV resize controller will consider the volume resize operation is done and then update the PV object’s capacity in K8s directly; If the return value is false, PV resize controller will leave kubelet to do the file system resize, and kubelet on worker node will call ExpandFS method of FlexVolume to finish the file system resize step(at present, only offline FS resize is supportted, online resize support is under community discussion here).

The return value of RequiresFSResize is collected from underneath volume driver when FlexVolume invokes init method of volume driver. The sample code of RequiresFSResize in FlexVolume looks like:

func (plugin *flexVolumePlugin) RequiresFSResize() bool {
  return plugin.capabilities.RequiresFSResize
}

And as a result, the FlexVolume type DriverCapability can be redefined as:

type DriverCapabilities struct {
  Attach           bool `json:"attach"`
  RequiresFSResize bool `json:"requiresFSResize"`
  SELinuxRelabel   bool `json:"selinuxRelabel"`
}

func defaultCapabilities() *DriverCapabilities {
  return &DriverCapabilities{
    Attach:           true,
    RequiresFSResize: true, //By default, we require file system resize which will be done by kubelet
    SELinuxRelabel:   true,
  }
}

ExpandFS

ExpandFS is another method to implement ExpandableVolumePlugin interface. This method allows volume plugin itself instead of kubelet to resize the file system. If volume plugin returns true for RequiresFSResize, PV resize controller will leave FS resize to kubelet on worker node. Kubelet then will call FlexVolume ExpandFS to resize file system once physical volume expansion is done.

As ExpandFS is called on worker node, volume driver can also take this chance to do physical volume resize together with file system resize as well. Also, current code only supports offline FS resize, online resize support is under dicsussion here. Once online resize is implemented, we can also leverage online resize for FlexVolume by ExpandFS method.

Note that ExpandFS is a new API for ExpandableVolumeDriver, the community ticket can be found here.

ExpandFS will call underneath volume driver expandfs method to finish FS resize. The sample code looks like:

func (plugin *flexVolumePlugin) ExpandFS(spec *volume.Spec, newSize resource.Quantity, oldSize resource.Quantity)  error {
  const timeout = 10*time.Minute
 
  call := plugin.NewDriverCallWithTimeout(expandFSCmd, timeout)
  call.Append(newSize.Value())
  call.Append(oldSize.Value())
  call.AppendSpec(spec, plugin.host, nil)
 
  _, err := call.Run()
 
  return  err
}

For more design and details on how kubelet resizes volume file system, please refer to volume resizing proposal at: /contributors/design-proposals/storage/grow-volume-size.md

Based on the above design, the resizing process for flex volume can be summarized as: * If flex volume driver does not support resizing, driver shall not implement expandvolume method and flex volume plugin will throw out error to stop the expand volume controller’s resizing process. * If flex volume driver supports resizing, it shall implement expandvolume method and at least, the volume driver shall be installed on master node. * If flex volume driver supports resizing and does not need file system resizing, it shall set “requiresFSResize” capability to false. Otherwise kubelet on worker node will call ExpandFS to resize the file system. * If flex volume driver supports resizing and requires file system resizing(RequiresFSResize returns true), after the physical volume resizing is done, ExpandFS will be called from kubelet on worker node. * If flex volume driver supports resizing and requires to resize the physical volume from worker node, the driver shall be installed on both master node and worker node. The driver on master node can do a non-op process for ExpandVolumeDevice and returns success message. For RequiresFSResize, driver on master node must return true. This process gives drivers on worker nodes a chance to make physical volume resize and file system resize together through ExpandFS call from kubelet. This scenario is useful for some local storage resizing cases.

Volume Driver Changes

Volume driver needs to implement two new interfaces: expandvolume and expandfs to support volume resizing.

For expandvolume, it takes three parameters: new size of volume(number in bytes), old size of volume(number in bytes) and volume spec json string. expandvolume expands the physical backend volume size and return the new size(number in bytes) of volume.

For those volume plugins who need file system resize after physical volume is expanded, the expandfs method can take the FS resize work. If volume driver set the requiresFSResize capability to true, this method will be called from kubelet on worker node. Volume driver can do the file system resize (or physical volume resize together with file system resize) inside this method

In addition, those volume drivers who support resizing but do not require fils system resizing shall set requiresFSResize capability to false:

if [ "$op" = "init" ]; then
        log '{"status": "Success", "capabilities": {“requiresFSResize”: false}}'
        exit 0
fi

UI

Expand FlexVolume size follows the same process as expanding other volume plugins, like GlusterFS. User creates and binds PVC and PV first. Then by using kubectl edit pvc xxx command, user can update the new size of PVC.

References