Proposal to fix Issue #22609
kube-apiserver currently has a command-line argument
specifying the number of api servers. This masterCount is used in the
MasterCountEndpointReconciler on a 10 second interval to potentially cleanup
stale API Endpoints. The issue is when the number of kube-apiserver instances
gets below or above the masterCount. If the below case happens, the stale
instances within the Endpoints does not get cleaned up, or in the latter case
the endpoints start to flap.
Each apiserver’s reconciler only cleans up for it’s own IP. If a new server is spun up at a new IP, then the old IP in the Endpoints list is only reclaimed if the number of apiservers becomes greater-than or equal to the masterCount. For example:
There is logic within the MasterCountEndpointReconciler to attempt to make the Endpoints eventually consistent, but the code relies on the Endpoints count becoming equal to or greater than masterCount. When the apiservers become greater than the masterCount the Endpoints tend to flap.
If the number endpoints were scaled down from automation, then the Endpoints would never become consistent.
The MasterCountEndpointReconciler does not meet the current needs for durability of API Endpoint creation, deletion, or failure cases.
Custom Resource Definitions were proposed, but they do not have clean layering. Additionally, liveness and locking would be a nice to have feature for a long term solution.
ConfigMaps were proposed, but since they are watched globally, liveliness updates could be overly chatty.
By porting OpenShift’s LeaseEndpointReconciler to Kubernetes we can use the Storage API directly to store Endpoints dynamically within the system.
CRD’s and ConfigMaps were considered for this proposal. They were not adopted for this proposal by the community due to technical issues explained earlier.
|1.9||stable||Change the logic in the current reconciler|
We could potentially reuse the old reconciler by changing the reconciler to count
the endpoints and set the
masterCount (with a RWLock) to the count.