Monitoring is split into two pipelines:
kubectl top). This pipeline is not intended for integration with third-party monitoring systems.
The architecture is illustrated in the diagram in the Appendix of this doc.
This document proposes a high-level monitoring architecture for Kubernetes. It covers a subset of the issues mentioned in the “Kubernetes Monitoring Architecture” doc, specifically focusing on an architecture (components and their interactions) that hopefully meets the numerous requirements. We do not specify any particular timeframe for implementing this architecture, nor any particular roadmap for getting there.
There are two types of metrics, system metrics and service metrics. System metrics are generic metrics that are generally available from every entity that is monitored (e.g. usage of CPU and memory by container and node). Service metrics are explicitly defined in application code and exported (e.g. number of 500s served by the API server). Both system metrics and service metrics can originate from users’ containers or from system infrastructure components (master components like the API server, addon pods running on the master, and addon pods running on user nodes).
We divide system metrics into
Service metrics can be divided into those produced by Kubernetes infrastructure components (and thus useful for operation of the Kubernetes cluster) and those produced by user applications. Service metrics used as input to horizontal pod autoscaling are sometimes called custom metrics. Of course horizontal pod autoscaling also uses core metrics.
We consider logging to be separate from monitoring, so logging is outside the scope of this doc.
The monitoring architecture should
We divide our description of the long-term architecture plan into the core metrics pipeline and the monitoring pipeline. For each, it is necessary to think about how to deal with each type of metric (core metrics, non-core metrics, and service metrics) from both the master and minions.
The core metrics pipeline collects a set of core system metrics. There are two sources for these metrics
These sources are scraped by a component we call metrics-server which is like a slimmed-down version of today’s Heapster. metrics-server stores locally only latest values and has no sinks. metrics-server exposes the master metrics API. (The configuration described here is similar to the current Heapster in “standalone” mode.) Discovery summarizer makes the master metrics API available to external clients such that from the client’s perspective it looks the same as talking to the API server.
Core (system) metrics are handled as described above in all deployment environments. The only easily replaceable part is resource estimator, which could be replaced by power users. In theory, metric-server itself can also be substituted, but it’d be similar to substituting apiserver itself or controller-manager - possible, but not recommended and not supported.
Eventually the core metrics pipeline might also collect metrics from Kubelet and Docker daemon themselves (e.g. CPU usage of Kubelet), even though they do not run in containers.
The core metrics pipeline is intentionally small and not designed for third-party integrations. “Full-fledged” monitoring is left to third-party systems, which provide the monitoring pipeline (see next section) and can run on Kubernetes without having to make changes to upstream components. In this way we can remove the burden we have today that comes with maintaining Heapster as the integration point for every possible metrics source, sink, and feature.
We will build an open-source Infrastore component (most likely reusing existing technologies) for serving historical queries over core system metrics and events, which it will fetch from the master APIs. Infrastore will expose one or more APIs (possibly just SQL-like queries – this is TBD) to handle the following use cases
In addition, it may collect monitoring metrics and service metrics (at least from Kubernetes infrastructure containers), described in the upcoming sections.
One of the goals of building a dedicated metrics pipeline for core metrics, as described in the previous section, is to allow for a separate monitoring pipeline that can be very flexible because core Kubernetes components do not need to rely on it. By default we will not provide one, but we will provide an easy way to install one (using a single command, most likely using Helm). We described the monitoring pipeline in this section.
Data collected by the monitoring pipeline may contain any sub- or superset of the following groups of metrics:
It is up to the monitoring solution to decide which of these are collected.
In order to enable horizontal pod autoscaling based on custom metrics, the provider of the monitoring pipeline would also have to create a stateless API adapter that pulls the custom metrics from the monitoring pipeline and exposes them to the Horizontal Pod Autoscaler. Such API will be a well defined, versioned API similar to regular APIs. Details of how it will be exposed or discovered will be covered in a detailed design doc for this component.
The same approach applies if it is desired to make monitoring pipeline metrics available in Infrastore. These adapters could be standalone components, libraries, or part of the monitoring solution itself.
There are many possible combinations of node and cluster-level agents that could comprise a monitoring pipeline, including cAdvisor + Heapster + InfluxDB (or any other sink) * cAdvisor + collectd + Heapster * cAdvisor + Prometheus * snapd + Heapster * snapd + SNAP cluster-level agent * Sysdig
As an example we’ll describe a potential integration with cAdvisor + Prometheus.
Prometheus has the following metric sources on a node: * core and non-core system metrics from cAdvisor * service metrics exposed by containers via HTTP handler in Prometheus format * [optional] metrics about node itself from Node Exporter (a Prometheus component)
All of them are polled by the Prometheus cluster-level agent. We can use the Prometheus cluster-level agent as a source for horizontal pod autoscaling custom metrics by using a standalone API adapter that proxies/translates between the Prometheus Query Language endpoint on the Prometheus cluster-level agent and an HPA-specific API. Likewise an adapter can be used to make the metrics from the monitoring pipeline available in Infrastore. Neither adapter is necessary if the user does not need the corresponding feature.
The command that installs cAdvisor+Prometheus should also automatically set up collection of the metrics from infrastructure containers. This is possible because the names of the infrastructure containers and metrics of interest are part of the Kubernetes control plane configuration itself, and because the infrastructure containers export their metrics in Prometheus format.