node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver. node-problem-detector can either run as a DaemonSet or run standalone. It also runs as a Kubernetes Addon enabled by default in the GKE cluster.
There are tons of node problems could possibly affect the pods running on the node such as:
Currently these problems are invisible to the upstream layers in cluster management stack, so Kubernetes will continue scheduling pods to the bad nodes.
To solve this problem, we introduced this new daemon node-problem-detector to collect node problems from various daemons and make them visible to the upstream layers. Once upstream layers have the visibility to those problems, we can discuss the remedy system.
A problem daemon is a sub-daemon of node-problem-detector. It monitors a specific kind of node problems and reports them to node-problem-detector.
A problem daemon could be:
Currently, a problem daemon is running as a goroutine in the node-problem-detector binary. In the future, we’ll separate node-problem-detector and problem daemons into different containers, and compose them with pod specification.
List of supported problem daemons:
|KernelMonitor||KernelDeadlock||A system log monitor monitors kernel log and reports problem according to predefined rules.|
|AbrtAdaptor||None||Monitor ABRT log messages and report them further. ABRT (Automatic Bug Report Tool) is health monitoring daemon able to catch kernel problems as well as application crashes of various kinds occurred on the host. For more information visit the link.|
|CustomPluginMonitor||On-demand(According to users configuration)||A custom plugin monitor for node-problem-detector to invoke and check various node problems with user defined check scripts. See proposal here.|
Tell us about a new Kubernetes application
Never miss a thing! Sign up for our newsletter to stay updated.
Discover and learn about everything Kubernetes