Last Updated on August 1, 2021
The idea of a stripped down operating system that only includes enough to start a container platform was pioneered by CoreOS a long time ago.
Recently there has been a bit of a resurgence in this area with Talos pushing the boundaries of what a host operating system should include in a very Kubernetes specific context.
Today we’ll take a look at some of the features of Container Linux (formerly CoreOS and now continued as FlatCar Linux by Microsoft), RancherOS (EOL), Talos, k3os and LinuxKit and discuss if there is any benefit to using these versus installing Kubernetes on a standard Linux distribution.
For the impatient here’s the raw data and link to the Google sheet. A more subjective discussion and interpretation will follow below.
Before we begin let’s take a brief detour into what an operating system really is. A blog entitled “Sorry, Linux. Kubernetes is now the OS that matters” rustled a few jimmies not so long ago. This prompted some fascinating discussion.
A traditional view of an operating system is something that abstracts hardware and allows us to start and interact with applications. In a Kubernetes centric world it’s easy to see why there has been a lively debate on this subject.
In fact, if I was merely looking for clicks I’d come out with claims like..
“Kubernetes is the new Linux”
“Kubelet is the new Kernel”
“kubectl is the new GNU base system”
“Helm is the new Yum”
“Operators are the new Config Management tools”
There is some partial truth to these outrageous claims but it’s clearly not the full story. At the heart of all of these Kubernetes centric operating systems there’s still a Linux kernel.
Where things get ambiguous is what else is actually then included in the base operating system.
Does it need a full init system? Possibly not.
Does it need a package management system? Not if we only run containers.
What about user-land binaries? Do we even need SSH? In the case of Talos you don’t even log into the OS any more as everything is managed over a GRPC API.
By reducing what’s contained in the base operating system we shift that complexity up a layer into Kubernetes. Does this improve security and reduce operational cost? I don’t know and I’m not sure anyone can prove it does or doesn’t.
It does mean however that we no longer need to know about Yum and Apt. Instead we learn how to package and manage applications on Kubernetes instead. The benefit being that we manage everything in a uniform and consistent way.
For those of us who have fully drank the Kubernetes kool-aid it’s certainly an attractive proposition. A shift towards an immutable operating system with a read-only root filesystem that boots our container platform and lets us manage our applications and dependencies via a single API.
I guess that explains why people are working on these projects.
Now we’ve dealt with the value, or perhaps the perceived value, let’s have a look at some of these projects and their current status.
Firstly, let’s mention a few of these. I won’t review or give recommendations on these as they are specific to which provider you use. If you are on AWS have a look at BottleRocket. And for GCP have a look at their Container Optimised OS. I couldn’t find one for Azure and I can only assume this was the reason they acquired FlatCar Linux so I’d expect one to be announced sooner or later.
Edit: CoreOS forked into FlatCar Linux and then the company was subsequently acquired by Microsoft. It remains to be seen whether this corporate sponsorship can renew the once awesome popularity of this project.
I got quite excited by this a few years ago and in all honesty I’m happy we never adopted it. There was always a concern that the company would go bust or get acquired, that the project would get ruined, and we’d be left on a dead-end platform. I have no idea what happened with the CoreOS that was merged into OKD but suffice to say it’s all bundled up into a very Redhat (IBM) specific option now. The other fork into Flatcar Linux is now owned by Microsoft. Microsoft has a bit of a bumpy history in terms of acquisitions. I remember when they acquired Deis and then most of their products went quietly away. On the other hand they bought Github (not technically open source) and haven’t totally ruined it. Similarly they bought Citus a while ago and they still seem to be operating independently. So there is a bit of hope for Kinvolk (the company who maintained Flatcar Linux).
Edit: Unfortunately RancherOS is now EOL and won’t be supported. There is a fork called BurmillaOs that intends to continue development.
I’m not sure why Rancher haven’t updated their Github repo to clearly state this is EOL. There’s a single line of text at the top of their Docs page stating “RancherOS 1.x is currently in a maintain-only-as-essential mode”. I incorrectly assumed at first glance that there was a 2.x version or something to replace it. I can understand that maintaining a Linux distro is a lot of overhead for probably no financial return. I believe Rancher itself uses Ubuntu as the default distro. Rancher has such an awesome reputation in the open source community I’m not sure why they aren’t being totally transparent by putting a big bold EOL message on the Github repo. I’d be pretty annoyed if I found the repo, started using it, and then some time later realised it was abandoned in terms of new features by the company who created it
Talos has taken a unique approach to what an operating system should include. Their ideas around managing nodes via a GRPC API are very interesting. Since I last looked at container optimized distributions it seems these guys have progressed rapidly and have built a production grade option that solves a lot of the security and resiliency issues inherent in normal distributions. It’s worth reading their documentation to get a feel for how it all works.
They have no package manager by default which is great. There is also no SSH and everything is done over gRPC which is a very modern and security centric way of managing nodes. It is also totally immutable which makes it far more reliable. It also has a very cool concept of looking at the operating system through an API. You can query nodes using their CLI to pull back important information. I have always agreed with the idea of banning SSH to nodes but have usually relented as it is not practical to debug production issues without it. With a few of the features Talos provides this may no longer be an issue.
k3os is limited by the scope of k3s itself which is focused on being a small single binary Kubernetes for running on edge devices or inside CI pipelines. I personally want to try using this as a private Gitlab runner using the Gitlab Helm chart. Similarly, if I wanted to install Kubernetes on a Raspberry Pi or edge device this is the option I’d go with.
LinuxKit lets you build operating system images with any customizations you want. There’s a Kubernetes specific project that I’ve referenced in the table above that demonstrates how to create an image with Kubernetes 1.10 baked inside.
I’m not convinced many people want to actually build their own operating system although I may be wrong. It’s a cool proof of concept and I’m sure there are valid reasons why you’d want to bake immutable images but I suspect many would prefer the convenience of using a distribution from an actively maintained project.
What do I usually use at work? Always a standard Linux distribution. Then we go through all of the hassle of hardening it and accept the problems associated with configuration drift and downtime that can cause. I believe if CoreOS had continued as a separate company I’d see a lot more of that being used. However, that turned into a massive train wreck and set the industry back a few years.
Nowadays I would probably give Talos a look. Talos has all the things I mention as being good (upgrade operators for the OS and Kubernetes versions, production maturity) and added benefits like support for R Pi, so you can run the same Vanilla K8s and OS on every platform or cloud provider. These guys seem to be taking the idea of a Kubernetes specific operating system seriously and they even wrote a comprehensive blog about why they are different.
Here’s hoping Talos can gain enough traction that it becomes a more compelling option in future.
Tell us about a new Kubernetes application
Never miss a thing! Sign up for our newsletter to stay updated.
Discover and learn about everything Kubernetes