Kubecon A session on how to hack into a Kubernetes cluster was among the highlights of a Kubecon where the main events were generally bland and corporate affairs, perhaps indicative of the technology now being a de facto infrastructure standard among enterprises.
Kubecon Europe took place online last week with more than 27,000 attendees, according to Chris Aniszczyk, CTO of the Cloud Native Computing Foundation (CNCF), which hosts the Kubernetes project among many others.
That is a substantial increase on the reported 13,000 or so at last year’s event, which was also virtual. Kubernetes is huge, and if there was an underlying theme at the event it was that Kubernetes is becoming the standard runtime platform.
There was plenty of strong technical content at the event, though attendees were left in no doubt that Kubernetes is big business and there was a dry corporate flavour to much of the keynote content along with the usual mutual backslapping.
CNCF introduced 27 new members, and observability specialist New Relic became a Platinum member, highlighting the significance of the OpenTelemetry project for collecting and analysing metrics, logs and traces from Kubernetes deployments. New Relic’s Zain Asgar joined the CNCF Governing Board. Asgar is CEO of Pixie Labs, acquired by New Relic in December 2020, and Pixie, a native Kubernetes observability product, has been open-sourced and will be contributed to CNCF.
“We wanted to make the observability product ubiquitous… it’s very hard to have a commercial offering that’s going to get to play everywhere,” Asgar told us.
“The goal behind Pixie is for it to be a vendor-neutral thing that everyone can use.” The commercial aspect is that Pixie is a data source that New Relic’s platform can consume, and the company also hosts Pixie Cloud as an option for managing the technology.
Spotify walked off with a “CNCF End User Award” for its work on Backstage, software that makes it easier to manage multiple services and share information. Spotify has 1,600 engineers, 14,000 software components and 1,400 microservices in production, according to web engineer Emma Indal who spoke at Kubecon, which explains why it came up with Backstage, and maybe why the Spotify app is no longer the simple, quick affair for streaming music that it was when first became popular.
Hacking Kubernetes: a story
As so often, the best content was not in the keynotes but in low-profile sessions. A highlight was a short piece on Hacking into Kubernetes by Ellen Körbes, head of product at Title, and Tabitha Sable, systems security engineer at Datadog. Körbes played the part of a developer at a fictional company where Sable was grandly called “Director of DevSecOps Enforcement”.
The story began when Körbes was annoyed by another developer using her port on the cluster. “I’m not calling the security people, they’re not fun, I’ll do this on my own,” she said.
She had limited RBAC (role-based access control) rights to the cluster, but that did not stop her. She got a shell on a pod that ran in a namespace with higher permissions, and performed the necessary command from there. The breach was discovered, but Körbes sat back and thought: “If the development cluster was out of commission all day, I would get the rest of the day off.”
She spotted CVE-2019-11253, “improper input validation in the Kubernetes API server… allows authorized users to send malicious YAML or JSON payloads, causing the API server to consume excessive CPU or memory, potentially crashing and becoming unavailable.”
DevSecOps ups the security to control its wayward developers but Körbes disliked being spied on and decided to go in and delete her logs. “Nobody is auditing anything.” Enter CVE-2020-15257 – “the containerd-shim API is improperly exposed to host network containers.” Körbes figured: “If I use a vulnerability in something Kubernetes is running on top of, I can bypass all Kubernetes security completely.”
A reverse shell and a bit of (unpublished) code later, she was in. Kubernetes vulnerabilities “don’t come around very often, but when they do they can ruin your day,” she mused. There is more: we will not spoil the story completely as it will be published for all to enjoy from 14 May.
“I struggled a lot to learn how to make talks engaging. The way to keep people engaging is with story,” explained Körbes at the wrap-up later, while Sable said: “We realised, Kubernetes security is complex because it’s the union of Linux security and network security and usually cloud provider security, and also Kubernetes has its own additional layer of complication there especially around RBAC and tying your shoes together with RBAC… I believe this is the first public demonstration of that Containerd exploit against Kubernetes.”
That was a great session, and also a neat illustration of what remains the big issue with Kubernetes: its complexity makes it hard to learn and easy to get wrong. There is no consensus on how this will be resolved, or whether it should be. We spoke to Mark Boost, CEO of Civo, a UK company offering hosted Kubernetes based on the lightweight K3S distribution (about which we hear more and more).
Despite the company’s focus on Kubernetes, Boost said he thinks fewer organisations will tangle with it directly in future. “Kubernetes is a great product but in the future it will be more under the hood, still be running Kubernetes, but there’ll be these layers on top which are just doing management on top to make things simple.”
Do we then end up back at Heroku, a revolutionary service when it was launched in 2007 as a way to run Ruby applications in the cloud (it has evolved since to support other runtimes) without managing the infrastructure? “In some ways, we do,” said Boost.
It seems that while many agree that using Kubernetes could and should be easier, other users would rather put up with the complexity for flexibility and control. “As more teams start modernising their applications, anything you can do to lower the cognitive cost of entry is good,” said Justin Turner, director of engineering at H-E-B, speaking at a Kubecon panel on the future of cloud native development.
“But there is a point where if you put too much abstraction on top of it, you lose a lot of control. You lose the ability to run operators… if we had too many layers of abstraction it may be hard to understand that those options are available.”
Jason McGee, CTO of IBM Cloud, said: “The lesson of Kubernetes is that there’s a diversity of workloads. People are moving towards an as-a-service consumption model and Kubernetes is evolving to have different personalities on how you consume the platform depending on what you are trying to do. Heroku, or the Cloud Foundry style of push code, lots of people want that. But maybe one of the lessons of that generation was that the platform doesn’t do everything.
“To me the power of Kubernetes is, if I’m building a simple app I can use that style, if I need to drop down and mess with the details of the application run stateful things, I can do that, all in one environment. I think we’ll add that to the ways Kubernetes is consumed. The question is whether we’ll do that in one way or whether there’s going to be 35 ways for that to happen.”
Most likely 35 ways, which makes the consensus around Kubernetes itself all the more remarkable. “For the first time in the industry we have standardised on the infrastructure with Kubernetes being that de facto control plane,” said Aniszczyk. ®