There is no shortage of complexity when it comes to building a platform based on Kubernetes. There are countless examples of DevOps and platform engineering teams who fall prey to easily misconfigured Kubernetes clusters and cloud-native technologies. Not to mention the challenge around security posture given the pace at which Kubernetes is released. There are several startups tackling problems on the security front, and I’d say there has been an uptick in focus in the community within the last few years now that Kubernetes is the clear de-facto leader in the space. The cloud-native ecosystem is sprawling, and I decided to dive into the cloud-native security ecosystem to understand where someone new to this space should dig in first.

So, where do you even start?

 

Kubernetes and the cloud-native ecosystem is a large ocean to tread water in (see for yourself) and the specific ecosystem around security and compliance is no small pond either.

 

Maybe you’re asking yourself the same thing I was, which was “isn’t there a good place where I can get the basic map of what I should be digging into?” Well, I’m hoping with this blog I’ll be able to help you along this path of where to start in the cloud native security ecosystem.

The 4 C’s of Cloud Native Security

A great place to start is on the Kubernetes documentation site where it lays out the concept of security across a layer cake of control points. This layer cake approach is a common representation of thinking about security usually referred to as “defense in depth”. There’s a lot to digest within the 4C’s so I encourage readers to dig in separately, however, this blog should help you organize and summarize the main concepts. The four C’s are:

Code – Specifically this layer refers to the application code. This layer starts with the application developers and teams alike adopting best practices in their codebase. Of course, this varies whether you are writing in Golang, Rust, Python, JavaScript or other languages.

Container – The Container step generally can refer to all touch points for a container, container image or container runtime. This means that when application teams are ready to build, test and ship their applications inside container images, a lot of attack surfaces must be considered.

Cluster – Cluster can be generic, but Kubernetes is the de facto here so as a core tenant for cloud native security this layer looks specifically at Kubernetes. This layer considers the cluster components and components that run inside a cluster (aka applications). Not all applications are business-facing e.g., a logging or metrics service can still be considered an application that runs on the cluster.

Cloud /Co-location /Datacenter – Kubernetes runs somewhere, and that somewhere is where this layer is focused. Of course, this can be public or private cloud and the security practices are in many cases much more mature than those surrounding cloud native architecture so its best to refer to those.

Code

Application teams shouldn’t stuff their code or “what was running in a VM” into a container image and hope DevOps teams magically make their applications secure. It isn’t enough to focus on one layer of the overall cake to create a well-oiled secure cloud-native architecture. This starts with security considerations at the code level. Software Composition Analysis (SCA) and Software Bill of Materials (SBOM) are broad terms to define and inspect all “ingredients” of your application. During these processes application teams can become aware of potentially vulnerable software libraries, packages, and binaries as well as topics such as thinking about how to avoid hard coding secrets or putting vulnerable information into unencrypted loggers. Having these assessments also helps with future CVEs (common vulnerabilities and exposures) that your app may have. Along with these assessments, development teams can follow language specific best practices and use static code analysis tools. Golang docs have a great reference to best practices which points out things like not defining packages that you don’t need. Having un-used but imported packages can be one way to let an unnecessary attack vector exist. Also, using TLS whenever communication outside of your application is needed is also an important factor, so make sure to investigate your language specific security implementations.

Container

Once the application is developed and ready to be containerized, it’s usually time for your application development teams to work with your build pipeline. This often means at the end of your pipeline a tested and validated container image is ready to be shipped off to test, dev, staging or production environments. This process has various aspects where security best practices can come into play. At the beginning, applications are often defined as a container image manifest (like a Dockerfile), or something to tell _how_ to build the software into a portable image. It’s important to understand your image layers and configurations. As an example, when creating a Dockerfile, use trusted and signed base layers that also have been checked for vulnerabilities. Also, don’t define secrets or unneeded permissions or ports in the Dockerfile that don’t get used. Next, once an image is built, it’s usually uploaded to an image registry where they can be pulled to runtime environments. Image registries themselves, especially public ones, can contain malicious images so it’s a good idea authenticate and frequently scan your images and protect from hidden malware in image registries.

Luckily there are a lot of tools out there to help with some of these steps. There are various image scanning tools available that can scan built images for code vulnerabilities and CVEs before ever approving them for production. Container tooling around image signing also exists so make sure to investigate your specific platform for this as well.

Cluster

I think when most folks think about security and Kubernetes, they are thinking about the Kubernetes cluster itself and this is where we focus _most_ of the concern.

The first place to start with the cluster is access to the cluster or cluster API itself. Using TLS for all cluster components is key to cluster security posture so make sure any installation and configuration tooling you are using allows to you manage necessary certificates for this. Once APIs are setup behind TLS, a next step would be to look at authentication methods, that is, how does the cluster identify a user and if they access to this cluster? Larger orgs will connect OIDC or LDAP servers so existing organizational teams and structures can be used to control access. Next comes authorization which is taking an authorized user and knowing what they are allowed to do within the cluster (create, delete, update …). This is most commonly done with RBAC but there are other methods so read all options that your cluster platform offers.

Moving beyond this, we can start to focus on specific Kubernetes nodes. Kubernetes nodes run a process called the Kubelet which receives instructions from the master (control) nodes on what applications to run after scheduling decisions have been made. This Kubelet should also run its communications over secure and authenticated APIs. The Kubelet then has the responsibility to run the application on the specific node and there are many options to consider here to do so securely. Some basic options to consider is taking advantage of resource quotas, limit ranges, security context configuration, pod security standard admissions, kernel module blacklisting and network policies. Security contexts include configuration such as container/pod UID, GID, SELinux objects, privileges, AppArmor profiles, Seccomp filters for system calls and more. There is too much to dig into for this blog but check out the links provided below to get a better operational sense of how this might work.

Check out these labs on working with AppArmor profiles and Security Context on Kubernetes.

Working with Kubernetes AppArmour Profiles

Working with Kubernetes Security Contexts

Container runtimes are not built equally. Kubernetes can allow selection of runtime classes during application runtime to allow for more secure options to run those applications. One reference to this is Kata containers which allows applications to run more like tiny efficient virtual machines with lightweight hardware virtualization and a separate kernel. This is in contrast to a traditional container where applications share a host kernel and are isolated with namespaces and cgroups.

Cloud

Cloud security varies from cloud to cloud or data center to data center so take a look at this list of cloud provider security centers to dive in more. Some generic things to consider are, adopting a principle of least privilege, making sure you’re not using password authentication and keeping your kernels patched. Some items in the cloud native space to consider here are:

• Try to encrypt all storage at rest, whether this is for persistent volumes for applications on worker nodes or Etcd on the master nodes.  Utilize Persistent Volume (PV) level encryption as well as storage pool encryption.
• Configure filesystem group and user configuration along with correct AccessMode for volumes.
• Store backups securely in object storage, configurations such as object lock can go a long way to product against things like ransomeware attacks.
• Think about utilizing an immutable OS for your Kubernetes nodes such as Bottlerocket and RHOS (formerly CoreOS).
• Actively monitor your cloud infrastructure during runtime, understand what normal operation looks like to help identify abnormal operations which can help identify active security concerns.
• Secure network access to infrastructure such as your control plane nodes, etcd nodes and application nodes.

Conclusion

Cloud native security is a massive topic which architects, practitioners and developers all need to be aware of.  In light of this Id like to leave you with the following

1. Continuous learning is essential and as developers and practitioners, we need to start by determining and focusing on our current security posture and improving from there.
2. Cloud native security needs to be considered from the very start.  If, as its often happens, it ends up being an afterthought it will be infinitely harder to introduce, and you will find yourself saddled with technical debt.
3. If you're head is swimming dont worry, it’s unlikely you’re responsible for all four of the layers.  My advice is to start by picking one of the four Cs to focus on and begin double-clicking.

Good luck and stay secure!