How to Create Local Kubernetes Dev Environments

I saw a post on Reddit this week that was labeled a ‘n00b question’ that felt like it revealed a much more basic question with Kubernetes: how do you define a local dev environment?

What is the best way to have my frameworks (back-end and front-end) working, with the ability to edit files on them in realtime, without having to rebuild images, restart pods, etc.

The question really stems from a starting place that this is really easy to do with a single containerized workload, e.g. with Docker.

With docker-compose, it's all real simple, e.g. if I work on a django project, I will just create a couple of compose files with a few containers (python, postgres, nodejs, nginx, etc.) with different targets for dev and production, use a bind mount (projectdir/django_app/ -> /django_app inside the container), edit the code, then use CI/CD tools to build images, push to a private repo, etc.

When trying to answer the user’s question, I thought of kind: kind is a tool for running local Kubernetes clusters using Docker container “nodes”. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. The Signadot team uses kind to test whether something will work on multiple versions of Kubernetes. However in my experience kind is not a tool for doing the live reloads that this user, and others, are hoping to use.

At least two tools are specifically targeted at this use case.

Tilt

Tilt is an ephemeral environment tool intended for small teams. Tilt is a tool to automate build and deploy processes for microservices. Technically, it can handle so much more than just a simple controller, but it serves this purpose well.

Gergely Brautigam has a nice writeup on using Tilt for local testing.

Skaffold

Skaffold is a command line tool that facilitates continuous development for container based & Kubernetes applications. Skaffold handles the workflow for building, pushing, and deploying your application, and provides building blocks for creating CI/CD pipelines.

The looming cloud

One potential solution for scaling local development is to use a cloud-based Kubernetes environment, such as AWS EKS or GKE. This allows for larger clusters with more resources, and can provide a more production-like environment for testing and development. However, this approach may not be feasible for smaller teams or those with limited budgets. You’re also going to be dealing with longer build and push times to test small code changes. Further, as your team grows will be that a cloud-based testing environment will end up getting ‘fought over’ as everyone wants to test their changes at the same time.

Scaling local development

While these are all great tools, they’re not all going to scale to very large engineering teams or super-large clusters. One redditor with a 5 node dev cluster sees it quickly use all available storage. Further, as discussed in my recent writeup on large teams testing on Kubernetes, while these tools will prevent you having to re-build your containers with each local code change, over time the daily process of updating all the other containers in the cluster and building the whole thing will start to impact developer velocity.

‍