A self-managed Kubernetes cluster on AWS with kops (and the gossip DNS trap)
From the IAM group to the first kubectl get nodes: how we set up Kubernetes clusters we run ourselves on AWS.

Some projects don’t justify a Kubernetes cluster, and some projects don’t justify anything else. This article is about the second case: a client with a handful of containerized services, a modest and predictable workload, and the need to be able to move everything elsewhere without rewriting the way applications get published. Nothing huge: one machine coordinating the cluster, a handful of machines doing the work, a budget counted in tens of euros a month.
For this kind of cluster we use kops, short for Kubernetes Operations: a command-line tool that builds a “plain” Kubernetes, meaning one with no proprietary additions, starting from a configuration file kept in an S3 bucket. Here you’ll find the procedure we follow, the reasoning behind each choice and the obstacles we ran into. One of those is a big one, a deprecation that in 2026 puts existing kops clusters out of the game, and it affects ours too.
Two terms that will come up often, for anyone who doesn’t live inside Kubernetes every day. The control plane is the brain of the cluster: it knows what should run and where, and it answers commands. The worker nodes are the machines the applications actually run on. The control plane can go down without the applications stopping, but while it’s down the cluster accepts no changes.
Why kops and not EKS
Fair question: AWS already offers a managed Kubernetes. Why not use it? Four reasons.
The same cluster on AWS, Google Cloud or Azure. This is the one that weighs most, and it’s a real capability rather than a detail for insiders: kops doesn’t only work with AWS. The same tool and the same procedure create a cluster on Google Cloud or on Azure, and what comes out is always Kubernetes put together the same way. Our clients aren’t all on the same platform, often for reasons that have nothing to do with technology: commercial agreements, credits to burn, internal policy. If tomorrow a project has to start somewhere else, we don’t begin from scratch and we don’t have to learn a different managed service. With EKS, that expertise only counts inside AWS.
Cost, on small clusters. The EKS control plane costs around $73 a month per cluster, always, no matter what runs on it. On a three-machine cluster that’s almost as much as the machines themselves: the spend doubles to run the very same applications. With kops the control plane is a machine like any other and you pay for it as such. On a large cluster the reasoning flips and that figure becomes negligible.
Control. With kops we can get onto the machines, change settings, and choose which version of Kubernetes to install and when to upgrade it. With EKS the control plane is a closed box that AWS updates on its own schedule. For many projects that closed box is an advantage, because it takes work off your hands; when something doesn’t behave as it should, it’s exactly the opposite.
No constraints. What comes out of kops is Kubernetes exactly as the community publishes it. The same configuration files would work on another cloud provider or on a server in the office.
Then there’s the most mundane reason of all: we already have kops clusters in production, with the expertise and the procedures well established.
The trade is clear. In exchange for the savings and the control, running it is on us: upgrades, maintenance, backups. At the end of the article you’ll find who, in our view, would be better off turning that trade down.
Prerequisites: IAM and the state store
Before it can do anything at all, kops needs two things: AWS credentials that are broad enough and an S3 bucket to keep the cluster state in.
Let’s start with the credentials. The official documentation asks for seven AWS managed policies. We attach them to a dedicated group and hook a dedicated user to that group: one group and one user per cluster, never shared.
# CAREFUL: make sure you have exported the credentials for the right account.
# It is the easiest way to create a cluster in the wrong place.
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export CLUSTER=my-fancy-cluster
aws iam create-group --group-name $CLUSTER
for policy in AmazonEC2FullAccess AmazonRoute53FullAccess AmazonS3FullAccess \
IAMFullAccess AmazonVPCFullAccess AmazonSQSFullAccess \
AmazonEventBridgeFullAccess; do
aws iam attach-group-policy \
--policy-arn arn:aws:iam::aws:policy/$policy \
--group-name $CLUSTER
done
aws iam create-user --user-name $CLUSTER
aws iam add-user-to-group --user-name $CLUSTER --group-name $CLUSTER
aws iam create-access-key --user-name $CLUSTER
It’s worth remembering that we have just handed that user very broad permissions: IAMFullAccess and AmazonS3FullAccess, so it can create roles and read any bucket in the account. In an account dedicated to a single cluster the risk is acceptable; in an account shared with other resources it needs weighing, and the right approach is narrowing the policies by hand. That is not a short job, and it’s why the official documentation doesn’t propose it as the normal route. We prefer keeping the account separate: it’s simpler, and cost separation comes along for free.
The second thing is the state store: an S3 bucket with versioning enabled, where kops keeps the complete description of the cluster. Versioning is not a detail to skip over: if a bad edit ends up in that file, the previous versions are the only way back.
export REGION=eu-north-1
aws s3api create-bucket \
--bucket ${CLUSTER}-state-store \
--region $REGION \
--create-bucket-configuration LocationConstraint=$REGION
aws s3api put-bucket-versioning \
--bucket ${CLUSTER}-state-store \
--versioning-configuration Status=Enabled
A note for anyone working in the us-east-1 region: there the --create-bucket-configuration parameter must be left out, because that region is the historical starting point of the AWS APIs and naming it explicitly makes the call fail.
The trap: gossip DNS is gone
Now for the part that sets this article apart from any kops guide written before 2025. If you have a kops cluster in production it’s worth reading even if you haven’t created a new one in years.
Historically kops offered two ways for the machines in the cluster to find the control plane. The first was a real DNS zone, hosted on Route 53, with the cluster named something like k8s.example.com. The second was gossip DNS, which you enabled simply by giving the cluster a name ending in .k8s.local: a direct communication network between the machines, which passed the control plane address around without relying on external DNS. Very convenient, since there was no zone to register and no domain to buy. For years it was the road every guide took, and it’s the one that shows up in our own oldest internal notes.
Gossip has been removed. Here’s the timeline, pieced together from the kops documentation:
- kops 1.29: gossip is declared deprecated and new clusters start using None-DNS automatically.
- kops 1.35:
kops create clusterno longer creates gossip clusters, not even if you ask. A name ending in.k8s.localnow produces a None-DNS cluster. Only clusters born under earlier versions are still gossip. - kops 1.36: gossip clusters move to a mixed mode. The control plane keeps using gossip, while the workers connect directly to the load balancer that exposes it.
- kops 1.37: gossip disappears completely. Clusters still using it have to be converted with kops 1.36 before they can move to the next version.
The reasons for the removal are solid, and they describe a risk your cluster was exposed to as well. Gossip relied on external libraries that are no longer maintained, run moreover by a process with elevated privileges on every machine. On top of that, for the machines to recognise each other, very broad AWS account permissions were needed even on plain workers: permissions that made the structure of the cluster readable to anyone who could reach the role attached to a machine.
Gossip has been replaced by None-DNS (--dns=none), available since kops 1.26. It covers the same use cases, meaning clusters with no external DNS zone available, including the AWS regions where Route 53 doesn’t exist. The difference is that there’s nothing left for the machines to talk about: the workers connect directly to the control plane’s load balancer, and the cluster publishes no DNS records at all.
As we write, the stable release is kops 1.36.1 and 1.37 is in alpha. Anyone with gossip clusters has a window of time ahead, not an open-ended postponement.
Converting an existing gossip cluster
This is the part that hits close to home, because our older clusters use gossip. The good news is that the procedure is shorter than the word “migration” suggests, and above all the cluster name doesn’t change: certificates, state store and kubectl configuration stay the same, and the way you connect doesn’t change either.
The conversion has to be done with kops 1.36, before upgrading kops to the next version.
# 1. Cluster already upgraded to kops 1.36: apply and move to the mixed mode
# (this removes gossip from the worker machines)
kops reconcile cluster --yes
# 2. kops edit cluster: disable gossip by setting the DNS topology to None
spec:
topology:
dns:
type: None
On AWS there’s one more constraint to know about before pressing enter: None-DNS requires a Network Load Balancer in front of the control plane. If the cluster still uses a Classic Load Balancer, the same edit also needs:
spec:
api:
loadBalancer:
class: Network
# 3. Apply: this replaces the control plane machines, which stop using gossip.
# kops removes the now-unused dns-controller component by itself (before
# 1.36 it had to be deleted by hand).
kops reconcile cluster --yes
# 4. Check
kops validate cluster --wait 10m
One last detail, of the kind that makes the validity check fail in a puzzling way: the gossipConfig and dnsControllerGossipConfig fields are deprecated and no longer have any effect, but their mere presence makes the configuration get rejected. If they’re still in your cluster’s file, take them out before upgrading.
Watch out for step 3, because it replaces the control plane machines. On a cluster that only has one, for the whole duration of the replacement the cluster accepts no commands: applications already running keep working, but you can’t publish new versions. Schedule it in a maintenance window rather than doing it on impulse on a Thursday afternoon.
Creating the cluster
From here on the article assumes a new cluster on kops 1.36, so in None-DNS mode. Let’s begin with the variables, which we keep in a script per cluster instead of trusting them to the terminal’s memory:
export AWS_ACCESS_KEY_ID=... # keys for the kops user created above
export AWS_SECRET_ACCESS_KEY=...
export NAME=myfancy.cluster.k8s.local
export KOPS_STATE_STORE=s3://my-fancy-cluster-state-store
export ZONES=eu-north-1a
export CONTROL_PLANE_COUNT=1
export CONTROL_PLANE_SIZE=t3.medium
export CONTROL_PLANE_VOLUME_SIZE=48
export NODE_COUNT=1
export NODE_SIZE=t3.medium
export NODE_VOLUME_SIZE=48
Before picking the zone it’s worth asking AWS which ones are available in the region, rather than going from memory:
aws ec2 describe-availability-zones --region eu-north-1
And then the command itself:
kops create cluster \
--name $NAME \
--cloud aws \
--zones $ZONES \
--control-plane-count $CONTROL_PLANE_COUNT \
--control-plane-size $CONTROL_PLANE_SIZE \
--control-plane-zones $ZONES \
--control-plane-volume-size $CONTROL_PLANE_VOLUME_SIZE \
--node-count $NODE_COUNT \
--node-size $NODE_SIZE \
--node-volume-size $NODE_VOLUME_SIZE \
--networking calico \
--topology private
Let’s look at what sits behind each choice, because this is where you decide how the cluster will be shaped for years to come.
--name with the .k8s.local suffix. On kops 1.36 this form gives you a None-DNS cluster, no longer a gossip one. If instead you have a Route 53 zone and want a real name, say k8s.example.com, just pass it and kops creates the DNS records. A real name is reachable from outside without any tricks; .k8s.local needs nothing registered.
--control-plane-count 1 and a single zone. Let’s put it plainly: this makes the control plane a single point of failure. If that machine breaks, the cluster stops accepting commands until kops replaces it. Applications already running on the workers keep going, because Kubernetes is designed to survive the loss of the control plane, but in the meantime the cluster is deaf to any request. For a small production environment it’s a compromise we take knowing what it costs; for a critical service you move to three control plane machines spread across three zones. It’s also why --control-plane-zones wants an odd number of zones: to make a decision, the cluster needs a majority of the copies to be reachable.
--control-plane-size t3.medium. Our first attempt was t3.small and we changed it almost immediately: 2 GB of memory is at the edge for running all the components that coordinate the cluster. It’s also worth remembering that machines in the t3 family have variable performance, because they build up credits while they’re idle and, once those run out, computing power gets throttled. For a control plane that’s always under load that’s behaviour to keep an eye on, and if the cluster grows it’s better to move to a family with steady performance such as m6i or c6i.
--networking calico. A warning here, because the situation has changed: kops today installs cilium unless you say otherwise, no longer calico. Both are valid and they’re the two most widespread networking systems on kops clusters. We stay on calico for consistency with the clusters we already have, but for a new cluster you may as well take what kops offers on its own. Anyone arriving from a dated guide and expecting calico “because it’s the default” is wrong today.
--topology private. The machines end up in private networks, with no public IP address, and reach the internet through a NAT Gateway. It’s the right topology for production and it has two costs worth budgeting for. The first is the NAT Gateway itself, billed both by the hour and by traffic: we’re around $30-40 a month purely for existing. The second is that connecting to a machine over SSH needs an intermediate hop, the so-called bastion, which kops can create by adding --bastion. With the public topology the machines have public addresses, the NAT isn’t needed and the spend goes down: for a test environment that’s a perfectly sensible choice.
Adjusting the configuration
kops create cluster doesn’t create anything on AWS: it just writes the cluster description into the state store. So this is the right moment to fix it up before anything actually comes to life. The command kops edit cluster ${NAME} opens that file in your editor and, on save, writes it back to S3.
The two entries we add nearly every time are these:
spec:
awsLoadBalancerController:
enabled: true
certManager:
enabled: true
managed: false
The first, awsLoadBalancerController, installs the AWS Load Balancer Controller, the component that turns requests to publish services into real AWS load balancers. We didn’t add it for elegance: we needed a load balancer with both TCP and UDP ports on the same address, and with the old component, the one built into the long-standing AWS integration, a service declaring TCP and UDP ports together simply cannot be created. The limitation isn’t on AWS’ side, its load balancers handle mixed ports perfectly well: it’s the old component that doesn’t know how to ask for them. The new one does, and as an added bonus it also stops creating a Classic Load Balancer for every exposed service, which is the dated option and costs more for the same features.
The second, certManager with managed: false, is the line that raises the most doubts. The controller above needs cert-manager, the tool that handles certificates inside the cluster. Writing enabled: true tells kops “cert-manager is there, the other components can count on it”; adding managed: false says “but don’t install it yourself, we’ll take care of it”. We do it so we can decide which version of cert-manager to install and how to configure it, with the same tooling we use for the rest of the cluster, instead of depending on kops’ timing. If you don’t want that responsibility, managed: true has kops install cert-manager directly and works just as well. What matters is that one of the two routes is followed all the way: with enabled: true, managed: false and no cert-manager installed, the load balancer controller won’t start.
Instance groups: where the applications land
This is the part guides tend to skip, and the one we use most in day-to-day work.
kops doesn’t think in terms of individual machines but of instance groups, sets of machines sharing the same characteristics. kops create cluster sets up two of them, one for the control plane and one for the workers, but nothing stops you from having five, each with its own configuration. It’s the mechanism that lets you say “these applications run on machines with plenty of memory, those others on cheap ones”.
export IG_NAME=service
kops create ig $IG_NAME --name $NAME --subnet $ZONES
The command opens your editor on the new group. There are only a few fields that matter:
spec:
machineType: t3.medium
minSize: 1
maxSize: 3
nodeLabels:
fancypixel.it/node-group: service
The nodeLabels are the key to everything: labels kops applies to the group’s machines as they are born, which let an application ask where it wants to run, through a nodeSelector:
# deployment.yaml
spec:
template:
spec:
nodeSelector:
fancypixel.it/node-group: service
A note on names, because getting them wrong wastes time: the label prefix has to be a valid domain, and the kubernetes.io/ and k8s.io/ prefixes are reserved. Use them and the request gets rejected.
The same fields can be changed on groups that already exist:
kops edit ig --name $NAME nodes-eu-north-1a
kops edit ig --name $NAME control-plane-eu-north-1a
And here’s the most effective way to keep the spend down: taking minSize and maxSize to 0. The group shuts down all its machines, its definition stays in the state store, and bringing it back to 1 has it running again in a couple of minutes. For groups dedicated to periodic work, say powerful machines needed only for an overnight job, that means paying for them only while they work.
There is a side effect that isn’t obvious, though, and it’s the most common way to get yourself into trouble: if you zero out all the workers, the cluster’s own service components have nowhere left to run. The first is CoreDNS, which handles internal name resolution, and without CoreDNS the validity check fails. More on that shortly.
Applying, and waiting
Up to this point we’ve only written files to S3. Now we move on to real resources:
kops update cluster ${NAME} --admin --yes
The --yes is what separates the simulation from reality: without it, kops just prints what it would do and stops there. Looking at the list before proceeding is a healthy habit. The --admin, on the other hand, adds an administrator user to the local kubectl configuration, so you can start working straight afterwards.
Then you wait, and not briefly: it takes twenty or thirty minutes for the cluster to be genuinely ready. In the meantime AWS creates the network, the subnets, the NAT, the load balancer and the machine groups; the instances boot, the control plane starts answering and the workers register themselves.
kops validate cluster --wait 10m
kubectl get nodes
Since version 1.32 kops offers a command that makes the two-step sequence of older scripts unnecessary: kops reconcile cluster --yes applies the changes and replaces control plane and workers one after another, in a single operation. It’s what we use when working on clusters that already exist. The old pair of commands is still valid when you want to keep the two moments separate:
kops update cluster ${NAME} --yes # applies the configuration to the infrastructure
kops rolling-update cluster --yes # replaces the machines to apply changes that
# require new instances
The distinction matters: changing a group’s machine type doesn’t touch the instances already running. update updates the template new machines will be created from, but until you run rolling-update (or reconcile) the old ones stay in service. More than once we’ve wondered why the machine was still a t3.small.
When validation fails because of CoreDNS
This is what you’ll run into if you follow the advice about zeroing out the groups you don’t need: kops validate reports that CoreDNS isn’t ready. The control plane machines carry a taint that blocks ordinary applications from starting, and if the workers are at zero CoreDNS has nowhere left to run.
The structural fix is keeping at least one worker switched on. If you need a temporary remedy to get through some maintenance, you can remove that taint:
# temporary: lets the service components run on the control plane
kubectl taint nodes <node-name> node-role.kubernetes.io/control-plane:NoSchedule-
The trailing dash is what removes the taint instead of adding it. Put it back as soon as the cluster is back to normal, because letting applications run on the control plane means risking that one memory-hungry process brings down the brain of the cluster.
Getting access to the cluster
Since Kubernetes 1.19 username and password authentication is gone: access goes through certificates and tokens, and kops takes care of generating the configuration.
kops export kubeconfig ${NAME} --admin
The --admin option produces an administrative credential with a limited lifetime: 18 hours unless you say otherwise. That’s not an oversight, it’s the right call: an administrator credential that never expires, sitting on a developer’s laptop, sooner or later ends badly. When it expires, you run the command again.
A practical habit that has saved us trouble more than once: when working across several clusters it’s better not to write into the default kubectl configuration, but to keep one per cluster.
KUBECONFIG="$(pwd)/config" kops export kubeconfig --admin --name $NAME
This way the active cluster is always the one belonging to the directory you’re working in, and you never fire a destructive command at a cluster other than the one you had in mind.
Day-to-day maintenance
Labelling machines by hand, when you need to single out one machine rather than a whole group:
kubectl label nodes ip-172-20-51-42.eu-north-1.compute.internal \
fancypixel.it/node-name=fancy-1
# to remove a label: the key name followed by a dash
kubectl label nodes ip-172-20-51-42.eu-north-1.compute.internal \
fancypixel.it/node-name-
With one caveat: labels applied this way belong to that specific machine and disappear along with it. If AWS replaces it, the label is gone. The ones that need to survive instance turnover belong in the group’s nodeLabels.
Looking at the cluster the way you need to. The usual output of kubectl get nodes reports name, status, role, age and version: the right information for a generic cluster, not much use when the question is “which group is serving what, and on which machines”. With custom-columns you build the table you’re actually interested in:
kubectl get nodes -o custom-columns="\
NODE_GROUP:metadata.labels.fancypixel\.it/node-group,\
NAME:.metadata.name,\
INSTANCE:.metadata.labels.node\.kubernetes\.io/instance-type,\
VERSION:.status.nodeInfo.kubeletVersion"
What comes out is a table that answers the everyday questions in one go: how many machines each group has running, whether it’s really using the instance type you think, and whether after a replacement every machine reached the new version. It’s the first command we run whenever something doesn’t add up.
Upgrading the cluster, which is the work you wouldn’t have with EKS:
kops upgrade cluster ${NAME} --yes # moves the configuration to the version you want
kops reconcile cluster --yes # applies and replaces control plane and workers
kops validate cluster --wait 10m
kops 1.36 supports Kubernetes up to 1.36 and has dropped 1.30, so the usable range runs from 1.31 to 1.36. The general rule is simple: the highest supported Kubernetes version matches the kops version, plus the two before it. In other words, if you fall behind on kops you fall behind on Kubernetes too, and catching up several versions at once is far more painful than staying current.
Still on 1.36 there are other removals that can block an upgrade, and it’s better to find out now than halfway through: Amazon Linux 2, Ubuntu 20.04 and Debian 10 are no longer supported and clusters using them need moving to a newer system first; anyone who has customised the containerd configuration has to move to version 3 of the TOML format.
The bill
Let’s see where the money goes on a small kops cluster. These are orders of magnitude based on AWS list prices: they vary from region to region, so read them as proportions rather than as a quote.
- EC2, meaning control plane and workers. On
t3.mediumwe’re around $30 a month per machine running continuously. It’s the line that dominates the spend, and the one that zeroing out non-essential groups acts on directly. - NAT Gateway, only with the
privatetopology: around $30-40 a month purely for existing, plus traffic. On a two-machine cluster it can become the second largest line, which makes--topology publica serious alternative for test environments. - Network Load Balancer in front of the control plane: a few tens of dollars a month.
- EBS, the 48 GB disk on each machine: a few euros each with gp3 volumes.
- S3, where the state store lives: a few megabytes, a figure that barely registers.
The honest comparison with EKS goes like this: you save the $73 a month of the managed control plane, but you give part of it back paying for the machine your own control plane runs on. On a small cluster the balance still favours kops, but it’s less of a win than the managed-service price alone suggests, and above all it doesn’t account for the time you’ll spend on upgrades.
Cleaning up
If you followed this article to try kops out, the most important thing to do now is take the cluster down. A cluster in private topology costs money purely by existing: NAT Gateway and load balancer are billed by the hour even when not a single application is running on top, and they’re the classic surprise for anyone who leaves a Friday afternoon experiment switched on.
The good news is that kops knows how to undo what it built, and it does it with a single command. First, though, it’s worth removing from the cluster the AWS resources it didn’t create, but which appeared while the cluster was running: the load balancers of the exposed services and the disks requested by the applications.
kubectl delete svc --all-namespaces --field-selector spec.type=LoadBalancer
kubectl delete pvc --all --all-namespaces
Then comes the real command. Without --yes it behaves as a simulation: it lists everything that would be deleted and stops.
# simulation: read the list
kops delete cluster --name=$NAME --state=$KOPS_STATE_STORE
# and only then, for real
kops delete cluster --name=$NAME --state=$KOPS_STATE_STORE --yes
Always run it without --yes first, and read that list. This isn’t fussiness: the command asks for no confirmation and can’t be undone, and that simulation is your only chance to notice you loaded another cluster’s variables by mistake.
Two things survive the deletion, and need removing by hand if the goal is getting back to the clean account you started from:
- The state store bucket. It has versioning enabled, so emptying it of the current files isn’t enough: before you can delete it, every previous version has to be removed as well.
- The IAM group, the user and its access key created at the beginning. In order: detach the policies from the group, remove the user from the group, delete the key, then delete user and group.
What we took away from it
The point isn’t that kops is better than EKS. The point is that the cost of managed Kubernetes is almost fixed, while the cost of a small cluster isn’t, and below a certain threshold that fixed part dominates the spend. Add to that the ability to create the same cluster on AWS, Google Cloud or Azure with a single tool, and for the kind of work we do kops is still the right choice.
Who, in our view, would be better off leaving it alone:
- Anyone without a person on the team willing to upgrade Kubernetes. With kops the upgrades are your responsibility, they come round every few months, and putting them off only makes things worse. If that sentence made you wince, pay the $73 and sleep well.
- Anyone who needs high availability without wanting to design it. A single control plane is a single point of failure, and taking it to three across three zones triples that line of the bill. EKS includes control plane availability in the price.
- Anyone living inside the AWS ecosystem. If the plan is to lean heavily on managed services and the platform’s native integrations, with EKS that road is already paved. With kops you get to the same results, but by doing more work.
There’s a broader lesson too, and it’s the one that convinced us to write the article: infrastructure notes age worse than code. Our internal procedure said “create a gossip cluster” and it worked for years, until that line became first discouraged, then impossible to follow, and in the end a debt to pay on clusters in production. Code that no longer compiles tells you immediately; a command that keeps working while everything around it changes tells you on the day the upgrade refuses to start.
The upside is that a kops cluster can be rebuilt: its complete description lives in a bucket, and what you’ve read here is the sequence to make it again from scratch. In the next article we’ll cover how we put applications on top of it, from publishing services to certificates and releases. See you then.