[Practical Kubernetes FinOps: How Karpenter and OpenCost Actually Cut Your Cloud Bill]

Your EKS bill went up 18% last quarter while request volume stayed flat. Finance wants to know which team is responsible, and the platform team cannot answer, because the cluster arrives as a single line on the cloud invoice. At the same time, the nodes sit at roughly a third of their requested CPU, so you are paying on-demand rates for capacity that no workload is using. This is the gap between knowing the bill went up and being able to do something about it, and it is the gap this guide is meant to close.
This guide is for DevOps, SRE, and platform engineers who already run Kubernetes in production and are comfortable with deployments, requests, limits, and autoscaling, but have not yet connected resource metrics to dollars in a way that survives a rollout. Engineering managers and CTOs who have heard the FinOps pitch and want a grounded technical plan rather than another dashboard tour will get value here, too.

We cover five things: where a cluster actually loses money, how to configure Karpenter so it removes waste instead of causing incidents, how to install OpenCost and turn cost data into team-level accountability, a reproducible 30-day plan to get a measurable reduction without breaking SLOs, and the trade-offs that bite teams who optimize too hard. Karpenter and OpenCost are two of the CNCF projects widely adopted for Kubernetes cost control, and most of the work is in wiring them into a feedback loop rather than installing either one.

A few terms up front:

• Kubernetes FinOps is the practice of attributing cloud spend to the teams and workloads that drive it, and then adjusting engineering decisions based on that attribution. 
• Consolidation is Karpenter actively removing or replacing under-used nodes to pack workloads more tightly. 
• Cost allocation is the process of splitting a cluster invoice across namespaces, workloads, and labels. 
• Idle cost is the part of the bill you pay for reserved capacity that no workload is using.

Why Kubernetes FinOps is an engineering problem

Copy link

Cost reports tell you the bill. They do not change it. A dashboard that shows spend per namespace is useful for a meeting, but the number only moves when an engineer changes a request value, swaps an instance type, deletes an orphaned volume, or lets a node be reclaimed. That is the core reason Kubernetes FinOps belongs to the platform team and not only to finance: every lever that reduces the bill is a configuration change with a blast radius.

The work runs as a loop with four steps. You measure real usage against what workloads reserve, you attribute the cost of that gap to a team or service, you change something (requests, instance policy, or scheduling), and you verify the change against the next billing window before moving on. Skip attribution, and you cannot prioritize. Skip verification, and you cannot tell an optimization from a regression. Most failed Kubernetes cost optimization efforts stall because they stop after the first dashboard, which is the measure step, and never close the loop.

OpenCost owns the measure and attribute steps. Karpenter owns a large part of the change step, specifically the node layer, where a surprising amount of waste lives. Right-sizing requests owns the other large part. Treating FinOps cloud cost management as a tooling purchase is the common mistake. The tools are necessary, but the durable savings come from the loop they enable, run on a regular cadence by people who own the workloads.

Where your cluster actually burns money

Copy link

Before touching either tool, it helps to know the shape of the waste, because the fixes are different and the savings are very unevenly distributed. In most production clusters we work on, a small number of patterns account for nearly all of the recoverable spend.

The single largest source is usually the gap between requested and used resources. Kubernetes schedules on requests, and OpenCost (correctly) charges a workload for the greater of its request and its usage, so a pod that reserves 2 CPU and uses 0.3 is billed as if it needs 2. Multiply that across a few hundred pods, and you are paying for a cluster two or three times larger than the work requires. This is also why Kubernetes node monitoring on its own is misleading: a node can look "full" on requests while running nearly idle on actual usage.

The second source is node-level waste: on-demand instances for workloads that tolerate interruption, oversized or older-generation instance types, and idle headroom left over after a scale-down that never reclaimed the emptied nodes. This is exactly the layer Karpenter is built to attack.

The table below maps the common patterns to the signal that exposes them and the fix that removes them. Treat the savings as directional; the actual numbers depend entirely on your current headroom and workload mix.

A practical starting point for the requests-versus-usage check, before OpenCost is even installed, is to look at how much of each node's allocatable capacity is reserved.

# query/node-pressure.sh
# kubernetes node monitoring starting point: how much of each node's
# allocatable capacity is reserved by requests, versus live usage.
kubectl describe nodes | grep -A5 "Allocated resources"
kubectl top nodes   # live usage; requires metrics-server installed

If requested CPU sits well above live usage across most nodes, you have found the cheapest savings in the cluster, and you have found them before spending a cent on tooling. Note that kubectl top shows live usage, not historical peaks. For right-sizing, you want the P95 or P99 over a representative window, which means Prometheus and Grafana (or the metrics stack OpenCost reads from), not a single snapshot. OpenCost's job later is to turn that request-versus-usage gap into a dollar figure per team so the right people prioritize fixing it.

Configuring Karpenter to cut node waste without breaking production

Copy link

Karpenter is a node lifecycle controller that watches for unschedulable pods, provisions instances that exactly fit them, and removes or replaces nodes when they are no longer the cheapest way to run the current workloads. It is a CNCF project with a mature AWS provider; Azure (AKS) has its own provider, and a Cluster API provider is in earlier stages. The configuration below is AWS-specific. The scheduling and disruption concepts carry across providers, but EC2NodeClass and Spot mechanics are AWS only, so check your provider's docs before copying instance references.

Two custom resources do the work. A NodePool describes what nodes are allowed (instance types, architectures, capacity type) and how aggressively Karpenter may disrupt them. An EC2NodeClass describes the AWS-specific node template (AMI, subnets, security groups, IAM role). The single most important design decision is to split NodePools by how much disruption a workload can tolerate, rather than running one pool for everything.

One prerequisite to settle before any of this: if you already run the Cluster Autoscaler, do not let both controllers manage the same nodes. Karpenter and the Cluster Autoscaler overlap in responsibility, and running them against the same node groups produces fights over scale-up and scale-down. Plan the migration so Karpenter owns its own NodePools while you retire the corresponding Autoscaler-managed groups, and understand that overlap before you start.

Before the YAML below, here is the environment the examples were written against. Treat it as a reproducibility anchor, not a recommendation to stay on these versions, and not a claim that this exact combination was certified in a lab.

kubernetes            1.33
karpenter             v1 API (karpenter.sh/v1), chart 1.13.x
opencost              1.120.x
kube-prometheus-stack 71.x   (Prometheus + kube-state-metrics)
helm                  3.x

Validate exact versions against the CRDs and Helm charts you actually install before applying anything. One compatibility rule matters most: the Karpenter controller and the NodePool/EC2NodeClass CRDs must come from the same chart release. Mixing a controller from one version with CRDs from another has broken NodePool schemas across minor versions, so treat every Karpenter upgrade as a planned change, not a helm upgrade you run on a Friday.

Splitting NodePools by disruption tolerance

Copy link

The cost-optimized pool packs standard, multi-replica, stateless services. It uses WhenEmptyOrUnderutilized consolidation, allows Spot, and keeps instance flexibility wide so Karpenter can pick the cheapest fit and survive a Spot pool drying up.

The two NodePool files below are a reference pattern, not a complete Karpenter installation. They assume Karpenter is already installed and that an EC2NodeClass named default exists with working subnet, security group, AMI, and IAM role discovery. Applying these two files alone will not bring up Karpenter.

# karpenter/nodepool-cost-optimized.yaml
# Reference pattern: validate field names against the CRDs in your installed
# Karpenter chart before applying. The v1 API (karpenter.sh/v1) is assumed.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: cost-optimized
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        # Wide families plus minValues keeps several instance types in play.
        # Narrowing this is the most common anti-pattern: it quietly raises
        # cost and increases how often Spot capacity is interrupted.
        # minValues is version-sensitive: confirm your installed CRD supports it.
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
          minValues: 3
        # Gt 5 means instance generation 6 and newer. Use Gte 5 if you also
        # want to allow 5th generation; Gt 5 quietly narrows the capacity pool.
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h   # rotate nodes every 30 days for patching and drift hygiene
  limits:
    cpu: "1000"           # hard ceiling so a runaway workload cannot scale the bill
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
      - nodes: "10%"      # never disrupt more than 10% of the pool at once
      # Karpenter schedules are UTC; timezones are not supported. Adjust this
      # window to your team's actual business hours expressed in UTC.
      - nodes: "0"        # freeze underutilized consolidation during business hours
        schedule: "0 9 * * mon-fri"
        duration: 8h
        reasons: ["Underutilized"]

The stable pool is the safety valve. Single-replica services and stateful workloads schedule here through a matching toleration; it is on-demand only, and it consolidates only WhenEmpty, so Karpenter never evicts a lone replica to save money.

# karpenter/nodepool-stable.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: stable
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["m", "r"]
          minValues: 2
      taints:
        - key: workload-class
          value: stable
          effect: NoSchedule   # only pods that tolerate this land here
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    consolidationPolicy: WhenEmpty   # only reclaim nodes with zero workload pods
    consolidateAfter: 5m

Making aggressive consolidation safe

Copy link

WhenEmptyOrUnderutilized is where the savings are, and it is also where outages come from if the workloads are not ready for it. Two safeguards are non-negotiable on any pool that consolidates underutilized nodes. The first is a PodDisruptionBudget on every service that matters. Karpenter respects PDBs during voluntary disruption, so a PDB gives consolidation an application-level availability guardrail; without one, eviction control is far weaker and depends only on the deployment's own replacement behavior and spread.

# workloads/payments-pdb.yaml
# Without this, consolidation can drain every replica of a service at once.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments
spec:
  minAvailable: 2   # keep at least 2 healthy replicas through any disruption
  selector:
    matchLabels:
      app: payments

The second is the karpenter.sh/do-not-disrupt annotation for work that must not be interrupted mid-run, such as long batch Jobs. It blocks voluntary disruption (consolidation, drift, and expiry) for the pod's lifetime. It does not block manual node deletion or external interruption, so it is a hint to Karpenter, not a hard lock.

# workloads/nightly-report-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-report
spec:
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      containers:
        - name: report
          image: registry.example.com/nightly-report:1.4.2

he anti-patterns worth naming explicitly: a single NodePool for every workload class, instance requirements so narrow that Karpenter has no cheaper option and Spot has no diversity, and WhenEmptyOrUnderutilized on services with no PDB and no topology spread. Each of these turns a cost win into an incident. When the bill does not drop after enabling consolidation, the usual cause is overly tight requests: if every pod reserves more than it uses, there is no slack for Karpenter to reclaim, and the node layer cannot fix a problem that lives in the workload spec.

Before you enable consolidation on a production cluster, confirm the prerequisites are in place. Run this after the explanations above, not as a substitute for them:

• Workloads carry accurate requests (right-sized in the previous step)
• Critical services have PodDisruptionBudgets with correct selectors
• Workloads are labeled by team or owner for later attribution
• An EC2NodeClass exists and has been tested on a non-prod cluster
• Any existing Cluster Autoscaler interaction is understood, and the migration is planned
• A non-prod pilot of the NodePools has completed without incident

Making cost visible with OpenCost

Copy link

OpenCost is the CNCF incubating project for real-time Kubernetes cost monitoring. It was originally built by Kubecost and donated to the CNCF, and it is the open-source allocation engine plus the vendor-neutral specification behind it. It reads pod resource consumption from Prometheus, prices it against live cloud pricing, and attributes the result to clusters, namespaces, controllers, services, pods, and labels. For compute it charges the greater of a pod's request or its usage, which is the honest accounting that makes the overprovisioning from the previous section show up as money owed by a specific team.

Installing it against your Prometheus

Copy link

OpenCost is Helm-only now; the standalone manifests were removed. It needs an existing Prometheus with kube-state-metrics, node-exporter, and cAdvisor, which a standard kube-prometheus-stack install already provides. Point OpenCost at that Prometheus rather than running a second one, using the official Helm chart.

# install/opencost-values.yaml
# Field names vary by chart version; run `helm show values opencost/opencost`
# to confirm against the version you install.
opencost:
  prometheus:
    internal:
      enabled: false
  exporter:
    defaultClusterId: prod-eu
    env:
      - name: PROMETHEUS_SERVER_ENDPOINT
        # For sharded or HA Prometheus, use a global query endpoint
        # (Thanos, Cortex, or Mimir), not a single Prometheus pod, or
        # exported cost data will be intermittent.
        value: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
  ui:
    enabled: true

# install/opencost.sh
helm repo add opencost https://opencost.github.io/opencost-helm-chart
helm repo update
helm install opencost opencost/opencost \
  -n opencost --create-namespace \
  -f install/opencost-values.yaml

Cost allocation is available through the API immediately. The query below returns the last seven days of spend grouped by namespace, which is the first number worth putting in front of teams.

# query/allocation-by-namespace.sh
# Cost allocation for the last 7 days, grouped by namespace.
kubectl -n opencost port-forward deployment/opencost 9003:9003 &
# or: kubectl -n opencost port-forward svc/opencost 9003:9003  (if your chart exposes that service)
curl -s "http://localhost:9003/allocation/compute?window=7d&aggregate=namespace" | jq

Turning allocation into accountability

Copy link

Namespace-level numbers are a start, but most organizations bill back by team, not by namespace, and one team often owns several namespaces. The fix is label hygiene: agree on a team (or owner) label, get it onto every workload, and aggregate on it. Cost data is only as good as the labels under it, so this is worth doing before you rely on the numbers in any planning conversation.

Once the label is in place, aggregate by it, and include idle so each team also sees the reserved capacity it is not using. Aggregating by controller takes you from namespace-level to workload-level decisions, which is where right-sizing actually happens.

# query/allocation-by-team.sh
# Cost per team, including the idle (reserved-but-unused) capacity each owns.
curl -s "http://localhost:9003/allocation/compute?window=7d&aggregate=label:team&includeIdle=true" | jq

# Cost per workload controller (Deployment, StatefulSet, etc.) for right-sizing targets.
curl -s "http://localhost:9003/allocation/compute?window=7d&aggregate=controller" | jq

The honest caveat on Kubernetes cost monitoring with OpenCost: by default, it uses public on-demand list pricing, so the absolute figures do not reflect your negotiated discounts, committed-use, or Savings Plans. This is the weakest part of running the open-source engine on its own. The relative allocation between teams and the trend over time are reliable and are what you actually need for Kubernetes cost optimization; treat the absolute total as approximate and reconcile it against your real cloud bill, or wire in the provider's billing integration when you need accurate totals.

Once allocation is trustworthy, the operational pattern is a short recurring FinOps session, every two weeks works well, where the team with the largest allocation or the largest idle cost walks through the top few line items and commits to one change. That keeps Kubernetes cost monitoring from becoming a dashboard nobody opens.

A practical 30-day cost-reduction playbook

Copy link

This is the sequence we run on a new engagement. It is deliberately ordered so the safe, high-yield changes come first, and the riskier node migration comes after, so you can measure its effect.

In week one, measure. Install OpenCost against the existing Prometheus, get the team label onto workloads, and capture a baseline of cost by team and idle cost by namespace. Do not change anything yet; you need a clean before-picture to verify against later.

In week two, right-size requests. This is the cheapest and safest saving, and it usually dwarfs everything else. Use observed usage to pull requests down toward P95 plus a sensible headroom, one workload class at a time, watching for throttling and OOM kills. This step alone often recovers more than the node-layer work that follows, because it removes the idle reservation that everything downstream is priced against.

In week three, introduce Karpenter on the node layer. Roll out the cost-optimized and stable NodePools on a non-production cluster first, confirm that consolidation behaves and that PDBs hold, then promote to production. Move stateless, multi-replica services to the Spot-capable pool and keep single-replica and stateful work on the stable pool. Migrating gradually, one pool at a time, keeps the blast radius small.

In week four, verify and set the cadence. Compare the new cost-by-team numbers against the week-one baseline, confirm SLOs held, and establish the recurring FinOps session so the loop keeps running after the initial push. The reproducible part is the loop, not a one-time cleanup.

Run this checklist before each production rollout step, after the explanations above are understood and not as a substitute for them:

• Every service that matters has a PodDisruptionBudget
• Multi-replica services have topology spread across zones
• Single-replica and stateful workloads are tainted onto the stable pool
• Long-running Jobs carry the do-not-disrupt annotation
• NodePool limits are set so a runaway workload cannot scale the bill
• Disruption budgets cap churn and freeze consolidation during business hours
• Karpenter controller and CRD versions come from the same chart release
• A cost baseline is captured in OpenCost before the change

Define what success looks like before you start, so the verify step has something to measure against. Fill in the placeholders with targets that fit your cluster:

• Idle cost down by X% against the week-one baseline
• Request-to-usage ratio improved for the top 10 workloads by spend
• No SLO regression on the affected services
• No increase in OOMKills or CPU throttling
• No Karpenter-related incident during the window
• Team-level cost report reviewed at least twice

Risks, trade-offs, and how not to over-optimize

Copy link

Every lever in this guide has a failure mode, and the teams that get burned are usually the ones who pushed one lever too far.

Spot capacity is the headline saving, and AWS prices it well below on-demand (the discount is frequently large but varies by instance type, region, and time, so treat any single percentage as directional and check current Spot pricing for your own types). The trade-off is interruption. Spot is right for stateless, replicated, interruption-tolerant work and wrong for a single-replica database. Keep the boundary in the NodePool split, not in good intentions.

Consolidation churn is the second trap. Aggressive consolidateAfter values and wide budgets make Karpenter reshuffle the cluster constantly, which is disruptive even when no single eviction violates a PDB. Slower consolidation and business-hours freeze windows trade a little theoretical saving for a lot of stability, and that is usually the right trade in production.

Right-sizing has its own edge. Cut requests too close to observed usage, and you remove the headroom that absorbs traffic spikes, which shows up as CPU throttling and OOM kills under load. Size to a realistic high percentile with headroom, not to the average.

The meta-risk is optimizing past the point of return. The first round of work, right-sizing and basic consolidation, typically recovers most of the recoverable spend. Chasing the last few percent with ever-tighter budgets and exotic instance mixes adds operational fragility for diminishing savings. The recommended approach in most production deployments is to take the large, safe wins, set up the loop, and stop when the engineering time costs more than the cloud spend it saves.

If you take nothing else from this section, take the short list of what not to do early:

• Do not start with Spot for critical or single-replica workloads.
• Do not reduce requests based on averages; use a high percentile with headroom.
• Do not enable aggressive consolidation before PDBs and topology spread are in place.
• Do not treat OpenCost absolute totals as finance truth until billing integration is configured; trust the relative allocation and the trend.

A short troubleshooting reference for the most common symptoms:

What to decide after reading this

Copy link

If you take three decisions from this guide, take these. First, split your NodePools by disruption tolerance before you turn on aggressive consolidation; the stable-versus-cost-optimized boundary is what lets you save money without evicting the workloads that cannot take it. Second, right-size requests before blaming the node layer, because the largest and safest saving in most clusters lives in the gap between what pods reserve and what they use, and no node autoscaler can reclaim slack that does not exist. Third, run the loop on a cadence, since Kubernetes finops is a recurring engineering practice and not a one-time cleanup, and the tools only pay off when measure, attribute, change, and verify keep cycling.

Karpenter and OpenCost are widely adopted CNCF projects for Kubernetes cost control because they make those three decisions executable: OpenCost gives every dollar an owner, Karpenter changes the node layer safely, and right-sizing closes the gap they both depend on. Real finops cloud cost management is the discipline of using them together, on a schedule, with people who own the workloads in the room.

If you can see the overspend but do not have a team with the time or the confidence to walk through these steps safely, that is the kind of work we do in our Expert Kubernetes Services: auditing the current cluster, designing Karpenter policies, setting up OpenCost, and running a pilot optimization with your team. Most teams that come to us already have a working Kubernetes setup and just want someone to run the Kubernetes cost optimization loop alongside them, from metrics and configuration through to a measurable reduction.