Cloud

Kubernetes Cost Optimization: From $12k/month to $4k Without Sacrificing Uptime

A practical guide to right-sizing your K8s cluster, using spot instances safely, and setting up VPA/HPA so you only pay for what you need.

CloudJune 30, 202610 min read

Last year we were paying $12,400/month for a Kubernetes cluster running a logistics platform. Three months later, after methodically applying these techniques, the bill was $3,800. Here's exactly what we did.

Step 1: Right-Sizing with VPA

Most K8s deployments are massively over-provisioned. Developers set resource requests conservatively to avoid OOM kills, and nobody ever revises them. VPA (Vertical Pod Autoscaler) fixes this automatically.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Off"  # Start in Off mode — just collect recommendations
  resourcePolicy:
    containerPolicies:
    - containerName: api
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 2Gi

Run VPA in 'Off' mode for 2 weeks first. Check `.status.recommendation` to see what it would set. We found 70% of pods were using <30% of their requested CPU.

Step 2: Spot Instances for Stateless Workloads

Spot instances (AWS) or Preemptible VMs (GCP) are 60-90% cheaper than on-demand. The catch: they can be terminated with 2-minute notice. For stateless API pods, this is fine with proper disruption budgets.

# Node pool with spot instances
apiVersion: v1
kind: NodePool
spec:
  template:
    spec:
      nodeClassRef:
        name: spot-node-class
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]  # Fallback to on-demand
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

# PodDisruptionBudget — always keep 80% of pods up
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: "80%"
  selector:
    matchLabels:
      app: api-server

Step 3: HPA with Custom Metrics

Default CPU-based HPA is too slow to react to traffic spikes. We switched to request-per-second metrics via Prometheus, which scales 3x faster.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"  # Scale when > 100 req/s per pod
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down

Cost Savings Summary

Right-sizing with VPA: -$2,100/month (removed 40% of unused CPU/memory)
Spot instances for API tier: -$3,800/month (moved 70% of workload to spot)
HPA tuning (fewer idle replicas): -$1,900/month
Karpenter bin-packing (fewer nodes): -$800/month
Total savings: $8,600/month (69%)

KubernetesAWSCostDevOps

← Back to Blog