Why Kubernetes isn't for every project

The system was one Next.js application talking to one Postgres database. In production, it used a median of 12 millicores of CPU. We ran it on two GKE Autopilot clusters, one for development and one for production, each with its own ingress controller, certificate manager, Helm releases, monitoring, and rollback tooling. Production also had autoscalers and a disruption budget.

Over two days in October 2026, we moved both environments to Cloud Run. The migration removed about $146 per month in cluster management fees before workload costs, deleted roughly 2,500 lines of infrastructure code and documentation, and replaced Kubernetes rollout machinery with a managed revision system. Then we investigated the application's recurring performance problems and found the part that Kubernetes could never scale for us: five missing database indexes.

Kubernetes was doing its job. The problem was that we had given it a job this application did not need.

The platform around one container

The application itself was ordinary: Next.js 14 with standalone output, Prisma, and Postgres 15 on Cloud SQL. It listened on port 3000. Background cron jobs and a report exporter already ran as Cloud Run Jobs, so we had quietly decided that serverless was good enough for everything except the web server.

The web server lived in two GKE Autopilot clusters in asia-southeast1. The separation was deliberate. Production had different deletion protection, certificates were managed through a cluster-scoped issuer, and development had seen 847 commits in eight weeks. Keeping frequent development deployments isolated from production sounded sensible.

But cluster isolation came with a full stack in each environment. Terraform and Helm installed nginx-ingress and cert-manager. We reserved static IPs, created workload identities, and granted a human container administrator access so Helm could create ClusterRoles. The app repository carried separate deployments, services, ingresses, and certificate issuers for development and production. Production added an HPA, a VPA, and a PodDisruptionBudget.

That was only the visible platform. We also maintained a secret injection script, a 300-line interactive rollback script, a cluster cost calculator, kubectl cheat sheets, secret rotation documentation, and a 500-line monitoring dashboard written around GKE metrics. Alerts watched for restarts, evictions, and application errors filtered by cluster name.

A deployment fetched an environment blob from Secret Manager, validated its shape, converted it into a Kubernetes secret, applied the manifests, changed the image, added a change-cause annotation, and waited up to ten minutes for the rollout. The annotation had to happen after the image change so our own rollback script could explain each revision.

We reserved 616 millicores and used 12

The production workload requested 616 millicores of CPU. Its measured usage was 12m at p50, 50m at p95, and 75m at peak. Depending on the point in the distribution, we reserved about 8 to 50 times what the app used. Development requested 2 GiB of memory and peaked at 0.61 GiB.

Autopilot bills requested CPU and memory rather than actual use. It also provisions nodes to fit those requests, which makes the request-versus-capacity metric sit near 100 percent. That number looks healthy while saying almost nothing about waste. We had to build a custom dashboard around usage divided by request to see the useful ratio.

The dashboard contained an older comment about a pod sitting at 2 vCPU while using 1 millicore. We had already diagnosed the pattern. Kubernetes had turned right-sizing a tiny web process into an ongoing operational task, and getting it wrong cost money without creating an obvious failure.

The fixed cost made the mismatch worse. GKE charged $0.10 per cluster-hour, about $73 per month. Two clusters meant about $146 per month before either one ran a pod. Workload charges and two reserved IPs came on top. The second cluster bought useful isolation, but separate Cloud Run services provided the isolation without a second control-plane fee.

Most of the infrastructure supported Kubernetes itself

Leaving GKE showed how much code existed to support the platform rather than the product. One application cleanup removed 1,618 lines and added 69. One infrastructure cleanup removed another 932 lines and added 48. Together they deleted the Kubernetes manifests, rollback and cost scripts, GKE Terraform module, custom dashboard, and most of the supporting documentation.

Cloud Run did not need direct replacements for most of it. Google's front end replaced nginx-ingress. Managed certificates replaced cert-manager and the ClusterIssuer. Cloud Run's autoscaling removed the HPA and VPA, while its revision model made our change-cause annotations and rollout-history workaround unnecessary. Rollback became one command that moves traffic to an immutable revision.

gcloud run services update-traffic production-service \
  --to-revisions=<revision>=100 \
  --region=asia-southeast1

Secrets became simpler too. Kubernetes' env-file handling had forced us to write strict validation around quotes and multiline values. On Cloud Run, we mounted the secret as a file and loaded it through Node's dotenv parser. A custom compatibility layer disappeared.

The footguns were not hypothetical

The strongest case for simplifying the system came from problems we had already hit, or narrowly avoided.

• Deleting a manifest from Git did not delete the live Kubernetes object. When we reverted a realtime voice service, its Deployment, HPA, ingress, PDB, and Service could have remained in the cluster until someone deleted them manually. Git and the cluster could disagree indefinitely because kubectl apply was not using --prune.
• A production pod reached 9.66 GiB of memory on August 28. It had requested 4 GiB but had no limit, so it kept growing until the node ran short and evicted it. The growth was off-heap, beyond the protection of the V8 heap cap, and needed its own alert and runbook.
• GKE deletion protection lived in Terraform state rather than as a switch we could change with gcloud. Removing a protected cluster required one apply to disable protection and another to delete it.
• The Helm provider depended on the cluster endpoint declared inside the module being removed. Once the module disappeared from configuration, Terraform could not forget its Helm releases because the provider configuration was gone. We removed the release state by hand for both clusters.
• The two cluster modules declared the same project-level administrator binding for one person. Destroying the development module's copy could have removed that person's production access, so we had to detach the duplicate state without destroying the underlying grant.

None of these are arguments that Kubernetes is broken. They are reminders that its state model, provider graph, resource lifecycle, and permissions remain our responsibility even when Google manages the nodes. That responsibility can be justified. For one small HTTP service, ours was difficult to justify.

The bottleneck was one table scan

After the move, we compared our alerts with the hardware that was actually under pressure. Production Cloud SQL had one vCPU and 3.75 GB of memory. Connections peaked at 29 out of 100, memory at 62 percent, and disk at 53 percent. CPU was the exception: it reached 95 to 100 percent in 48 of 864 ten-minute windows over six days.

The spikes followed two repeatable workloads. Every night, tradeCheckReportOutletAvailabilityTimeSeries issued 12,002 per-outlet queries averaging 418 ms. Those queries consumed 72 percent of database time in the window because TradeCheckReport.outletId had no index.

Every afternoon, a CSV export read about 1.29 million rows in 2,590 batches. Per batch, an Attachment query with 500 IDs took 1.33 seconds, and a TradeCheckReportTransportation query took 0.53 seconds. Their join columns were not indexed either. An outlet availability chart averaged 4.5 seconds, and range queries on Releasing.pickedUpAt took 1.7 to 2.9 seconds.

We added five indexes with CREATE INDEX CONCURRENTLY, then updated the Prisma schema so the code described the production database. We created the indexes concurrently without taking the application offline.

This changed how I thought about the whole migration. The HPA could add web pods, and the VPA could resize them, but neither could make a sequential scan use an index. More application capacity would only let the web tier send expensive queries to the database faster. For many CRUD and B2B systems, the database does the hard work while the stateless web process is a relatively thin shell. We had concentrated our scaling work on the web tier rather than the database bottleneck.

The cutover was deliberately boring

We did not switch production in one leap. Cloud Build deployed the same image to GKE and Cloud Run, so both environments ran in parallel. We changed DNS from an A record to a CNAME pointing at Google's host. If the new service failed, rollback was a DNS change while the cluster was still alive.

The domain mapping took about an hour to become ready, almost all of it spent waiting for Google to issue the managed certificate. Once it was ready, traffic moved to Cloud Run. We removed development first, then lifted production's deletion protection and removed the production cluster after the new service had settled.

Cloud Run was not free of ownership mistakes. CI owned the image and secret mount while Terraform owned the rest of the service. Our first Terraform apply after migration removed the secret volume that Cloud Build had added, and development returned 503 errors because DATABASE_URL was missing. We fixed the contract by telling Terraform to ignore the volume field. Moving platforms removed Kubernetes-specific complexity; it did not remove the need to define who owns each setting.

What Cloud Run made worse

Cloud Run fit this workload better, but the trade was not one-sided.

• HTTP/1 request bodies are limited to 32 MiB. Our nginx configuration allowed 50 MB, so uploads in the gap now fail.
• Memory is a hard ceiling. A GKE pod with a 4 GiB request could burst above it until node pressure caused an eviction. A Cloud Run instance that exceeds 4 GiB is killed, and choosing more than 4 GiB also requires at least 2 vCPU.
• CPU is throttled between requests by default. Work that continues after sending the response slows down unless we pay for always-on CPU.
• WebSockets have a maximum 60-minute request, no sticky sessions, and consume one concurrency slot each. Scale-in gives roughly ten seconds of shutdown grace, far below the 630 seconds our GKE pod once had. Long voice sessions can be cut off.
• A separate realtime service behind path-based routing would need an HTTPS load balancer, adding roughly $18 per month instead of using a free domain mapping.
• Cloud Run domain mappings remain a preview feature without an SLA. Certificate issuance also required us to leave the DNS record unproxied while Google validated it.

Production now runs with 1 vCPU, 4 GiB of memory, a minimum of one instance, and a maximum of six. Development uses 1 vCPU and 2 GiB, with no minimum instance, so it can scale to zero. Production is still an always-on service and therefore still costs money. The confirmed saving is the removal of the two cluster fees; a fair Cloud Run cost comparison needs a full billing cycle.

When I would still choose Kubernetes

I would choose Kubernetes for a system with many cooperating services, especially when network policy, service discovery, sidecars, or a mesh solve real problems. It also makes sense for long-lived consumers, heavy streaming or WebSocket traffic, StatefulSets, GPUs, custom operators, unusual scheduling requirements, and workloads that need a long termination window.

The economics also change at sustained high utilization. Reserved capacity can beat request-based pricing when the machines stay busy. A platform team can spread cluster fees and operational expertise across many services. Portability and multi-cloud requirements may be worth the extra abstraction too.

Our application had none of those properties. It was one stateless HTTP container beside a managed database, and most of the time its traffic fit inside a small instance. In that shape, a cluster was overhead paid in money, code, and attention.

A checklist before creating the cluster

Before putting a new application on Kubernetes, I would ask:

• How many independently deployed services exist today, rather than on the roadmap?
• Does the workload need pods, nodes, sidecars, custom networking, or long-lived processes, or does it only need to answer HTTP requests?
• Who will own request sizing, upgrades, policies, observability, state drift, and incident response?
• What fixed cluster cost will we pay at zero traffic, and how many applications share it?
• Where is the measured bottleneck? If it is the database, will scaling the application tier help?
• Which Cloud Run limits would the product actually hit?

Kubernetes gives a team control over almost every layer between source code and a machine. That control is valuable when the product needs it. This application did not. Removing the clusters did more than lower our bill: it let us stop tuning a 12-millicore web process and spend the time on the database queries users were waiting for.