Two Weeks on TPU: The Bill Says More Than the Benchmarks

A friend recommended Google's TPU to me, saying they'd managed to push down training costs on their end. I had the team go open a quota, and after nearly two weeks of back-and-forth, we gave it a try to see how well it actually works.

First, let me be clear about what this thing is. TPU is Google's in-house custom chip, technically called an ASIC, meaning a chip built for one specific class of tasks, not like a GPU that can do everything. It was first announced at Google I/O in 2016, designed by Broadcom, used internally at Google as early as 2015, and only opened up to the public on the cloud in 2018. The latest product line splits into two tracks: TPU 8t for training and TPU 8i for inference.

We didn't buy hardware. Google only confirmed in April 2026 that it had started delivering TPU hardware to some customers, and at our scale we can't get in line, so we went with spinning up TPU VMs on Google Cloud and renting by the hour.

Just opening the quota took over a week, requiring us to explain the use case, scale, and roughly how long we'd run. That experience was so-so, but understandable — when compute is tight, everyone's like this.

The real hurdle was migration. Our original training scripts were written in PyTorch, and TPU is smoothest with JAX and TensorFlow; PyTorch has to go through a compiler called XLA. Three people on the team, and getting the first script running took about four days, with the biggest blocker being unsupported operators and not-very-friendly error messages. Once we got past that, it actually went smoothly.

The pleasant surprise was communication. The most annoying part of multi-card training is setting up your own interconnect and tuning all that all-reduce stuff; with TPU, the chips are directly connected, and you basically don't have to deal with it in code. What you save here is people.

Let me start with the pros — energy efficiency. The latest TPU v7, codenamed Ironwood, is said to match Nvidia's B200 in memory capacity and have an edge in energy efficiency. Google published first-gen data years ago: 15 to 30 times the performance of CPUs and GPUs at the time, and an order of magnitude better performance per watt. That benchmark is a bit old, but the direction hasn't changed. Large clusters are also less hassle — once you scale up, you don't have to build your own wheels for interconnect and scheduling. The billing structure is clear too: you rent by chip-hour, not the murky mess of building your own data center.

There are three cons. What I care about most is the tight ecosystem lock-in — painful to migrate in, painful to migrate out, a strategic-level problem. Small models aren't worth it: no matter how strong the compute, once the model is small enough, the bottleneck is data loading, not computation, so renting it is a waste. And you can't get the hardware — building your own is basically hopeless, you just have to accept the cloud-rental form.

My judgment is: it depends, and in a fairly conservative way.

Large model scale, long-term repeated training, and someone on the team willing to grind through JAX — if any one of these three is missing, the ROI doesn't add up. Conversely, for small-to-medium teams still in rapid trial-and-error on models, or those already deeply invested in the PyTorch ecosystem, it's not suitable. The cost of one migration is likely higher than the compute money you save. I ran the numbers on our end: at our current volume, the payback period is about a year, assuming training tasks stay continuously saturated.

One thing needs to be clear: TPU can't replace GPU. It's more like a path to push down unit cost on specific tasks. Jensen Huang himself has said that in the ASIC space, Google's TPU is a special existence. Coming from a competitor's mouth, that carries different weight.

When I wrote about edge devices before, I said adaptation and toolchains matter more than benchmarks. Applied to TPU, that judgment hasn't changed.

If you really want to migrate, the performance hurdle is actually easy to clear — migration cost is what you need to calculate clearly in advance.