I ran the numbers with inference and training combined

I compared Verda's set of inference and frontier model training infrastructure with the two separate clusters we originally bought, and actually ran it once. It took about an afternoon plus an evening.

First, why I looked at it. Two months ago I wrote a piece saying that separating training and inference is the most expensive waste in the RL era. At the time my judgment was simple: training clusters pursue throughput and high-speed interconnect, inference clusters pursue low latency and high concurrency, the hardware configurations on both sides are naturally different, so most teams just buy two sets. The problem is that on the RL path, training and inference run alternately, one side runs while the other stops, and the stopped side is pure idle. From a financial perspective, idle GPUs are the most expensive thing, they're depreciating at the same time and occupying cash flow at the same time.

Verda's selling point is erasing this boundary, the same set of resources can handle both training and inference. AI Infra refers to the underlying facilities that provide compute, network, and storage for large models, and the model runs on top of it.

The activation process was simpler than I thought. Register, bind a card, choose a region, then enter the console. Resources are divided by instance specs, there are training-leaning models with large VRAM and high interconnect, and cost-effective models leaning toward inference, and the network column marks bandwidth. My first reaction was, isn't this just putting two price lists side by side. Clicking further down, it gives a training-inference consistency option, meaning the same batch of data, the same environment, after training you can switch directly to inference without repackaging the image or reconfiguring dependencies.

I've really suffered on this point. In the past when training and inference were separated, the most annoying thing was environment drift, the hardware itself wasn't the problem. Dependencies that ran fine on the training side would error out when moved to the inference machine, and upon checking it was a CUDA version or some library mismatch, and fiddling with it took half a day. This kind of work produces no business value, but the man-hours are genuinely burned.

I took a small open-source model and ran a small task, first doing a few rounds of fine-tuning with GRPO. Let me explain GRPO here, it's a reinforcement learning method whose core is having the model generate a set of answers to the same question, then comparing within the group, whoever scores above average gets rewarded, saving the overhead of separately training a scoring model. After running, I switched directly to inference, requesting the same model, the latency I measured was acceptable, no obvious cold start.

There were two places where I got stuck. One was data upload, my dataset wasn't big, but the upload channel speed fell short of expectations, later I switched regions and it was better. The other was quota, the default concurrency given wasn't high, I submitted a ticket to raise it, and waited most of a day. Neither of these is a product defect, they're common process friction in cloud services, but if you're in a hurry, you need to reserve time in advance.

The pleasant surprise was billing. It charges by actual occupied duration, instances stopped don't incur charges, and the interval of switching between training and inference doesn't require restarting the machine. I punched it into a calculator, if converted by our previous two clusters' idle rate, the book cost of the same batch of work can be pushed down. How much exactly depends on your idle ratio, this number differs for every team, I can't give a unified answer.

There are downsides too. First, the console's information density is high, first time in it's easy to get lost, it took me over ten minutes to match the billing rules with the specs. Second, its network requirements aren't low, if your data is in a local data center, the upload segment will become a bottleneck, best to put the data in object storage in the same region first. Third, on ecosystem it's still fairly general-purpose, if you heavily depend on a specific adaptation layer of some domestic framework, migrating over requires writing some glue code yourself.

Conclusion here. Who it suits: teams doing inference models, running RL post-training, with frequent alternation between training and inference, especially those who bought two clusters themselves and whose idle rate never comes down. Who it doesn't suit: teams that only occasionally run inference, or whose model scale is very small and a single machine is enough, using this set is overkill, the math doesn't work out.

Next I plan to migrate a small segment of the online inference pipeline over for canary testing, to see long-term stability and the billing curve. One afternoon of testing doesn't prove much, whether cash flow is healthy needs a few quarters to tell.