After a Week of Batch Inference, I Shut Down the Real-Time Pipeline

As someone who's never touched batch inference, I gave Together AI's Batch Inference a try. The name sounds intimidating, but it's really just piling up a bunch of requests and throwing them over all at once, letting it run slowly, and giving you the results when done. It's a different path from real-time calls.

Day one, getting set up had no real barrier — register, grab an API key, similar to other inference services. What actually tripped me up was the format. Batch inference doesn't take a JSON array you casually throw together, it wants JSONL, the kind of file with one request per line. My first version stuffed the whole list into one big JSON, submitted it and got an error right away, and it didn't tell me which line was the problem — I flipped through the docs for a while before it clicked. After fixing it and submitting, the task went into the queue, and then it was just waiting. It doesn't return synchronously; after submitting you can only poll the status, or wait for it to give you a result file. Day one I got stuck on this, staring at the screen thinking it would spit out text in seconds like a real-time API, but nothing came.

Day three, once the volume went up, I finally felt where it saves. Calling one by one is slow, and not all the slowness is the model inference itself. Every request sent from the service has to establish a connection, handshake, authenticate, serialize, wait for the response, deserialize — this chain of repeated overhead is invisible when volume is small, but once volume grows it's substantial. Batching spreads that layer thin. On my end, for the same work, the batch run finished faster than sending one by one, and the cost column looked a lot better too.

But it's also picky about data. If a batch has a few malformed entries mixed in, the whole batch result comes out incomplete, and you have to go back, pick out the dirty data and rerun. A couple days ago I came across an article about cross-vendor batch processing, and there was a line I noted down.

You can, at the application orchestration layer, collect requests from different model vendors into the same batch processing pipeline and execute them in parallel, instead of letting each request fight its own battle.

I'm not at that level of complexity yet, but the idea is right. Most of the benefit of batch processing comes from the organizing you do up front.

A week later, I moved all the periodic work over — nightly reports, historical data backfill, the stuff that isn't latency-sensitive, throw it in and forget it. Real-time APIs are kept only for things like conversations that need instant responses.

Putting the two usages side by side.

	Real-time calls	Batch inference
Return method	Synchronous, seconds	Asynchronous, poll after submit
Throughput	Limited by concurrency	Noticeably higher after batching
Cost	Higher unit price	Lower unit cost
Good for	Conversation, interaction	Reports, backfill, evaluation

It's suited for teams that have periodic, large-volume, latency-insensitive inference tasks, and whose data itself is tidy. If you're doing real-time conversation, or your task volume is too small to be worth setting up a submission pipeline, or your data is very dirty and you don't want to clean it first — don't bother. My day-one usage of casually throwing an array in was basically wasted effort.

Looking ahead, I'd guess batch inference will go from a money-saving alternative to the default path. Real-time inference stays for interaction, and the rest — post-training data generation, evaluation, reports — will increasingly go through async batch processing. By then, whether you can use batch processing might affect your bill more than which model you pick.