Running Inference Without a GPU: A Beginner's First Steps

Last month I helped a client with inference-side selection, and for the first time I seriously got hands-on with Euclyd's non-GPU architecture system. At the time my first reaction was the same as most people's: no graphics card, so isn't that just brute-forcing it with CPU—how fast can it be? But after opening an account and running it for two weeks, my thinking changed a bit. This post walks through my path from zero to getting results, for people who've never touched it.

First let me explain three terms, otherwise the rest is all jargon.

Inference is the process after a model is trained where you ask a question and it answers. Training is like going to school, inference is like going to work. School only looks at whether you test well; work looks at how many people you can serve per minute.

Quantization is storing the numbers in a model in a coarser way. The benefit is the model gets smaller and runs faster; the cost is occasionally answering more roughly. Currently 4-bit compression is the most cost-effective tier—I looked through several comparison materials and the conclusions were all consistent.

Throughput and latency: one is how many people can be answered per second, the other is how long one person waits from asking to getting an answer. These two numbers often fight each other.

From opening an account to running it through, six steps

1. After registering and entering the console, it asks you to choose a compute spec. Newbies should just pick the smallest tier—don't worry about parameters for now, just enough to run through.

2. Find the model management section and mount an already-compressed small model. After the progress bar finishes, the status changes from "loading" to "ready." This may take a few minutes, so don't refresh repeatedly.

3. Use the built-in dialog box to send the first sentence. I usually have it "condense the following passage into three sentences," then paste in a news article. Seeing it spit out one character at a time means the link is working.

4. Switch to the API page and copy that key and request address. Send a request once from the command line; the expected result is a JSON response. This step is for connecting to your own system later, but even if you don't connect, run it once to confirm.

5. Adjust concurrency from 1 to 8 and watch how those two numbers move. In my testing, latency basically didn't change between 1 and 4, and started rising at 8.

6. Finally, look at the monitoring panel, focusing on the memory bandwidth column.

Three pitfalls, I stepped in two

The first is weight format. Compressed models come in several formats; GPTQ and GGUF are the two most common—the former leans toward the graphics card route, the latter is more universal. Mount the wrong one and it either errors out directly or loads and then does nothing, with no prompt on the interface at all. Newbies are most likely to burn an afternoon here.

The second is pushing concurrency too hard. As concurrency goes up, at some point latency suddenly jumps a tier, and you have to find that point yourself. The core contradiction is that removing the graphics card saves procurement cost, but the bottleneck itself doesn't disappear—it just moves from VRAM to memory bandwidth. This is the same logic as my earlier post on reducing weight in collaborative robots: what you save doesn't necessarily land where you want to save.

The third is don't pull the context to the maximum right away. The longer the context, the more it has to reread for every character it answers, and latency rises faster than intuition suggests.

There's a saying I somewhat agree with: most problems in production environments are flaws in system design.

After running it through, the next step is to try two things. One is to put the same model on this system and on the machine with a graphics card you have on hand, run the same batch of questions, and compare latency and cost per thousand calls. The other is to slowly push concurrency up and find the point where latency starts to spike—that point is basically the actual capacity of your line.

The scenarios I tested are on the lighter side, all short-text Q&A. How long-document summarization and such would perform on it, I haven't tried yet. Anyone who's run it can chime in.