
Full-Stack Inference Compute Solution: From an Empty Machine to the First Inference Result
A friend recommended Sunrise's full-stack inference compute solution, saying it cleans up the model deployment mess pretty well. I took an afternoon last week to try it — from an empty machine to the first inference result. Verdict up front: it runs, but the first half hour is basically spent wrestling with the environment.
First, let's clarify two terms. Inference is the process where, after a model is trained, you ask it something and it answers — the same thing as typing in a chat box. A full-stack compute solution packages the chip, driver, scheduling software, and model service into one whole set; you just call it, and you don't have to assemble the middle layer yourself.
Why try this. I'd previously assembled a stack myself: find a machine with a card, install drivers, install the inference framework, pull model weights, start the service, then wrap an interface layer myself to expose it. There's a technical article about GPUStack in the Ascend community, and one line stuck with me:
Although the mainstream official inference engine MindIE's multi-machine distributed inference performs reasonably well, the configuration process is extremely complex — from environment preparation and configuration initialization to parameter details, every step requires extra caution, otherwise it's very easy to fail deployment due to a missed detail or configuration error.
That's about Ascend, but it's about the same on any platform. The pitfalls I hit myself are basically all in that sentence.
Let's put the two approaches side by side.
| Stage | DIY | Full-stack solution |
|---|---|---|
| Driver and framework versions | Align yourself, easiest to screw up | Bundled in the image |
| Pulling model weights | Manual download, format conversion | Pick from a list |
| Starting the service | Assemble startup parameters | Fill in a few numbers |
| External interface | Wrap a layer yourself | OpenAI-compatible interface given directly |
| Adding machines to scale | Reconfigure everything | Add a node |
The biggest difference is the last row. With DIY, adding a machine means redoing every step above; with the full-stack approach, it's one more node, and the service side worries less.
Hands on, from login to result.
1. Log into the console, find Compute Nodes in the left menu. First glance at the node status column — online is green. If offline, don't go further; it's most likely network or the driver isn't up.
2. Go to Model Service, click New Service. Pick a 7B-class small model from the model list. Don't go for the biggest right away — reason below.
3. It'll ask you for two parameters. Concurrency, meaning how many requests it can handle at once; max length, meaning the most characters it can spit out in one go. Use defaults for both, don't touch them yet.
4. Click Deploy and wait for the status to go from Creating to Running. For the 7B scale, mine was ready in a few minutes.
5. Once the service is up, the page gives you an address and a key. OpenAI-compatible interface means the code you originally wrote for OpenAI works directly after swapping the address and key — not a single line of call logic needs changing.
Then verify. On the same machine I first hit the local address:
curl http://
-H "Authorization: Bearer
-d '{"model":"
The returned JSON has a content field, and the chain is connected.
Pitfall one: network. The service is up on the machine, but you can't connect from outside. Don't rush to change configs — first curl the local address on the machine. If local works, the service is fine, and the rest is all network policy.
Pitfall two: picking too big a model. The first time I went straight for a big one, ran out of memory, the service came up then crashed, and the logs only gave an out-of-memory. Get 7B running first, then scale up — chain verification and performance verification are two different things, don't mix them.
Pitfall three: concurrency. The default is on the high side, and latency starts jittering under load testing. In my testing, cutting it to half stabilized single-request latency. This number depends on both the card and the model — don't copy someone else's.
From login to the first result returned, the actual platform operations took under an hour. The big savings were in the one to two hours of version alignment. The 7B model's single-request first-token latency was in the few-hundred-millisecond range, and several rounds of back-and-forth showed no stutter. Pitfall two cost the most time — rushing to a big model directly, back-and-forth restarts wasted about forty minutes.
I wrote a post on inference cost at the end of July; that one did the math, this one is about whether it can actually land. People who work on chips tend to habitually look at which tasks a solution pushes into the software layer. I spent a few years on 5G basebands, so I'm sensitive to this: with the same compute, how well the scheduling is written directly determines whether you can get around the power wall.
After learning this, you can try two next things. One, push concurrency up and see where that card's latency inflection point is — that number is more useful than a benchmark score. Two, hook up your own business interface, like throwing a customer service conversation at it for classification, and see how long end-to-end takes.
Model deployment will likely become as widespread in the next year or two as moving to the cloud once was. Installing it isn't hard; the hard part is managing it — being able to clearly state the boundaries of cost, concurrency, and stability. That's all.
Physix Frontier