On-device LLM deployment stalls first at model conversion

Spent the weekend messing with this edge-plus-cloud inference product from North Pole Bear Core (Beiji Xiongxin) and hit quite a few pitfalls. The name is a bit convoluted, so I'll split it into two halves by usage: one half runs on the board, the other goes over the network.

My scenario isn't complex—small industrial quality inspection tasks. Images come in, get a first-pass screening, and anything uncertain gets sent up. I previously played with ESP32 for three months and know that the edge-side game is not easy, so I kept my expectations very low.

What I got was an evaluation kit plus SDK. Setting up the environment went fairly smoothly—the cross-compilation toolchain has ready-made scripts, and getting the demo running took half an afternoon. The real hurdle was model conversion. I took an open-source small-parameter multimodal model and tried to convert it. The toolchain supports mainstream quantization, but operator coverage is incomplete—the first conversion threw a bunch of unrecognized ops. Checked the docs; it says "supports common structures." What counts as common, you have to find out yourself. This is exactly the same wording problem I complained about before—one set of claims on the promo page, another set in the SDK repo's documentation.

Switched to a model with a simpler structure and it converted. After quantizing to low bits, the drop was noticeable—on the same test set, local results were a notch worse than cloud. I can accept that; both parameters and precision were cut. What surprised me was latency: a local inference is basically on the order of tens of milliseconds, no waiting for the network—the advantage in weak-network scenarios like tunnels and basements is real. I also hooked up the cloud path; under normal network it's two to three hundred milliseconds, and when it jitters it can go past a second—the difference is obvious.

The cloud's advantage is a higher capability ceiling—complex problems still have to be sent up. So what I ended up with is a hybrid approach: the edge handles most requests first, and uncertain ones get forwarded to the cloud. This combination has been repeatedly mentioned in industry discussions as the engineering optimum for most products, and from what I've run, I agree with that judgment.

Downsides must be mentioned too. First, the deviation between documentation and actual behavior—especially in model conversion; time spent on pitfalls was longer than writing business code. Second, debugging tools are on the sparse side—when it crashes locally, unlike the cloud you can't pull logs; you can only print bit by bit over serial. Third, the team threshold—this isn't something you plug in and use; someone needs to understand cross-compilation, quantization, and operator adaptation. On my side it was cobbled together by force; a smaller team would most likely get stuck.

Advantages are latency and data not leaving the local device—solid. Data like medical, financial, and factory process parameters shouldn't be sent out in the first place; edge is the only solution.

Conclusion depends on the situation. For automotive, industrial, IoT—weak network or data-sensitive, with an embedded person on the team—it's worth adopting. If you just want to add Q&A capability to an app, a cloud API is cheaper and easier; no need to mess with boards.

Looking ahead a year, I guess pure edge and pure cloud will both move toward the middle—edge handles wake-up and first-pass screening, cloud handles fallback; this line will become the default architecture. Those doing selection now should abstract the interfaces early—don't weld your business to one side.