An embodied AI model software stack, stuck at the calibration step

A friend recommended the Perceptron "robot perception-reasoning-execution software" stack to me, saying their team used it to save themselves the work of building a pipeline from scratch. I happened to have a desktop grasping task on hand where I needed to compare several approaches side by side, so I applied for a trial and ran it on and off for two weeks.

First, what it is. It's embodied large-model software — in plain terms, it splits "look once, think once, output an action" into three stages: perception, reasoning, execution. The perception end turns images into object poses and scene representations, the reasoning end uses a vision-language model to break a plain-language sentence into several sub-tasks, and the execution end translates those sub-tasks into joint targets. What they're selling is the interfaces for each of these three stages, plus the glue layer in between.

What I got approved was a Python SDK plus a doc, environment Ubuntu with ROS2. You calibrate the camera yourself. I'm familiar with the hand-eye calibration flow — I wrote a post a while back about powering up a joint module for the first time, where I mentioned how annoying zero-point alignment is. Same principle here, just moved from joints to cameras.

I like to run through simulation first before going to real hardware. I've used the Digua Uranus setup for a while — it learns motions and environment changes from real interaction data, supports multiple embodiments and multi-view continuous rollouts, and the closed-loop evaluation correlation is said to reach 0.98. I don't fully trust that number, but it's good enough as a pre-filter, at least to screen out obviously bad policies and save real-hardware time.

Getting started

The first demo was desktop grasping. Give it "put the cup on the plate on the left," the reasoning end breaks it into four steps, the execution end outputs joint trajectories, and it worked on the first try. Smoother than I expected.

The problem was timing. The reasoning step is on the order of hundreds of milliseconds, the control loop is on the order of a hundred hertz — the two frequencies aren't even in the same order of magnitude. What it outputs is action tokens, and the token rate is not the same thing as the control period — you have to interpolate yourself in between. The docs are very light on this; I only went back to check after seeing jitter.

Calibration was the second pit. The SDK examples use simulation intrinsics, and porting them straight to real hardware is wildly off. After recalibrating and rerunning, the grasp success rate finally looked reasonable. That back-and-forth cost me about two days.

Pitfalls

There's an article discussing embodied RSI that asks a sharp question: is the robot self-evolving, or just grinding the same problem set over and over. My two weeks lean toward the latter — the model itself isn't bad, but change the environment or the embodiment and everything ground before has to start over.

The third pit is failure feedback. The plan from the reasoning layer is basically open-loop; execution failures don't get fed back, so you have to wrap a state machine around it yourself. After switching embodiments the commands also need realignment — consistent with my earlier judgment that reusing old commands only makes things messier.

And the docs. Overall research-style: methods explained clearly, parameters under-documented, several key hyperparameters I reverse-engineered from the example code. In a lab that's no big deal; in an engineering team it's a scheduling risk.

Conclusion, plainly: it depends — I lean toward recommending it to teams that are already established.

The good: the three layers are cleanly separated, interfaces are clear, action representation is unified, and there's a decent foundation for cross-embodiment transfer. What it saves is exactly the glue code I least want to write — that's worth a lot.

The bad: engineering maturity is clearly lacking. Real-time behavior, failure recovery, monitoring — you have to fill all that in yourself. Calibration is basically unassisted, and the docs aren't friendly to newcomers.

Who it's not for: anyone who wants plug-and-play, or teams with nobody to handle calibration and the comms layer — they'll most likely stall at step two.

It's a good foundation, not a finished product.