The foundation of physical AI: what usually gets stuck isn't the model
Community Discussion · General

The foundation of physical AI: what usually gets stuck isn't the model

PM YuanPM Yuan6d ago2026/11/13 156 views

A friend recommended Qingyan Tech's physical AI infrastructure and foundation models, saying it was worth trying out to see if it's actually any good. I didn't take it too seriously at first. The term "physical AI" has been shouted from the rooftops all year, but most so-called infrastructure, when you break it down, is just a simulation environment bundled with data annotation tools.

What won me over was another thing he said: they're doing geometry-and-physics-driven general object representation. In plain terms, it means letting the model know this is a cup, and also know how heavy the cup is, where its center of gravity is, and which way it'll topple if you push it.

I come from an imaging AI background, so I have a bit of an obsession with representation. The hardest step for imaging models is getting the model to understand what shape and material something actually is in 3D space—segmentation accuracy is secondary. So I took the thing I know best and tried it out.

The process went like this. They gave me a test environment—not the kind where you click around on a webpage, you have to read the docs, set up the environment, and run examples first. I got stuck on the very first step. The example used a standard set of rigid bodies, but my data was irregular bone structures reconstructed from CT, with noisy surfaces and non-closed boundaries. Feeding it in directly, the object representation step produced a mushy shell.

After talking with their engineers, I learned the preprocessing step can't be skipped. That geometric representation requires the input to be a watertight mesh—meaning the model has to be a closed surface with no holes. My data needed mesh repair first.

That step took me about an afternoon. After fixing it and running again, the output was genuinely different. It outputs a shape along with a set of physical properties—mass distribution, collidable boundaries. I used a virtual gripper to push it, and the direction it fell was close to what I expected. That experience was kind of interesting, because back in imaging, the model outputs a 3D structure and that's it—nobody cares whether it falls over or not.

The pleasant surprise was the verifiable learning part. Simply put, every operation during training, the system can judge right or wrong on its own—no need for a human to check one by one. For product people this is hugely valuable. In healthcare and robotics, the most expensive thing is having people annotate and review; compute is actually cheap. Apple's HALO ran 360 tasks across six unfamiliar environments with an average success rate of 91%, relying on massive simulation data self-verifying. That approach is right.

But I need to be clear about who this isn't for.

If you don't have real physical scenarios and data on hand, this thing is basically worthless to you. It's not a model where you sign up and it generates a video. It requires you to have objects, tasks, and clear success/failure criteria. When I tried it, what tripped me up was having to first figure out what counts as "correct"—it had little to do with model capability.

A few other issues. The docs are thin—lots of concepts you have to ask people about; simulation accuracy and speed are at odds—crank up accuracy and a single run takes forever, lower it and the conclusions aren't trustworthy. The "simulated inaccurately, computed too expensively" complaint from that earlier team—I felt it in this environment; the ecosystem is still early, with limited hardware and formats you can connect to.

I can't give a verdict on cost—I used the test environment and didn't get to the business side. Last month I helped an imaging team set up GPU cloud compute, similar approach: get it running on on-demand machines first, don't rush to scale up.

Next I plan to run a surgical instrument model I have through the full pipeline—from scanning to representation to a simple grasping strategy—to see if it can hold up on small, smooth, non-rigid objects. Rigid bodies are easy; soft tissue and instruments are the real test.

2 replies

?
Ctrl + Enter to reply
Zhi Wei
Zhi Wei5d ago

The watertight mesh step is so real. Our CT-reconstructed bone data is also full of holes—just repairing them scares off half the people.

Tian Ji
Tian Ji5d ago
Reply to Zhi Wei

I also got stuck for a whole afternoon on the watertight mesh step with CT bone data. You can't get output until you fix that mass distribution and collision boundary; feeding it in with holes unfilled just gives you a mushy shell.