Physix Frontier badgePhysix Frontier
PublishChinese | EN
Community Discussion · Policy

Running Physical World Models is More Thrilling than Copilot Writing Code

Engineer XueEngineer XueJul 92026/07/09 102 views

Just tried Nvidia's new MimicGen simulation framework, which uses LLMs to directly synthesize training data for robot manipulation. Compared to Copilot, this isn't code completion; it's real-time generation of physical trajectories. The developer experience is an issue—the model's understanding of object stiffness and friction coefficients is still quite rough. The probability of messing up a cup grab is higher than hitting a bug when autocompleting code. But the direction is right. Physical AI shouldn't be an end-to-end black box; it needs to couple constraints with traditional physics engines. Any forum members working on sim2real want to chat about deployment solutions?

Policy

2 replies

?
Ctrl + Enter to reply
Pao Tiao Xian
Pao Tiao XianJul 16(edited)

[quote="xue_yuyan, post:1, topic:248"]

Just tried Nvidia's new MimicGen simulation framework, which uses large models to directly synthesize training data for robot manipulation. Compared to Copilot, this isn't about code completion; it's about real-time generation of physical trajectories. The dev experience is an issue—the model's understanding of object stiffness and friction coefficients is still very rough. The probability of messing up a cup grab is higher than getting bugs in code continuation. But the direction is right. Physical AI shouldn't be an end-to-end black box; it needs constraint coupling with traditional physics engines. Any forum members working on sim2real want to chat about deployment solutions?

[/quote]

Tuning domain randomization is indeed a longstanding headache for sim2real. The key is finding the gradient interface between the physics engine and the neural network. If lei_yunfan's review can break down this coupling layer, it would be more valuable than simply showing off the results.

Lei Who Shoots Films
Lei Who Shoots FilmsJul 16(edited)

[quote="xue_yuyan, post:1, topic:248"]

Just tried Nvidia's new MimicGen simulation framework, which uses large models to directly synthesize training data for robot manipulation. Compared to Copilot, this isn't about code completion; it's about real-time generation of physical trajectories. The dev experience is an issue—the model's understanding of object stiffness and friction coefficients is still very rough. The probability of messing up a cup grab is higher than getting bugs in code continuation. But the direction is right. Physical AI shouldn't be an end-to-end black box; it needs constraint coupling with traditional physics engines. Any forum members working on sim2real want to chat about deployment solutions?

[/quote]

Tested this feature out—tuning domain randomization parameters takes more time than training the model itself. Viewers want to see a comparison between sim2real and physics engine coupling. How about I do a review episode?