From a Video Clip to a Robot in Motion: How the Party OS Path Works

I spent two days trying out RoboParty's Party OS. Conclusion up front: worth using, especially if you already have a humanoid robot and some demonstration motions on hand; but it's not something that gives you an interface after installation—it's more like a pipeline, and going in empty-handed you'll get stuck at the first step.

Party OS is a robot operating system made by RoboParty Lab, supporting the development and operation of applications like home service robots. The route described in the repository is very direct:

Chaining together data collection and generation, motion retargeting, imitation learning and unsupervised learning, and gradually expanding to human–object interaction and human–system.

Translated into plain language. Teaching a humanoid robot a motion involves three hurdles. The first: record the human motion, called data collection. The second: translation. A human's arms and a robot's arms don't match in joint count or length, so you have to convert the human motion into how many degrees each robot joint should rotate—this is called motion retargeting, similar to adapting piano sheet music for guitar. The third: let the robot find patterns from a bunch of demonstrations itself, rather than you writing rules one by one—this is called imitation learning. Party OS's value is chaining these three hurdles into one pipeline, writing all the glue code in between for you.

Walkthrough: from a video to joint trajectories

My borrowed environment was a small team doing humanoids; the machine was a second-hand modified one with low degrees of freedom and few movable joints. The whole process took me a day and a half, half of which was spent stepping on pitfalls.

1. Prepare materials. A human motion—motion capture is easiest; without mocap, use monocular video. I used the latter; the cost will be explained later.

2. Set up the environment. Ubuntu plus Python, following the repository README step by step. It's fairly tightly bound to the ROS version—I installed the wrong version and the retargeting script wouldn't even start, and the error reported had nothing to do with the actual cause.

3. Do skeleton mapping. This is pure manual work. Map the human skeleton joints one by one to the robot's joints, written as a mapping table. The most time-consuming step in the whole process, and the least automatable.

4. Run retargeting, output a trajectory file of joint angles.

5. First replay in simulation—don't rush to the real machine. Watch two things: whether there's mesh penetration, and whether joints exceed limits.

6. On the real machine, slow down to about 30% speed for the first run. My first run here, the robot self-locked halfway through raising its arm.

7. Once the trajectory is satisfactory, feed it as demonstration data into the imitation learning stage. Only at this step does it start to "learn"—everything before was just "transferring."

Three pitfalls I stepped on

The first is coordinate systems. Human skeleton data generally has Y-axis up, while the robot base has Z-axis up—a ninety-degree difference. My first run, the robot was lying down in simulation. First confirm the up axis on both sides, then touch other parameters.

The second is joint limits. Retargeting only calculates angles; it doesn't care whether your robot can achieve that angle. A human elbow can bend backward a bit; a robot can't. Leave a safety margin for each joint in the mapping table—it's easier than adjusting parameters afterward.

The third is foot slipping. Human motions have no contact constraints—the retargeted legs trace beautifully in the air, but start slipping once they hit the ground. Especially obvious with video sources—monocular estimated depth is inherently inaccurate. In my testing, the results from mocap and video paths differed quite a lot; use mocap if you can.

Also, frame rates from different sources are inconsistent, so motions will speed up and slow down erratically—don't skip the timestamp alignment step.

After it works, what to try next

My suggestion is don't touch the human-object interaction layer yet. Compressing the closed loop of "collect one motion, retarget, reproduce on real machine" to within two or three hours is far more useful than adding one more feature. Once the loop is stable, then try motions with objects, like grabbing a cup—at that point you'll find the difficulty shifts entirely from retargeting to contact and force control. A few days ago when I wrote about Weiyi's inspection system I said something similar—the bottleneck in inspection isn't at the model layer, and here it's the same: the bottleneck isn't in the algorithm.

Looking ahead two or three years, what this type of robot OS competes on is foundational work like data pipelines and real-machine alignment—the algorithms themselves are public. Whoever can compress the time from "one demonstration to one successful reproduction" to a few hours will stake out the ecosystem first.