Producing the motion doesn't mean getting the task right

For embodied large models like operational-intelligence ones, the most awkward part right now isn't the hand — it's the ledger. Whether the motion can be performed, you can see in public demos; but how do you prove it did it right in your scenario — nobody gives you an answer on that.

I've been using one for a month, running my own batch of grasping tasks. Success rate on the test set looks decent; swap in a different batch of materials, change the lighting at the workstation, and the number slides down.

How much it drops I won't pin down. The direction is clear: down. This is exactly what I ran into doing fraud detection. The model's metrics look great on historical samples; the fraudsters switch tactics and rule coverage is immediately exposed. The model didn't get dumber. The eval set never covered that path, so on unfamiliar samples performance naturally drops.

Problem is in the definition. It counts "picked it up" and "picked it up correctly" as one bucket — the ones gripped crooked, dropped halfway and caught again, all counted as success. Report that number and it's like merging fraud cases and normal cases to compute a pass rate — you can't attribute the false-positive rate at all.

More than the motion library, what embodied models should be filling in right now is a tiered definition of outcomes. Which tier counts as success, which as failure-but-recoverable, which as hard failure — nail that down first, then talk about capability ceilings. Otherwise the smoother the demo, the more nervous you feel in deployment.

I mentioned something similar in my ev¹ post last week: teaching it one commonly traveled path is more useful than letting it try every path. Probably the same logic in the embodied field.

How do you all evaluate embodied models — by task success rate, or by recoverable failure rate?