Getting Started with Simulation Data Assets: One Week to Master Synthetic Data Augmentation

A friend recommended synthetic data assets to me, so I tried them out to see if they're actually useful. I have a detection model where long-tail scenarios never get filled in, and real collection takes weeks per round, so I figured I'd give it a shot.

First, what it is. Synthetic data assets are data computed by physics engines and rendering pipelines — images, annotations, depth maps — not shot by cameras. The core use is data augmentation: you have only a small amount of real samples, and the platform generates a large batch based on them to fill in rare scenarios. Annotations are just boxes telling the model what's at what location in the image.

Below is a log of the three time periods I actually went through.

Day one, feed in the seed data. After registering, go into the console, click New Project on the left, fill in the project name, choose the task type — for detection tasks pick object detection; pick wrong and the annotation format won't be recognized later. Go into the project and find Dataset, upload seed data — that is, the small amount of real samples you have. Upload images and annotation files together; I uploaded COCO format json. Halfway through uploading it prompted that categories weren't mapped — that's the first pitfall. The platform's category system and the class names in your annotations are two different things. Without mapping, all generated boxes come out empty. In Category Mapping, match the class names one by one. If that's too much hassle, just keep one class to get it running first. Then create a generation task — weather, lighting, occlusion intensity are all parameterized, you can change just one variable, and fill in quantity and resolution as needed. My first round I set it very small. After submitting it's queued, not immediate — on my end I waited about an hour before seeing the first batch.

Day three, quality check — don't rush to accept everything. The preview interface lets you flip through directly. Check three things: whether boxes fit tightly, whether lighting has jumps, and whether anything is physically unreasonable, like objects clipping through each other. In my batch, a small portion of boxes were noticeably off. Looking back, it was because a few seed images had blurry annotations to begin with. Going back to the seed set, removing dirty samples and re-running is faster than fixing them one by one in the results. The QC page supports batch filtering, sorted by score, lowest first. The more you open up the parameters, the better the diversity, but the more unreasonable samples too, and rework goes up with it. Don't be greedy on the first round.

A week later, mix the passing synthetic data into the original training set at a small ratio and run a round. In my testing, recall on long-tail categories improved, but once the synthetic ratio got high, real-machine performance actually dropped. That's the sim-to-real gap — looks great in simulation, doesn't hold up in reality.

| Dimension | Real collection | Synthetic augmentation |

| Collection cycle | Weeks | Hours |

| Long-tail coverage | Hit or miss | Parameters specifiable |

| Annotation cost | High | Amortized over generation volume |

| Distribution fit | Naturally consistent | Needs tuning to converge |

My approach is to add a little each round, keep an eye on the real test set, and roll back if it drops.

When Zibianliang released X2Real, they made a point: real-machine evaluation is costly and slow, and can't sustain high-frequency testing. I agree with that.

I used to think synthetic data was a stopgap, but now it looks more like a way to pull training and evaluation into the same reproducible environment.