Before letting a flying robot avoid obstacles on its own, don't rush to real hardware

I compared taking a machine straight into the woods to test, versus first getting the perception, planning, and control chain working in simulation before going to the real machine — ran both sides. Conclusion up front: the cost of one prop strike in the former is enough to let you tune parameters for three rounds in front of your computer.

Autonomous flight in complex environments breaks down into three things — seeing obstacles, deciding where to go, and executing the motion — jargon: perception, planning, control. The most common beginner mistake is tuning control parameters right off the bat, when actually the coordinate frames aren't aligned yet, so everything downstream is wasted effort. I wrote a post a few days ago about joint module calibration — the same thinking applies here: align the coordinate frames first, then talk about control.

Let me explain a few terms. Point cloud is just a bunch of spatial points scanned out by lidar, like tracing an object's outline with raindrops. Autonomous flight means no remote stick — the machine decides where to fly and how fast.

In the prep phase, before powering on, confirm the props aren't installed backwards — mount a CW/CCW prop wrong and it flips the instant it takes off. After connecting the ground station, check the attitude angles first — at rest, roll and pitch should be near 0; if they're off, do a level calibration first. From my testing, skip this step and it'll keep drifting in one direction during hover.

Then go into simulation. Pick a scene with a few pillars and low obstacles — don't start with a forest. In settings.json, turn on both camera and lidar, confirm the corresponding Enable flags are true, run the official Python example, have it fly to the first waypoint, and check whether moveToPositionAsync returns normally. For obstacle avoidance, start with the dumbest rule — if point cloud appears within some meters ahead, stop and replan, cap speed under 1 m/s, and first see whether it can get around. Expected behavior: the machine flies smoothly to the point, stops, then decelerates before the pillar, sidesteps, and moves forward again — the motion will be a bit jerky, that's normal.

Let me note a few pitfalls. If it keeps timing out on connection, it's probably that the IP and port on the simulation side and the script side don't match, or you ran the script before the Unreal window finished loading — wait until the scene is fully rendered before connecting. If the camera feed is all black, check whether camera_name matches what's written in settings — if the name doesn't match, you get an empty image. If the point cloud is empty, either lidar isn't on, or the coordinate transform is written backwards — I ran into a case where the sign on the attitude angle was wrong, and the machine thought it was flying upside down. There's also the case where it flies fine in simulation but drifts the moment it's on the real machine — that's the sim2real gap: wind, ground effect, sensor noise, none of which exist in simulation.

There's a paper in Science Robotics that uses ultrasound instead of cameras and lidar, on the grounds that the latter two fail in darkness and thick fog. That approach reminds me of one thing — don't assume your sensors will definitely work in the target environment.

Next step is to throw the same obstacle avoidance strategy into a simulation scene with wind disturbance and run it again, see whether the thresholds need loosening, and only after that do low-speed test flights on the real machine. What these products will compete on going forward is whether a strategy tuned in simulation can be transferred to the real machine with low loss — the flight controller itself can't create much of a gap anymore.