Comparing Precision of Autopilots for Survey Missions - The Results - Blogs

Hi,

We’ve reached the final chapter in our series of articles in which we compare the precision of different drone autopilots from various manufacturers.

All drones were flown on a typical area scan route created using UgCS flight planning software and compared in terms of how accurately can they stick to survey lines on horizontal and vertical planes.

Drones flown in these tests range from hobby-level drones such as Pixhawk with Arducopter firmware and popular consumer-grade drones such as DJI Phantom 4 to more professional drones like the DJI M600 RTK. This list even includes a very expensive drone with Kestrel autopilot that is used in the military and by law enforcement agencies - the Lockheed Martin Indago.

Since precision had to be measured numerically, Python scripts were written which compared the .kml file from each flight to the .kml file of the test mission. Only points from straight survey lines were used, points from turns (when transitioning from one survey line to the next) were ignored. Then the closest horizontal and vertical distances from each point in flight were compared to the test route. The table below presents the average and maximum horizontal and vertical deviations.

Autopilot / turn type	Turn type	AVG Horizontal Error	MAX Horizontal Error	AVG Vertical Error	MAX Vertical Error
DJI A2	stop and turn	0.62	2.7	0.45	1.71
DJI A2	bank turn	1.61	9	0.44	2.26
DJI A2	adaptive bank turn	1.03	5.13	0.35	1.66
3DR Iris - Pixhawk with PX4 fw	straight	0.32	0.91	1.27	1.96
3DR Iris - Pixhawk with Arducopter fw	straight	0.3	0.77	0.15	0.41
3DR Iris - Pixhawk with Arducopter fw	spline	0.54	1.74	0.34	0.57
DJI Naza-M V2	adaptive bank turn	1.04	3.29	0.62	1.96
DJI Naza-M V2	stop and turn	2.56	9.14	0.41	1.38
DJI Naza-M V2	bank turn	1.03	4.11	0.57	2.18
MikroKopter Quad XL		0.86	3.19	0.93	2.66
DJI Phantom 3	adaptive bank turn	0.16	0.55	0.09	0.23
DJI Phantom 3	stop and turn	0.18	0.53	0.13	0.63
DJI M600	stop and turn	0.28	1.04	0.08	0.19
DJI M600	adaptive bank turn	0.31	1.02	0.08	0.19
DJI M600 RTK	adaptive bank turn	0.14	1.19	0.03	0.1
DJI M600 RTK	stop and turn	0.1	0.45	0.03	0.1
DJI Phantom 4	stop and turn	0.31	1.36	0.11	0.3
DJI Phantom 4	adaptive bank turn	0.36	0.74	0.19	0.41
DJI Inspire 1	adaptive bank turn	0.16	0.53	0.36	0.85
DJI Inspire 1	stop and turn	0.26	1.12	0.26	0.85
Microdrone MD 4-200		0.58	2.31	9.76	11.49
LM Indago		0.67	3.05	4.73	17.81
Yuneec H520	stop and turn	0.1	0.34	0.62	0.8

As seen from the table, two drones with the lowest horizontal average error of 0.1 meters are the DJI M600 RTK and Yuneec H520. On a vertical plane, however, the DJI M600 RTK proved to be more accurate with an average vertical error of 0.03 meters.

It is also worth mentioning that Pixhawk autopilot with Arducopter firmware while being one of the cheapest drone setups in this list, proved to be quite accurate with an average horizontal error of only 0.3 meters and average vertical error of 0.15 meters.

Please note that these tests were not made under ideal circumstances. Environmental factors such as wind and temperature were not identical in all tests so it is likely that they had some impact on the results.

If you have any questions or ideas for more tests we could do, make sure to leave a comment.

Newest version of UgCS mission planning software can be downloaded here: https://www.ugcs.com/en/page/download

Wishing you safe and precise flights,

UgCS Team

Very cool. However, if the error is calculated on how well the resulting log sticks to the plan (if I understood this correctly), then you have two major factors that confuse things:

1 - the accuracy of the onboard GPS and, most importantly, how much variation there is between readings. A GPS that gives readings jumping around by a metre every second will result in hopeless error no matter how good the overall setup (it will be feeding you generous streams of error while sitting on the ground). As an extreme example, let's say you have a really lazy GPS that gives you location rounded to the nearest 10 metres. Your error will end up being very small, even though your drone will likely be flying like drunk. Ideally, you should have a completely independent setup for measuring (say, the same autopilot for all tests with RTK GPS, simply powered up and logging, nothing else).

2 - The hardware and tuning of the DIY vehicles. Quoting a mass produced drone model should be sufficient, they all have identical hardware and software or close enough. DIY ones nominally using the same autopilot can be completely different. What motors/ESC for what weight, what props, what GPS, what tuning params?A build with aggressive params, high thrust to weight and high performance GPS/baro will stick to the flight plan as if possessed. My bet is that if you run the same test for a bunch of random builds of Ardupilot or PX4 (or even DJI professional autopilots on random builds) you are likely to encounter similar variation as you got between the different manufacturers.

Overall, while very interesting, I would argue that this is not so much an autopilot comparison as a build comparison and its weakest point is error self reporting as opposed to an identical logging payload across tests. A really cool autopilot comparison would be to see how different autopilots drive the exact same vehicle as measured by a dedicated logger identical across tests.

You need to be a member of diydrones to add comments!

Join diydrones

Comments

Alexey Dobrovolskiy February 6, 2019 at 7:35am

With DJI M600 RTK - we used DJI RTK kit of course.
Tersus GNSS February 6, 2019 at 4:33am

Hi, there, Which RTK system you are using? thanks.
Alexey Dobrovolskiy January 23, 2019 at 3:03pm

I would like to say some word to protect Lockheed Martin reputation)))

We received our Indago's at winter. It was around -15 centigrees, strong wind from Baltic and snow.

We were impressed that it flies promised 45 minutes even in such conditions (with payload). Indago still the only drone which we had in our hands which flies exactly same time as promised by manufacturer.

From numbers promised by another vendors usually we have to deduct 20-30% to get real flight time.

Indago is a great drone for purpose it was build - video surveillance, scouting, SAR missions. But not for photogrammetry ))).
Andreas Gazis January 23, 2019 at 12:55am

Sure guys, by all means, your results can be valuable and are definitely welcome. I am just pointing out what, I think, is a very easy mod to do if you are planning on doing any future tests. Well done either way.
Kristaps Brass January 23, 2019 at 12:52am

Hi Andreas,

Thanks for raising these points, both of them are valid.

It is a good idea to have a separate autopilot with RTK GPS onboard, that would make the results more accurate and comparable. By no means are these tests we did perfect, same way that temperature and wind conditions were not consistent which also had some impact on the results. We just wanted to give an insight on approximately what precision could you expect.

We did give information on DIY vehicles, however. Most of these drones were mass produced. One which was assembled by us was the DJI Naza-M V2. It was sitting on an F450 frame with standard parameters. The other vehicle which you could call DIY was the 3DR Iris+ with Pixhawk. Even though it is a mass produced drone and was assembled in the factory, Ardupilot and PX4 firmware of course allow for modification. However the parameters again were standard ones aside from calibrations.
Andreas Gazis January 22, 2019 at 11:15pm

Very cool. However, if the error is calculated on how well the resulting log sticks to the plan (if I understood this correctly), then you have two major factors that confuse things:

1 - the accuracy of the onboard GPS and, most importantly, how much variation there is between readings. A GPS that gives readings jumping around by a metre every second will result in hopeless error no matter how good the overall setup (it will be feeding you generous streams of error while sitting on the ground). As an extreme example, let's say you have a really lazy GPS that gives you location rounded to the nearest 10 metres. Your error will end up being very small, even though your drone will likely be flying like drunk. Ideally, you should have a completely independent setup for measuring (say, the same autopilot for all tests with RTK GPS, simply powered up and logging, nothing else).

2 - The hardware and tuning of the DIY vehicles. Quoting a mass produced drone model should be sufficient, they all have identical hardware and software or close enough. DIY ones nominally using the same autopilot can be completely different. What motors/ESC for what weight, what props, what GPS, what tuning params?A build with aggressive params, high thrust to weight and high performance GPS/baro will stick to the flight plan as if possessed. My bet is that if you run the same test for a bunch of random builds of Ardupilot or PX4 (or even DJI professional autopilots on random builds) you are likely to encounter similar variation as you got between the different manufacturers.

Overall, while very interesting, I would argue that this is not so much an autopilot comparison as a build comparison and its weakest point is error self reporting as opposed to an identical logging payload across tests. A really cool autopilot comparison would be to see how different autopilots drive the exact same vehicle as measured by a dedicated logger identical across tests.
3D Robotics

Chris Anderson January 22, 2019 at 7:18am

Such interesting data! Thanks for doing this.

It's telling that by far the worst result was from the biggest Aerospace company on the list, Lockheed Martin (LM Indago). Open source software on cheap hobby airframes beats aerospace technology costing 100 times as much. DIY FTW ;-0

Also the Yuneec H520 is running PX4

This reply was deleted.