Making maps with the help of small UAVs seems to be easy enough. However, when geometric accuracy is critical, how do you determine and state the estimated accuracy of your mapping product? One way is to establish a test bed with a redundancy of accurately surveyed check points which are used as independent bench marks against which the map can be compared. In this picture we show a comparison between high accuracy GPS derived check point coordinates and corresponding coordinates as derived from our 3D model. Note that for illustration purposes the horizontal errors were scaled by a factor of 1000.
The 3D model was created from 261 aerial images (13mm GSD) captured by a Sony a6000 camera with 16mm fixed lens. The camera was carried by a Pixhawk equipped modified Steadidrone quadcopter. The flying time was 10 minutes and, using our V-Map system (http://www.microaerialprojects.com/v-map/ ), we could accurately survey the 12 ground control points in about 15 minutes.
Using 105 check points in our error analysis we can now confidently claim that when we map similar terrain with this particular method we can realistically expect to achieve the following accuracy:
Horizontal approximate circular error at 95% confidence 0.028m
Vertical Accuracy at 95% confidence: 0.039m
Not bad, especially when considering that the 4.75ha map can be delivered the very next day after image acquisition.
Comments
As always, very nice work Walter!
Very interesting. Have you run any tests in tall grass or another non-hard landcover? You essentially tested absolute accuracy of the system but my guess is that both horizontal and vertical accuracy radically decreases in a tall vegetation. I've done a lot of research and my thesis looking at LIDAR accuracy in tidal wetlands and even with lasers we struggle to find the ground surface.
@Martin: Maximum horizontal error was 0.041m, maximum vertical error was 0.063m. But a point by point comparison is not really the whole story here. Due to the extremely high density of points on the surface - some 1800 of them per square meter - and the fact that their errors appear to be random - you get a much more accurate model than you could economically get from RTK. The test field was chosen because it contains permanent road markings that can be used for ground control and check points (hence no need for target maintenance) and because it represents a surface that is typical for a lot of as built survey work on roads etc. The size of the job is small but that is where the niche lies for UAVs - small jobs ordered today and delivered tomorrow. Also, there is enough data in those 4.75ha to keep a CAD jockey busy for at least a day.
@Marek: The camera was fixed to the air frame. Hence all images are near nadir. Pitch angles up to 8 degrees, roll values up to 4 degrees - as derived from Agisoft's PhotoScan. The job was controlled by 12 classic ground control points (which were excluded from the error analysis).
@me: For this exercise I used the mean of two independent dual frequency GPS post processed solutions (carrier phase ambiguities resolved) to obtain coordinates for the GCPs as well as the check points. You are absolutely right when you say that my errors are within what you would expect from a classical RTK survey. In fact, the SfM results could be used to verify the GPS survey - that is how good they actually are. A huge amount of survey work is done by RTK and this little test shows that SfM is just as good, if not better than RTK. Now go try and tell that to the survey profession. ;-)
Is that a pun, Dan?! Funny thing was that we did the survey during summer and we almost died from the heat trapped in this super insulated space!
Good job! We are always looking for data like this to prove to our clients how accurate UAV modeling really is. This is great as its done by someone "out of house!"Given that RTK GPS is at best 3cm accuracy in terms of heighting, the results are very close. I think a more definitive comparison would be using a theodolite to measure the control for such a small base line. Our test areas with using theodolite placed control are excellent. Obviously you will get zero discrepency at the actual control point as it is used in the transformation of the model but in those areas between tn the control and the areas extrapolated outside of the control will give you errors. As I always say. the only errors you can get in photogrammetry are control point accuracy, miss identification of control points and insufficient over/side lap. Otherwise it is super accurate. We once gave 2mm interval contours for a concrete slab in a frozen produce warehouse produced from a camera mounted onto a trolley 8m high, pushed on a predetermined "flight path". The client didn't believe us that our results were accurate enough. To prove it, we marked the low spots on our map and translated and marked them on the slab. Then we threw a buck of water over the slab and it settled right in our marks.
Depending on how you surveyed the GCPs, you're within the error of the test data. RTK GPS would have precisions on that level. Are you running dual-frequency RTK on the UAS, using targets, or simple L1 code positioning on the UAS?