Palentier: Using UAV Open Source Software to Map with Kites

Palentier is an Open Source software that was originally created by me and a good friend (M. Stange) for processing aerial images from un-manned aerial vehicles (UAVs) like the MikroKopter.  Palentier can process aerial photographs into georeferenced and scaled images which accurately reflect the location and size of objects in the real world.  These images can be exported to Google Earth or to GIS software, like ArcGIS, for review and analysis.

 

Our current attempt is to modify the software so that it could be useable by the Kite Aerial Photography (KAP) community. Several weeks ago, I asked the KAP forum for help with test images and rig GPS data. They really came through and I now have results from using Palentier with KAP imagery.  The system was tested using data from four places (Northern England, and Northern Libya, Southern Texas, and West Virginia).  All had good results and all but one (West Virginia which I don't have permission to share publicly) are shown here.

 

Nicholas Kirby provided several dozen images from his KAP work south of Tripoli in Libya.  The building in the photographs is a Roman mausoleum.   Nick provided Qstar GPS data from his location, the kite, and a stationary point (his vehicle).  Primarily the GPS data from the kite was used in Palentier's processing.

 

Google Earth File (~3.5 Mb)

 

After days of poor wind, Lindsey Kemp successfully  photographed an area near Eldwick, England.  He used i-gotu GPSes to record locations for both the kite rig and anchor.  Again, the kite's GPS data was the primary locational data used in Palentier.

 

Google Earth File (~2 Mb)

 

I used a ground based handheld GPS (Garmin) that I carried on my person, to process this aerial photograph of my crews excavating a burned rock feature near Del Rio, Texas. 

 

Google Earth File (~1.6 Mb)

I'd like to point out that you can realistically measure things in this image using the measure tool in Google Earth.  The  excavation unit is four meters long north to south.

 

The fourth project was conducted near Canaan Heights, West Virginia for USGS biogeographers.  They are using KAP and Palentier to map micro-floral plant communities in wetland environments where other means of mapping are too costly or too slow. ( I currently do not have permission to share their images but will post them here in the future if they grant it.)

 

Lessons Leared:

 

Camera Parameters

Palentier was originally tested using Canon cameras.  Canon cameras have a very rich EXIF header that  many other manufacture's lack.  When tested on non-Canon cameras, I found that some of the information about the camera's configuration had to be manually added.  To address this, we added new input fields into Palentier's "Options".  The main parameter that is lacking from non-Canon cameras is the metric sensor size.  www.dpreview.com keeps sensor size information for most modern cameras on the "Specifications" section of its website.

 

Ground Elevation and Altitude Override
For Palentier to work properly, it needs two things: one, the general elevation of the ground being photographed and two, the approximate height of the camera above that ground. To accommodate for this a new "Ground Elevation" option was added to the software.  We have found a glitch in the code that does not allow for the proper manual selection of the elevation.  Until we can fix this, a temporary work around  is to use the "Altitude Override" option and enter the approximate altitude the camera was flown at.

 

Compass Override and Compass Declination

The test data had only approximate declination information for the camera (which way the top of the camera was point at the time a photograph was taken).  From my own experience this is not unusual because the rig tends to have a lot of play in it as the wind buffets the camera constantly.  To account for this, a "Compass Override" option was created.  Unfortunately, there is a glitch in this code as well, and it does not work properly, yet.  Luckily, the "Compass Declination" option does essentially the same function.  So, this option was used to rotate the images until the approximate north in the photographs matched true north.

 

Offset North and South

The GPSes used in these tests, (i-gotu GT120, Garmin Map 60Cx, and Qstarz)  are only accurate to between 10 and 15 meters.  This low level of accuracy can throw the image locations off a bit.  To allow the user to "bump" the photographs into place, the "Offset North" and "Offset East" options are used.  The user can export a set of photographs, see that it needs to be moved one direction or another, pick the appropriate options and re-export for proper fit.

 

General Issues

The biggest issue challenge a user might have with using Palentier and KAP imagery is that it is not fully automatic.  When Palentier is used with MikroKopters or other UAVs, all the data  for geo-location and rectification are automatically generated and the process is as simple as a few clicks of the mouse.  Using Palentier with KAP involves more user input and may require multiple exports to Google Earth as well as fine tuning of parameters in Palentier.  This can be a little cumbersome. Furthermore, since there is lot of variance between individual photographs  taken from a kite, it will often be best for a user to adjust a single image at a time instead of being able to bulk process dozen of images at once.

 

Palentier does not correct the distortion in photographs.  All photographs have some level of distortion resulting from aberrations in manufacturing and some by design (like fisheye lens).  Distortion is also present in ortho photography when the camera is not pointed straight down.  The nature of KAP (using cheap cameras and wind driven flight) can exaggerate some of these distortion issues problems.  What this means to the user is that it may not be possible to have every portion of a kite aerial image lineup exactly with Google Earth (but it should still be pretty darn close).  Alignment problems can arise when there is great variability in the elevation of the ground (like when photographing the top of a mountain and the valley below).  We compensate for these alignment issues by processing the images in photogrammetry software, such as Leica Photogrammetry Suite, (which Palentier supports export to) but are not currently directly handled by Palentier.

 

Want to help?

Palentier is Open Source and meant to be an asset we can all take advantage for  fun and in our work. Any thoughts on how to improve the software are always welcome. I am not a very good programmer but I do understand the principals of photogrammetry.  If you are a programmer and interested in further developing the software,  we need your help.

 

Many Thanks
I am very grateful to Nick Kirby and Lindsey Kemp for sharing their aerial photographs and for taking the time to collect GPS data from their rigs.  You are both awesome and your help is genuinely appreciated.

 

Thank you!

 

This posting is also available on my blog: palentier.blogspot.com

Views: 2850

Reply to This

Replies to This Discussion

Good stuff! Mark could you add a ArduPilotMega and GPS to your kite setup just to log data? Not for navigation but to speed up the image processing . . . ??

Most problems for me seem to be caused by brief diagnostics when matching the files.

Please find included:

trigger1.gpx

(it is a translation of trace.txt.TRIG1-001.nmea made using GPICSYNC).

 

a jpeg file

_6335.JPG

 

Note there is 2374s time difference between UTC and camera time:

example:

Camera time
14:20:32

GPS time
13:00:06

 

This is why I have entered in the window

Options/Time difference: -2374 or 2374

In a my case:

Selecting 'Match' gives no matches.

 

BTW there is sa typo in the dialog box:

Time Difference

 

using

Palentier_Beta_v017p0.jar

 

Any ideas?

 

www.aerialrobotics.eu/qwerty/PalentierTest.zip

 

Krzysztof, It looks like there are a couple of things going on.  The actual time difference is -91,224 seconds (1 day, 80 minutes, and 26 second).  You camera's EXIF data has the date as Dec 5 and the GPS is Dec 4. Also, when I review the track log, it appears that the actual moment the UAV was above the target is different (my guess is 12:56:13 ).  If the time sync isn't the problem, it might be that there is a Lat/Long to UTM conversion problem causing this...

 

The other issue is related to the elevation field in the GPX header.  It looks like you're returning absolute elevation (altitude) above sea level.  For Palentier to work, it needs the approximate height above ground level.

 

I hope that helps.  The software obviously needs some work.   I appreciate your comments and testing the software.

 

-Mark

My autopilot stores the altitude above takeoff level, among others.

So this is close enough to AGL on flat areas - I will have simply to generate a custom gpx on my own here. Thanks for the tip about 1 day difference - completely missed it.

Krzysztof, you're very welcome.  FYI, I just but a newer version of Palentier online. -Mark

testing v0.19p0

 

minor thing:

loading

'

<?xml version="1.0" encoding="UTF-8"?>
<gpx
 version="1.0"
creator="GPSBabel - http://www.gpsbabel.org"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.topografix.com/GPX/1/0"
xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd">
<time>2010-12-15T18:34:12Z</time>
<bounds minlat="50.146015000" minlon="22.411716667" maxlat="50.149910000" maxlon="22.433950000"/>
<trk>
<trkseg>
<trkpt lat="50.149521667" lon="22.432130000">
  <altitude>200</altitude>
<time>2010-12-04T13:00:06Z</time>
  <course>94.000000</course>
  <speed>13.611111</speed>
  <fix>3d</fix>
  <sat>9</sat>
  <hdop>1.000000</hdop>
  <vdop>1.000000</vdop>
  <pdop>1.000000</pdop>
</trkpt>
</trkseg>
</trk>
</gpx>

'

several times sometimes yields

Time: 1291464006000

sometimes

Time: 1291464006001

 

minor thing:

cannot copy-paste from the console (CTRL-C)

There is something FUNDAMENTALLY STRANGE on how Palentier reads date and time from a JPG file.

 

If you right-click on windows explorer you can see:

Photo creation date 2010-12-04 14:16 (in EXIF)

The file has last modification date 2010-12-04 14:16:14

AcdSee shows EXIF metadata cretion date 2010-12-04 14:16:14

 

Palentier v0.19 shows

2010-12-05 14:13:50

 

AcdSee shows both dates, one is camera time, probably the time that is when downloading the data from the camera using USB. I think the key is to use Image Digitization Date/Time.

 

If you use command exif from imagemagick,

EXIF tags in '_6335.JPG':             0      1      EXIF   GPS    Interop
  0x0000 GPS tag version                 -      -      -      *      -
  0x0001 InteroperabilityIndex           -      -      -      *      *
  0x0002 InteroperabilityVersion         -      -      -      *      *
  0x0003 East or West Longitude          -      -      -      *      -
  0x0004 Longitude                       -      -      -      *      -
  0x0005 Altitude reference              -      -      -      *      -
  0x0006 Altitude                        -      -      -      *      -
  0x0010 GPS Img Direction Reference     -      -      -      -      -
  0x0011 GPS Img Direction               -      -      -      -      -
  0x00fe New Subfile Type                -      -      -      -      -
  0x0100 Image Width                     -      -      -      -      -
  0x0101 Image Length                    -      -      -      -      -
  0x0102 Bits per Sample                 -      -      -      -      -
  0x0103 Compression                     -      *      -      -      -
  0x0106 Photometric Interpretation      -      -      -      -      -
  0x010a Fill Order                      -      -      -      -      -
  0x010d Document Name                   -      -      -      -      -
  0x010e Image Description               *      -      -      -      -
  0x010f Manufacturer                    *      -      -      -      -
  0x0110 Model                           *      -      -      -      -
  0x0111 Strip Offsets                   -      -      -      -      -
  0x0112 Orientation                     *      -      -      -      -
  0x0115 Samples per Pixel               -      -      -      -      -
  0x0116 Rows per Strip                  -      -      -      -      -
  0x0117 Strip Byte Count                -      -      -      -      -
  0x011a x-Resolution                    *      -      -      -      -
  0x011b y-Resolution                    *      -      -      -      -
  0x011c Planar Configuration            -      -      -      -      -
  0x0128 Resolution Unit                 *      -      -      -      -
  0x012d Transfer Function               -      -      -      -      -
  0x0131 Software                        *      -      -      -      -
  0x0132 Date and Time                   *      -      -      -      -
  0x013b Artist                          -      -      -      -      -
  0x013e White Point                     -      -      -      -      -
  0x013f Primary Chromaticities          -      -      -      -      -
  0x014a SubIFD Offsets                  -      -      -      -      -
  0x0156 Transfer Range                  -      -      -      -      -
  0x0200 JPEGProc                        -      -      -      -      -
  0x0201 JPEG Interchange Format         -      -      -      -      -
  0x0202 JPEG Interchange Format Lengt   -      -      -      -      -
  0x0211 YCbCr Coefficients              -      -      -      -      -
  0x0212 YCbCr Sub-Sampling              -      -      -      -      -
  0x0213 YCbCr Positioning               *      -      -      -      -
  0x0214 Reference Black/White           -      -      -      -      -
  0x02bc XML Packet                      -      -      -      -      -
  0x1000 RelatedImageFileFormat          -      -      -      -      -
  0x1001 RelatedImageWidth               -      -      -      -      -
  0x1002 RelatedImageLength              -      -      -      -      -
  0x828d CFARepeatPatternDim             -      -      -      -      -
  0x828e CFA Pattern                     -      -      -      -      -
  0x828f Battery Level                   -      -      -      -      -
  0x8298 Copyright                       -      -      -      -      -
  0x829a Exposure Time                   -      -      *      -      -
  0x829d FNumber                         -      -      *      -      -
  0x83bb IPTC/NAA                        -      -      -      -      -
  0x8649 Image Resources Block           -      -      -      -      -
  0x8773 InterColorProfile               -      -      -      -      -
  0x8822 Exposure Program                -      -      -      -      -
  0x8824 Spectral Sensitivity            -      -      -      -      -
  0x8827 ISO Speed Ratings               -      -      -      -      -
  0x8828 OECF                            -      -      -      -      -
  0x9000 Exif Version                    -      -      *      -      -
  0x9003 Date and Time (original)        -      -      *      -      -
  0x9004 Date and Time (digitized)       -      -      *      -      -
  0x9101 Components Configuration        -      -      *      -      -
  0x9102 Compressed Bits per Pixel       -      -      *      -      -
  0x9201 Shutter speed                   -      -      *      -      -
  0x9202 Aperture                        -      -      *      -      -
  0x9203 Brightness                      -      -      -      -      -
  0x9204 Exposure Bias                   -      -      *      -      -
  0x9205 MaxApertureValue                -      -      *      -      -
  0x9206 Subject Distance                -      -      -      -      -
  0x9207 Metering Mode                   -      -      *      -      -
  0x9208 Light Source                    -      -      -      -      -
  0x9209 Flash                           -      -      *      -      -
  0x920a Focal Length                    -      -      *      -      -
  0x9214 Subject Area                    -      -      -      -      -
  0x9216 TIFF/EP Standard ID             -      -      -      -      -
  0x927c Maker Note                      -      -      *      -      -
  0x9286 User Comment                    -      -      *      -      -
  0x9290 SubsecTime                      -      -      *      -      -
  0x9291 SubSecTimeOriginal              -      -      *      -      -
  0x9292 SubSecTimeDigitized             -      -      *      -      -
  0x9c9b XP Title                        -      -      -      -      -
  0x9c9c XP Comment                      -      -      -      -      -
  0x9c9d XP Author                       -      -      -      -      -
  0x9c9e XP Keywords                     -      -      -      -      -
  0x9c9f XP Subject                      -      -      -      -      -
  0xa000 FlashPixVersion                 -      -      *      -      -
  0xa001 Color Space                     -      -      *      -      -
  0xa002 PixelXDimension                 -      -      *      -      -
  0xa003 PixelYDimension                 -      -      *      -      -
  0xa004 RelatedSoundFile                -      -      -      -      -
  0xa20b Flash Energy                    -      -      -      -      -
  0xa20c Spatial Frequency Response      -      -      -      -      -
  0xa20e Focal Plane x-Resolution        -      -      *      -      -
  0xa20f Focal Plane y-Resolution        -      -      *      -      -
  0xa210 Focal Plane Resolution Unit     -      -      *      -      -
  0xa214 Subject Location                -      -      -      -      -
  0xa215 Exposure Index                  -      -      -      -      -
  0xa217 Sensing Method                  -      -      *      -      -
  0xa300 File Source                     -      -      *      -      -
  0xa301 Scene Type                      -      -      -      -      -
  0xa302 CFA Pattern                     -      -      -      -      -
  0xa401 Custom Rendered                 -      -      *      -      -
  0xa402 Exposure Mode                   -      -      *      -      -
  0xa403 White Balance                   -      -      *      -      -
  0xa404 Digital Zoom Ratio              -      -      *      -      -
  0xa405 Focal Length In 35mm Film       -      -      -      -      -
  0xa406 Scene Capture Type              -      -      *      -      -
  0xa407 Gain Control                    -      -      -      -      -
  0xa408 Contrast                        -      -      -      -      -
  0xa409 Saturation                      -      -      -      -      -
  0xa40a Sharpness                       -      -      -      -      -
  0xa40b Device Setting Description      -      -      -      -      -
  0xa40c Subject Distance Range          -      -      -      -      -
  0xa420 Image Unique ID                 -      -      -      -      -
  0xa500 Gamma                           -      -      -      -      -
  0xc4a5 PRINT Image Matching            -      -      -      -      -

 

you see the problem is that Palentier usesexif tag

 0x0132 Date and Time

instead of

  0x9004 Date and Time (digitized)

 

Not to mention that the debug outptut for parsing GPX:

Time: 1291554830000 says almost nothing compared to JPG debug output.

OK I have implemented custom offset in my software: +day,+hour,+minute,+second

so it is easier to play with.

General impression:

In Canon S45 the time used by Palentier is somethign not reflecting photo digitization time and teh results are hopeless.

Out of 343 latlon marks in gpx, out of 357 JPEGS loaded, only 157 mapped into positions. As expected, as teh time offset was chosen usting the last few photos, they are in positions. The rest is almost there, but not quite.

 

I really need more verbose JPEG parsing in such a way I can capture it from Palentier. At the moment even the Java console output is too short to hold all data if opening even 50 jpegs.

Not enough to find a pattern among lost chains.

 

Stress testing results:

kmz of 157 photos at 600pix res easily reached 87MB as expected, exploding all was left running from GoogleEarth.

Reduced to 320pix resolution and exploded the app again, kmz is 27MB but memory usage more than 1 gigabyte from gogole earth.

Lesson: GE is NOT a GIS database.

 

Apparently the software is guessing well the camera parameters, but the timing is dubious, probably because Palentier was not used so far for processing long flights or maybe we still have a secret issue in how Canon S45 writes exif.

So retried with 42 photos and there were only 20 matches.

Because of roll stabilised head in Pteryx and with heading estimator, the results 'show potential for improvement',

but could have been much better with working timestamp matching.

 

FYI

GPICSYNC 1.2 is able to parse jpeg data no problem, no coordinates are lost

Attachments:

One more bug:

if you move the Palentier_Beta_v019p0.jar to another directory, and run it, all settings are lost.

Most probably because the actual application path is added to the registry key name when saving those settings.

Quite annoying feature because you can 'lose' settings without knowing it.

The settings were not lost. You just did not copy the config file over with it. (userconfig.xml)

Ahh thanks. Right.

In the next few days I should have another testable version for everyone to use.

The Gui is completely redone, some of the processing errors have been fixed (hopefully) and all the data will be editable. There will be more coming for it after this though as I try to make it 'smarter' for matching and such. I have a couple bright ideas for that.

 

Once I finish with this, comment and clean up the code some I will be posting the source for anyone to download and use.

 

One comment, the times mentioned below are accurate. All times are in seconds. For every GPX time that is the same it gets incremented by 1 millisecond which does not affect the outcome at all. Most GPS's have 2 entries per second. Thus the 0, and 1. But this allows for other, more accurate entries at a later time.

RSS

© 2014   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service