Developer

The CanberraUAV Outback Challenge team has put together a package that will allow anyone to try the image recognition system we developed for our successful 2012 OBC entry. We hope that by releasing this in an easier to use form that other OBC teams and anyone interested in image recognition in UAVs will be able to reproduce what we've done and build upon it for their own systems.

The fundamental problem that our system is meant to solve is providing real-time recognition of objects on the ground from a UAV, while requiring only a low bandwidth radio link between the aircraft and the ground station. A lot of UAVs are run using analog video links from the plane to the ground, but we think that is not a good way to do many search and rescue and environmental tasks. The problems with an analog video link are:

  • the quality of the video is often greatly degraded by being sent over the analog link, making it hard to spot small objects
  • the task of an operator staring at a video screen for hours is much too hard. The operator may notice something flash by, then has to decide whether to rewind (possibly missing something else in the search) or to skip the object
  • in many (most?) parts of the world analog video links may violate local radio licensing regulations

We think the solution is for the aircraft to do an initial pass of image recognition to find "interesting" objects on the ground, and then to show small (low bandwidth) thumbnails of those objects on the ground station, overlaid on a satellite map of the area. The operator can then select which of these thumbnails to look at more closely, bringing down a full high resolution image around that object from the plane over the telemetry link when needed. When using this method the operator gets a complete overview of the search area, and can quickly focus on areas of interest, using the human to decide which of the objects that the computer shows up are worth investigating. That was the strategy we successfully used in our 2012 OBC flight.

Up to now the image recognition code we developed for that competition was fairly deeply embedded in the MAVProxy module we used, making it hard for other teams or UAV researchers to run the algorithm over their own data to see how it performs. So to address that we have now put together a set of packages that can be installed on Linux, Windows or MacOS to try the system for yourself.

The new tool is called geosearch.py, and is a standalone UAV search tool that operates offline on a set of images, using either EXIF positioning data in the images, or a MAVLink telemetry log for geo-referencing. It runs the same code that we use in our real-time system embedded in the aircraft, but converted to run as a local GUI tool on a desktop computer. Note that the algorithms we use are tuned for speed. They are not state of the art computer vision algorithms, but they are fast, allowing them to run in real-time on a low power ARM Linux box embedded in an aircraft. We originally tried to use algorithms from the excellent OpenCV system, but found them too slow for our application. We do however use some OpenCV helper functions in our code to make the system a bit simpler.

Release of data from our OBC flight

As a companion to this tool we have released the images captured from our OBC 2012 flight for anyone to download, along with the telemetry logs and other files needed to reproduce the result. The images are kindly hosted by CSIRO in Australia, and are available from the CSIRO Outback Challenge website.

On that site you will find a set of 20 zip files containing the raw images from our flight. If you download and extract those zip files you will end up with a directory called 'raw' which contains 17042 pgm image files. Those files are 8 bit raw Bayer grid files of 1280x960, which are the raw images from our PtGrey Chameleon camera. We're hoping those files will be useful to anyone wanting a decent sized set of UAV images for research, or for training an algorithm for the next Outback Challenge.

(NOTE: as of the time of writing the CSIRO site has the images as 17k separate image files, those should be converted to the 20 zip files for easier download soon).

We also provide a tool called pgm_convert.py which will take those pgm raw images and convert them to other formats, most commonly to lossless PNG files.

How to try our system for yourself

If you watch the video and want to try our system on your own data, or on the data from our OBC flight, then you will need to install a set of python packages that provide the basic system tools we use in our code.

On Windows you need Python 2.7, plus a set of extra python packages. To make life easier for windows users we have collected all of the packages you need in this directory. Note that this includes both the packages needed to run our tools, and the packages needed to build our tools for yourself for when you want to modify the code.On Linux installing the basic python tools is a bit easier. For example, on an Ubuntu Linux system the following command should be enough:

  sudo apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-httplib2 libjpeg-turbo-progs

If you are using MacOS, then you should install a suitable Python 2.7 package first, then use easy_install to add the required additional packages.

Once you have the basic tools installed, the next step is to install the 3 CanberraUAV python packages, pymavlink, MAVProxy and cuav. You can get those packages from the python packaging index:

That will allow you to download tar balls or windows installers for each package. If you are going to modify the code, then you will be better off downloading the source from our github repository.

To build the packages yourself (instead of using the pre-built packages), you would use the following commands in each of the 3 package directories. For Linux or MacOS, use this:

  python setup.py build install --user

for Windows use this:

  python setup.py build -c mingw32 install

Note that for building on WIndows you also need to install the MinGW package.

How it all works

To see how it all works, you can either read the code (just follow the above links to our code repository) or you can watch a presentation I gave in January 2012 which explains the algorithm to some extent (video and slides).

We'd also encourage you to ask questions on the CanberraUAV mailing list, or on the Outback Challenge discussion group.

How much bandwidth does it use?

The aim of this system is to allow for real-time UAV search with only a low bandwidth telemetry link to the aircraft. It is possible to run the system with a 64 kbit/s ISM band radio, such as the excellent RFD900 radio.

To make this possible we developed a new network transport protocol called block_xmit (mp4-video, ogg-video, slides, implementation), which allows us to intermix low bandwidth thumbnail images and python objects with normal MAVLink telemetry over a single radio link. The system also scales up to higher bandwidth links, so if you happen to have a high bandwidth ethernet bridge (such as a Ubiquity Bullet) in your aircraft, and you are close enough to the ground station to have a good link, then the system will transmit real-time images that gives the effect of a digital video link. That capability can be seen in the demonstration video above.

The block_xmit protocol also prioritizes images based on the image score from the recognition algorithm, so you can set quite a low threshold for what images to send, and if a high scoring object (such as Joe!) turns up it will jump the queue in the radio link and be shown to the operator quickly.

For our aircraft we combine both types of radio link, with the results of our imaging system coming over both a 5.8GHz Ubiquity ethernet bridge and a 900MHz RFD900 telemetry radio (with smaller thumbnail sizes to save bandwidth). That gives us redundancy in case a particular radio technology does not perform well on the day.

Geo-referencing

Included in the system is a geo-referencing system that uses the MAVLink telemetry stream along with image timestamps to work out the geographic location (latitude/longitude) of any region of interest in the image. These are displayed on the map in various formats (degrees decimal, degrees/minutes/seconds and UTM grid coordinates).

If you instead have a camera with a built-in GPS that tags images with EXIF geo-referencing tags the the code can instead use those for geo-referencing. That tends to result in much less accuracy however, as the EXIF tags don't contain nearly as much information about the aircraft attitude, which makes a big difference.

Also note that our current system uses local timestamps in the raw pgm filenames to work match the time to the MAVLink log. That is a mistake which we will fix in a future release. It means that if you are using our OBC data in a timezone other than Queensland you need to use the time offset option in the geosearch.py program to adjust the time to match the times in the MAVLink log. For example, if you are perth, which is 2 hours west of Queensland, then you would use an offset of -7200 seconds.

Next Steps

Some parts of our algorithm were tuned for the expected object sizes of the OBC competition (about 2 meters by 0.5 meters) and for the altitude and images sizes we used. We'd like to parametrize those aspects of the code to make it easier to use for a wider variety of tasks and camera systems.

We'd also like to make it easier to plug in different scoring filters. The recognition algorithm works in two stages, a first stage that finds anything unusual in the image, then a second stage that converts that region to HSV colour space and applies some simple heuristics to score the region. We'd like users to be able to easily plugin different scoring systems to suit their own image search task, as not everyone is looking for a lost hiker wearing blue jeans and a yellow shirt!

We are also interested in adapting our system to thermal cameras, such as the Tamarisk 640, and we hope to do some testing with that system later this year.

Please share your improvements

If you use this code in your own system then please share your improvements! You can clone our repository on github, or post an issue, or just discuss your changes on our mailing list. If we work together we can create a system which will end up saving lives, so please don't just keep your improvements to yourself.

E-mail me when people leave their comments –

You need to be a member of diydrones to add comments!

Join diydrones

Comments

  • Is there a "How to Setup Guide" for getting this program to work real time from the plane to the GCS instead of "Post Processing"? I have all the required electronics, but Im not code savy in Python (slowly a work in progress) and Im trying to get it up and running. Can anyone help me out please? I have it installed and running on both Windows 8 and the Panda Board ES. Just need to figure out how to get it to run without the GUI and have it transmit through the RFD 900 to the GCS. 

  • sorry there is a typo

    libcd1394-22-dev

    is libdc1394-22-dev

  • I have just installed the new version to a Linux server and here are the steps I used to get it to work.  As a dabbler in Linux I had a lot of help from Tridge and Chris - thanks.  This is a dummies guide for people like me.

    1.  Install lubuntu 12.04 downloadable from here http://www.lubuntu.net/tags/lubuntu-1204 - I used the 12.04LTS version as several of the needed libraries are broken in the 13.10 version.  I also use lubutu as the Unity interface in real ubuntu causes a lot of headaches.  lubuntu is Ubuntu with all the extra crap stripped off and a has a light interface that works on PCs or servers that are graphically challenged.  I used the 64 bit version however this should work for the 32 bit version as well

    2.  Open a terminal and paste in the following

         sudo apt-get update

         sudo apt-get python-setuptools

         sudo apt-get install python-pip

         sudo apt-get install git

    This installs all the tools that you will need to install the rest of the packages.

    Now for the required libraries - cut and paste the following into the terminal 

           sudo apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-        httplib2 libjpeg-turbo-progs

           sudo apt-get install libusb-1.0.0 libusb-1.0.0-dev libcd1394-22-dev python-dev libturbojpeg libjpeg-dev

    This should be the complete list if you installed 12.04 - now install the programs using pip:

            sudo pip install pymavlink

            sudo pip install MAVProxy

    Now you should be able to run this to install the demo

            sudo pip install cuav

    However at the time of writing this the pip package is not working.  Try it and see - if it works good if it errors out run this workaround:

          git clone git://github.com/tridge/cuav.git
          cd cuav
          sudo python setup.py build install

    The files are installed in the following directory:

    Installing cuav_benchmark.py script to /usr/local/bin
    Installing geosearch.py script to /usr/local/bin
    Installing geotag.py script to /usr/local/bin
    Installing cuav_lens.py script to /usr/local/bin
    Installing agl_mission.py script to /usr/local/bin
    Installing pgm_convert.py script to /usr/local/bin

    To run the demo just enter in a terminal window

         cd /usr/local/bin

         python geosearch.py

    A gui screen should pop up and you can enter the parameters you need and off you go.

  • Worked this one out as well.  Just had to run geosearch.py from a terminal window instead of clicking on the icon.  I now have both the windows and linux versions working perfectly.

    BTW the log files in the CSIRO repository are different from your presentation.

    The only one that works with the images is flight.log

  • Hi Tridge - running the app in Linux the map and mosaic come up but it does not scan through files that I point to.

    Are the errors logged anywhere - I looked in /var/log and could not find any errors

  • Moderator

    I also found while installing this on an ARM based Ubuntu installation that  libturbojpeg is also needed to get cuav to install.

    sudo apt-get install libturbojpeg

  • Hi - Figured it out.  The problem is that on mint linux the -dev packages are not installed.

    Had to run:

    sudo apt-get install python-dev libjpeg-dev

    cuav then ran with easy install.

  • Hi Andrew,

    Got this error both when I used easy_install and just running the install from where I downloaded the files:

    stephen@ImagingTest ~/Downloads/setuptools-1.1.6 $ sudo easy_install cuav
    Searching for cuav
    Reading https://pypi.python.org/simple/cuav/
    Best match: cuav 1.0.8
    Downloading https://pypi.python.org/packages/source/c/cuav/cuav-1.0.8.zip#md5=f...
    Processing cuav-1.0.8.zip
    Writing /tmp/easy_install-aOTNPi/cuav-1.0.8/setup.cfg
    Running cuav-1.0.8/setup.py -q bdist_egg --dist-dir /tmp/easy_install-aOTNPi/cuav-1.0.8/egg-dist-tmp-KHHjNB
    image/scanner.c:6:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: Setup script exited with error: command 'i686-linux-gnu-gcc' failed with exit status 1

    Have installed MAVlink and pymavlink fine 

    Any ideas?

  • Hi Tridge,

    I am busy trying to run the image search algorithm as per the instructions in this blog, everything appears to be good the Map window loads with Map and grid, however an error is thrown in the python cli that says

    Traceback (most recent call last):
      File "C:\Python27\Scripts\geosearch.py", line 196, in <module>
        process([opts.directory])
      File "C:\Python27\Scripts\geosearch.py", line 78, in process
        wp.load(opts.mission)
      File "C:\Python27\lib\site-packages\pymavlink\mavwp.py", line 221, in load
        raise MAVWPError("Unsupported waypoint format '%s'" % version_line)
    pymavlink.mavwp.MAVWPError: Unsupported waypoint format '<?xml version="1.0" enc
    oding="UTF-8"?>'

     

    It would appear I don't have the correct Waypoint file to run this, any help greatly appreciated.

    Regards

    Rob

  • There seems to be one more dependency on Ubuntu 13.04:

    sudo apt-get install libusb-1.0-0-dev

    (for libusb.h)

This reply was deleted.