The CanberraUAV Outback Challenge team has put together a package that will allow anyone to try the image recognition system we developed for our successful 2012 OBC entry. We hope that by releasing this in an easier to use form that other OBC teams and anyone interested in image recognition in UAVs will be able to reproduce what we've done and build upon it for their own systems.
The fundamental problem that our system is meant to solve is providing real-time recognition of objects on the ground from a UAV, while requiring only a low bandwidth radio link between the aircraft and the ground station. A lot of UAVs are run using analog video links from the plane to the ground, but we think that is not a good way to do many search and rescue and environmental tasks. The problems with an analog video link are:
- the quality of the video is often greatly degraded by being sent over the analog link, making it hard to spot small objects
- the task of an operator staring at a video screen for hours is much too hard. The operator may notice something flash by, then has to decide whether to rewind (possibly missing something else in the search) or to skip the object
- in many (most?) parts of the world analog video links may violate local radio licensing regulations
We think the solution is for the aircraft to do an initial pass of image recognition to find "interesting" objects on the ground, and then to show small (low bandwidth) thumbnails of those objects on the ground station, overlaid on a satellite map of the area. The operator can then select which of these thumbnails to look at more closely, bringing down a full high resolution image around that object from the plane over the telemetry link when needed. When using this method the operator gets a complete overview of the search area, and can quickly focus on areas of interest, using the human to decide which of the objects that the computer shows up are worth investigating. That was the strategy we successfully used in our 2012 OBC flight.
Up to now the image recognition code we developed for that competition was fairly deeply embedded in the MAVProxy module we used, making it hard for other teams or UAV researchers to run the algorithm over their own data to see how it performs. So to address that we have now put together a set of packages that can be installed on Linux, Windows or MacOS to try the system for yourself.
The new tool is called geosearch.py, and is a standalone UAV search tool that operates offline on a set of images, using either EXIF positioning data in the images, or a MAVLink telemetry log for geo-referencing. It runs the same code that we use in our real-time system embedded in the aircraft, but converted to run as a local GUI tool on a desktop computer. Note that the algorithms we use are tuned for speed. They are not state of the art computer vision algorithms, but they are fast, allowing them to run in real-time on a low power ARM Linux box embedded in an aircraft. We originally tried to use algorithms from the excellent OpenCV system, but found them too slow for our application. We do however use some OpenCV helper functions in our code to make the system a bit simpler.
Release of data from our OBC flight
As a companion to this tool we have released the images captured from our OBC 2012 flight for anyone to download, along with the telemetry logs and other files needed to reproduce the result. The images are kindly hosted by CSIRO in Australia, and are available from the CSIRO Outback Challenge website.
On that site you will find a set of 20 zip files containing the raw images from our flight. If you download and extract those zip files you will end up with a directory called 'raw' which contains 17042 pgm image files. Those files are 8 bit raw Bayer grid files of 1280x960, which are the raw images from our PtGrey Chameleon camera. We're hoping those files will be useful to anyone wanting a decent sized set of UAV images for research, or for training an algorithm for the next Outback Challenge.
(NOTE: as of the time of writing the CSIRO site has the images as 17k separate image files, those should be converted to the 20 zip files for easier download soon).
We also provide a tool called pgm_convert.py which will take those pgm raw images and convert them to other formats, most commonly to lossless PNG files.
How to try our system for yourself
If you watch the video and want to try our system on your own data, or on the data from our OBC flight, then you will need to install a set of python packages that provide the basic system tools we use in our code.
On Windows you need Python 2.7, plus a set of extra python packages. To make life easier for windows users we have collected all of the packages you need in this directory. Note that this includes both the packages needed to run our tools, and the packages needed to build our tools for yourself for when you want to modify the code.On Linux installing the basic python tools is a bit easier. For example, on an Ubuntu Linux system the following command should be enough:
sudo apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-httplib2 libjpeg-turbo-progs
If you are using MacOS, then you should install a suitable Python 2.7 package first, then use easy_install to add the required additional packages.
Once you have the basic tools installed, the next step is to install the 3 CanberraUAV python packages, pymavlink, MAVProxy and cuav. You can get those packages from the python packaging index:
That will allow you to download tar balls or windows installers for each package. If you are going to modify the code, then you will be better off downloading the source from our github repository.
To build the packages yourself (instead of using the pre-built packages), you would use the following commands in each of the 3 package directories. For Linux or MacOS, use this:
python setup.py build install --user
for Windows use this:
python setup.py build -c mingw32 install
Note that for building on WIndows you also need to install the MinGW package.
How it all works
To see how it all works, you can either read the code (just follow the above links to our code repository) or you can watch a presentation I gave in January 2012 which explains the algorithm to some extent (video and slides).
We'd also encourage you to ask questions on the CanberraUAV mailing list, or on the Outback Challenge discussion group.
How much bandwidth does it use?
The aim of this system is to allow for real-time UAV search with only a low bandwidth telemetry link to the aircraft. It is possible to run the system with a 64 kbit/s ISM band radio, such as the excellent RFD900 radio.
To make this possible we developed a new network transport protocol called block_xmit (mp4-video, ogg-video, slides, implementation), which allows us to intermix low bandwidth thumbnail images and python objects with normal MAVLink telemetry over a single radio link. The system also scales up to higher bandwidth links, so if you happen to have a high bandwidth ethernet bridge (such as a Ubiquity Bullet) in your aircraft, and you are close enough to the ground station to have a good link, then the system will transmit real-time images that gives the effect of a digital video link. That capability can be seen in the demonstration video above.
The block_xmit protocol also prioritizes images based on the image score from the recognition algorithm, so you can set quite a low threshold for what images to send, and if a high scoring object (such as Joe!) turns up it will jump the queue in the radio link and be shown to the operator quickly.
For our aircraft we combine both types of radio link, with the results of our imaging system coming over both a 5.8GHz Ubiquity ethernet bridge and a 900MHz RFD900 telemetry radio (with smaller thumbnail sizes to save bandwidth). That gives us redundancy in case a particular radio technology does not perform well on the day.
Included in the system is a geo-referencing system that uses the MAVLink telemetry stream along with image timestamps to work out the geographic location (latitude/longitude) of any region of interest in the image. These are displayed on the map in various formats (degrees decimal, degrees/minutes/seconds and UTM grid coordinates).
If you instead have a camera with a built-in GPS that tags images with EXIF geo-referencing tags the the code can instead use those for geo-referencing. That tends to result in much less accuracy however, as the EXIF tags don't contain nearly as much information about the aircraft attitude, which makes a big difference.
Also note that our current system uses local timestamps in the raw pgm filenames to work match the time to the MAVLink log. That is a mistake which we will fix in a future release. It means that if you are using our OBC data in a timezone other than Queensland you need to use the time offset option in the geosearch.py program to adjust the time to match the times in the MAVLink log. For example, if you are perth, which is 2 hours west of Queensland, then you would use an offset of -7200 seconds.
Some parts of our algorithm were tuned for the expected object sizes of the OBC competition (about 2 meters by 0.5 meters) and for the altitude and images sizes we used. We'd like to parametrize those aspects of the code to make it easier to use for a wider variety of tasks and camera systems.
We'd also like to make it easier to plug in different scoring filters. The recognition algorithm works in two stages, a first stage that finds anything unusual in the image, then a second stage that converts that region to HSV colour space and applies some simple heuristics to score the region. We'd like users to be able to easily plugin different scoring systems to suit their own image search task, as not everyone is looking for a lost hiker wearing blue jeans and a yellow shirt!
We are also interested in adapting our system to thermal cameras, such as the Tamarisk 640, and we hope to do some testing with that system later this year.
Please share your improvements
If you use this code in your own system then please share your improvements! You can clone our repository on github, or post an issue, or just discuss your changes on our mailing list. If we work together we can create a system which will end up saving lives, so please don't just keep your improvements to yourself.
@Jamie - no probs. BTW I have a hack job I did on geosearch.py where I deleted all the gui parts and just runs on the command line.
It runs the search and then copies the thumbnails with something in them to a different directory. If you have the setup it will also FTP the thumbnails to a destination of your choice.
I just remembered, the GUI that comes up is not sized correctly for you to view the entire screen, I've been meaning to mention this to Tridge, but what Ive done so far is tab from a given field and count the number of tabs until it reaches the top field again. Go back to the same starting point and subtract 2 tabs from the number you counted and tab that many times. Hit Enter and it should start.
@ Stephen Gloor, I just wanted to say thank you for the step by step post you put up earlier. I was able to get it running on my PandaBoard ES running Ubuntu 12.04 successfully. (In the GUI form of course) but its a step in the right direction. The only thing I didnt get to work was the actual run command in the Terminal. My CUAV pkg installed differently I guess, but I did a search for it, found it and was able to run it with no problems.
@ Ryan Sales, save yourself the headache. Go to this link http://uav.tridgell.net/geosearch/ click on the geosearch-0.4.zip to download the file. Also download the libjpeg-turbo-1.3.0. Once these have downloaded, unzip the files. Now, you can open the Geosearch-0.4 folder and scroll to find the geosearch.py. Double click on it and it will open and run the program
OK. This is how I make it work:
Booted RPi on a fresh image: 2013-09-25-wheezy-raspbian image
- expand filesystem
- boot to text console with login
- change pi passwd
- enable camera
- enable ssh
login as pi and: sudo passwd root
login as root, attach network cable and reboot with network
ssh as root from pc with putty, then:
- apt-get upgrade
---- cuav instalation----
- apt-get install python-setuptools
- apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-httplib2
- nano /etc/apt/sources.list
------add these two repos----
deb http://zarrenspry.info/repos/apt/raspbian testing main
deb-src http://zarrenspry.info/repos/apt/raspbian testing main
close and save Ctrl-X, Y
- wget -O - http://zarrenspry.info/zarrenspry.gpg.key|apt-key add -
- apt-get update
- apt-get install libturbojpeg libturbojpeg-dev libjpeg-turbo-tools
----additional Packages if needed----
(These will replace files provided by Independent JPEG Group and may cause problems with other software.)
- apt-get install libturbojpeg8 libturbojpeg8-dev
- apt-get install python-dev libusb-1.0.0-dev libdc1394-22-dev (@moglos missing that libdc1394-22-dev prevents you from finishing compiling)
- apt-get install python-imaging-tk
- easy_install PIL
- git clone git://github.com/tridge/cuav.git
- cd cuav
- nano setup.py
- remove "-mfpu=neon"]
- python setup.py build install
---- cuav end install ----
apt-get install tightvncserver
install tightvncclient to PC and connect to RPI
to test your intalation try this script replacing with your photo:
nano imageproc.py: (copy and paste lines bellow)
import numpy, os, time, cv, sys, math, sys, glob
from cuav.lib import cuav_util
from cuav.image import scanner
from cuav.lib import cuav_mosaic, mav_position, cuav_joe, cuav_region
from cuav.camera import cam_params
from MAVProxy.modules.mavproxy_map import mp_slipmap
from MAVProxy.modules.mavproxy_map import mp_image
im_orig = cuav_util.LoadImage('Your image here')
(w,h) = cuav_util.image_shape(im_orig)
im_full = im_orig
im_full = numpy.ascontiguousarray(cv.GetMat(im_full))
img_scan = im_full
(sw,sh) = cuav_util.image_shape(img_scan)
regions = scanner.scan(img_scan)
regions = cuav_region.RegionsConvert(regions, cuav_util.image_shape(img_scan), cuav_util.image_shape(im_full))
composite = cuav_mosaic.CompositeThumbnail(cv.GetImage(cv.fromarray(im_full)), regions)
img_view = img_scan
(wview,hview) = cuav_util.image_shape(img_view)
mat = cv.fromarray(img_view)
viewer = mp_image.MPImage(title='Image', can_zoom=True, can_drag=True)
for r in regions:
cv.CvtColor(mat, mat, cv.CV_BGR2RGB)
run it with python imageproc.py
If all goes right you can run geosearch.py, but this won't help much if don't have the required files/directories, see Tridgell demo video on the top of the page
I'm still having trouble installing cuav.
I've tried sudo easy_install cuav, and
When I run the last line I get the following error. (Running sudo easy_install cuav results in less text, but the last three lines are the same)
Actually that was Tridge's workaround. Also there is this from Chris when he had problems with turbojpeg
"I have had problems with this before, but in Ubuntu 12.4 the packaged libturbojpeg worked. But not for me on 13.10, so I had to compile my own from sources.
Thanks for all your help guys (this stuff really does my head in when it doesn't work). I'll have a chance to look at that later tonight. I'll let you know how I get on.
When I did sudo pip install cuav I had a lot of issues as well; I followed Stephen Gloor's earlier post and it worked for me.
Below is Stephen Gloor's workaround: Thanks for this!
git clone git://github.com/tridge/cuav.git
sudo python setup.py build install
The files are installed in the following directory:
Installing cuav_benchmark.py script to /usr/local/bin
Installing geosearch.py script to /usr/local/bin
Installing geotag.py script to /usr/local/bin
Installing cuav_lens.py script to /usr/local/bin
Installing agl_mission.py script to /usr/local/bin
Installing pgm_convert.py script to /usr/local/bin
To run the demo just enter in a terminal window
A gui screen should pop up and you can enter the parameters you need and off you go.
what Debian release are you using?
I'm SUPER new to the programmatic side of this house. I'm using a Windows 8.0 machine. I downloaded and installed all of the Python modules.
Can anyone provide with with the process to go from python installation through to running the program? I'm not sure how to actually get the CanberraUAV packages installed on my machine so I can run it.
Right now my need is to run the system to post process images (no live connection right now).
Thank you so much!