The CanberraUAV Outback Challenge team has put together a package that will allow anyone to try the image recognition system we developed for our successful 2012 OBC entry. We hope that by releasing this in an easier to use form that other OBC teams and anyone interested in image recognition in UAVs will be able to reproduce what we've done and build upon it for their own systems.
The fundamental problem that our system is meant to solve is providing real-time recognition of objects on the ground from a UAV, while requiring only a low bandwidth radio link between the aircraft and the ground station. A lot of UAVs are run using analog video links from the plane to the ground, but we think that is not a good way to do many search and rescue and environmental tasks. The problems with an analog video link are:
- the quality of the video is often greatly degraded by being sent over the analog link, making it hard to spot small objects
- the task of an operator staring at a video screen for hours is much too hard. The operator may notice something flash by, then has to decide whether to rewind (possibly missing something else in the search) or to skip the object
- in many (most?) parts of the world analog video links may violate local radio licensing regulations
We think the solution is for the aircraft to do an initial pass of image recognition to find "interesting" objects on the ground, and then to show small (low bandwidth) thumbnails of those objects on the ground station, overlaid on a satellite map of the area. The operator can then select which of these thumbnails to look at more closely, bringing down a full high resolution image around that object from the plane over the telemetry link when needed. When using this method the operator gets a complete overview of the search area, and can quickly focus on areas of interest, using the human to decide which of the objects that the computer shows up are worth investigating. That was the strategy we successfully used in our 2012 OBC flight.
Up to now the image recognition code we developed for that competition was fairly deeply embedded in the MAVProxy module we used, making it hard for other teams or UAV researchers to run the algorithm over their own data to see how it performs. So to address that we have now put together a set of packages that can be installed on Linux, Windows or MacOS to try the system for yourself.
The new tool is called geosearch.py, and is a standalone UAV search tool that operates offline on a set of images, using either EXIF positioning data in the images, or a MAVLink telemetry log for geo-referencing. It runs the same code that we use in our real-time system embedded in the aircraft, but converted to run as a local GUI tool on a desktop computer. Note that the algorithms we use are tuned for speed. They are not state of the art computer vision algorithms, but they are fast, allowing them to run in real-time on a low power ARM Linux box embedded in an aircraft. We originally tried to use algorithms from the excellent OpenCV system, but found them too slow for our application. We do however use some OpenCV helper functions in our code to make the system a bit simpler.
Release of data from our OBC flight
As a companion to this tool we have released the images captured from our OBC 2012 flight for anyone to download, along with the telemetry logs and other files needed to reproduce the result. The images are kindly hosted by CSIRO in Australia, and are available from the CSIRO Outback Challenge website.
On that site you will find a set of 20 zip files containing the raw images from our flight. If you download and extract those zip files you will end up with a directory called 'raw' which contains 17042 pgm image files. Those files are 8 bit raw Bayer grid files of 1280x960, which are the raw images from our PtGrey Chameleon camera. We're hoping those files will be useful to anyone wanting a decent sized set of UAV images for research, or for training an algorithm for the next Outback Challenge.
(NOTE: as of the time of writing the CSIRO site has the images as 17k separate image files, those should be converted to the 20 zip files for easier download soon).
We also provide a tool called pgm_convert.py which will take those pgm raw images and convert them to other formats, most commonly to lossless PNG files.
How to try our system for yourself
If you watch the video and want to try our system on your own data, or on the data from our OBC flight, then you will need to install a set of python packages that provide the basic system tools we use in our code.
On Windows you need Python 2.7, plus a set of extra python packages. To make life easier for windows users we have collected all of the packages you need in this directory. Note that this includes both the packages needed to run our tools, and the packages needed to build our tools for yourself for when you want to modify the code.On Linux installing the basic python tools is a bit easier. For example, on an Ubuntu Linux system the following command should be enough:
sudo apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-httplib2 libjpeg-turbo-progs
If you are using MacOS, then you should install a suitable Python 2.7 package first, then use easy_install to add the required additional packages.
Once you have the basic tools installed, the next step is to install the 3 CanberraUAV python packages, pymavlink, MAVProxy and cuav. You can get those packages from the python packaging index:
- https://pypi.python.org/pypi/pymavlink
- https://pypi.python.org/pypi/MAVProxy
- https://pypi.python.org/pypi/cuav
That will allow you to download tar balls or windows installers for each package. If you are going to modify the code, then you will be better off downloading the source from our github repository.
To build the packages yourself (instead of using the pre-built packages), you would use the following commands in each of the 3 package directories. For Linux or MacOS, use this:
python setup.py build install --user
for Windows use this:
python setup.py build -c mingw32 install
Note that for building on WIndows you also need to install the MinGW package.
How it all works
To see how it all works, you can either read the code (just follow the above links to our code repository) or you can watch a presentation I gave in January 2012 which explains the algorithm to some extent (video and slides).
We'd also encourage you to ask questions on the CanberraUAV mailing list, or on the Outback Challenge discussion group.
How much bandwidth does it use?
The aim of this system is to allow for real-time UAV search with only a low bandwidth telemetry link to the aircraft. It is possible to run the system with a 64 kbit/s ISM band radio, such as the excellent RFD900 radio.
To make this possible we developed a new network transport protocol called block_xmit (mp4-video, ogg-video, slides, implementation), which allows us to intermix low bandwidth thumbnail images and python objects with normal MAVLink telemetry over a single radio link. The system also scales up to higher bandwidth links, so if you happen to have a high bandwidth ethernet bridge (such as a Ubiquity Bullet) in your aircraft, and you are close enough to the ground station to have a good link, then the system will transmit real-time images that gives the effect of a digital video link. That capability can be seen in the demonstration video above.
The block_xmit protocol also prioritizes images based on the image score from the recognition algorithm, so you can set quite a low threshold for what images to send, and if a high scoring object (such as Joe!) turns up it will jump the queue in the radio link and be shown to the operator quickly.
For our aircraft we combine both types of radio link, with the results of our imaging system coming over both a 5.8GHz Ubiquity ethernet bridge and a 900MHz RFD900 telemetry radio (with smaller thumbnail sizes to save bandwidth). That gives us redundancy in case a particular radio technology does not perform well on the day.
Geo-referencing
Included in the system is a geo-referencing system that uses the MAVLink telemetry stream along with image timestamps to work out the geographic location (latitude/longitude) of any region of interest in the image. These are displayed on the map in various formats (degrees decimal, degrees/minutes/seconds and UTM grid coordinates).
If you instead have a camera with a built-in GPS that tags images with EXIF geo-referencing tags the the code can instead use those for geo-referencing. That tends to result in much less accuracy however, as the EXIF tags don't contain nearly as much information about the aircraft attitude, which makes a big difference.
Also note that our current system uses local timestamps in the raw pgm filenames to work match the time to the MAVLink log. That is a mistake which we will fix in a future release. It means that if you are using our OBC data in a timezone other than Queensland you need to use the time offset option in the geosearch.py program to adjust the time to match the times in the MAVLink log. For example, if you are perth, which is 2 hours west of Queensland, then you would use an offset of -7200 seconds.
Next Steps
Some parts of our algorithm were tuned for the expected object sizes of the OBC competition (about 2 meters by 0.5 meters) and for the altitude and images sizes we used. We'd like to parametrize those aspects of the code to make it easier to use for a wider variety of tasks and camera systems.
We'd also like to make it easier to plug in different scoring filters. The recognition algorithm works in two stages, a first stage that finds anything unusual in the image, then a second stage that converts that region to HSV colour space and applies some simple heuristics to score the region. We'd like users to be able to easily plugin different scoring systems to suit their own image search task, as not everyone is looking for a lost hiker wearing blue jeans and a yellow shirt!
We are also interested in adapting our system to thermal cameras, such as the Tamarisk 640, and we hope to do some testing with that system later this year.
Please share your improvements
If you use this code in your own system then please share your improvements! You can clone our repository on github, or post an issue, or just discuss your changes on our mailing list. If we work together we can create a system which will end up saving lives, so please don't just keep your improvements to yourself.
Comments
Hi Michael - I had some of the same problems with the ARM version of libturbojpeg - this is the reply I got from Tridge - hope it helps:
"you do need an updated mavproxy, but that isn't the problem shown in
your build. The problem is the libturbojpeg library which hasn't been
built correctly for use in a shared object. I'd forgotten that I rebuilt
libturbojpeg when I got my Odroid-U3.
My rebuilt version is here:
http://uav.tridgell.net/Odroid/
Run these commands to fix:
cd /opt
wget http://uav.tridgell.net/Odroid/libjpeg-turbo.tar.gz
tar xzf libjpeg-turbo.tar.gz
echo "/etc/ld.so.conf.d/turbojpeg.conf" > /etc/ld.so.conf.d/turbojpeg.conf
ldconfig
After that you should be able to build. Sorry for not remembering this!"
Sorry I completely forgot about this until I saw your post on the Canberra UAV site.
OK... now I have installed libjpeg_turbo-1.3.1 which I got from the sourceforge page.
I am no able to compile cuav v1.3.2 successfully, and I see the cuav-1.3.2-py2.7-linux-armv6l.egg in the /usr/local/lib/python2.7/dist-packages folder.
I can import cuav in the python shell, however when I try to import the scanner module (from cuav.image import scanner), I get the following error:
libturbojpeg.so.0 - cannot open, no such file
But I can see this file in the /opt/libjpeg-turbo/lib32 folder
There appears to be some issue that the scanner.so file is looking up the wrong location for the libjpeg file.... very confusing!
Further to the above comment - I read another of Stephen Gloor's comments, and have installed the ldc1394 module with the following command (sude get-apt install libdc1394-22-dev).
Now the setup of the CUAV module gets stuck on the compilation of the scanner module - fata; error turbojpeg.h no such file or directory.
So I think the issue is I can't find the right way to install the libjpeg-turbo module.
Hi Tridge,
I have the CUAV modules working on my Windows PC, and they work great. I am trying to integrate this system into our Raspberry Pi but I'm having some problems.
I followed the steps that you outlined above, and it all works fine (except from libjpeg-turbo-progs, but I have installed libjpeg-dev) as per Stephen Gloors comments.
When I try to install cuav package (sudo python setup.py build install --user) it starts installing but gets stuck on the Chameleon_util - it says that it cannot find -ldc1394
Are you able to help me figure out what is going wrong? I noted as well in your comment that you had to disable 'neon vector optimisation' to get this working on the Raspberry Pi - does this have anything to do with it?
Thanks
Michael
This is great stuff Tridge. Could you please explain what sort of on board computer did you use? Also, how did you make communication between flight computer and AutoPilot (APM)?
@parvupv
There is an error - I found it too when my EXIF tags did not line up
Apart from the os.time error there also is one when the exif tags from FV-5 are used. I had to change the code to this:
try:
t = time.mktime(m['Exif.Image.DateTime'].value.timetuple())
except Exception:
try:
t = time.mktime(m['Exif.Photo.DateTimeOriginal'].value.timetuple())
except Exception:
t = t = os.path.getmtime(filename)
In FV-5 the time EXIF tag is different - I use the time taken rather than the time digitised.
@ Tridgell
I think there is an error in cuav/lib/mav_position.py
line 347 should read t = os.path.getmtime(filename)
Please correct me if I'm wrong. Thank you
@Jamie Glover
It's a bit tricky to get it to run with the window size, but I was able to figure it out. Thanks so much. I THINK I have it up and running. The only thing I noted in the command prompt window was that there were a bunch of pickle errors. Seems like the script is hard coded to a location C drive location.
Many thanks!
@ moglos
Of course the ] should end the remaining line. the line should be:
extra_compile_args=["-std=gnu99", "-O3"]
Do you have the directories/files required by the geosearch.py input screen? I'd be interested in a set of files for testing too ...
@ parvupv
Wow, thank you for those detailed instructions. I followed them from start to finish. I learnt a lot, and got it working up to the point where I get the geosearch.py pop up window asking for the various directories/options etc. That's my next step. Thanks so much for your help.
One question, you wrote to: remove "-mfpu=neon"]
Should I remove the closing square bracket too? I removed the comma that appeared immediately before that text and I left the closing square bracket, it looked like that ended the list.