The CanberraUAV Outback Challenge team has put together a package that will allow anyone to try the image recognition system we developed for our successful 2012 OBC entry. We hope that by releasing this in an easier to use form that other OBC teams and anyone interested in image recognition in UAVs will be able to reproduce what we've done and build upon it for their own systems.
The fundamental problem that our system is meant to solve is providing real-time recognition of objects on the ground from a UAV, while requiring only a low bandwidth radio link between the aircraft and the ground station. A lot of UAVs are run using analog video links from the plane to the ground, but we think that is not a good way to do many search and rescue and environmental tasks. The problems with an analog video link are:
- the quality of the video is often greatly degraded by being sent over the analog link, making it hard to spot small objects
- the task of an operator staring at a video screen for hours is much too hard. The operator may notice something flash by, then has to decide whether to rewind (possibly missing something else in the search) or to skip the object
- in many (most?) parts of the world analog video links may violate local radio licensing regulations
We think the solution is for the aircraft to do an initial pass of image recognition to find "interesting" objects on the ground, and then to show small (low bandwidth) thumbnails of those objects on the ground station, overlaid on a satellite map of the area. The operator can then select which of these thumbnails to look at more closely, bringing down a full high resolution image around that object from the plane over the telemetry link when needed. When using this method the operator gets a complete overview of the search area, and can quickly focus on areas of interest, using the human to decide which of the objects that the computer shows up are worth investigating. That was the strategy we successfully used in our 2012 OBC flight.
Up to now the image recognition code we developed for that competition was fairly deeply embedded in the MAVProxy module we used, making it hard for other teams or UAV researchers to run the algorithm over their own data to see how it performs. So to address that we have now put together a set of packages that can be installed on Linux, Windows or MacOS to try the system for yourself.
The new tool is called geosearch.py, and is a standalone UAV search tool that operates offline on a set of images, using either EXIF positioning data in the images, or a MAVLink telemetry log for geo-referencing. It runs the same code that we use in our real-time system embedded in the aircraft, but converted to run as a local GUI tool on a desktop computer. Note that the algorithms we use are tuned for speed. They are not state of the art computer vision algorithms, but they are fast, allowing them to run in real-time on a low power ARM Linux box embedded in an aircraft. We originally tried to use algorithms from the excellent OpenCV system, but found them too slow for our application. We do however use some OpenCV helper functions in our code to make the system a bit simpler.
Release of data from our OBC flight
As a companion to this tool we have released the images captured from our OBC 2012 flight for anyone to download, along with the telemetry logs and other files needed to reproduce the result. The images are kindly hosted by CSIRO in Australia, and are available from the CSIRO Outback Challenge website.
On that site you will find a set of 20 zip files containing the raw images from our flight. If you download and extract those zip files you will end up with a directory called 'raw' which contains 17042 pgm image files. Those files are 8 bit raw Bayer grid files of 1280x960, which are the raw images from our PtGrey Chameleon camera. We're hoping those files will be useful to anyone wanting a decent sized set of UAV images for research, or for training an algorithm for the next Outback Challenge.
(NOTE: as of the time of writing the CSIRO site has the images as 17k separate image files, those should be converted to the 20 zip files for easier download soon).
We also provide a tool called pgm_convert.py which will take those pgm raw images and convert them to other formats, most commonly to lossless PNG files.
How to try our system for yourself
If you watch the video and want to try our system on your own data, or on the data from our OBC flight, then you will need to install a set of python packages that provide the basic system tools we use in our code.
On Windows you need Python 2.7, plus a set of extra python packages. To make life easier for windows users we have collected all of the packages you need in this directory. Note that this includes both the packages needed to run our tools, and the packages needed to build our tools for yourself for when you want to modify the code.On Linux installing the basic python tools is a bit easier. For example, on an Ubuntu Linux system the following command should be enough:
sudo apt-get install python-wxgtk2.8 python-matplotlib python-numpy python-pyexiv2 python-opencv python-httplib2 libjpeg-turbo-progs
If you are using MacOS, then you should install a suitable Python 2.7 package first, then use easy_install to add the required additional packages.
Once you have the basic tools installed, the next step is to install the 3 CanberraUAV python packages, pymavlink, MAVProxy and cuav. You can get those packages from the python packaging index:
That will allow you to download tar balls or windows installers for each package. If you are going to modify the code, then you will be better off downloading the source from our github repository.
To build the packages yourself (instead of using the pre-built packages), you would use the following commands in each of the 3 package directories. For Linux or MacOS, use this:
python setup.py build install --user
for Windows use this:
python setup.py build -c mingw32 install
Note that for building on WIndows you also need to install the MinGW package.
How it all works
To see how it all works, you can either read the code (just follow the above links to our code repository) or you can watch a presentation I gave in January 2012 which explains the algorithm to some extent (video and slides).
We'd also encourage you to ask questions on the CanberraUAV mailing list, or on the Outback Challenge discussion group.
How much bandwidth does it use?
The aim of this system is to allow for real-time UAV search with only a low bandwidth telemetry link to the aircraft. It is possible to run the system with a 64 kbit/s ISM band radio, such as the excellent RFD900 radio.
To make this possible we developed a new network transport protocol called block_xmit (mp4-video, ogg-video, slides, implementation), which allows us to intermix low bandwidth thumbnail images and python objects with normal MAVLink telemetry over a single radio link. The system also scales up to higher bandwidth links, so if you happen to have a high bandwidth ethernet bridge (such as a Ubiquity Bullet) in your aircraft, and you are close enough to the ground station to have a good link, then the system will transmit real-time images that gives the effect of a digital video link. That capability can be seen in the demonstration video above.
The block_xmit protocol also prioritizes images based on the image score from the recognition algorithm, so you can set quite a low threshold for what images to send, and if a high scoring object (such as Joe!) turns up it will jump the queue in the radio link and be shown to the operator quickly.
For our aircraft we combine both types of radio link, with the results of our imaging system coming over both a 5.8GHz Ubiquity ethernet bridge and a 900MHz RFD900 telemetry radio (with smaller thumbnail sizes to save bandwidth). That gives us redundancy in case a particular radio technology does not perform well on the day.
Included in the system is a geo-referencing system that uses the MAVLink telemetry stream along with image timestamps to work out the geographic location (latitude/longitude) of any region of interest in the image. These are displayed on the map in various formats (degrees decimal, degrees/minutes/seconds and UTM grid coordinates).
If you instead have a camera with a built-in GPS that tags images with EXIF geo-referencing tags the the code can instead use those for geo-referencing. That tends to result in much less accuracy however, as the EXIF tags don't contain nearly as much information about the aircraft attitude, which makes a big difference.
Also note that our current system uses local timestamps in the raw pgm filenames to work match the time to the MAVLink log. That is a mistake which we will fix in a future release. It means that if you are using our OBC data in a timezone other than Queensland you need to use the time offset option in the geosearch.py program to adjust the time to match the times in the MAVLink log. For example, if you are perth, which is 2 hours west of Queensland, then you would use an offset of -7200 seconds.
Some parts of our algorithm were tuned for the expected object sizes of the OBC competition (about 2 meters by 0.5 meters) and for the altitude and images sizes we used. We'd like to parametrize those aspects of the code to make it easier to use for a wider variety of tasks and camera systems.
We'd also like to make it easier to plug in different scoring filters. The recognition algorithm works in two stages, a first stage that finds anything unusual in the image, then a second stage that converts that region to HSV colour space and applies some simple heuristics to score the region. We'd like users to be able to easily plugin different scoring systems to suit their own image search task, as not everyone is looking for a lost hiker wearing blue jeans and a yellow shirt!
We are also interested in adapting our system to thermal cameras, such as the Tamarisk 640, and we hope to do some testing with that system later this year.
Please share your improvements
If you use this code in your own system then please share your improvements! You can clone our repository on github, or post an issue, or just discuss your changes on our mailing list. If we work together we can create a system which will end up saving lives, so please don't just keep your improvements to yourself.
Took some jpeg images from a recent search and ran them through the geosearch.py
script and the results were amazing,
Scanned 124 images in 90 sec, another example scanned 362 images in 4 min.
AGL: 120m and Resolution 12 MPxl with GPS in EXIF
Running Ubuntu 14.04 LTS Linux on VirtualBox on laptop & exported the DISPLAY to Windows XMing Server.
Picked up some other people in the search area, plus some blue 25L plastic containers.
Equipment: DJI Inspire1 running MapsmadeEasy App to Survey the Search area.
Search area in South Africa
For anyone trying to install the latest CUAV - tridge must have added some stuff as I had to install a new package to get it to work.
You need to add these new packages:
sudo apt-get install libxml2 libxml2-dev libxslt1-dev zlib1g-dev
And then lxml will actually compile and run - run the following now:
pip install lxml
Yep - forget it. Unless you can build your own version of a library just stick to the 32 bit 14.04 - it works fine there.
This was from Chris about getting it working on 13.10 x64 which probably also applies to 14.04
I have had problems with this before, but in Ubuntu 12.4 the packaged libturbojpeg worked. But not for me on 13.10, so I had to compile my own from sources.
Since I'm on a 64 bit laptop, libturbo-jpeg was installed to /opt/libjepg-turbo/lib64, but cuav tries to link to /opt/libjpeg-turbo/lib/libturbojpeg.so. I fixed that by making a link
sudo ln -s /opt/libjpeg-turbo/lib64 /opt/libjpeg-turbo/lib
That was enough to get everything building, but not running..."
Having trouble installing cuav it on ubuntu14 64bit
user@ubuntu14:~/cuav$ sudo python ./setup.py build install --user
building 'cuav.image.scanner' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/opt/libjpeg-turbo/include -Icuav/camera/include -I/usr/include/python2.7 -c cuav/image/scanner.c -o build/temp.linux-x86_64-2.7/cuav/image/scanner.o -std=gnu99 -O3
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
cuav/image/scanner.c: In function ‘score_one_region’:
cuav/image/scanner.c:850:25: warning: assignment from incompatible pointer type [enabled by default]
(*pixel_scores) = PyArray_FromDims(2, dims, NPY_DOUBLE);
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/opt/libjpeg-turbo/include -Icuav/camera/include -I/usr/include/python2.7 -c cuav/image/imageutil.c -o build/temp.linux-x86_64-2.7/cuav/image/imageutil.o -std=gnu99 -O3
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/cuav/image/scanner.o build/temp.linux-x86_64-2.7/cuav/image/imageutil.o -L/opt/libjpeg-turbo/lib -lturbojpeg -o build/lib.linux-x86_64-2.7/cuav/image/scanner.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libturbojpeg.a(libturbojpeg_la-turbojpeg.o): relocation R_X86_64_32 against `.data' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libturbojpeg.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Latest version of cuav from github.
Any ideas for a quick fix?
Just sent a test email to the one in your post - if it works email me back and I can send it to you. Otherwise I can give it to Nic if you still see him at all. I am in PerthUAV
Hi Stephen - that's a very kind offer, thanks a lot!
If it's easier for you perhaps you could just send the SD card across, or maybe copy the file system image from the SD card and upload it?
Otherwise if you're happy I can reimburse you for the cost of the unit and postage. Or we can return it to you at Kingaroy.
Thanks again, let me know what would work best for you - h2joe2014 @ gmail.com
I can send you one that worked on the BBB when we were using it before we switched to the Odroid.
Yep, basically just running a stripped down version of geosearch.py
Are you trying to run geosearch.py or are you writing your own?
Many things for having a look at this and for your suggestion.
I have done the following:
-removed the original previously installed libjpeg-turbo (sudo rm -rf <folder>)
-followed tridges steps above
-note I had to use: sudo sh -c "echo '/opt/libjpeg-turbo/lib' > /etc/ld.so.conf.d/turbojpeg.conf" - the sudo sh -c is to get the command to execute properly, and I assume the first part of the command should actually be the link to the directory with the libjpeg-turbo shared objects
-followed the rest of the steps
Now when I import the scanner module in python it no longer says 'cannot find libturbojpeg.so', however it now restarts the program while it is loading the file.
What a doozy! I think it might be linking properly now, but for whatever reason importing the scanner module is now causing the program to crash..