Thermal Imaging for your Drone on a Budget

This blog will describe how to add thermal imaging to your drone without breaking the bank! The above photo shows my drone with the thermal imaging camera, the FlirOne device, which is an inexpensive, lightweight thermal imaging camera designed for Android or IPhone devices.  

A flight test of this setup is shown here

The drone platform is arbitrary, as you could follow the instructions here to add thermal imaging to any drone, either copter or fixed-wing UAVs. The platform I chose is a quad copter that is 3-d printed and the printer files for this model are located here.

The FlirOne camera actually has two cameras; a thermal imaging camera, and an HD (1440x1080) camera for mixing the thermal images with the background. The thermal imaging camera has a resolution of 160x120, with a frame rate of 10fps. This a a relatively slow and low resolution, however, this is a 'budget' camera. Higher resolution cameras are VERY expensive and much heavier which balloons the price of the drone also because you need a larger drone. This camera can be mounted on any drone, as it only weighs 36 grams. 

In my design, I have a 3rd camera, a Raspberry PI 2 Cam, that has a much higher frame rate for FPV viewing in flight, however this is not necessary unless you want a higher frame rate. The FlirOne has an HD camera built in which works perfectly well if you can stand the low frame rate. 

I ended up mounting both the FlirOne and the PI Camera on a 3-d printed dummy GoPro, using these 2-Axis Gimbals

Setup Architecture

The FlirOne is designed to work with Android, however, strapping a phone or android device to your drone is not that practical compared to using a Raspberry PI.  Fortunately, there are some good hackers out there that figured out the FlirOne USB interface and they were kind enough to publish the driver for Linux using Video-For-Linux (V4L2).  In my configuration, I chose to use GStreamer to stream the video over WiFi to the GCS (Ground control station) as shown in the following diagram:

From the above diagram, you can see there are 3 video streams available. If you are using a PI cam, you can disable the second video stream from the FlirOne, or alternatively, use only the FlirOne for both normal video and thermal imaging. 

Step 1) Configure Raspberry PI WiFi

In order to get video from your drone, you will need to have a wireless connection of some kind, presumably WiFi. It is beyond the scope of this blog to describe how to connect a Raspberry PI to a WiFi ground station. Please refer to this link for documentation on how to setup WiFi. 

For my configuration, I am using a Netis 2561 500mW, 5GHZ WiFi Dongle, with an antenna tracker on the GCS. Using this adapter with a tracker gives several miles range, which makes the drone usable for search and rescue situation where thermal imaging is valuable.

Step 2) Install Linux Driver for the FlirOne 

The FlirOne camera has a dual video stream over a single USB connection, so in order to utilize both streams, you need to install the Linux loopback driver to split the single USB source into two Linux devices. To install the loopback driver you will need to build it using the development tools, or download a copy that matches your firmware revision. 


Please refer to this forum for some additional information on how to setup a Linux system for the FlirOne.  To build the loopback driver use the following commands on your Raspberry PI. Note: you must connect your PI to the internet first.

sudo apt-get install linux-headers-rpi
sudo wget https://raw.githubusercontent.com/not... -O /usr/bin/rpi-source && sudo chmod +x /usr/bin/rpi-source && /usr/bin/rpi-source -q --tag-update
rpi-source
sudo apt-get install bc
rpi-source

git clone https://github.com/umlaeute/v4l2loopback
cd v4l2loopback
make
make install

Note: installing the headers is optional if you already have an image with the kernel source. The kernel source is not included with the standard Raspbian image.

Next, you must build the FlirOne driver which is located here. Unpack this driver and rum the 'make' command supplied with the package. 

The files associated with this project can also be found here.  A compiled version of the loopback driver and the FlirOne driver is included in this link, however the loopback driver won't work unless you are using the exact same version of the Raspberry PI firmware as my configuration. You will probably need to compile the loopback driver locally. I may publish the entire image if there is interest, but it's 16Gb in size.

Step 3) Configure Startup Script to Start Streams

To make the streams start automatically on boot, I use the startup script "rc.local" in the directory /etc.  I have included a copy of this file in the dropbox folder.

Here are the key commands:

sudo modprobe v4l2loopback devices=5

cd /home/pi/flir8p1-gpl
sudo ./flir8p1 Rainbow.raw &
sleep 5

#Video
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.1.1 port=9001 &
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=160,height=128, framerate=1/10 ! rtpvrawpay ! udpsink host=192.168.1.1 port=9002 &

The first command loads the loopback module, configuring it for 5 possible devices. In my config, I am actually only using 3, but if you happen to have more devices, you will need to allow for them.

The second command starts the FlirOne driver that connects the USB output to the loopback driver. It will run in the background continuously.

The final two commands are to pipe the output of the two FlirOne cameras to the ground station over WiFI. I am using the IP address 192.168.1.1 and port 9001 and 9002 on my GCS PC, however you may want to use your own setup with different IP addresses and ports.

Step 4) Configure your Ground Control Station to Display the Video

The final step in getting all of this to work is the GCS. You must install GStreamer on your GCS. If you are not familiar with GStreamer, you can download an installer here.

To display the standard video stream use this command:

gst-launch-1.0 udpsrc port=9001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink

To display the thermal image, use this command:

gst-launch-1.0 udpsrc port=9002 !application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)160, height=(string)128, colorimetry=(string)BT601-5, payload=(int)96 ! rtpvrawdepay ! autovideosink

If all goes well, you should see both video streams on your GCS PC like this:

Happy Flying!

Views: 13542

Comment by Pascal P. on May 6, 2017 at 2:09am

Nice contribution Patrick, as usual from you !

DIY always at its best.

Comment by benbojangles on May 6, 2017 at 5:10am

Automated thermal signature detection/notification during flight next! :)

Comment by Rick on May 6, 2017 at 3:39pm

I gather the Netis WF2561 is on the GCS.

What WiFi dongle are you using on the quad?

TIA,

Rick

Comment by Patrick Duffy on May 6, 2017 at 8:48pm

@Rick, the Netis WiFi dongle is on the quad.  The GCS has a tracker with a Ubiquity high-gain dish antenna. You can see it on the video.

Comment by Patrick Duffy on May 6, 2017 at 8:50pm

@benbojangles, your certainly welcome to work on the signature detection if you want to contribute. OpenCV would be a good choice. 


Moderator
Comment by Bill Piedra on May 6, 2017 at 10:16pm

Patrick I can add higher resolution thermal camera support using a Raspberry Pi or Odroid XU4 SBC.  I tried to publish the code here but they wouldn't let me.  Cost of the SEEK Thermal Camera is $200 or less and can be found on Amazon.   I'm happy to share the disk image that runs it as a ROS Topic in Ubuntu 14.04.

Please let me know.  Also, I know work as a Developer Advocate for Qualcomm.  I'm going to get the camera working there, but the USB is a big problem.


Moderator
Comment by Bill Piedra on May 6, 2017 at 10:41pm
Here are a couple of links to the cameras from Seek on Amazon for 204x156 15hz Camera:

https://www.amazon.com/Seek-Thermal-CompactPro-Imaging-Android/dp/B...

And the hard to find cable: https://www.amazon.com/gp/huc/view.html?ie=UTF8&newItems=C9GCZW...

With a Raspberry Pi, you can have a thermal camera working on your drone for $184.00. The software needs work, but I'm available as a consultant. One app I'm working on is a thermal livestock counter. I can trigger and geotag with Pixhawk so you can make a thermal map, and it can be used as a sensor for ROS.

As a ROS Sensor, I'm going on ABC Nightline to demo a drone I built that uses the thermal signature of a person in the water, and then flies toward them to deliver an inflating life preserver. The story from ABC will air around the end of May.

Just thought I'd put in my two cents.

I can help with the software, and I have tons of it, including a camera trigger for Pixhawk.

Let me know if you want the disk image. It's about 16GB, but includes ROS, The SEEK Compact 204x156 camera drivers. ROS preinstalled to use the camera as a ROS Sensor, MAVROS, and a bunch of sample code that I've downloaded, compiled and tested.

If you're not an experience linux programmer, I can sell you one for $600 installed on your drone. I sold a bunch as a beta but spent a ton of time teaching remedial computing. When I install them myself they work PERFECTLY.

Bill
Comment by Patrick Duffy on May 6, 2017 at 10:59pm

@Bill, Colaboration is certainly welcome. Info on how to interface the Seek camera would be great. That link you provided for the Seek camera on Amazon is priced at $499.  Where are you getting the $184 figure? I looked at the Seek platform, but Flir had an SDK that was open to the public, but Seek did not. I think they are going to release an SDK but I haven't checked lately.  If you have code, that would be great, I would be interested in trying it out, but I probably won't do much for a while considering another $500 for a camera to test it. 


Moderator
Comment by Bill Piedra on May 6, 2017 at 11:02pm

Sorry the link I posted is for the Compact Pro, which I do not have running as a ROS Topic yet.  This is the link I meant to post: 

https://www.amazon.com/dp/B00NYWAHHM/ref=sr_ph_1?ie=UTF8&qid=14...

Do you have a dropbox where I can put a 16GB Disk Image? 


Moderator
Comment by Bill Piedra on May 6, 2017 at 11:04pm

The cable I use is this one: https://www.amazon.com/gp/product/B00MEBEOW8/ref=ox_sc_act_title_1?...

Odroid XU4 also requires a 4A 5V power, I run it on a 7A UBEC.

Comment

You need to be a member of DIY Drones to add comments!

Join DIY Drones

© 2019   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service