This blog will describe how to add thermal imaging to your drone without breaking the bank! The above photo shows my drone with the thermal imaging camera, the FlirOne device, which is an inexpensive, lightweight thermal imaging camera designed for Android or IPhone devices.
A flight test of this setup is shown here
The drone platform is arbitrary, as you could follow the instructions here to add thermal imaging to any drone, either copter or fixed-wing UAVs. The platform I chose is a quad copter that is 3-d printed and the printer files for this model are located here.
The FlirOne camera actually has two cameras; a thermal imaging camera, and an HD (1440x1080) camera for mixing the thermal images with the background. The thermal imaging camera has a resolution of 160x120, with a frame rate of 10fps. This a a relatively slow and low resolution, however, this is a 'budget' camera. Higher resolution cameras are VERY expensive and much heavier which balloons the price of the drone also because you need a larger drone. This camera can be mounted on any drone, as it only weighs 36 grams.
In my design, I have a 3rd camera, a Raspberry PI 2 Cam, that has a much higher frame rate for FPV viewing in flight, however this is not necessary unless you want a higher frame rate. The FlirOne has an HD camera built in which works perfectly well if you can stand the low frame rate.
I ended up mounting both the FlirOne and the PI Camera on a 3-d printed dummy GoPro, using these 2-Axis Gimbals.
Setup Architecture
The FlirOne is designed to work with Android, however, strapping a phone or android device to your drone is not that practical compared to using a Raspberry PI. Fortunately, there are some good hackers out there that figured out the FlirOne USB interface and they were kind enough to publish the driver for Linux using Video-For-Linux (V4L2). In my configuration, I chose to use GStreamer to stream the video over WiFi to the GCS (Ground control station) as shown in the following diagram:
From the above diagram, you can see there are 3 video streams available. If you are using a PI cam, you can disable the second video stream from the FlirOne, or alternatively, use only the FlirOne for both normal video and thermal imaging.
Step 1) Configure Raspberry PI WiFi
In order to get video from your drone, you will need to have a wireless connection of some kind, presumably WiFi. It is beyond the scope of this blog to describe how to connect a Raspberry PI to a WiFi ground station. Please refer to this link for documentation on how to setup WiFi.
For my configuration, I am using a Netis 2561 500mW, 5GHZ WiFi Dongle, with an antenna tracker on the GCS. Using this adapter with a tracker gives several miles range, which makes the drone usable for search and rescue situation where thermal imaging is valuable.
Step 2) Install Linux Driver for the FlirOne
The FlirOne camera has a dual video stream over a single USB connection, so in order to utilize both streams, you need to install the Linux loopback driver to split the single USB source into two Linux devices. To install the loopback driver you will need to build it using the development tools, or download a copy that matches your firmware revision.
Please refer to this forum for some additional information on how to setup a Linux system for the FlirOne. To build the loopback driver use the following commands on your Raspberry PI. Note: you must connect your PI to the internet first.
sudo apt-get install linux-headers-rpi
sudo wget https://raw.githubusercontent.com/not... -O /usr/bin/rpi-source && sudo chmod +x /usr/bin/rpi-source && /usr/bin/rpi-source -q --tag-update
rpi-source
sudo apt-get install bc
rpi-source
git clone https://github.com/umlaeute/v4l2loopback
cd v4l2loopback
make
make install
Note: installing the headers is optional if you already have an image with the kernel source. The kernel source is not included with the standard Raspbian image.
Next, you must build the FlirOne driver which is located here. Unpack this driver and rum the 'make' command supplied with the package.
The files associated with this project can also be found here. A compiled version of the loopback driver and the FlirOne driver is included in this link, however the loopback driver won't work unless you are using the exact same version of the Raspberry PI firmware as my configuration. You will probably need to compile the loopback driver locally. I may publish the entire image if there is interest, but it's 16Gb in size.
Step 3) Configure Startup Script to Start Streams
To make the streams start automatically on boot, I use the startup script "rc.local" in the directory /etc. I have included a copy of this file in the dropbox folder.
Here are the key commands:
sudo modprobe v4l2loopback devices=5
cd /home/pi/flir8p1-gpl
sudo ./flir8p1 Rainbow.raw &
sleep 5
#Video
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.1.1 port=9001 &
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=160,height=128, framerate=1/10 ! rtpvrawpay ! udpsink host=192.168.1.1 port=9002 &
The first command loads the loopback module, configuring it for 5 possible devices. In my config, I am actually only using 3, but if you happen to have more devices, you will need to allow for them.
The second command starts the FlirOne driver that connects the USB output to the loopback driver. It will run in the background continuously.
The final two commands are to pipe the output of the two FlirOne cameras to the ground station over WiFI. I am using the IP address 192.168.1.1 and port 9001 and 9002 on my GCS PC, however you may want to use your own setup with different IP addresses and ports.
Step 4) Configure your Ground Control Station to Display the Video
The final step in getting all of this to work is the GCS. You must install GStreamer on your GCS. If you are not familiar with GStreamer, you can download an installer here.
To display the standard video stream use this command:
gst-launch-1.0 udpsrc port=9001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink
To display the thermal image, use this command:
gst-launch-1.0 udpsrc port=9002 !application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)160, height=(string)128, colorimetry=(string)BT601-5, payload=(int)96 ! rtpvrawdepay ! autovideosink
If all goes well, you should see both video streams on your GCS PC like this:
Happy Flying!
Comments
@Bill. OK. I will email you and will pay using PayPal. Thanks.
The answer is YES I have a camera that will trigger via the Pixhawk. Geotagging is done manually in mission planner by pressing control F and using the geotag feature. This is the Wiki Page. http://ardupilot.org/copter/docs/common-geotagging-images-with-miss...
I've actually stopped selling the camera for now because I already have more than 80 beta users, but I would really like to test your application, so I can sell you a system that I can guarantee to work using the SEEK COMPACT.
I am going to do a crowdfunding campaign very soon - I am going to appear on ABC Nightline demonstrating a ROS Application using the Thermal Camera. The crowdfund will shortly before then. I expect a lot of media coverage but cannot sell any cameras during that campaign.
It would be best if I configured the pixhawk and camera for you. I would otherwise require charging you a consulting fee for my time, as explaining it some someone else can take hours sometimes. I know how to do it so it works every time if I have the Pixhawk and GPS.
If you're interested in buying a camera, please email me directly at billrockus@gmail. The price will be higher and you will either have to send me your pixhawk or I can purchase one for you. There is a trick to this that involves syncing the clocks.
You will have to act quickly and I will not be able to deliver for at least 30 days. I would also need your shipping address. There are no returns for full refunds of this type of system, but I will work with you if you are dissatisfied in any way. Any refund would be partial and take up to 60 days.
(Too many people have bought a camera, copied the SD card, and then returned it so they can use it with their own computers and cameras)
If you contact me at billrockus@gmail.com we can do everything via PayPal or bank wire transfer.
Thanks for your interest. The system you will be purchasing will include the ROS Build. For the simple triggering of the camera for geotagging, ROS is not required. I will do things like count livestock when we're done.
Please let me know your thoughts,
Bill Piedra
PS: To give you some of my credentials, I was a contestant in the DJI Developer Challenge, I was mentioned by Brendan Schulman during his speech to the US Senate. I've appeared on TV more than 40 times and I was on all 10 Episodes of the Science Channels TV Show DRONED.
@Bill, you have SEEK Thermal camera trigger for Pixhawk? Does it means that you interface this SEEK thermal camera using MAVROS or Mavlink and use the telemery port of Pixhawk? I am very interested to buy a camera from you if I could geotagged the thermal images, auto or manually.
I liked FLIR's SDK, well done, but their camera is no where near as good as the Seek. Their RGB mixing can be done here with the seek but the codes requires a little modication:
https://github.com/PhroZenOne/heatseeker
It is designed to display the image on a Pi display, but that can be cut out easily. It does a nice job combining an RGB and Thermal picture similar to the way FLIR does.
I have a lepton sensor I'm not using? Interest in buying it?
The cable I use is this one: https://www.amazon.com/gp/product/B00MEBEOW8/ref=ox_sc_act_title_1?...
Odroid XU4 also requires a 4A 5V power, I run it on a 7A UBEC.
Sorry the link I posted is for the Compact Pro, which I do not have running as a ROS Topic yet. This is the link I meant to post:
https://www.amazon.com/dp/B00NYWAHHM/ref=sr_ph_1?ie=UTF8&qid=14...
Do you have a dropbox where I can put a 16GB Disk Image?
@Bill, Colaboration is certainly welcome. Info on how to interface the Seek camera would be great. That link you provided for the Seek camera on Amazon is priced at $499. Where are you getting the $184 figure? I looked at the Seek platform, but Flir had an SDK that was open to the public, but Seek did not. I think they are going to release an SDK but I haven't checked lately. If you have code, that would be great, I would be interested in trying it out, but I probably won't do much for a while considering another $500 for a camera to test it.
https://www.amazon.com/Seek-Thermal-CompactPro-Imaging-Android/dp/B...
And the hard to find cable: https://www.amazon.com/gp/huc/view.html?ie=UTF8&newItems=C9GCZW...
With a Raspberry Pi, you can have a thermal camera working on your drone for $184.00. The software needs work, but I'm available as a consultant. One app I'm working on is a thermal livestock counter. I can trigger and geotag with Pixhawk so you can make a thermal map, and it can be used as a sensor for ROS.
As a ROS Sensor, I'm going on ABC Nightline to demo a drone I built that uses the thermal signature of a person in the water, and then flies toward them to deliver an inflating life preserver. The story from ABC will air around the end of May.
Just thought I'd put in my two cents.
I can help with the software, and I have tons of it, including a camera trigger for Pixhawk.
Let me know if you want the disk image. It's about 16GB, but includes ROS, The SEEK Compact 204x156 camera drivers. ROS preinstalled to use the camera as a ROS Sensor, MAVROS, and a bunch of sample code that I've downloaded, compiled and tested.
If you're not an experience linux programmer, I can sell you one for $600 installed on your drone. I sold a bunch as a beta but spent a ton of time teaching remedial computing. When I install them myself they work PERFECTLY.
Bill
Patrick I can add higher resolution thermal camera support using a Raspberry Pi or Odroid XU4 SBC. I tried to publish the code here but they wouldn't let me. Cost of the SEEK Thermal Camera is $200 or less and can be found on Amazon. I'm happy to share the disk image that runs it as a ROS Topic in Ubuntu 14.04.
Please let me know. Also, I know work as a Developer Advocate for Qualcomm. I'm going to get the camera working there, but the USB is a big problem.
@benbojangles, your certainly welcome to work on the signature detection if you want to contribute. OpenCV would be a good choice.
-
5
-
6
-
7
-
8
-
9
of 9 Next