This blog will describe how to add thermal imaging to your drone without breaking the bank! The above photo shows my drone with the thermal imaging camera, the FlirOne device, which is an inexpensive, lightweight thermal imaging camera designed for Android or IPhone devices.
A flight test of this setup is shown here
The drone platform is arbitrary, as you could follow the instructions here to add thermal imaging to any drone, either copter or fixed-wing UAVs. The platform I chose is a quad copter that is 3-d printed and the printer files for this model are located here.
The FlirOne camera actually has two cameras; a thermal imaging camera, and an HD (1440x1080) camera for mixing the thermal images with the background. The thermal imaging camera has a resolution of 160x120, with a frame rate of 10fps. This a a relatively slow and low resolution, however, this is a 'budget' camera. Higher resolution cameras are VERY expensive and much heavier which balloons the price of the drone also because you need a larger drone. This camera can be mounted on any drone, as it only weighs 36 grams.
In my design, I have a 3rd camera, a Raspberry PI 2 Cam, that has a much higher frame rate for FPV viewing in flight, however this is not necessary unless you want a higher frame rate. The FlirOne has an HD camera built in which works perfectly well if you can stand the low frame rate.
I ended up mounting both the FlirOne and the PI Camera on a 3-d printed dummy GoPro, using these 2-Axis Gimbals.
The FlirOne is designed to work with Android, however, strapping a phone or android device to your drone is not that practical compared to using a Raspberry PI. Fortunately, there are some good hackers out there that figured out the FlirOne USB interface and they were kind enough to publish the driver for Linux using Video-For-Linux (V4L2). In my configuration, I chose to use GStreamer to stream the video over WiFi to the GCS (Ground control station) as shown in the following diagram:
From the above diagram, you can see there are 3 video streams available. If you are using a PI cam, you can disable the second video stream from the FlirOne, or alternatively, use only the FlirOne for both normal video and thermal imaging.
Step 1) Configure Raspberry PI WiFi
In order to get video from your drone, you will need to have a wireless connection of some kind, presumably WiFi. It is beyond the scope of this blog to describe how to connect a Raspberry PI to a WiFi ground station. Please refer to this link for documentation on how to setup WiFi.
For my configuration, I am using a Netis 2561 500mW, 5GHZ WiFi Dongle, with an antenna tracker on the GCS. Using this adapter with a tracker gives several miles range, which makes the drone usable for search and rescue situation where thermal imaging is valuable.
Step 2) Install Linux Driver for the FlirOne
The FlirOne camera has a dual video stream over a single USB connection, so in order to utilize both streams, you need to install the Linux loopback driver to split the single USB source into two Linux devices. To install the loopback driver you will need to build it using the development tools, or download a copy that matches your firmware revision.
Please refer to this forum for some additional information on how to setup a Linux system for the FlirOne. To build the loopback driver use the following commands on your Raspberry PI. Note: you must connect your PI to the internet first.
sudo apt-get install linux-headers-rpi
sudo wget https://raw.githubusercontent.com/not... -O /usr/bin/rpi-source && sudo chmod +x /usr/bin/rpi-source && /usr/bin/rpi-source -q --tag-update
sudo apt-get install bc
git clone https://github.com/umlaeute/v4l2loopback
Note: installing the headers is optional if you already have an image with the kernel source. The kernel source is not included with the standard Raspbian image.
Next, you must build the FlirOne driver which is located here. Unpack this driver and rum the 'make' command supplied with the package.
The files associated with this project can also be found here. A compiled version of the loopback driver and the FlirOne driver is included in this link, however the loopback driver won't work unless you are using the exact same version of the Raspberry PI firmware as my configuration. You will probably need to compile the loopback driver locally. I may publish the entire image if there is interest, but it's 16Gb in size.
Step 3) Configure Startup Script to Start Streams
To make the streams start automatically on boot, I use the startup script "rc.local" in the directory /etc. I have included a copy of this file in the dropbox folder.
Here are the key commands:
sudo modprobe v4l2loopback devices=5
sudo ./flir8p1 Rainbow.raw &
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.1.1 port=9001 &
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=160,height=128, framerate=1/10 ! rtpvrawpay ! udpsink host=192.168.1.1 port=9002 &
The first command loads the loopback module, configuring it for 5 possible devices. In my config, I am actually only using 3, but if you happen to have more devices, you will need to allow for them.
The second command starts the FlirOne driver that connects the USB output to the loopback driver. It will run in the background continuously.
The final two commands are to pipe the output of the two FlirOne cameras to the ground station over WiFI. I am using the IP address 192.168.1.1 and port 9001 and 9002 on my GCS PC, however you may want to use your own setup with different IP addresses and ports.
Step 4) Configure your Ground Control Station to Display the Video
The final step in getting all of this to work is the GCS. You must install GStreamer on your GCS. If you are not familiar with GStreamer, you can download an installer here.
To display the standard video stream use this command:
gst-launch-1.0 udpsrc port=9001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink
To display the thermal image, use this command:
gst-launch-1.0 udpsrc port=9002 !application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)160, height=(string)128, colorimetry=(string)BT601-5, payload=(int)96 ! rtpvrawdepay ! autovideosink
If all goes well, you should see both video streams on your GCS PC like this:
HI, great effort. would you please tell me more about the tracker? how to get or build one?
@AI B, are you using the Raspberry PI? The method I use to control the camera is through the PI. Why connect the camera to the PX4? I connect the PX4 to the Raspberry PI if I want the flight controller to trigger taking pictures. The FLIR Vue Pro is pretty expensive, I built one using the ThermApp camera over USB, costs about $900.
@Patrick, In our case, we're constrained by cost, weight, and space so the small module works best for us; as shown here. And, the PX4 firmware already have a driver for it.
Without those constraints, then we probably had used/mounted the FLIR Vue Pro instead of the Flir One since it can connect directly to the Pixhawk
@AI B, it will probably work. You will need to write your own code to connect to the stream, but it shouldn't be too difficult. Seems like the cost is more if you use this module, compared to the USB camera they sell online though. What is the advantage of using this module compared to the USB camera?
@Patrick, Instead of the usb adapter cable, can one use this board?
Hello Juliasn, the Flir camera is connected over USB, you need an adapter cable. Streaming is done with GStreamer via WiFi to the ground station. You can see the gstreamer commands used in the blog. There is an android or windows app to play back the live stream.
can you please specify how did you connect the flir one to the RPI and how did you streamed the quads image ?
Okay, thank you very much for checking that out. I thought I was going crazy. Will report it to Bill.
@healthyfatboy, I cannot extract it. The file appears to be corrupted.