This blog will describe how to add thermal imaging to your drone without breaking the bank! The above photo shows my drone with the thermal imaging camera, the FlirOne device, which is an inexpensive, lightweight thermal imaging camera designed for Android or IPhone devices.
A flight test of this setup is shown here
The drone platform is arbitrary, as you could follow the instructions here to add thermal imaging to any drone, either copter or fixed-wing UAVs. The platform I chose is a quad copter that is 3-d printed and the printer files for this model are located here.
The FlirOne camera actually has two cameras; a thermal imaging camera, and an HD (1440x1080) camera for mixing the thermal images with the background. The thermal imaging camera has a resolution of 160x120, with a frame rate of 10fps. This a a relatively slow and low resolution, however, this is a 'budget' camera. Higher resolution cameras are VERY expensive and much heavier which balloons the price of the drone also because you need a larger drone. This camera can be mounted on any drone, as it only weighs 36 grams.
In my design, I have a 3rd camera, a Raspberry PI 2 Cam, that has a much higher frame rate for FPV viewing in flight, however this is not necessary unless you want a higher frame rate. The FlirOne has an HD camera built in which works perfectly well if you can stand the low frame rate.
I ended up mounting both the FlirOne and the PI Camera on a 3-d printed dummy GoPro, using these 2-Axis Gimbals.
Setup Architecture
The FlirOne is designed to work with Android, however, strapping a phone or android device to your drone is not that practical compared to using a Raspberry PI. Fortunately, there are some good hackers out there that figured out the FlirOne USB interface and they were kind enough to publish the driver for Linux using Video-For-Linux (V4L2). In my configuration, I chose to use GStreamer to stream the video over WiFi to the GCS (Ground control station) as shown in the following diagram:
From the above diagram, you can see there are 3 video streams available. If you are using a PI cam, you can disable the second video stream from the FlirOne, or alternatively, use only the FlirOne for both normal video and thermal imaging.
Step 1) Configure Raspberry PI WiFi
In order to get video from your drone, you will need to have a wireless connection of some kind, presumably WiFi. It is beyond the scope of this blog to describe how to connect a Raspberry PI to a WiFi ground station. Please refer to this link for documentation on how to setup WiFi.
For my configuration, I am using a Netis 2561 500mW, 5GHZ WiFi Dongle, with an antenna tracker on the GCS. Using this adapter with a tracker gives several miles range, which makes the drone usable for search and rescue situation where thermal imaging is valuable.
Step 2) Install Linux Driver for the FlirOne
The FlirOne camera has a dual video stream over a single USB connection, so in order to utilize both streams, you need to install the Linux loopback driver to split the single USB source into two Linux devices. To install the loopback driver you will need to build it using the development tools, or download a copy that matches your firmware revision.
Please refer to this forum for some additional information on how to setup a Linux system for the FlirOne. To build the loopback driver use the following commands on your Raspberry PI. Note: you must connect your PI to the internet first.
sudo apt-get install linux-headers-rpi
sudo wget https://raw.githubusercontent.com/not... -O /usr/bin/rpi-source && sudo chmod +x /usr/bin/rpi-source && /usr/bin/rpi-source -q --tag-update
rpi-source
sudo apt-get install bc
rpi-source
git clone https://github.com/umlaeute/v4l2loopback
cd v4l2loopback
make
make install
Note: installing the headers is optional if you already have an image with the kernel source. The kernel source is not included with the standard Raspbian image.
Next, you must build the FlirOne driver which is located here. Unpack this driver and rum the 'make' command supplied with the package.
The files associated with this project can also be found here. A compiled version of the loopback driver and the FlirOne driver is included in this link, however the loopback driver won't work unless you are using the exact same version of the Raspberry PI firmware as my configuration. You will probably need to compile the loopback driver locally. I may publish the entire image if there is interest, but it's 16Gb in size.
Step 3) Configure Startup Script to Start Streams
To make the streams start automatically on boot, I use the startup script "rc.local" in the directory /etc. I have included a copy of this file in the dropbox folder.
Here are the key commands:
sudo modprobe v4l2loopback devices=5
cd /home/pi/flir8p1-gpl
sudo ./flir8p1 Rainbow.raw &
sleep 5
#Video
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=640,height=480 ! jpegenc ! rtpjpegpay ! udpsink host=192.168.1.1 port=9001 &
gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,width=160,height=128, framerate=1/10 ! rtpvrawpay ! udpsink host=192.168.1.1 port=9002 &
The first command loads the loopback module, configuring it for 5 possible devices. In my config, I am actually only using 3, but if you happen to have more devices, you will need to allow for them.
The second command starts the FlirOne driver that connects the USB output to the loopback driver. It will run in the background continuously.
The final two commands are to pipe the output of the two FlirOne cameras to the ground station over WiFI. I am using the IP address 192.168.1.1 and port 9001 and 9002 on my GCS PC, however you may want to use your own setup with different IP addresses and ports.
Step 4) Configure your Ground Control Station to Display the Video
The final step in getting all of this to work is the GCS. You must install GStreamer on your GCS. If you are not familiar with GStreamer, you can download an installer here.
To display the standard video stream use this command:
gst-launch-1.0 udpsrc port=9001 ! application/x-rtp,encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink
To display the thermal image, use this command:
gst-launch-1.0 udpsrc port=9002 !application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:0, depth=(string)8, width=(string)160, height=(string)128, colorimetry=(string)BT601-5, payload=(int)96 ! rtpvrawdepay ! autovideosink
If all goes well, you should see both video streams on your GCS PC like this:
Happy Flying!
Comments
Some reviews on Amazon say the CompactPro is a disappointment. Don't know if it's correct, but one review said the frame rate is only 7fps.
The ethz libseekthermal library can also be used as a ROS node if you're interested in robotics applications:
https://github.com/ethz-asl/seekthermal_ros
Don't know if this helps or not:
https://github.com/zougloub/libseek
This one has support for the CompactPro
https://github.com/maartenvds/libseek-thermal
https://github.com/ethz-asl/libseekthermal
The Seek Thermal camera is amazing in relation to the price. The problem is the lack of functioning drivers. "eevblog" has thousands of posts about different attempts. It's a mess.
If you've found working drivers please post a link here!
It would be very thankful not to go through thousands of posts in the eevblog forum. Many thanks! Many thanks! Many thanks!
@Noli @patrick - somewhere on DIYDrones I posted some instructions for someone else on how to use RC10 DO_SET_RELAY from the mission planner camera trigger to run a python script on the pi. I also sync the clock after connecting via mavlink on startup. It works well for the Pi camera or the thermal camera.
After you take the pictures,use CTRL-F in mission planner to inject the geo data to the time stamped pics.
@Noli @patrick - somewhere on DIYDrones I posted some instructions for someone else on how to use RC10 DO_SET_RELAY from the mission planner camera trigger to run a python script on the pi. I also sync the clock after connecting via mavlink on startup. It works well for the Pi camera or the thermal camera.
After you take the pictures,use CTRL-F in mission planner to inject the geo data to the time stamped pics.
@Noli, You need a micro USB female to USB male cable. You can buy them on Amazon. For photos from the FlirOne, you can grab a frame out of the video at any instant, like a trigger from a PixHawk or GCS. I have not code this, but its very possible to create such function. Currently I am just recording the video.
@Patrick, I am interested of FlirOne with Rasberry Pi 3 for Aerial Thermal Photo Mapping. Can the FlirOne be trigger with USB cable? Or this is doable with Rasberry Pi Linux? The idea is to trigger FLirOne using USB cable attached to Pixhawk FC. The aerial thermal photos will be saved in RPi SD or SSD.
@Patrick, I have used both pi 2 & pi zero as (gstreamer) receiver with no terrible frame drops. I use up to 720p, and adjust bitrate, lowering until I find a good stream rate without major image deterioration. i figure it will be neat to compile qtgstreamerhud for pi and connect to HD goggles, maybe after then also add headtracking to control gimbal pan/tilt.
As per Jetson, i'm receiving flight video and experimenting/learning using cnn such as yolo & ssd, just to see what it can and cannot do.
-
4
-
5
-
6
-
7
-
8
of 9 Next