This summer, during my summer internship at Navstik Labs, I took up a project to implement vision-based object tracking and following, using a gimbal mounted camera on a drone. There are already some ready-to-fly drones that offer such a capability. However, it is still quite complex to integrate this feature with your own custom built drones.

Now anybody can guess that tracking and following objects using gimbal will need some OpenCV code and a PID controller; what else could possibly be required! And, that is all I had to do. However, the challenge was to get it up and running on the hardware, and provide a user-friendly interface for live video-feed, object-selection and controller-tuning, on a remote machine (web interface). All this had to be done in a matter of days, not months, by a person with no prior experience of working with drones (me!).

Thanks to FlytPOD/FlytOS platform, I did not have to spend any time interfacing autopilot, gimbal, camera with companion computer, or installing/configuring ROS, OpenCV to work with it, not even setup the wifi link. FlytPOD/FlytOS provided all of that, out of the box. That was a huge timesaver!

Drone with camera, 3 axis Gimbal and FlytPOD

Just some connections between gimbal and FlytPOD and I was ready to roll. I had APIs to command gimbal and drone from ROS, access camera feed in OpenCV, a high-bandwidth wifi-link between FlytPOD and my laptop to monitor live data and set control parameters.

Starting off with OpenCV, I got a color and shape based object detector working. So the detection is handled by Hough circle transform. Later, I also used Zdenek Kalal's TLD algorithm, ported to FlytOS, which uses machine learning to track objects in video stream. Then I put together an onboard app, which calls ROS services for the FlytGimbal API from FlytPOD. Next step was to extract the centroid of the object which would lead me to angular setpoints for gimbal.

FlytGimbal: Web-app for selecting objects to track and gain tuning

Implementing and tuning PID controller was the real experimental stuff, which could have taken me a lot of time, had there not been the web-app to configure my settings during testing. One of my fellow interns got me this nice looking web-app, FlytGimbal, within a day, using RESTful APIs available in FlytOS. Now from my browser, I could select any object from video feed with just a mouse drag, watch the detected false objects or tune my PID gains. FlytOS took care of all the data-plumbing between my ROS code and this web-app.

Now, with this I can ask the gimbal/camera to keep looking at my dog or car, while the drone is free to fly all over the place. I am planning to take this work ahead to solve interesting problems, like, having a drone follow me visually, or land on a moving object, etc.

Read more about FlytOS, FlytPOD here.

Here is link to code for web-app and OpenTLD algorithm ported to FlytOS.

FlytPOD is live on Indiegogo, support us and grab your own Flight Computer.