This summer, during my summer internship at Navstik Labs, I took up a project to implement vision-based object tracking and following, using a gimbal mounted camera on a drone. There are already some ready-to-fly drones that offer such a capability. However, it is still quite complex to integrate this feature with your own custom built drones.
Now anybody can guess that tracking and following objects using gimbal will need some OpenCV code and a PID controller; what else could possibly be required! And, that is all I had to do. However, the challenge was to get it up and running on the hardware, and provide a user-friendly interface for live video-feed, object-selection and controller-tuning, on a remote machine (web interface). All this had to be done in a matter of days, not months, by a person with no prior experience of working with drones (me!).
Thanks to FlytPOD/FlytOS platform, I did not have to spend any time interfacing autopilot, gimbal, camera with companion computer, or installing/configuring ROS, OpenCV to work with it, not even setup the wifi link. FlytPOD/FlytOS provided all of that, out of the box. That was a huge timesaver!
Drone with camera, 3 axis Gimbal and FlytPOD
Just some connections between gimbal and FlytPOD and I was ready to roll. I had APIs to command gimbal and drone from ROS, access camera feed in OpenCV, a high-bandwidth wifi-link between FlytPOD and my laptop to monitor live data and set control parameters.
Starting off with OpenCV, I got a color and shape based object detector working. So the detection is handled by Hough circle transform. Later, I also used Zdenek Kalal's TLD algorithm, ported to FlytOS, which uses machine learning to track objects in video stream. Then I put together an onboard app, which calls ROS services for the FlytGimbal API from FlytPOD. Next step was to extract the centroid of the object which would lead me to angular setpoints for gimbal.
FlytGimbal: Web-app for selecting objects to track and gain tuning
Implementing and tuning PID controller was the real experimental stuff, which could have taken me a lot of time, had there not been the web-app to configure my settings during testing. One of my fellow interns got me this nice looking web-app, FlytGimbal, within a day, using RESTful APIs available in FlytOS. Now from my browser, I could select any object from video feed with just a mouse drag, watch the detected false objects or tune my PID gains. FlytOS took care of all the data-plumbing between my ROS code and this web-app.
Now, with this I can ask the gimbal/camera to keep looking at my dog or car, while the drone is free to fly all over the place. I am planning to take this work ahead to solve interesting problems, like, having a drone follow me visually, or land on a moving object, etc.
Read more about FlytOS, FlytPOD here.
Here is link to code for web-app and OpenTLD algorithm ported to FlytOS.
FlytPOD is live on Indiegogo, support us and grab your own Flight Computer.
Comments
Yes, the gimbal is one of those chinese models.
The gimbal can only accept PWM commands, and does not give feedback of its current orientation.
I try to minimize the error between center of frame and object's centroide. First step is to convert this pixel distance into angle targets, which in next step are fed to gimbal in steps. (assuming that gimbal has limited max velocity). P and D gains control this step size.
As far as I know I haven't implemented any tracking filter yet. I would love to hear your ideas how they can improve the things in this case.
Thanks.
thx for the response
I was asking because the gimbal looks like the readily available China gimbals with STorM32 controller. May I assume it's that controller?
Besides all that cool CV stuff, at the end it gives you some coordinates to point at, or target angles. Are you feeding that directly to the gimbal, or do you in addition apply some sort of a tracking filter, e.g. alpha-beta-gamma filter, or a tracking control structure?
Thanks OlliW!
Right now there is separate gimbal controller to stabilize the gimbal. FlytPOD is giving angle commands to this gimbal controller according to object in the scene.
While FlytPOD could also support that, we are focusing on getting the overall thing done with less hassle.
now this is really very cool, fantastic job, sir
may I ask about the gimbal, do you use a separate gimbal controller, or is that also handled by the FlytPOD electronics?
-
1
-
2
of 2 Next