Controling a copter by image recognition

Hi all, here we would like to share a recent progress on our image recognition based copter controller.

The system uses a tablet and a transmitter. The tablet detects the copter's position by recognizing the red circle marker painted on the copter, and try to maintain the copter's position within the center of the tablet's display (Green circled area on tablet's display) via a Bluetooth -> PPM adapter attached to the trainer port of the transmitter (Futaba 14SG). The control is a simple PID loop, using the difference of red marker and green center shown on display as Err for P, and the change of position for D, with slightly applied I. As shown in the video attached, with a fan blowing aside and deliberately deviated course by hand, once the switch is turned on, the system automatically recovers the copter into the center and maintain it there.

This is an initial try, currently it only control the X and Y on a flat plane, and we are trying to add Z (Depth) and Yaw axis by adding markers and detecting the change in marker size. The whole detection and control is done within the Tablet (Nvidia Shield Tablet), the embedded back camera of the tablet gives a good result inside the room, and we will give it a try soon in outside. The image recognition uses OpenCV, which runs in approx. 20fps (@1080) with a noticeable time-lag. The time-lag causes certain instability on the control loop, to overcome such problem, we are trying to implement an approximated delayed-feedback model into the control loop.

The copter side uses a pixhawk running APM 3.2 in AltHold mode. The system was developed for infrastructure investigation copters, e.g. under a bridge or beside buildings where GPS signal is weak and wind blows randomly, in such case one can fix the tablet on a tripod, shooting and control the copter. We met Randy some weeks ago, when he suggested us to use Mavlink instead of RC, and we would modify the system and make it even simple. This is an open source project by enRoute Co. Ltd., a member of Dronecode from Japan, we will put the source onto GitHub when the Depth and Yaw control is implemented.



E-mail me when people leave their comments –

You need to be a member of diydrones to add comments!

Join diydrones


  • Hi, Kai Yan. Recently , I am trying to control my copter move X ,Y plane in the loiter mode without GPS.  Now , I have created a MAVLink message to input position offset , but the next step I got no idea. Could you share this project or give me some tips?? Thanks for your help. 

  • What edge detection are you using with OpenCV?

  • Now 13MP MIPI camera is available from e-con for Jetson TK1 kit.


    e-con Systems' e-CAM130_CUTK1 - 13mp camera board can be interfaced to the Jetson kit through 4-lane MIPI CSI-2 interface.
  • WZP?
    "The tablet detects the copter's position by recognizing the red circle marker painted on the copter, and try to maintain the copter's position within the center of the tablet's display"
    2D projection positioning? Very freaky =)
    But if u find money for one or two more chinose tablest and try real 3D pilotage it may be very interesting experiments!

  • @Jiro, actually not sure yet, but hopefully there will be more choice on cameras other than USB at the time Tegra X1 Jetson comes out, USB host controller is a bottle-neck in any patform

  • How are you expecting from X1 processor power ?

    Very exciting on your development work.


  • @Chris Thanks a lot :D

    @Jiro Jetson TK1 is a nice platform, however, its USB host controller somehow limits the bandwidth and 1080 video input is not smooth, maybe its not tuned up yet.  We are collaborating with NVIDIA and looking forward to test the Tegra X1 version of Jetson platform. E-CON camera is very nice btw.

    @Randy, thanks! The position difference on display is dependent on depended on Z (distance from copter to camera in depth direction), thus the PID parameter should be a function of Z. If using the constant parameter set there will be overshooting when copter is closer to the camera. The parameter set is also effected by the specific lens the tablet is using, I am trying to make a simple way to customize and adjust such parameters.

    @Daniel In case of trouble we can immediately turn off the signal source by the switch on transmitter, so initially we used transmitter + trainer port input. But now it's getting stable so at some point we will migrate to Mavlink directly. 

  • Very impressive! Was there any reason you tapped into the trainer port of the transmitter instead of sending mavlivk commands(rc override) directly from the tablet using telemetry radios?
  • Excellent control! The e-con80 looks interesting.
  • Hattori-san,

    Yes it is, We are testing Jetson + e-con80.

This reply was deleted.