Hello Companion Experimenters

Based on Randy's  Red_Balloon_Finder project, this is the implementation of this fascinating project on a Raspberry Pi 2.

This is the perfect companion computer project and it covers the whole spectrum of the autonomous flight:

1- High level processing language: Python + OpenCV

2- Full integration of the Companion Computer (CC) with Flight Control (FC) and Ground Control Station (GCS)  using MavProxy on a mix of serial and UDP communication

3- Integration of the Balloon_Finder with Software In The Loop (SITL)

4- Usage of Drone API  allowing all sort of automated flight control

5- Perfect introduction to OpenCV using a Drone Specific Application that is relatively easy to program and configure with PYTHON

6- Creation of a complete standardized system image dedicated to this application so it can be used by all the interested experimenters as a training tool to get into vision controlled autonomous flight 

Here is the proposed system:

It is based on the Companion Computer Architecture described here .

This is composed of 3 major building blocs:

1- FC: Flight Control , this is basically the Autopilot

2- CC : Companion Computer, which process the high level computation

3-GCS: Ground Control Station, this is Mission Planner, or QcGroundControl , mavlink --console or any GCS

The FC is interfacing with these signals:

A) Manual or Backup control links from the Radio Control. Transmission type: PWM,PPM,sBus, etc.  Freq. 2,4 Ghz

B)Telemetry & Control to and from CC using  UART .  Signal type: Mavlink.    Direct-Connect.

C) Interface with various sensors and actuators: PWM - UART - I2c - SPI - USB -GPIO

The CC is interfacing with these signals:

A) Main DOWNLINK to GCS. Transmission type WIFI, LTE, Other. Freq. 5 Ghz (WIFI).  Bandwidth (6 - 600 Mbps)

B)Telemetry & Control to and from FC using UART.

C) Interface with Camera    Using USB2  for Logitech or  CSI2 for RasCamera

The GCS is interfacing with these signals:

A) Main DOWNLINK to from CC.

B) Optional connection to Internet using LTE or HotSpot

C) On the computer side, numerous inearfaces can be connected:

    i. Immersive googles

    ii. Joystick

    iii. Gesture Interface (embedded tablet IMU)

    iv You Name It

Views: 2216

Replies to This Discussion

These are the next steps:

a) Build an inventory of all the required programs, drivers, tools to create the project's image;

b) Test, Optimise and integrate all pieces of the puzzle into a working system;

c) Document the various shell programs and services-daemon involved to get the system running;

d) Describe the workflow and inner operations of the Balloon_Finder program and how it interact with the other programs;

e)Document method and procedures to fly the balloon_finder

f)Expand this project by adding more features and complexity, Here are some examples:

  - Optimize code to get better accuracy, speed and hit rate

  - Add gimbal control so the camera can scan while flying

  - Add moving object tracking

  -Add multiple object recognition

Well that is a lot of work, but Randy has already did the most of it , so I will try my best to follow its foosteps... Hope you will !

I'd suggest you fork this: https://github.com/diydrones/companion   and then commit your script/s as per the heirarchy ( ie RPI2/Raspbian for Randy and whoever wants to help the Raspberry PI effort, and Odroid_XU4/Ubuntu for Bills efforts.  )   then push to your personal repo/s and make a pull-request.   :-) 

CHAPTER 1 BUILDING THE IMAGE

References:
https://github.com/rmackay9/ardupilot-balloon-finder
http://dev.ardupilot.com/wiki/raspberry-pi-via-mavlink/


A) GET IMAGE = Raspberrypi.org
B) LOAD IMAGE = Win32 Disk Manager
C) Start system an go Menu - Raspberry pi configuration
- System = Expand Image
- Interfaces == camera and others
- Performance High + GPU : 256
- Set Locales
D) Reboot

E) sudo rpi update (if new RPI)


F) ===========Tools & Packages============
sudo apt-get update
sudo apt-get install screen python-wxgtk2.8 python-matplotlib python-opencv python-numpy python-dev libxml2-dev libxslt-dev gstreamer1.0 mplayer2
sudo apt-get install hostapd dnsmasq iptables iw

sudo easy_install pip
sudo pip install pymavlink mavproxy cherrypy imutils
sudo pip install --upgrade droneapi

TEST:
gst-inspect-1.0 == 223 plugins , 1185 features (or more)
python
import cv2 as cv
import numpy as np
__cv__.version ==2.4.9.1
__np__.version ==1.8.2
crtl-d

G)============Apps======================
INFO:
http://dev.ardupilot.com/wiki/odroid-via-mavlink/#Red_Balloon_Finder

mkdir /home/pi/Flight_Director
cd Flight_Director
git clone https://github.com/rmackay9/ardupilot-balloon-finder.git

NOTE: to run your python script as a program, you also need to set a shebang for Python in the first line.
#!/usr/bin/env python
And give execution privileges to it:
sudo chmod +x /home/pi/Flight_Director/ardupilot-balloon-finder/scripts/*.py
Then you should be able to simply run from anywhere, just by typing name of progam.

CHAPTER 2 CONFIGURATION AND SCRIPTS

H)============ Configuration and Scripts ===================

1- ADD PATH:
cd ~/.
sudo nano .bashrc
#---insert this at end of file----
export PATH=$PATH:/home/pi/Flight_Director/ardupilot-balloon-finder
export PATH=$PATH:/home/pi/Flight_Director/ardupilot-balloon-finder/scripts
export PYTHONPATH=$PYTHONPATH:/home/pi/Flight_Director/ardupilot-balloon-finder/scripts

source ~/.bashrc
or you can use the shorter version of the command:

. ~/.bashrc

TEST with: echo $PATH and echo$PYTHONPATH


2- START the DroneAPI automatically:
sudo nano /home/pi/Flight_Director/MyCopter/.mavinit.scr
add:
module load droneapi.module.api
api start/home/pi/Flight_Director/ardupilot-balloon-finder/scripts/balloon_strategy.py


3- AUTOSTART-MAVPROXY
edit the /etc/rc.local file,
adding the following lines just before the final “exit 0” line:
(
date
echo $PATH
PATH=$PATH:/bin:/sbin:/usr/bin:/usr/local/bin
export PATH
cd /home/pi
screen -d -m -s /bin/bash mavproxy.py --master=/dev/ttyAMA0 --baudrate 57600 --aircraft MyCopter
) > /tmp/rc.log 2>&1
exit 0

NOTE:
Whenever the RPi connects to the Pixhawk, three files will be created in the /home/pi/MyCopter/logs/YYYY-MM-DD directory:
mav.parm : a text file with all the parameter values from the Pixhawk
flight.tlog : a telemetry log including the vehicles altitude, attitude, etc which can be opened using the mission planner (and a number of other tools)
flight.tlog.raw : all the data in the .tlog mentioned above plus any other serial data received from the Pixhawk which might include non-MAVLink formatted messages like startup strings or debug output

If you wish to connect to the MAVProxy application that has been automatically started you can log into the RPi and type:
screen -x


CHAPTER 3 WIFI CONFIGURATION


 WIFI configuration files (Just Scripts for the moment == No Services)


MOST IMPORTANT INFORMATION:

-The chipset have to be AP capable: https://wireless.wiki.kernel.org/en/users/drivers
- On the 5Ghz frequency band, if the country code is not set, hostapd might not start
- If some frequencies are disabled, make sure your driver is set to use the right regulatory domain.
- You can see the current one with: iw reg get
If it says “country 00?, you need to set it manually, in /etc/default/crda.

nano /etc/default/crda
# Set REGDOMAIN to a ISO/IEC 3166-1 alpha2 country code so that iw(8) may set
# the initial regulatory domain setting for IEEE 802.11 devices which operate
# on this system.
#

REGDOMAIN=US

--------------------------INTERFACES--------------------------

This setting allows the RPI to act as WIFI Acces Point

Please set interface according to values set when typing: ifconfig -a

nano /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp


# WiFi
auto wlan0
iface wlan0 inet static
hostapd /etc/hostapd/hostapd.conf
address 192.168.8.1
netmask 255.255.255.0

# Ethernet/RNDIS gadget (g_ether)
iface usb0 inet static
address 192.168.7.2
netmask 255.255.255.0
network 192.168.7.0
gateway 192.168.7.1

usage: service networking restart

----------------------HOSTAPD------------------

Complete this line in the hostapd file

nano /etc/init.d/hostapd

DAEMON_CONF=/etc/hostapd/hostapd.conf

----------------

Set interface according to values set in ifconfig -a

Set country_code according to https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2

Try to set channel to an unused frequency

-----ACCESS POINT @ 2Ghz:--------

nano /etc/hostapd/hostapd.conf

interface=wlan0

driver=nl80211
ssid=BBB_MINI_2Ghz
channel=1
ignore_broadcast_ssid=0
country_code=US
ieee80211d=1
hw_mode=g

macaddr_acl=0
max_num_sta=10
auth_algs=1
ignore_broadcast_ssid=0
rsn_preauth=1
rsn_preauth_interfaces=wlan1
wpa=3
wpa_passphrase=1234567890
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

--------ACCESS POINT @ 5Ghz:----------------

nano /etc/hostapd/hostapd.conf

interface=wlan0
driver=nl80211
ssid=BBB_MINI_5Ghz
channel=48
ignore_broadcast_ssid=0
country_code=US
ieee80211d=1
hw_mode=a
ieee80211n=1
ht_capab=[HT20][SHORT-GI-20][HT40-][SHORT-GI-40]
macaddr_acl=0
max_num_sta=10
auth_algs=1
ignore_broadcast_ssid=0
rsn_preauth=1
rsn_preauth_interfaces=wlan0
wpa=3
wpa_passphrase=1234567890
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

HOW TO TEST HOSTAPD
/etc/hostapd hostapd.conf (-dd = option verbose )

service hostapd restart


----------------------IP FORWARD -------------------
nano /etc/sysctl.conf

net.ipv4.ip_forward=1


----------------------DNS MASQ -------------------------

nano /etc/dnsmasq.conf


interface=lo,wlan0
no-dhcp-interface=lo
dhcp-range=192.168.8.20,192.168.8.40,255.255.255.0,12h

--if using usb--

interface=usb0
dhcp-range=192.168.7.1,192.168.7.1,12h

usage: /etc/init.d/dnsmasq {start|stop|restart|force-reload|dump-stats|status}


---------------------------IP TABLES ---------------------------
type:
iptables -t nat -A POSTROUTING -s 192.168.8.0/24 ! -d 192.168.8.0/24 -j MASQUERADE

copy to file
iptables-save > /etc/iptables.rules

-------------------------RC LOCAL---------------------------
We need to activate NAT in the built-in firewall of Linux to make sure the traffic going out uses the external address as its source address and thus can be routed back. It can be done for example by appending the following line to the file /etc/rc.local

nano /etc/rc.local

Option A) Manual entry:
iptables -t nat -A POSTROUTING -s 192.168.8.0/24 ! -d 192.168.8.0/24 -j MASQUERADE


Option B) Restore from file
iptables-restore < /etc/iptables.rules

REBOOT

CHECK DRIVERS/SERVICES/Light should blink on the wifi card

CHECK CONNECTION STATUS= tail -f /var/log/syslog


CHAPTER 4 CAMERA & GSTREAMER & V4L2

==================c920============================
https://wiki.matthiasbock.net/index.php/Logitech_C920,_streaming_H....

TOOLS:
lsusb == list usb devices
ls /dev/vid* == list video devices
lsmod == show drivers loaded on the kernel

v4l2-ctl --list-formats
v4l2-ctl --all

set camera pixelformat to H264 and various resolutions
v4l2-ctl --device=/dev/video0 --set-fmt-video=width=320,height=240,pixelformat=1
v4l2-ctl --device=/dev/video0 --set-fmt-video=width=640,height=480,pixelformat=1
v4l2-ctl --device=/dev/video0 --set-fmt-video=width=800,height=600,pixelformat=1

==================GSTREAMER=================================
640x480 - TCP
gst-launch-1.0 -v v4l2src device=/dev/video0 ! h264parse \
! video/x-h264, width=640, height=480, framerate=30/1 ! rtph264pay pt=96 config-interval=10 \
! gdppay ! tcpserversink host=192.168.2.108 port=5000


640x480 - UDP (have to set destination adress)
gst-launch-1.0 -v v4l2src device=/dev/video0 ! h264parse \
! video/x-h264, width=640, height=480, framerate=30/1 ! rtph264pay pt=96 config-interval=10 \
! gdppay ! udpsink host=192.168.2.18 port=5000

LOCAL DISPLAY
gst-launch-1.0 -v v4l2src device=/dev/video0 ! h264parse \
! video/x-h264, width=640, height=480 ! omxh264dec ! fbdevsink sync=false

Local with RGB
gst-launch -v v4l2src device=/dev/video0 ! ffmpegcolorspace ! video/x-raw-rgb,width=320,height=240,framerate=15/1 ! fbdevsink sync=false


SIMPLE TEE DEMO
gst-launch videotestsrc ! ffmpegcolorspace ! tee name=t ! queue ! autovideosink t. ! queue ! fbdevsink

TEE == TCP & LOCAL SCREEN
gst-launch-1.0 -v v4l2src device=/dev/video0 ! tee name=t \
! queue ! h264parse ! video/x-h264, width=640, height=480, framerate=30/1 \
! rtph264pay pt=96 config-interval=10 ! gdppay ! tcpserversink host=192.168.2.108 port=5000 t. \
! queue ! h264parse ! video/x-h264, width=640, height=480 ! omxh264dec ! fbdevsink sync=false

CHAPTER 5 Hookup - Setup - Lab Test


Software In The Loop -SITL -Testing
Ubuntu based computer is running SITL:
sim_vehicle.sh -v ArduCopter -f X -j 2 --out=192.168.2.108:14550 --aircraft=balloon --quadcopter --console --map


Testing the mavproxy on the RPI (add .18)
Using tcp connection on port 5763 == Telemetry port (Set Adress of the PI and the port of the master)
mavproxy.py --master tcp:192.168.2.18:5763 --aircraft MyCopter

Configure Balloon_finder
load python colour_finder.py
adjust setting to filter balloon
save setting by sliding bottom cursor
a file balloon.finder.cnf will be saved on root
the parameters can be copied/merged into the balloon_finder.cnf on the script directory


Testing the balloon_finder
module load droneapi.module.api
api start /home/pi/Flight_Director/ardupilot-balloon-finder/scripts/balloon_strategy.py

on console type:
mode loiter
arm throttle
rc 3 1800 ===get to 15-20 alt.
rc 3 1500 === stabilize height
mode guided

==camera starts
==sequence begins
== ''the hunt is on''
mode rtl
control-c to quit

The video sequence (camera and openCV overlay) is saved in AVI files in root (5Fps)
Playback : mplayer balloonn-yyy-mm-dd-hh-min.avi


more to follow......

Oh good, you've got at least parts of it running!

Well, I would say most of it... :-)
What I/We really need is the logic behind the strategy == the introduction chapter that explain how the mission is planned, how the way points interacts with the 360 degree scan, and when does the mission stop : at the last way point , after "n" balloons hit , time limit or rtl when low battery.

I can decypher most of the state machine but i am missing the global strategy.... If any :-))

P.s. I made a major progress tonight on the camera piping, i am getting closer to the full speed split between opencv and UDP sink..... Yeah!!

OK, so at a high level ardupilot (running on the Pixhawk) accepts commands from a GCS or companion computer whenever it's in GUIDED mode or during missions (I.e. AUTO) when the NAV_GUIDED mission command is being executed (that command may have been renamed to GUIDED_ENABLED).  So the balloon-popper runs quietly watching the state of ardupilot and when it sees it has switched to one of these modes it starts issuing commands vi dronekit.  It only sends two types of command, DO_CONDITIONAL_YAW (to change the copters heading) and SET_GLOBAL_POSITION (to control velocity in 3dimensions).

When activated, the balloon-popper commands the vehicle to turn 360 degrees looking for the largest target.  It then turns towards that heading, double checks the big target is still there and then starts sending velocity commands towards it.  It uses a couple of PID controllers to turn the balloon's positionwithin the camera's views into perpendicular velocity commands (I.e. left-right and up-down).  It always maintains a fixed velocity forward.  So basically it keeps flying towards the target at a fixed speed (defined in the config file) but then moves up-down, left-right trying to get the balloon in the middle of the frame.

Once it loses sight for about 3 seconds it gives up and sends velocity commands to stop the vehicle.

CHAPTER 6

BUILDING A NEW KERNEL FOR V4L2LOOPBACK TO GET MULTIPLE CAMERA SOURCES

 DO NOT RPI UPDATE ONCE COMPLETED (apt-get update is OK)


starting with a new image (kernel 4.1.17)
sudo rpi update (if new RPI)
sudo apt-get update
sudo apt-get install dkms build-essential gcc-4.7
sudo rm /usr/bin/gcc
sudo ln -s /usr/bin/gcc-4.7 /usr/bin/gcc

#==check gcc vor 4.7
gcc--version
uname -a = version and wget appropriate kernel header -- 4.1.17

wget https://www.niksula.hut.fi/~mhiienka/Rpi/linux-headers-rpi/linux-he...
sudo dpkg -i linux-headers-4.1.17-v7+_4.1.17-v7+-2_armhf.deb

get from adress below release 9.1
https://github.com/umlaeute/v4l2loopback/releases
expand on home/pi
sudo make
sudo make install

---comments from developers---
on RPI the implementation is not fully functionnal becauses of the missing kernel headers
This is a workaround using insmod instead of modprobe == need to reboot to modify setups.

v4l2loopback requires the videodev module to be loaded manually:
cd ./v4l2loopback-release
sudo modprobe videodev
sudo make modprobe
then
sudo insmod ./v4l2loopback.ko

make script or integrate in rc.local
sudo nano load.sh
sudo modprobe videodev
sudo insmod ./v4l2loopback.ko devices=1 exclusive_caps=0
sudo chmod 777 +x load.sh


check:
lsmod == show driver installed
ls /dev/vid* == show new video devices

==============================================================
SETUP RPI2: Note: sudo chown -R pi /home/pi (optionnal)
http://dev.ardupilot.com/wiki/raspberry-pi-via-mavlink/


sudo apt-get install ==note: If we implement just the basic v4l2 capture and redirect = gstreamer0.10-plugins-base gstreamer0.10-plugins-good
(sudo apt-get install gstreamer-tools gstreamer0.10-plugins-base gstreamer0.10-plugins-good gstreamer0.10-plugins-bad gstreamer0.10-plugins-ugly)
(sudo apt-get install gstreamer1.0 )

TEST:
gst-inspect == xxx plugins , xxx features
=============================================================

=================V4L2LOOPBACK======================
BASIC TEST NO CAMERA:
ls /dev/video* = (Note down the new devices: let X be the number of the first new device.)
v4l2-ctl -D -d /dev/videoX = show config
gst-launch videotestsrc ! v4l2sink device=/dev/videoX = feed target on virtual
mplayer -tv device=/dev/videoX tv:// = see target on mplayer


Some utility examples:
v4l2loopback-ctl set-fps 30 /dev/video0
v4l2loopback-ctl set-caps "video/x-raw-yuv, width=640, height=480" /dev/video0

=======Camera feed to virtual video port============

resolution has to be in conformance with v4l2 setup (see v4l4 utils)
gst-launch -v v4l2src device=/dev/video0 ! ffmpegcolorspace ! video/x-raw-rgb,width=320,height=240,framerate=15/1 ! v4l2sink device=/dev/video1
gst-launch -v v4l2src device=/dev/video0 ! ffmpegcolorspace ! video/x-raw-yuv,width=640,height=480,framerate=15/1 ! v4l2sink device=/dev/video1


Local with RGB
gst-launch -v v4l2src device=/dev/video1 ! ffmpegcolorspace ! video/x-raw-rgb,width=320,height=240,framerate=15/1 ! autovideosink sync=false
mplayer -tv device=/dev/video1 tv://

=========Streaming over WIFI:=========

3 Options are possible:

A) WIFI Unicast & Broadcast with encoders like Gstreamer
B) Reflector : A web Page on the CC where any http based app like broswer or Misson Planner HUD can directly connect like mjpg-stremae - ffmstreamer - motion
C) WifiBroadcast, that is tweeking the maintenance session to pass packet without encapsulation for faster speed and lower latency.


All these methods are opening a large field of experimentation.
Patrick Duffy is doing mostly Gstreamer over wifi using ubuquity equipment, and gets pretty good results.
See its forum: http://diydrones.com/xn/detail/705844:Comment:1799906
befinitiv is developping the WifiBroadcast stuff, you can read his forum:
http://diydrones.com/forum/topics/3-km-hd-fpv-system-using-commodit...
and web: https://befinitiv.wordpress.com/wifibroadcast-analog-like-transmiss...


I do not intend to get deeper in this for the balloon_finder. Thes cheapest, quickest way to get the image on GCS will be retained.
So far it is either Gstreamer UDP reflecting on VLC on GCS, or low res http with ffmpeg/mjpeg-streamer/motion.

Final decision following test and benchmark.

Hi Patrick, nice work on all this information it is very helpful! 

I'm trying to setup routing on my rPi2 correctly for the following situation. The rPi connects to office router (with internet) via wlan and gets IP address 192.168.178.33. I also a device connected to the rPi2 eth0 port which is on a different IP range. Below should describe this,

Device (192.168.0.168) <> eth0 rPi2 (192.168.0.100) <> wlan0 rPi2 (192.168.178.33) <> workstation (192.168.178.25)

I want to be able to ping/talk to the device (192.168.0.168) from the workstation (192.168.178.33). I'm having a hard time figuring out how I should setup the routing on the rPi. 

I have set net.ipv4.ip_forward=1 in /etc/sysctl.conf

Then I should just have to setup the correct routing in iptables? Any suggestions?

RSS

© 2019   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service