This blog is a continuation of my previous post.
How to build a High-Definition FPV UAV using a Rasperry PI with HD camera, using a high speed WiFi link
This post will discuss how to use GStreamer and Mission Planner together to display the HD video with a HUD (Head-Up-Display).
Note: I have only tested this feature on Windows so the instructions given here are for Windows only.
To give proper credit, the HUD created here was borrowed from APM Planner, a Qt-Based app similar to Mission Planner. The HUD part was created from the Qt codebase QML HUD created by Bill Bonney who is on the APM Planner development team. To make the HUD work with the background video, I used a GStreamer library called "QtGStreamer" which integrates GStreamer plugins with painting on a Qt widget. This library is available on the GStreamer website.
The end-result is dynamically added to Mission Planner using the plug-in architecture.
In the previous posts I discussed used a Raspberry PI and a High-speed WiFi link using GStreamer on the PI and the ground station PC. To get the HUD to work, you need to already have a successful link with the video on your ground station.
Here are the steps to follow to install the plugin:
1) Install Mission Planner.
2) Download and install GStreamer from this link. Use the x86 version, the x86_64 version will NOT work. (Use the default path 'C:\GStreamer' when installing). When installing GStreamer, select 'Custom' install and select ALL plugins to be installed.
3) Follow the steps in the previous blog noted above to get your video stream working.
4) Download and the MSI installer from this link. and run the installer.
If all went well, you should have the plugin installed.
Open Mission Planner and navigate to the "Flight Data" page and right-click on the map. You should see a menu item called "GStreamer HUD" as shown below:
Select this menu item and the following screen should appear:
In the upper-left corner is a context menu. Here is where you enter your GStreamer Pipeline string. If you had the video displaying without the HUD using a valid pipeline, enter it here.
Note: The GStreamer Pipeline string should be exactly the same as the string you used before, but WITHOUT the final video sink element. The video sink is the QtGStreamer element which will be added automatically by the plugin. The GStreamer pipe should therefore be the same, except remove the last element for the video sink.
Here is an example string I used on my setup:
udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264
If all is well, you can connect to your UAV and see the HUD elements moving. To change the HUD, right click on the display and select which elements you want to display. The default is to display everything shown here.
If anybody has problems, please post back and I'll update the blog in case I missed something, and you cannot get it to work.
Happy Flying!
Comments
@Paul, Please post the pipeline you are using on both sides. Changing the IP address could affect TCP connections but if you are using UDP to send packets to your ground station, you shouldn't need to modify the pipeline on the ground.
Hi,
I can no longer get the HUD to show video. I had it working before, but all I did was change the IP address of the Pi from which the video is streaming. I changed the IP address as well in he pipeline field in HUD but I keep getting the "Could not open resource for reading" message in the CLI screen.
I appreciate any help to figure out what I'm doing wrong again. Thanks,
Paul
Thanks Patrick.
i agree with your approach of displaying the armed and disarmed messages for only a few seconds. In the meantime I will do some reading on Qt QML.
Paul
@FlyingW, It is possible to add new items but you have to edit the QML files included with the install. It will require some technical knowledge of the Qt QML syntax. I can add that one for you and post another version next week. It is also possible to change the priority level of the messages so you see the "Armed" message appear for a few seconds if you just want to see when it changes instead of showing the armed state on the UI.
How do I add additional information to the stand-alone HUD? I want to see the ARM and DISARM states.
Thanks,
Paul
Only way I know how to make it work is if you have an IP address and Port for the stream. Then use this form: http://10.0.0.39:9090/stream/video.mjpeg
Once nice thing about the uv4l streaming server is several people or computers can monitor simultaneously. We normally have someone watching the stream while our TX board is running a deep neural net correlation.
Wow Ronald, Thanks for the detailed instructions; I will perform them very soon.
In the meantime I am already streaming h264 video from the Pi using raspivid, and can view it on Patrick's stand-alone HUD. Why couldn't I display the same stream in the integrated Mission Planner HUD? When I enter various attempts at the syntax to place in the "set MJPEG source" field, none of them work.
Thanks,
Paul
Streaming Video from Raspicam to web using UV4L Driver
Step 1: Material
- Raspberry Pi or Odroid-W Running Raspbian
- Raspicam
- Router
- Ethernet Cable or wireless usb card
- Windows Computer
- USB Keyboard and Mouse
- HDMI Monitor
Step 2: Install Raspbian
Download NOOBS on your computer and Extract to a Folder (https://www.raspberrypi.org/downloads/noobs/)
Format SD Card. On Windows, use SD Formatter (with format size adjustment “on”). On linux, use gparted (create one single Fat32 partition) (https://www.sdcard.org/downloads/formatter_4/index.html)
Copy all folders and files in NOOBS to SD Card
Install SD Card in Raspberry Pi
Connect Raspberry Pi to a monitor, keyboard, and mouse and power it up. Change Language to English (US). Install Raspbian.
Step 3: Set Configuration
Connect to Internet via WiFi or LAN Cable
Open the terminal and run:
$ sudo raspi-config
Under Boot Options, Select Console Autologin
Under Internationalizations Options, Select Locale, Timezone, and Wi-Fi Country
Under Enable Camera, Select “Enable”
Under Advanced Options, Select Memory Split and “256”
Under Advanced Options, Select SSH and “Enable”
Under Advanced Options, Select “Update”
Reboot Raspbian
Open the terminal and update the GPU Firmware:
$ sudo rpi-update
Step 4: Installing Raspicam Driver
To install the uv4l driver, open the terminal and run the following commands:
$ wget http://www.linux-projects.org/listing/uv4l_repo/lrkey.asc && sudo apt-key add ./lrkey.asc
Add the following line to the file /etc/apt/sources.list :
$ sudo nano /etc/apt/sources.list
deb http://www.linux-projects.org/listing/uv4l_repo/raspbian/ jessie main
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install uv4l uv4l-raspicam uv4l-raspicam-extras uv4l-server uv4l-uvc uv4l-xscreen uv4l-mjpegstream
$ sudo reboot
$ sudo apt-get autoremove #(to cleanup after reboot)
Source: http://www.linux-projects.org/uv4l/installation/
$ sudo ifconfig
And note the IP address of the device (eth0 if cabled or wlan0 if wireless)
Use this IP address with browser and ssh client (putty).
$ sudo shutdown now
Shut down the device
Step 5: Connect the Camera
Make sure the device is powered off.
Gently lift the plastic lid on the connector behind the Ethernet port. Do not use the other connector on the edge (that is used for the LCD and will fry the camera).
Plug the ribbon cable in with the stripped side aligned with the pins on the connector (see photo). Make sure it fits smoothly.
Gently push the lid back on and power the device back on.
Step 6: Start the streaming server
Note the IP address from step 4. You will need it for the browser and ssh client.
The server should start automatically on port 8080. Verify by using browser (http://<raspberry IP address>:8080). You should see the UV4L Streaming Server panel, or (http://<raspberry IP address>:8080/stream) to start streaming.
By default it uses mjpeg. To modify the settings, use the “configuration” option in the browser or edit the config file directly on the PI and then restart the service:
$ sudo nano /etc/uv4l/uv4l-raspicam.conf
$ sudo service uv4l_raspicam restart
Alternatively, you can run the server manually with different options. ssh into the PI, kill the existing process and start a new one.
$ sudo pkill uv4l
(Option 1) $ sudo uv4l -nopreview --auto-video_nr --driver raspicam --encoding mjpeg --width 640 --height 480 --framerate 20 --server-option '--port=9090' --server-option '--max-queued-connections=30' --server-option '--max-streams=25' --server-option '--max-threads=29'
(Option 2)$ sudo uv4l -nopreview --auto-video_nr --driver raspicam --encoding h264 --width 640 --height 480 --framerate 20 --server-option '--port=9090' --server-option '--max-queued-connections=30' --server-option '--max-streams=25' --server-option '--max-threads=29'
(Option 3)$ sudo uv4l -nopreview --auto-video_nr --driver raspicam --sched-rr --encoding h264 --width 640 --height 480 --framerate 20 --server-option '--port=9090' --server-option '--max-queued-connections=30' --server-option '--max-streams=25' --server-option '--max-threads=29'
As an alternative to manually setting the real time scheduling priority to the uv4l process each time, you can run the driver with the --sched-rr option, which requires root privileges by default:
Notes:
For ssh, the default login is pi / raspberry
Edit the line above line and choose only one format (mjpeg or h264)
The --port=9090 is the local IP port. You can use any port you like.
The --max-streams=25 is the maximum simultaneous streams.
Step 7: Configure your router
- Open a Web browser
- Connect to Router (password Kashmir4)
- Type your router IP address (http://192.168.1.1/ ).
- Router login. If you're using Linksys router, the user and pass could be "admin" and “password”
- Port forward your Raspberry Pi IP address.
If you are using Linksys router:
- Click "Applications & Gaming" Tab
- Click "Simple Port Forwarding" Tab
- Configure the following parameters:
External Port: 9090 (Public IP port to reach your Raspberry stream)
*Make sure your chosen port is Open with http://mxtoolbox.com/PortScan.aspx. If it's not, call your internet provider to open the port.
Internal Port: 9090 (Raspberry Pi streaming port)
Protocol: Both
To IP Address: 192.168.1.104 (Your local Raspberry Pi IP Address)
Enabled: checked
Resources
How to find your router IP address: http://portforward.com/networking/routers_ip_addre...
Port Forwarding: http://setuprouter.com/router/cisco/linksys-e1200/...
Step 8: Test
To see the streaming follow the next steps:
- Open a browser
- Type your Raspberry Pi IP followed by the external/public port (http://Your Raspberry Pi IP:9090/stream)
Step 9: Set Up for Stream on Boot
To set up your Raspberry to Stream on boot, add the following lines to etc/rc.local
sudo nano /etc/rc.local
sudo pkill uv4l
sudo uv4l -nopreview --sched-rr --auto-video_nr --driver raspicam --encoding mjpeg --width 640 --height 480 --framerate 20 --server-option '--port=9090' --server-option '--max-queued-connections=30' --server-option '--max-streams=25' --server-option '--max-threads=29'
Step 10: Stream/Record Over Mission Planner HUD
Open Mission Planner
Right Click on HUD
Under Video, Select MJPEG Source
Enter the URL to the mjpeg source:
http://10.0.0.39:9090/stream/video.mjpeg
Step 11: Stream/Record Over VLC Player
Open VLC Player
Media
Open Network Stream
http://10.0.0.27:9090/stream/video.mjpeg
Step 12: Record Video to micro SD Card
dd if=/dev/video0 of=video.mjpeg bs=1M & pid=$! ; sleep 10; kill $pid
Appendix A: Diagnostics
If you are unable to see the stream, run the command:
$ sudo raspistill -v -o test.jpg
If you see:
mmal: Camera is not detected. Please check carefully the camera module is installed correctly
Make sure you enabled the camera in raspi-config
Make sure the ribbon cable is plugged in the right way
If you see:
mmal: No data received from sensor. Check all connections, including the Sunny one on the camera board
Make sure you power down the PI before performing any of the actions below and power it back up afterwards to retest.
Make sure the ribbon cable is plugged in firmly into the connector.
Make sure the other side is plugged in firmly into the camera board.
Gently clean the cable end going into the PI.
If you have an extra cable, replace it.
If all else fails, you may have a bad camera :(
Appendix B: Backup/Restore SD Card
You can backup the SD card after you are done with the configuration and then flash another SD card as needed, to replace the original card. This can help save a lot of time. Power down the PI, remove the mini SD card and plug it into your PC (using SD or USB adapter).
On Windows:
Source: http://raspberrypi.stackexchange.com/questions/311/how-do-i-backup-my-raspberry-pi
On Linux:
In the example below, SD card is /dev/sdg - double check and modify as necessary
To Backup
$ sudo dd if=/dev/sdg of=pi_sd_image
To restore on a FAT32 formatted SD card:
$ sudo dd if=pi_sd_image of=/dev/sdg
Alternatively, you can use compression to save some space on your hard drive (will take longer):
Backup:
$ sudo dd if=/dev/sdg | gzip > pi_sd_image.gz
Restore:
$ sudo gzip -dc pi_sd_image.gz | dd of=/dev/sdg
Appendix C: Serial Console into PI
Using a USB to TTL cable (https://www.adafruit.com/product/954) can eliminate the need for a monitor/keyboard to initially configure the PI. Connect the cable as shown in picture and plug into USB on you PC.
Note: The red wire provides power to the PI. Make sure the mini USB is unplugged before connecting it. Alternatively, leave the red wire unconnected and lug in the mini USB to power the PI.
On Windows:
Download and install the drivers for the cable (https://cdn-shop.adafruit.com/product-files/954/PL2303_Prolific_DriverInstaller_v1_12_0.zip).
Download and install putty.
Plug the cable in.
Search “Devices and Printers” to find the COM port the cable is connected under (assume COM6 for this example)
Launch putty:
Under Connection type: click Serial
Under Serial line type COM6 (or whatever port is listed in 4.)
Under Speed type 115200
Optionally, save the parameters for easy access in the future
Under Saved Sessions type PI Serial (or another meaningful name)
Click Save
Click Open
On Linux:
Use putty as above or launch terminal and use:
$ screen /dev/ttyUSB0 115200
Ronald,
Please tell me more - I'm using a Rapsberry Pi with raspivid right now. With what commands or statement do I tell the native Mission Planner HUD to display the streaming video from the Pi?
Thanks,
Paul
There are many options for streaming video in the native Mission Planner. We set up a Raspberry Pi as uv4l streaming server onboard the aircraft. You can view/save the stream on your GCS via VLC Player, in Mission Planner, over any Android device, or in Tower.
-
2
-
3
-
4
-
5
-
6
of 39 Next