How to setup a Mission Planner HD-Video HUD using GStreamer and Mission Planner plugins

This blog is a continuation of my previous post.

How to build a High-Definition FPV UAV using a Rasperry PI with HD camera, using a high speed WiFi link

This post will discuss how to use GStreamer and Mission Planner together to display the HD video with a HUD (Head-Up-Display).

Note: I have only tested this feature on Windows so the instructions given here are for Windows only.

To give proper credit, the HUD created here was borrowed from APM Planner, a Qt-Based app similar to Mission Planner. The HUD part was created from the Qt codebase QML HUD created by Bill Bonney who is on the APM Planner development team. To make the HUD work with the background video, I used a GStreamer library called "QtGStreamer" which integrates GStreamer plugins with painting on a Qt widget. This library is available on the GStreamer website.

The end-result is dynamically added to Mission Planner using the plug-in architecture.

In the previous posts I discussed used a Raspberry PI and a High-speed WiFi link using GStreamer on the PI and the ground station PC. To get the HUD to work, you need to already have a successful link with the video on your ground station.

Here are the steps to follow to install the plugin:

1) Install Mission Planner.

2) Download and install GStreamer from this link. Use the x86 version, the x86_64 version will NOT work. (Use the default path 'C:\GStreamer' when installing). When installing GStreamer, select 'Custom' install and select ALL plugins to be installed.

3) Follow the steps in the previous blog noted above to get your video stream working.

4) Download and the MSI installer from this link. and run the installer.

If all went well, you should have the plugin installed.

Open Mission Planner and navigate to the "Flight Data" page and right-click on the map. You should see a menu item called "GStreamer HUD" as shown below:

Select this menu item and the following screen should appear:

In the upper-left corner is a context menu. Here is where you enter your GStreamer Pipeline string. If you had the video displaying without the HUD using a valid pipeline, enter it here.

Note: The GStreamer Pipeline string should be exactly the same as the string you used before, but WITHOUT the final video sink element. The video sink is the QtGStreamer element which will be added automatically by the plugin. The GStreamer pipe should therefore be the same, except remove the last element for the video sink.

Here is an example string I used on my setup:

udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264

If all is well, you can connect to your UAV and see the HUD elements moving. To change the HUD, right click on the display and select which elements you want to display. The default is to display everything shown here.

If anybody has problems, please post back and I'll update the blog in case I missed something, and you cannot get it to work.

Happy Flying!

@Paul, I have not observed the latency issue with the previous version of the HUD with the mavlink stream, but I could have introduced a bug in the last release. Let me run some tests and get back with you.

Here's the PI stream command:

raspivid -t 0 -w 1280 -h 720 -fps 40 -b 4000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse config-interval=1 ! rtph264pay ! udpsink host= $1 port=9000

This is for 720p. If you want lower latency, and can accept a lower resolution, 800x600 on my setup is sub 100ms. Better for FPV if you have goggles that are not HD, no need to send an HD data stream. Also, I would play with the frame rate/bit rate to see what kind of performance you can get. You need a good graphics card on your ground PC to get the best performance.

I could not get 48fps to work, but others have reported success with 48fps/6mps bitrate. My ground station PC is too slow I think, but you can give it a shot.

For the Nano/Rocket question, I am using two Rocket M5s, and the ground station has two spiral 14dbi antennas, one RHCP and one LHCP. My setup is custom, because I could not buy a LHCP antenna, so I rewound an RHCP one to be LHCP. It took a bit of soldering, but it works.

For the best performance without having to mess with your own antenna setup, I would suggest using the Nano on the ground. However, if you have circularly polarized antennas on your UAV, you get a 3db loss using the Nano. This is usually very acceptable and will work for a fair distance, probably 10km range, if you have good antennas on the UAV.

You need to be a member of diydrones to add comments!

Join diydrones

Comments

Patrick Duffy March 23, 2015 at 6:44pm

@Kevin, you need a clock utility that updates the display with a timed numerical output where you can point your camera and display the HUD window on the same PC. You can then capture the screen and look at the difference. If you are running with a raspberry PI and a similar GStreamer pipe with UDP, I can guess that it's around 120 - 140ms. Someone posted a link to the clock utility on another thread, but I have to go find it.
Kevin Goff March 23, 2015 at 4:53pm

Works like a charm on first try! Thanks for putting this together! I have very low visual latency. How can I measure the actual latency?
Patrick Duffy March 1, 2015 at 8:33pm

For anybody interested, I have posted a new version with some additional enhancements. The HUD now includes a 'home heading' indicator on the compass in case you get disoriented while flying FPV (happens to me occasionally) and you don't want to hit the RTL button. There are also about 10 other elements that can be optionally displayed, but are hidden by default. (Time-In-Air, GPS HDOP, lat/lon, dist to home, dist to waypoint, and a few others). Currently you have to edit the 'InformationIndicator.qml' file to display them, but in a near release I will put this option on the GUI.

I have uploaded the Windows (Stand-alone and MIssion Planner versions), to the dropbox. Android version will take a couple more days.
Amit Ganjoo February 27, 2015 at 7:11am

@Patrick, Thanks I will let you know how it goes
Patrick Duffy February 27, 2015 at 5:24am

@Amit, here'l the link to the android version:

https://www.dropbox.com/s/g38tv79z3ey8yks/QtGStreamerHUD.apk?dl=0

A note on the Android version, you need a pretty snappy phone to run this app if you are going to display HD. If you run into pixelation problems, try reducing the resolution.

QtGStreamerHUD.apk

Shared with Dropbox
Amit Ganjoo February 26, 2015 at 6:51pm

@Patrick I have been testing the windows version and that is working great but I cannot fin a link to the android version

Can you please reshare?

Thanks
Patrick Duffy February 20, 2015 at 5:37am

For anybody interested in running on XP, here's a link to the installer: https://www.dropbox.com/s/a2x3chw9ycyxlx4/GStreamerHUD_XP.msi?dl=0

Also, for the current windows version, I have redesigned the installation to include the dependencies for GStreamer so it's no longer necessary to install GStreamer separately. If you already have it installed, it should still work.

GStreamerHUD_XP.msi

Shared with Dropbox
Patrick Duffy February 15, 2015 at 6:40pm

FYI, I have posted some new version of the stand-alone app for both Windows and Android on the dropbox. Mainly this is an enhancement release. I have added the ability to scale the HUD and also change the font size of the info pane. I have tested the Android version on about 3 different types of devices, a couple of phones and a 7 inch tablet running Android 5 'Lollipop' and it works with pretty good latency. The HUD should work on anything back to Android SDK 14, but some devices it will choke no matter what version of Android is on the device because of some incompatibilities with Open GL and QtGStreamer. So if you try it out and it fails, let me know what kind of device you are running and what version of Android is on the device.

Otherwise, have fun, and let me know if you have problems.
Patrick Duffy February 13, 2015 at 9:52am

@Paul, I have not observed the latency issue with the previous version of the HUD with the mavlink stream, but I could have introduced a bug in the last release. Let me run some tests and get back with you.

Here's the PI stream command:

raspivid -t 0 -w 1280 -h 720 -fps 40 -b 4000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse config-interval=1 ! rtph264pay ! udpsink host= $1 port=9000

This is for 720p. If you want lower latency, and can accept a lower resolution, 800x600 on my setup is sub 100ms. Better for FPV if you have goggles that are not HD, no need to send an HD data stream. Also, I would play with the frame rate/bit rate to see what kind of performance you can get. You need a good graphics card on your ground PC to get the best performance.

I could not get 48fps to work, but others have reported success with 48fps/6mps bitrate. My ground station PC is too slow I think, but you can give it a shot.

For the Nano/Rocket question, I am using two Rocket M5s, and the ground station has two spiral 14dbi antennas, one RHCP and one LHCP. My setup is custom, because I could not buy a LHCP antenna, so I rewound an RHCP one to be LHCP. It took a bit of soldering, but it works.

For the best performance without having to mess with your own antenna setup, I would suggest using the Nano on the ground. However, if you have circularly polarized antennas on your UAV, you get a 3db loss using the Nano. This is usually very acceptable and will work for a fair distance, probably 10km range, if you have good antennas on the UAV.
Paul Hammond February 13, 2015 at 1:24am

Great thanks Patrick.

I also downloaded your standalone HUD application on windows. I think ihave a issue with my configuration because i can get video streaming with low latency, but as soon as i connect power to the pixhawk and it starts to stream mavdata into the RPI my video goes very slow.

Have you seen a similar issue with video having high latency when mavlink is streaming?

Can i ask for your pipeline that you run on your RPI? I saw something about using 48fps but would like to try a well tested pipeline.

Ill have another play with the android app later, ill see if i can connect a usb keyboard and make it easier to input the correct string. If I find any issues I will let you know.

Is the ubiquiti rocket m5 & ubiquiti nano m5 the way to go?

Thanks Paul

of 39

This reply was deleted.