This blog is a continuation of my previous post.
How to build a High-Definition FPV UAV using a Rasperry PI with HD camera, using a high speed WiFi link
This post will discuss how to use GStreamer and Mission Planner together to display the HD video with a HUD (Head-Up-Display).
Note: I have only tested this feature on Windows so the instructions given here are for Windows only.
To give proper credit, the HUD created here was borrowed from APM Planner, a Qt-Based app similar to Mission Planner. The HUD part was created from the Qt codebase QML HUD created by Bill Bonney who is on the APM Planner development team. To make the HUD work with the background video, I used a GStreamer library called "QtGStreamer" which integrates GStreamer plugins with painting on a Qt widget. This library is available on the GStreamer website.
The end-result is dynamically added to Mission Planner using the plug-in architecture.
In the previous posts I discussed used a Raspberry PI and a High-speed WiFi link using GStreamer on the PI and the ground station PC. To get the HUD to work, you need to already have a successful link with the video on your ground station.
Here are the steps to follow to install the plugin:
1) Install Mission Planner.
2) Download and install GStreamer from this link. Use the x86 version, the x86_64 version will NOT work. (Use the default path 'C:\GStreamer' when installing). When installing GStreamer, select 'Custom' install and select ALL plugins to be installed.
3) Follow the steps in the previous blog noted above to get your video stream working.
4) Download and the MSI installer from this link. and run the installer.
If all went well, you should have the plugin installed.
Open Mission Planner and navigate to the "Flight Data" page and right-click on the map. You should see a menu item called "GStreamer HUD" as shown below:
Select this menu item and the following screen should appear:
In the upper-left corner is a context menu. Here is where you enter your GStreamer Pipeline string. If you had the video displaying without the HUD using a valid pipeline, enter it here.
Note: The GStreamer Pipeline string should be exactly the same as the string you used before, but WITHOUT the final video sink element. The video sink is the QtGStreamer element which will be added automatically by the plugin. The GStreamer pipe should therefore be the same, except remove the last element for the video sink.
Here is an example string I used on my setup:
udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264
If all is well, you can connect to your UAV and see the HUD elements moving. To change the HUD, right click on the display and select which elements you want to display. The default is to display everything shown here.
If anybody has problems, please post back and I'll update the blog in case I missed something, and you cannot get it to work.
Happy Flying!
Comments
@Kevin, That's interesting since I have never had a problem using UDP for the video stream. Sure seems like there is a network issue if TCP works and the only thing you change is the protocol. Did you ever use a packet analyzer to see if the UDP was reaching the destination host/port ?
The situation that @Chris A is describing is essentially the same problem I was describing and still having. No errors or feedback of issues, just no video. I'm not using an i7 but it is a windows machine as the client. I kind of gave up and went back to TCP. TCP works ok for me since I increased the range using a UB Wisp station and an Engenius base station.
@Chris A. Are you trying to stream to the HUD or the native GStreamer window? If you get GStreamer to work without the HUD first, that will verify that you have a working setup with your network. You don't need to disable the firewall, just open the port you are using for UDP or TCP. For the GStreamer pipeline without the HUD you need to include a video sink element like this:
udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! d3dvideosink
Once you have that working, you can try the HUD with this pipeline:
udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264
If you use the stand-alone version of the HUD, it includes the GStreamer dependencies so it should work no matter what version you have installed locally. This is a change from the original release. If you have the Mission Planner plug-in version, then you still need the 32 bit version installed locally. Your codec pack should not have an effect. It sounds like you either have a network issue, or something wrong with the pipeline.
Patrick,
Really impressed with all the work you have done! What a great contribution to the hobby! Thank you!
Now on to my problems: I have set up gstreamer to stream from a networked Raspberry Pi 2 with Pi Cam. I am using an i7 machine on the receiving end with gstreamer 1.0 (non-64bit). I copy pasted your code and both the receiving side and transmitting side seem to initiate a pipeline without error. Unfortunately, i am getting no video, i.e. no window pops up to display the video feed. I do recall having installed gstreamer-SDK and the 64 bit version before prior to installing the non-64bit 1.4.5 version. I successfully uninstalled the incorrect versions prior to installing the correct version... but still no window with video in it. Could my k-lite codec pack be playing a role? Why won't the feed show on the receiving end? Thanks.
Also does the router firewall AND windows firewall have to be turned off for this to work?
@Steve, it looks like you have some conflicting software on your system. It should work without the 'autovideosink'. You might try downloading the latest version from the dropbox, uninstall the old HUD and re-install. If that does not work, check your path to make sure that other versions of GStreamer are not in your path. If you have the 64 bit version of GStreamer it will conflict.
@patrick
I did a little work to get the qml hud to work..I stripped out a couple things to get it to work so right now all it has is the play,pause,stop buttons..no qml hud right now. The issue is that when I hit play a separate video opens up and is not attached to the quick videosurface. Here is output from QT creator
QUrl( "file:PrimaryFlightDisplayWithVideoQML.qml" )
QML Status: 0
Failed to create qtquick2videosink. Make sure it is installed correctly
GStreamer Pipeline String = ""
GStreamer Pipeline String = "udpsrc port=5200 ! application/x-rtp, encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink"
The Pipeline command string has no unlinked video source element, cannot link pipeline. String = "udpsrc port=5200 ! application/x-rtp, encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink"
I got rid of autovideosink but then it has run-time error and doesn't load widget.
I don't get the qtqtuickvideosink ...I have qt-gstinstalled and all the examples run including qml-player. any ideas of what to try? The pipeline I'm using is :
gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240 ! jpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=5200 for sending (using my laptop webcam-just for testing)
@Patrick Great idea but still no UDP love with those strings with a wired connection. This is what loads on the client.
$ gst-launch-1.0 udpsrc port=5000 buffer-size=60000 ! application/x-rtp,encodin
g-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! d3dvid
eosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
I need to fire up wireshark and watch the traffic...but that is for another night. :)
@Kevin, the RPI should not block any traffic outbound, but I would suggest that you try a wired Ethernet connection between your GCS and the PI, just to eliminate any wireless issues. Once you have that working, you can add the wireless link.
@Patrick I turned off the Windows Firewall completely. My GCS is talking to the UAV via an ad hoc wireless network. I will investigate this to see if UDP packets are moving across or getting blocked with Wireshark. I *DO* have the the GCS attached to my home network at the same time with the built in WIFI card so maybe that is somehow interfering with the UDP traffic. I also haven't looked to see if I need to allow UDP outbound on the RPi?
@Kevin, there shouldn't be any reason why TCP works and UDP does not. Have you verified that you can send UDP packets on your network? Your GCS could be blocking UDP on your firewall.
-
18
-
19
-
20
-
21
-
22
of 39 Next