This blog is a continuation of my previous post.
How to build a High-Definition FPV UAV using a Rasperry PI with HD camera, using a high speed WiFi link
This post will discuss how to use GStreamer and Mission Planner together to display the HD video with a HUD (Head-Up-Display).
Note: I have only tested this feature on Windows so the instructions given here are for Windows only.
To give proper credit, the HUD created here was borrowed from APM Planner, a Qt-Based app similar to Mission Planner. The HUD part was created from the Qt codebase QML HUD created by Bill Bonney who is on the APM Planner development team. To make the HUD work with the background video, I used a GStreamer library called "QtGStreamer" which integrates GStreamer plugins with painting on a Qt widget. This library is available on the GStreamer website.
The end-result is dynamically added to Mission Planner using the plug-in architecture.
In the previous posts I discussed used a Raspberry PI and a High-speed WiFi link using GStreamer on the PI and the ground station PC. To get the HUD to work, you need to already have a successful link with the video on your ground station.
Here are the steps to follow to install the plugin:
1) Install Mission Planner.
2) Download and install GStreamer from this link. Use the x86 version, the x86_64 version will NOT work. (Use the default path 'C:\GStreamer' when installing). When installing GStreamer, select 'Custom' install and select ALL plugins to be installed.
3) Follow the steps in the previous blog noted above to get your video stream working.
4) Download and the MSI installer from this link. and run the installer.
If all went well, you should have the plugin installed.
Open Mission Planner and navigate to the "Flight Data" page and right-click on the map. You should see a menu item called "GStreamer HUD" as shown below:
Select this menu item and the following screen should appear:
In the upper-left corner is a context menu. Here is where you enter your GStreamer Pipeline string. If you had the video displaying without the HUD using a valid pipeline, enter it here.
Note: The GStreamer Pipeline string should be exactly the same as the string you used before, but WITHOUT the final video sink element. The video sink is the QtGStreamer element which will be added automatically by the plugin. The GStreamer pipe should therefore be the same, except remove the last element for the video sink.
Here is an example string I used on my setup:
udpsrc port=9000 buffer-size=60000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264
If all is well, you can connect to your UAV and see the HUD elements moving. To change the HUD, right click on the display and select which elements you want to display. The default is to display everything shown here.
If anybody has problems, please post back and I'll update the blog in case I missed something, and you cannot get it to work.
Happy Flying!
Comments
Here are two simplified ways to solve the missing QtGStreamer parseLaunch() function:
from m_pipeline = QGst::ElementFactory::parseLaunch(pData).dynamicCast<QGst::Pipeline>();
to m_pipeline = QGst::Parse::launch(pData).dynamicCast<QGst::Pipeline>();
and #include <QGst/Parse>.
Either method works but I don’t know if the original parseLaunch() code is different in some way. Method #2 is easiest because it doesn't require recompiling QtGStreamer. I wasn't aware of the existence of QGst::Parse::launch() until after my previous comment.
I have qt-gstreamer source from freedesktop.org which is the same as github.com/GStreamer/qt-gstreamer by George Kiagiadakis which is no longer maintained. Diane Trout has been maintaining github.com/detrout/qt-gstreamer which is used to send patches to freedesktop. I built both from George and Diane with no change in symptoms. Neither contain the parseLaunch() function and there is no evidence it was removed at some point. If you can take the time to update your repository, I'd like to give it a try. Even your old qt-gstreamer code might be better suited than the most recent that I found. Your precompiled QtGStreamerHUDApp with your qt-gstreamer binaries work fine for me. Your Qt5GStreamer-1.0.dll contains parseLaunch() but mine doesn't. The qt-gstreamer examples I built work fine with the qt-gstreamer I built. Has anyone else been able to build QtGStreamerHUDApp?
I'm working with a Raspberry Pi camera (which is not on a drone yet) with a zoom lens. Just for development, I had been using a gstreamer window on the side and then I saw that Dan Murray had been doing the same thing until he found your project. I'd be happy to share my lessons learned or contribute to a project when I put together something meaningful. I read the discussion about a community project for this but don't see that it has begun.
@Jeff, Have you built the code from the Freedesktop.org source? I don't think you should need a custom version now, as I think the bugs were fixed. The changes I made were hacks. I think I already updated my code to the latest version. I was not keeping the custom source for qtgstreamer in github. I can update my repository as I have not checked in any code recently so maybe that would help you.
What exactly are you working on? Maybe it would be of use to have you contribute your new features to the repository, or possibly I could help you make the changes you need.
You mentioned that you included your custom version of QtGstreamer in your source tree but I haven’t been able to find it after searching projects here and dropbox links mentioned in this thread. I’m hoping that might be key to solving my problems. Where can I find your modified QtGstreamer code?
Since my last comment, I found implementations for plugin_init() and gst_qt_quick2_video_sink_update_node() in qt-gstreamer but I haven’t yet figured out why they are not found upon compiling main.cpp.
I’m building on Windows 10 with QtCreator using all 32-bit components. I used QtGStreamerHUDAppVR from github because it said "QtGStreamerHUDApp_VR_Win32" and appeared to be the most recent (Aug 17, 2017). It has three variants of .pro files in it, one of them for XP and all with win32 scope. What code should be used to build QtGStreamerHUDApp for Windows? I have gstreamer source but it doesn’t contain the missing routines. I built QtGstreamer debug files and have started stepping through QtGStreamerHUDApp while debugging.
plugin_init(): I found code similar to the following in mavlink/qgroundcontrol/src/VideoStreaming/gstqtvideosink/gstqtvideosinkplugin.cpp:
static gboolean plugin_init (GstPlugin *plugin)
{
return gst_element_register (plugin, "qtquick2videosink", GST_RANK_NONE, GST_TYPE_QT_QUICK2_VIDEO_SINK);
}
gst_qt_quick2_video_sink_update_node(): I found code similar to the following in mavlink/qgroundcontrol/src/VideoStreaming/gstqtvideosink/gstqtquick2videosink.cpp:
static gpointer gst_qt_quick2_video_sink_update_node(GstQtQuick2VideoSink *self, gpointer node, qreal x, qreal y, qreal w, qreal h)
{
return self->priv->delegate->updateNode(static_cast<QSGNode*>(node), QRectF(x, y, w, h));
}
I’m still trying to adapt the code above for QtGStreamerHUDApp.
parseLaunch(): This is referenced in QtGStreamerHUDAppVR/GStreamerPlayer.cpp:
m_pipeline = QGst::ElementFactory::parseLaunch(pData).dynamicCast<QGst::Pipeline>();
This implies the function should be in qt-gstreamer/src/QGst/elementfactory.cpp so I was guessing it was in your custom version of QtGstreamer.
@Jeff Ward, Those references are to gstreamer, and qtgstreamer libraries. You will need the development version of gstreamer and the source code for qtgstreamer (on freedesktop.org) If you downloaded the VR version, that's for Android, not windows. What platform are you trying to build on? It's pretty complex to build for Android, you need the full SDK, and the Arm/ArmV7 toolchain.
I need to integrate some of the functionality of QtGStreamerHUDApp with other code. The main portions I need are gstreamer + HUD (instantiation primarily) and some basic supporting features. QtGStreamerHUDApp probably already contains the best practices for these so I'm trying to avoid reinventing the wheel. I wanted to figure out what I needed by stepping through key pieces with a debugger rather than by visually reviewing code. Right now, I'm stuck without definitions for plugin_init(), gst_qt_quick2_video_sink_update_node(), and parseLaunch() but I'm not sure how essential they are or how to get around not having them.
@Jeff Ward, Let me know what kind of customization you are looking for and I can possibly help you out. Building from source could be a pain.
What code should be used to build QtGStreamerHUDApp for Windows? I installed the precompiled binaries from GStreamerHUD.msi and ran QtGStreamerHUDApp.exe (built Jul 10,2016) which works. I need to customize it so I tried building the project defined by QtGStreamerHUDApp_5.4.1.pro (slightly modified for my paths) but encountered unresolved external symbols for plugin_init() and gst_qt_quick2_video_sink_update_node() in main.cpp so I commented references to them to see how far I could get. I may be using main.cpp meant for Android but found no alternative. I used QtGStreamerHUDAppVR from github because it said "QtGStreamerHUDApp_VR_Win32" and appeared to be the most recent (Aug 17, 2017). I also had to comment QGst::ElementFactory::parseLaunch() in GStreamerPlayer.cpp because of "no member named 'parseLaunch' in 'QGst::ElementFactory'". I built QtGstreamer but had no parseLaunch(). In this case, I remembered you said you’d modified QtGstreamer to create a custom version of it in 2015 but I haven’t found the code. Upon execution, I unsurprisingly received the message "Failed to create qtquick2videosink. Make sure it is installed correctly." which may be another issue.
Thanks for all the work you've done on this project and for sharing it.
-
1
-
2
-
3
-
4
-
5
of 39 Next