Synthetic camera offers FPV purely as a software solution

This is my second post in the series called "FlightZoomer 3 Innovation Previews". The first was this one: Maximized terrain awareness for drone pilots

The innovation highlight presented in the video above is the synthetic camera, that offers an FPV experience without many of the disadvantages of a classic FPV approach. In the video 10 advantages of the synthetic camera are listed and demonstrated:

  • No video data transmission needed -> less hardware, less complexity
  • Truly worldwide range if there is nothing more than 2.5G cellular connectivity
  • Highly improved reliability
  • Cut the Gordian knot of zero cost Full HD FPV
  • Principally no yello, no shaking
  • No fog, no clouds, no twilight, never night
  • Adjust the viewing angle during the flight
  • Adjust the camera tilt and yaw angle during the flight
  • Manual snap back view direction control (MSBVDC :-))
  • Integrated optional zero lag gimbal

Of course there are limitations too:

  • Close to the ground the synthetic view lacks accuracy.
  • 3D cities are limited, 100% world wide coverage is only offered as aerial image laid upon an 3D elevation model (which offers still remarkably stunning views, especially when cruising a bit higher)
  • The scenery stays static. You cant observe non-static objects, like traffic, cars,...

The solution is implemented using the MapControl of the UWP programming stack. It offers a comprehensive API to place a camera over the 3D landscape at any place and specify all the parameter, that we need (including pitch and bank angle). B.t.w. this is also the reason, why this solution is not easily portable from Windows to Android or IOS (in fact not doable at all): the Android or IOS map API do not offer the needed capabilities for this feature. Google does not support to place a camera at all at a particular position and altitude and Apple does not support the combination of setting pitch & bank while using 3D maps with an aerial image.

An important detail is the possibility, to additionally feed the video from a real FPV camera into the app. The real camera view can be placed on top of everything else as a small, moveable and resizable overlay. This feature perfectly complements the overall package, FlightZoomer offers.

Views: 903

Comment by Guy McCaldin on March 23, 2017 at 8:52pm

It's an interesting concept that will definitely find a few applications.  I have some constructive feedback based on my personal experience and understanding.

For hobby FPV, the whole point of flying a drone instead of using a simulation is the connection to the real world.  Close ground accuracy, dynamic scenery, weather, and aesthetics play a vital role.  As a result, I think developing it for this market is a dead end.

For professional surveying, consider it from an actionable information perspective.  What problem does a synthetic camera solve?  The main reason for an FPV camera in my experience is either to acquire information on specific features that can't be gleaned from existing satellite imagery, or dynamic changes to the survey area.  This information is used to adjust the flight plan.  A proper 3D terrain model is definitely valuable during the initial planning stage, but the current regulations in most countries require VLOS operations, so there isn't any additional information provided by a synthetic camera system during operation.  

Projecting sensor information (LWIR, multi/hyperspectral) onto the map in real time would be valuable, and a neat way to visualise the information.  It's something Procerus Technologies (Now owned by Lockheed Martin) have already accomplished with their OnPoint Vision system, but its terrain model wasn't as sophisticated, and it was heavily priced and subject to export controls.  This would be one application you could explore.

Another application I think is worth focusing on is as a GCS component for professional BVLOS operations.  It would be a small component of the overall information system required for proper situational awareness, as terrain avoidance is generally managed during the initial planning stage.  However, it would be extremely useful for failure events where you have to ditch.

Comment by Martin Rüedi on March 24, 2017 at 12:31am

Hi Guy

This is an accurate assessment about the use cases!

For Hobby FPV it could still provide some benefit as supplementary or backup system in case of outages, bad weather. Another cool feature would be setting a "click-to-fly" target on the 3D map. Other than a camera feed, the map based presentation offers sophisticated pixel-to-longitude/latitude/elevation (and vice versa) conversion methods. Exploiting this capability I can see a lot of nice ideas, that could be realized even beyond "click-to-fly".

For surveying I fully agree with what you write.

The look&feel goes a bit in the direction of the direction of the Kongsberg Iris UAS, some similar applications could be possible.

The projecting feature would be very easy doable using the custom tile source feature of the map API. In fact I am using this feature to calculate the terrain elevation map dynamically which is shown on the Navigation Display.

Comment by Hamish on March 24, 2017 at 12:52am
Wow Martin, that's a great idea! As you just said above, you could fly a real drone with just telemetry and no video feed, that's pretty innovative. It would greatly improve "return to home" operations.
Comment by gustavo on March 24, 2017 at 9:23am

your development is fantastic ! , congratulations and my respect to your skills .

i would definitely try the system on my Pixhawk drone , 

i  would need to buy a phone, is any Window phone that would recommend , does it has to be windows mobile 8?

just windows phone ?

may i use the mavlink radio telemetry at the same time using other port for BT mavlink to the phone?

instead of using BT mavlink, can i use my Wifi telemetry to the  phone ( im using it with an esp8266 + esplink )


Comment by Martin Rüedi on March 25, 2017 at 1:57am

Hi Gustavo

Currently the software supports this: BT MAVLink, Arducopter, Windows phone 8 or higher for the Sensor device.

Supporting Arduplane or other flavors of the telemetry connectivity would still require some extension in software and testing (= would not be ready immediately). Arduplane requires slightly different solution where Arducopter uses the GUIDED mode.

Publically available is version 2.1 (smartphone based groundstation) but if you are interested, I could offer you a beta version of the version 3 cockpit.

Comment by gustavo on March 25, 2017 at 5:11am

thank you Martin, i would have to find a pair of W8 phones first , will make contact as soon as have it covered ,.

Comment by gustavo on March 25, 2017 at 5:22am

could you please tell me if this phone would be ok?

Comment by Martin Rüedi on March 25, 2017 at 7:09am

Hi Gustavo

The phone is upgradeable to Windows Phone 8.1 so everything should work perfectly. I fact I never had an onboard device for more than about 40$. Any second hand WP8.1 device with Bluetooth should be fine.

Be careful about the hardware you need! You don't need a pair of phones! The groundstation device was only a phone up to FlightZoomer version 2.1. The version 3, which has the synthetic outside view and is announced above, is designed for the tablet form factor (or notebook, in fact any Windows 10 device, except phone). So what you need is one phone as onboard device, a second phone as tethering hub (can be just any smartphone out there, including Android, IOS devices) that is used by the Windows 10 groundstation tablet or notebook device to provide internet connection.

The whole communication chain is this:

Pixhawk <-> BT HC-06 <-> Onboard WP81 smartphone <-internet-> Relay Server (Windows PC at home) <- internet -> Any smartphone as tethering hub <-WiFi-> Windows 10 Tablet or PC as groundstation.

You might want to dig into the installation documentation on my homepage

Comment by gustavo on March 26, 2017 at 6:00am

very clear thanks

Comment by r3d_f1v3 on March 30, 2017 at 11:19am

Hi Martin. I think your software would be a very good backup and security system for hobby FPV flights, in case of fog, darkness, or video loss. The thing is, I don't fly multirrotors, so It would be very interesting if there was a version compatible with APM:Plane. I know a lot of other FPV plane-flying friends out there would also be interested in buying a plane version, so I think you have a huge market there...

One question: Could this system work without a mobile phone, using only a 3DR compatible SiK Radio link?



You need to be a member of DIY Drones to add comments!

Join DIY Drones


Season Two of the Trust Time Trial (T3) Contest 
A list of all T3 contests is here. The current round, the Vertical Horizontal one, is here

© 2020   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service