I decided to open a new thread to register some progress about this project. I am working on it in my spare time, and I feel excited about it.
This is a very preliminar video -- please disregard the pink view, I was using an infrared camera -- with a normal camera the video is way more clear and interesting. I also need to find better 180º (or better yet, 230º) lens. I am also willing to go full 360 degrees eventually.
So, it is an experimental Virtual Reality controlled drone with some AR features which I will show when I finish implementing them. The plan is to have one or more people walking and controlling the drone as if they were truly inside the physical aircraft. Much like the traditional FPV, but not limited to the video view only, as the crew will be able to interact and walk around freely while they fly.
I have lots of ideas to implement on this experiment. I plan to talk about them while they are implemented.
At first I wanted to show it only after I had a real flight video. But for a long time I had no opportunity to bring all the equipment to a secure place for a flight. With the pandemy, I can't go to a safe place where I feel secure with my Oculus Quest and all the related hardware with me. Hopefully the next video will be more interesting, as it will show a real flight with a non-infrared camera, plus all interactions from arming the drone to taking-off, flying around and landing back, while showing some AR overlays.
Please note that this is NOT a fake video, I can already control this quadcopter from start to end using an Oculus Quest. I was using my home, conventional 2.4ghz Wifi when I recorded the video, but I could also use Raw Wifi (much like Befinitiv's WifiBroadcast, although I do have my own implementation from scratch), or 4G. It is always low latency, using a simple UDP protocol made by me.
The flying hardware has a Raspberry Pi commanding a Pixhawk through serial Mavlink, while exchanging information with the VR headset through a ground-based Rpi (this ground Pi has a 1w sunhaus and a Yagi antenna for extended (Raw)Wifi range). The shown 3D part on Oculus Quest was done using Unity3D for now -- not sure if I will keep that path, though. Unity is a nice engine and quick to develop on, but it has its drawbacks when used on custom projects like this. I do have my own 3D engine and a second implementation of the same environment (except for the nice hands), which might take over in the future. My low-level engine is very lightweight and can also run on Webasm if desired. It does not have an editor yet, though, and is harder to use, so there is an indecision in that regard. =)
This is planned to be free and open-sourced, when it reaches a certain maturity level.