FlytSIM is a 3D simulator that helps in the development and testing of commercial drone applications using the FlytBase platform. FlytSIM is now available on cloud and can be accessed through your web browser.
Development of commercial drone applications involves integrating payloads, coding business specific logic and building custom user interfaces as web/mobile apps. To address this, FlytBase offers a platform with FlytPOD (advanced flight computer), that runs FlytOS (operating system for drones), and a set of developer tools, including FlytSIM (3D simulator).
FlytSIM offers a SITL simulation environment for testing user apps without the drone hardware. The drones and a virtual world is simulated using a ROS-Gazebo based setup. The autopilot in simulation uses the same control algorithms as on the FlytPOD making the behaviour closer to real flight.
FlytSIM has now taken to the cloud with Beta release of FlytSIM (cloud). It is a web-app which provides a 3D software simulation environment for drones, right from the comfort of your web browser. Applications developed using FlytAPIs (RESTful and websocket) can be tested in FlytSIM (cloud), before deploying on real drones. First login to the flytsim cloud portal. Then just direct the API calls in your app to FlytSIM (flytsim.flytbase.com) with your username as namespace. This will allow you to interface with the drone in FlytSIM from your app.
Taking FlytSIM to cloud had several motivations, including ease of access without having to install a virtual machine or a native Linux, ROS based setup. Secondly, web-app allows cross-platform support, allowing developers to develop apps on their platform of choice. Also, introducing updates and new features becomes seamless. This offers developers a pleasant drone application development experience, while keeping it safe, and saving them a huge amount of time and effort.
We are accepting signups for a limited time. Subscribe to this programhere, and we would revert to you with your subscription details.
When you hook up a new payload to your drone and you want to configure or tune the parameters, how about just opening a web browser and getting an interface to do exactly that. You can get that with the FlytBase platform and all you need to do is write a quick code in python and HTML/JS for your custom web app. This is only a part of the story.
FlytPOD has an onboard WiFi router and FlytOS includes a web server exposing FlytAPIs as RESTful and websocket calls. This enables remote devices to easily interact with the drone. And, you can make this experience even richer by building your own onboard web apps and defining custom RESTful endpoints.
FlytAPIs cover most of the core functionality required for any drone application and we keep striving to make it richer. However, as a drone developer, you might need a very specific feature for your application and FlytOS gives you the flexibility to implement it through onboard web apps.
Why is this helpful?
If you are a web developer or you know how to write simple web apps then, with FlytBase, you can also write apps for drones.
You can have a custom web app served directly from the drone. This gives you an interface that is cross-platform and is accessible from any ground device without needing a separate server or a native app
You can integrate the web app with the onboard driver for a new payload and get data or send commands to the payload by accessing the app in your mobile/laptop
You can build a fully featured app for your domain e.g. Agriculture or build a custom GCS or a quick Client demo or build simple utilities to expose new features through your app routes e.g. get onboard CPU % utilization, get console output (status messages), assign/get custom drone ID, transform position data to a custom coordinate frame and so forth.
FlytOS uses lightweight Flask microframework for onboard web apps. Custom user apps can be added as blueprints using the existing samples and documentation. You can write a complete web app with server side code in Python and Static content with HTML, CSS, JS.
FlytVision App
FlytConsole App
Currently, FlytConsole and FlytVision are available as onboard web apps which provide good demonstration of the use case and also serve as reference for building new apps. We also use the onboard web app option for quickly building demos e.g. April tags, Joystick etc
Cameras have become an integral part of autonomous drones, helping them navigate as well as providing some application specific insights like an object of interest. Detecting a desired object in the camera view allows the drone to take decisions like following the object or orbiting around it. Such capabilities are useful in several applications from photo/video shoots to surveys and search and rescue missions.
The previous article (Part 1) had glimpse of onboard object tracking module in FlytOS and this article will delve into details on the same. It uses relatively simple OpenCV based algorithms to detect and track an object in the field of view using attributes like color and shape for detection along with Kalman Filter for tracking. It also has support for ROS based OpenTLD library which needs to be separately installed. Along with tracking, there are APIs to make the drone follow the object being tracked. It uses a PD controller and currently assumes a downward looking camera.
FlytVision is an onboard sample web app which provides an interface for ground devices to access onboard video streams and is now extended to include object tracking and following. It also demonstrates how seamlessly onboard image processing fits in the overall framework and allows for data plumbing with ground devices.
First step is to stream the processed images from the object tracking module. Then select the Detect/Track mode in the app. Currently available modes:
Color: Uses HSV color, heuristics like change in distance and area of the object and Kalman Filter for tracking
Circle: Uses HSV color, Hough circle for circle detection and Kalman Filter for tracking
TLD: Uses ros_open_tld_3d library modified for integration with object tracking module
The object of interest can be selected on the video stream itself. Depending on the selected mode, corresponding attributes are detected and tracked in subsequent images. To follow the object, its distance from the image center is projected to ground and position setpoints are generated with a PD controller. The overall workflow:
The onboard modules and web app are first tested in simulation using FlytSim. While the parameters may need to be tuned again for real drone, simulation helps in validating the algorithm and overall functionality before the drone even takes off.
Several params have been exposed from the onboard object tracking module so that they can be tuned from the ground app for a given setup. These include HSV color ranges, Hough circle params, TLD params, controller gains and options to turn attitude compensation, tracking and follow modes On and Off.
Besides params, custom data sharing is required for indicating the region of interest as selected by a user in the video stream. This is achieved by publishing a new topic in the app and subscribing the same in the onboard object tracking module. Whenever a user selects a region by drawing a rectangle in the video stream, the corresponding coordinates are published in this topic.
The Inspect section in the app shows object centroid position, drone’s position and the setpoints being sent. These data streams are obtained by subscribing to them using FlytOS websocket APIs. The object tracking features can be accessed in your own custom app using the object tracking FlytAPIs.
In the demo video above we used a SJCam-4000 camera plugged into a FlytPOD and flew it on a hex-550 frame. The images are captured at 30 fps with 320 x 240 resolution. The onboard object tracking module ran at ~25 fps, had approx 75 % CPU core utilization and used color mode. The camera is rigidly attached to the frame without a gimbal. This requires the onboard attitude compensation for setpoint correction though the onboard video feed is still a bit shaky and smoother operation is possible with a gimbal.
We are exploring more capabilities to add like April tags recognition, gimbal control to keep object in the image center combined with follow and possibility of accelerated vision processing.
As drones continue to move from basic flight to commercial applications, the ability for onboard and offboard customizations becomes more and more important. A given application would likely introduce new payloads and require custom onboard business logic. This also leads to customizations in remote applications allowing for custom data plumbing between ground app and drone as well as targeted user interfaces for bettering the overall user experience. FlytOS provides a scheme for web/mobile applications that not only allows such customizations but also simplifies them by using standard tools.
FlytOS provides an onboard web server and exposes RESTful and websocket APIs. With this, there are following approaches for building web and mobile apps:
WEB:
Onboard Web App: FlytOS framework provides blueprint to write onboard web apps with server side code in python. Such an app is directly served from the drone and can be accessed on any ground device in a browser.
Offboard Web App: This requires an offboard web server which hosts and serves the web app and can be coded in the language of choice. This app internally makes REST and websocket calls to talk to the drone.
MOBILE:
Cordova/Phonegap: Frameworks like Cordova or Phonegap provide a way to build cross platform mobile apps using web technologies (HTML/CSS/JS). The code can be written and tested in browser with mobile oriented UI design and then built for a specific mobile platform e.g. Android.
Native App: This is conventional native mobile app development. The idea is to use FlytOS supported SDK/libraries and develop native Android or IOS apps.
The video above shows a sample joystick app for controlling the drone. It is an android app built with HTML/CSS/JS using the Cordova engine. The idea behind this app was to provide a simple interface that can enable anyone to fly a drone without prior flying experience. This app is built by our mobile app developer who also successfully tested it on a real drone while our pilots just stood by.
The app mainly has a set of buttons (HTML/CSS) that are mapped to REST calls (JS) at the backend and thereby allow the control of drone’s position and heading. We can also get data streams from drone to the app by subscribing to desired topics using the websocket APIs of FlytOS. This is a relatively simple app with only an offboard component but advanced applications can have both onboard and offboard components. In such a case, new data could be generated on either side which needs to be shared with the other component. This is possible in FlytOS by publishing and subscribing to new topics or by using new params.
The step by step procedure for building an offboard web app and a cordova based mobile app is documented and available in flytdocs. Onboard web apps documentation is currently in progress and will soon be updated in flytdocs. FlytConsole and FlytVision are inbuilt onboard web apps. The setup for native Android apps is currently under development and a how-to guide with a sample app will be put up after internal testing.
The code for the sample apps is available in github. This currently includes the above joystick app, a simple demo app with an onboard script, a sample shape tracing app and a GPS based Follow Me app. Further apps to follow soon include a joystick app to control multiple drones and a photo shots app for capturing shots like Orbit and Cable Cam. Please visit links below to learn more and get started with imagining and building your own web/mobile interfaces.
It has been a typical scenario for drones to have an onboard camera and a key requirement to transmit live camera feed to ground devices. As a next step we also want the drones to acquire the images into the onboard processing framework and apply some computer vision algorithms or even simple image processing to enhance the raw image. And then we would like the processed image stream to be sent to ground devices as well. Once we have the video stream in our laptop or mobile, why not interact with it and send the inputs back to the onboard computer? Welcome to FlytOS!
FlytOS is ROS based framework and ROS enables a cleaner modularized way to process and share data. FlytOS image pipeline is as follows:
The basic idea is to get all image data in the ROS pub-sub bus. Then an image processing module can subscribe to raw images and publish back processed images. A web video server can subscribe to any of the available image topics and stream it to ground devices on web as needed.
The images are captured from USB camera attached to FlytPOD or ODROID and are published on ROS using a cam driver. The ROS based web video server subscribes to an image topic and provides mjpeg stream over http. This stream can then be viewed in most modern browsers without any plugin or client software. FlytOS provides options through its web APIs to list available stream topics and start/stop a particular stream. You can also get a snapshot on demand. Further, you can interact with the image stream e.g. select a region of interest and send that information back to the onboard computer. We have tested this setup and it works well as can be seen in the demo video above.
However, there are several constraints when we think of drone video streaming. We should be able to
Capture the images from camera into the onboard processing framework like FlytOS for image processing (as against having camera as payload and transmitting images directly to ground)
Transmit raw as well as processed images to ground
Get a low latency/lag real time stream as stale data may not be helpful or sometimes even harmful
Have relatively low power/processing requirement to be suitable for onboard computer
Have relatively low bandwidth requirement
Get good quality image/video
Have adaptive streaming based on network bandwidth
The current setup in FlytOS serves the first two items well and reasonably tackles the next three points. It is not yet focused on HD quality and does not have adaptive streaming. Additionally, we wanted to stream the video to a web browser without the need for a client software or plugin for easy integration with web apps and cross platform accessibility. A good alternative web oriented implementation with real time focus could be WebRTC however it does not really support ARM Linux yet.
In any case, having multiple parallel streams being served directly from the drone to several ground devices is probably not an ideal scenario as it would put strain on the onboard computer and its network link. Alternate setup is to stream the images from the drone to a single computer on ground. This ground device would be a distribution server (preferable in cloud) that can then stream parallely to multiple client devices and can also be a WebRTC setup. FlytOS will continue to improve upon its solution and such a streaming architecture could be the next step.
Meanwhile, you can get started with FlytOS right away, test video streaming, try out sample apps and we would like you to give us your feedback and discuss your views on the next steps.
Note: The demo video above shows sample of Object Detection and Tracking. This is accomplished using an Object Tracking module in FlytOS. We would talk more about what is available and how it works in the next article. Stay tuned!.
