Last week, the Advanced Robotic Systems Engineering Laboratory (ARSENL) team at the Naval Postgraduate School, based in Monterey, California, successfully launched and flew 50 aircraft simultaneously.
It was nearing 110 degrees Fahrenheit on the runway at Camp Roberts, California as we carted the last rack of planes from the flight technicians’ tables over to the launch site. We had started flight preparations around 8:30 that morning, after arriving an hour earlier to set up shade structures, tables, and generators, and unpack some 68 planes we would prepare for flight that day. Knowing that the planes’ limited battery life would restrict our launch window to less than half an hour, and that we’d previously encountered last-minute sensor calibration problems and WiFi devices that went brain-dead in the heat and sun, we conservatively assumed that we would prepare four planes to fly three. Our goal this time was to fly 50.
The planes we fly are built from the Ritewing Zephyr II kit. These are EPO foam wings with elevons and room for an electric pusher motor. The plane is outfitted with Hitec digital servos, an O.S. motor driven by a Castle Creations ESC, and dual Thunder Power 5000 mAh LiPo batteries. In this configuration, our expected flight time is approximately 45-50 minutes, meaning that we needed to have all aircraft launched within about 30 minutes to have time to perform cooperative flight and to land safely before battery failsafes started to trigger on the autopilot, forcing an autonomous landing.
A custom PCB distributes power to and manages some of the wiring for the Pixhawk, which flies a lightly-modified branch of ArduPlane. To the right of the Pixhawk is an ODroid u3 companion computer (“payload” in our team’s parlance) running a stripped-down version of Ubuntu Linux. Our payload software is built on top of ROS, making it easy to add new functionality incrementally. While the Pixhawk manages keeping the plane airborne, the payload ingests autopilot pose and status, which it broadcasts to other planes and to the ground. It also accepts commands from the ground, which are either passed to the autopilot or used to control execution of a handful of payload-directed path planning algorithms, such as leader-follower, distributed search, and sequenced landing. These algorithms “drive” the plane by sending updated target latitude-longitude-altitudes to the autopilot, effectively moving a carrot in front of the plane.
The 900 MHz telemetry radios typically used to communicate with the Pixhawk are only a backup link in our architecture, and a tenuous one with nearly 70 of them powered at once. After observing difficulty establishing telemetry links during previous field tests, we measured signal-to-noise with 50 planes powered on the bench and saw that the default 915-928 MHz band was saturated. Following a suggestion from the DIY Drones community, we decided to split our available 902-928 MHz band into three equal-size sub-bands and place an equal number of planes into each sub-band. We reasoned that, even with the reduction in per-band bandwidth, some 22 to 23 planes (of the 68 prepped for flight) in a sub-band would hold up better than all planes in a single, wider band. This intuition came after observing that at most 50 frequency hopping channels could be supported in a band in the current radio firmware.
In any case, at most one plane would actively be controlled over the telemetry link, and only in an emergency. Our primary link with the fleet is an ad hoc 802.11n wireless network that connects each plane’s payload with other planes and our ground control stations. All of our command and status traffic is sent via UDP; as with MAVLink, our messages are designed to minimize statefulness and be tolerant to loss and blind retransmission.
Should a plane’s payload have failed or the network conditions become too lossy, one of the ground operators’ software had a button that would reconfigure an attached telemetry radio to the select plane’s sub-band and Net ID, and then open a MAVProxy session to that plane within a few seconds.
It took our diligent crew of flight techs close to four hours to prepare all the planes. Preparations comprise not only putting propellors on motors and making sure that all mechanical and electrical connections are secure, but also running through a set of software preflight checks. These checks have evolved over the past two years from a paper checklist - imagine checking the values of 40-odd autopilot parameters by hand - to a MAVProxy module, and finally to a standalone interface that works in concert with the payload and a central configuration server to perform the preflight process.
When a plane is first powered, the payload reaches out to the server and pulls down its mission configuration: autopilot parameters, waypoints, rally points, and geo-fence. The payload verifies that the autopilot matches these files and then sets status flags in a message broadcast back to the ground stations.
Meanwhile, the flight tech is busily performing sensor and actuator checks. Our Flight Tech Interface (FTI) software reports similar status lights to those in MAVProxy’s console module. A red “as” light indicates that the airspeed sensor must be calibrated; the flight tech places a bottle over the pitot tube and presses the “Cal Pressure” button in the FTI, and a few seconds later the light changes to green indicating completion. Then the servos and motor are checked. For this, we designed simple test stands that are staked into the ground. The plane is secured in the test stand, and then the flight tech once again sends a command from the FTI to the payload. First, they check the servos: the elevons execute a left roll, right roll, up pitch, down pitch, and return to level. Then, the technician arms the throttle and commands a run-up of the motor, which lasts about two seconds. Finally, a second tech picks the plane up and rotates it manually while the first tech verifies on the screen that the autopilot correctly reports roll and pitch.
Once done with the checklist, the flight tech clicks one last button, telling the plane to set a “flight ready” status bit so the plane reports as ready for flight. The plane is then placed in a rack built from two-by-fours and PVC pipe, which is moved 100 feet out to the launch site when full. Each rack carries ten planes.
It was getting close to 1 p.m., and some of the aircraft had been powered for multiple hours. Though the amount of power they consume on the ground is small, it isn’t nothing. It was hot, so we were also worried about electronics starting to fail. From our previous field test experiences, we had begun to suspect that heat was an issue for some onboard components including the wireless network devices, so we were continually edging racks of planes under shade structures to track the sun’s movement.
As the last planes were carted out and a few last-minute recalibrations were performed, we prepared our automated launcher for its big test run. We had previously used a bungee launch system. It was simple and reliable, though required a person to sprint after the bungee each time it was released, and pull some 25 pounds of force back to the aluminum rails from which the aircraft launched. Between the number of planes to launch and the heat, we were glad to see the new launcher ready for action.
The brainchild of two Masters students at the school, the Automated Multi-Plane Propulsion System (A.M.P.P.S.) comprises a chain wrapped around two cogs spaced eight feet apart, set between two parallel rails from which the aircraft launches. On the leading edge of the launcher is an electric motor that draws an impressive 200 amps when spun up at maximum rate. It is capable of accelerating a plane from rest to roughly 16 meters per second before it leaves the rails. Technically, it can do more, though we ran into problems with shearing launch hooks from the airframe.
With a few modifications to the motor controller’s acceleration profile and a redesigned, 3D printed adapter attaching the plane to the launch chain, we were reasonably confident that A.M.P.P.S. would launch all 50 planes successfully. Nonetheless, we kept the old launcher ready with a bungee pre-tensioned just in case of a failure.
Timothy Chung, lead faculty for the project, called over the radio, “All right team ARSENL, this is it. Everyone have fun, stay safe. Mission 36... commence.” This was the thirty-sixth mission of the week. Most of those missions had been single-plane checkout flights; every brand-new and newly-repaired plane goes through a short "mechanical checkout flight" before being reintroduced into the fleet. Our only multiple-plane missions during the week had been two ten-plane flights the day before, as practice runs for the 50-plane test. The largest number of planes we had flown previously was 30 in July; we had been increasing our numbers exponentially since our first multi-plane flight with the Zephyr platform last year.
Time to hustle. We loaded the first plane onto the launcher, and our launch operator used a variant of the FTI to command final configuration parameters. Since any plane might fail to launch, we needed to tell the plane which “slot” in the flight configuration it would fill (more on this later). She then commanded the throttle to arm, and signaled both the safety pilot - who had selected that aircraft on an RC transmitter - and another operator standing next to the launcher, that the aircraft was ready for launch. The second operator pressed two buttons on a gamepad controller wirelessly connected to the launcher, commanding it to launch the plane. Three safety beeps from the launcher, then a loud whirring as the chain suddenly accelerated, kicking the plane off into the air, and that long-seeming moment before the plane detected launch and started the throttle. The plane took off steeply toward its initial waypoint.
As the safety pilot confirmed stable flight, an intern carried the next plane over to the launcher. This time, there was some confusion between operators, and it took about a minute to get the second plane airborne. At a minute per plane, we would be landing the first planes as the final ones were being launched.
Fortunately, the next few planes launched quickly, at an average of just over 30 seconds per launch. The tenth plane aloft experienced some flight instabilities and the pilot had to pause the launch sequence so he could take manual control and land that plane. That cost us an extra minute. After that, the team found a good rhythm for the rest of the launch sequence. After the 50th launch, the launch operator - who had been keeping track of launches on paper - paused and asked, “is that all of them?” We all looked at each other, then quickly remembered that we had to launch a fifty-first plane to make up for the one that was landed earlier. It’s amazing how difficult it is to keep track of so many planes.
In all, we performed 51 launches, all using A.M.P.P.S., in 27 minutes 39 seconds, at an average of 33 seconds between launches. High-fives and handshakes commenced at the launch site, while operations were just ramping up inside a trailer parked 100 feet away.
Our flight configuration consisted of two “stacks” of 25 planes, with each “slot” in the stack separated 15 meters apart vertically. The two stacks were separated laterally so that the orbits and racetracks of one stack would not overlap with the other. Though we launched in altitude order, we designed the waypoint sequence with special “ingress” and “egress” points where planes could ascend and descend without entering the airspace of other planes. This altitude deconfliction meant that we could execute cooperative behaviors within a single stack in two dimensions, without having to worry about implementing collision avoidance due to separation in the third dimension. The launch operator sent the stack and slot configuration to each plane just prior to launch.
Example of a "stack" forming with 12 planes visible.
Each time another five or so planes entered the stack, a single operator sitting in a trailer placed away from the launch site would command them into groups and initiate cooperative behaviors. In fact, there were only two operators in the trailer: one was commanding the fleet as a whole, and the other was monitoring planes’ health.
We executed our leader-follower algorithm during this flight, due to its heavy vetting in previous field tests. This algorithm determines a single leader and commands all other planes in the group to follow it directly, while maintaining the altitudes set by the launch operator. The highest altitude plane in the group becomes the leader and enters a prescribed racetrack, while all other planes listen to its broadcast pose and compute following positions.
Author’s comment: Watching 24 planes stacked almost perfectly on top of one another, following a common lead plane, is pretty amazing. Each plane broadcasts its pose at roughly 8 Hz, but considering wireless contention and the relative orientation of antennas, follower planes may sometimes receive as little as 1-2 updates per second. The payload software extrapolates positions using the current pose and linear velocities only, so the fact that the stack performed so well under lossy network conditions continues to impress me.
Visualization in Gazebo using data captured in ROS on each payload. Aircraft are continuing to join the left stack while the rest of the stack are independently loitering at deconflicted altitudes. Those in the right stack are performing a leader-follower algorithm, with the highest aircraft acting as leader.
We had 9 minutes 45 seconds with all 50 planes airborne, performing cooperative behaviors in various groups. Due to our long preflight and launch sequence, batteries were quickly depleting in the air. We had to start bringing down planes.
With the safety pilot on standby, focusing on individual planes as quickly as possible, we commanded groups within a stack to execute a sequenced landing. Using this algorithm, planes order their landings according to altitude, lowest first, and each plane in sequence begins an autonomous landing. Our landing has each plane perform a descending loiter to 40 meters above the runway, then break into a glide slope until it finally skids to a halt on the asphalt. Each plane bears 3D printed skids that can easily be replaced when they ablate.
Landing 50 planes did not happen not without minor glitches. Because we ran close to the limit of the batteries getting all aircraft aloft and executing cooperative behaviors, a few planes triggered their low-battery failsafe and automatically executed a controlled autonomous landing alongside those performing the sequenced landing. This made for great video, though the safety pilot had to pick one to monitor and one to let land on its own. Scrolling through models on his transmitter took most of his attention, otherwise. A couple planes' ESCs also triggered their own low-voltage failsafe and cut the throttle, causing a controlled descent into a field adjacent to the runway. Thankfully, all but two of these planes had a soft landing, and all batteries were intact after landing - though some were retired due to their low remaining voltages.
Example of sequenced landing showing three aircraft inbound.
Planes able to make it to the runway faced another challenge. All planes were programmed with the same landing point, with the assumption that GPS and barometer inaccuracy would provide sufficient spread on landing. But with nearly 50 planes on the runway, on-deck collisions were unavoidable. Some of the video captured by GoPro cameras mounted to the nose of each plane show skidding into other planes, or coming down directly on top of other planes on the ground. Lesson learned for next time: add a bit more variation to the landing coordinates.
On August 27, 2015, at roughly 1:20 p.m., the ARSENL team achieved a major milestone in our research: we launched and flew 50 planes together, executing cooperative behaviors in flight, operated by only two humans during flight and a total team of 16 people.
The team that performed the flight (from left to right): Randy Bice, Kevin Jones, Marianna Jones, Ray Davis, Tyler Lippert, Michael Day, Evan Maki, Brian Le, Aaron Ray, Bob Been, Paul Epperson, Timothy Chung, Mike Clement, David Harach, Duane Davis, and EJ Montoya. This flight would not have been possible without the contributions of countless additional faculty, staff, students, and interns; ONR reservists; and the ongoing work of the DIY Drones community.
Photos courtesy of the ARSENL team and the Naval Postgraduate School Public Affairs Office. Find out more at arsenl.nps.edu.
Since the 50-plane mission, we've been focusing less on numbers and more on tactics. You can find some aerial video highlights from a recent event with aerial combat between two teams of 10 aircraft here:
Here we are essentially competing tactics software developed by two different teams against each other with identical fleets. While there were some interesting moments in the air, the only plane-on-plane contact occured during the mass landing.
Thanks Pat! That adapter looks similar to what we used. I can't speak to the current state of HIL in ArduPilot, but I'm a proponent of SITL and HIL testing in general and I'm sure there are some folks on this site that have more recent experience using both.
Best of luck with your demonstration!
Thanks Mike! Also great job on the demonstation, I'm working on a multiplane fixed-wing demo as well. I previously was thinking about this wifi adapter: https://www.amazon.com/gp/product/B0035GWTKK/ . How does this compare with the device you used?
On another note, I am having success with SITL and had planned to moved to HIL and finally live testing, but am having troubles with HIL. Did you use HIL and is it worth it?
We used an ALFA USB 802.11n device with an RP-SMA connector for an external antenna. The aircraft used 3 dBi omni antennas, and on the ground we used these: http://www.l-com.com/wireless-antenna-24-ghz-5-dbi-patch-wide-angle... No amplifiers were used, though we did consider adding some on the ground.
The ground antennas were mounted on a mast about 12-15 feet off the ground, with a 45 degree uptilt. Note that our ground station was off to one side of the flight arena, so we did not have aircraft flying on all sides of us.
As I recall, the farthest aircraft from the ground station were roughly 450 meters above and perhaps 200-300 meters laterally away.
We experienced reasonable loss rates for periodic UDP datagrams carrying telemetry from air to ground and both periodic heartbeats and event-driven commands from ground to air. TCP performance was rather poor, however; we did not attempt to remotely log in to any aircraft with all fifty aloft.
Did you guys have to use a wifi range extender or antenna? How did you use your adhoc network because from my knowledge, built in wifi only has about 10m range.
You're right, Silvio misunderstood my question.
I was very glad to see that someone is thinking into the same side as me:
«The primary purpose of the PCB is to minimize the rat's nest of wiring that is usually required for all the Pixhawk peripherals. Also, by removing the PCB with the Pixhawk and Odroid attached, everything that is needed for HITL is in one compact form. »
I also decide on the optimization of the system options to the single-board version.
1) BEC - The level of noise protection, the right of the current figures. Never use a voltage regulator to obtain.
2) Bus I2C - is for communication on board, not for signal transmission, in addition noise immunity and limited capacity of the tire
3) If possible, the current control instead of voltage or CAN
4) galvanic isolation where necessary and appropriate.
5) ultrasonic rangefinder - long used
6) Now the discussion regarding the correct GPS sensor
7) pressure sensor - the ones that are available I'm not very satisfied - there is a good idea to use but add the CAN communication system
The main effort - noise immunity and reliability in the failure of the product assembly.
say the reliability of the product - is the reliability of an untrusted site in this product ...
I think there may be some confusion between Sergii's question and Silvio's response. The PCB we use is not the Odroid, but the Odroid is mounted to the PCB and is powered by the PCB. The PCB is special purpose, specifically to fit our modified Zephyr II airframe. We do not make things to sell, but at some point we may open up the design for those that want to duplicate it on there own.
The picture is of an older version, but has most of the main parts as the latest version. Key features are that the voltage regulation for both the Pixhawk and the Odroid are built in (and are more tolerant of errant voltage spikes than the regulator built into the stock power module - another story), as well as voltage and current sensing for Pixhawk, I2C routing and the pressure transducer for airspeed. In this older model a blue heatsink can be seen. It is attached to a Schottky diode pair that allows both batteries to connect in parallel without any risk of back-current in case an under-trained operator plugs in a full and an empty battery. In parallel this would likely destroy the low battery and may even result in a fire. The diode pair eliminates that risk but siphons off about 3% of our total power. The Navy mandates that we use protection. In the newer versions of the PCB an SMT version of the diode pair is used eliminating the heatsink and opening up a little more room in the bay. The latest version of the PCB also includes mounting and wiring for a Maxbotix ultrasonic range finder, the plan being to gain precision on the autonomous landing touchdown point. It also includes power routing that lets us power the aircraft from shore power while the aircraft are in the stacks waiting for launch. This is one place where the diode pair pays off - by routing power into the PCB downstream of the diodes, and at a voltage above our 3-cell batteries, the batteries are not drained and are safe from over-charge. When you consider a 3-4 hour prep time for the fleet of 50, the first planes prepared have lost on the order of 20Wh of energy before they are launched, or about 20% of their available energy.
The primary purpose of the PCB is to minimize the rat's nest of wiring that is usually required for all the Pixhawk peripherals. Also, by removing the PCB with the Pixhawk and Odroid attached, everything that is needed for HITL is in one compact form.
@Sergii - The PCB is a custom design that we developed; I think there is a plan to post a description with the CAD files online at some point. Great question about the USB-UART; we decided on that because there wasn't an easy-to-use 1.8v reference voltage available on the U3. For the next generation, which may use the XU4, we have discussed putting a level converter directly on the PCB.
Can I purchase?
4Gmetry is ready to fly http://4gmetry.voltarobots.com/
Please forgive me for my english ...
Please let me know what kind of PCB in the photo on which the PIXHAWK & ODROID-XU4
who has made it ??? Can I purchase?
And another question, why do you use to communicate PIXHAWK & ODROID-XU4 transition USB - UART ??? easier, more beautiful and safer would be to use galvanic isolation UART 5V0 - 1V8
Thank you for your reply