Last week, the Advanced Robotic Systems Engineering Laboratory (ARSENL) team at the Naval Postgraduate School, based in Monterey, California, successfully launched and flew 50 aircraft simultaneously.
It was nearing 110 degrees Fahrenheit on the runway at Camp Roberts, California as we carted the last rack of planes from the flight technicians’ tables over to the launch site. We had started flight preparations around 8:30 that morning, after arriving an hour earlier to set up shade structures, tables, and generators, and unpack some 68 planes we would prepare for flight that day. Knowing that the planes’ limited battery life would restrict our launch window to less than half an hour, and that we’d previously encountered last-minute sensor calibration problems and WiFi devices that went brain-dead in the heat and sun, we conservatively assumed that we would prepare four planes to fly three. Our goal this time was to fly 50.
The Fleet
The planes we fly are built from the Ritewing Zephyr II kit. These are EPO foam wings with elevons and room for an electric pusher motor. The plane is outfitted with Hitec digital servos, an O.S. motor driven by a Castle Creations ESC, and dual Thunder Power 5000 mAh LiPo batteries. In this configuration, our expected flight time is approximately 45-50 minutes, meaning that we needed to have all aircraft launched within about 30 minutes to have time to perform cooperative flight and to land safely before battery failsafes started to trigger on the autopilot, forcing an autonomous landing.
A custom PCB distributes power to and manages some of the wiring for the Pixhawk, which flies a lightly-modified branch of ArduPlane. To the right of the Pixhawk is an ODroid u3 companion computer (“payload” in our team’s parlance) running a stripped-down version of Ubuntu Linux. Our payload software is built on top of
ROS, making it easy to add new functionality incrementally. While the Pixhawk manages keeping the plane airborne, the payload ingests autopilot pose and status, which it broadcasts to other planes and to the ground. It also accepts commands from the ground, which are either passed to the autopilot or used to control execution of a handful of payload-directed path planning algorithms, such as leader-follower, distributed search, and sequenced landing. These algorithms “drive” the plane by sending updated target latitude-longitude-altitudes to the autopilot, effectively moving a carrot in front of the plane.
The 900 MHz telemetry radios typically used to communicate with the Pixhawk are only a backup link in our architecture, and a tenuous one with nearly 70 of them powered at once. After observing difficulty establishing telemetry links during previous field tests, we measured signal-to-noise with 50 planes powered on the bench and saw that the default 915-928 MHz band was saturated. Following a suggestion from the DIY Drones community, we decided to split our available 902-928 MHz band into three equal-size sub-bands and place an equal number of planes into each sub-band. We reasoned that, even with the reduction in per-band bandwidth, some 22 to 23 planes (of the 68 prepped for flight) in a sub-band would hold up better than all planes in a single, wider band. This intuition came after observing that at most 50 frequency hopping channels could be supported in a band in the current radio firmware.
In any case, at most one plane would actively be controlled over the telemetry link, and only in an emergency. Our primary link with the fleet is an ad hoc 802.11n wireless network that connects each plane’s payload with other planes and our ground control stations. All of our command and status traffic is sent via UDP; as with MAVLink, our messages are designed to minimize statefulness and be tolerant to loss and blind retransmission.
Should a plane’s payload have failed or the network conditions become too lossy, one of the ground operators’ software had a button that would reconfigure an attached telemetry radio to the select plane’s sub-band and Net ID, and then open a MAVProxy session to that plane within a few seconds.
Preparation
It took our diligent crew of flight techs close to four hours to prepare all the planes. Preparations comprise not only putting propellors on motors and making sure that all mechanical and electrical connections are secure, but also running through a set of software preflight checks. These checks have evolved over the past two years from a paper checklist - imagine checking the values of 40-odd autopilot parameters by hand - to a MAVProxy module, and finally to a standalone interface that works in concert with the payload and a central configuration server to perform the preflight process.
When a plane is first powered, the payload reaches out to the server and pulls down its mission configuration: autopilot parameters, waypoints, rally points, and geo-fence. The payload verifies that the autopilot matches these files and then sets status flags in a message broadcast back to the ground stations.
Meanwhile, the flight tech is busily performing sensor and actuator checks. Our Flight Tech Interface (FTI) software reports similar status lights to those in MAVProxy’s console module. A red “as” light indicates that the airspeed sensor must be calibrated; the flight tech places a bottle over the pitot tube and presses the “Cal Pressure” button in the FTI, and a few seconds later the light changes to green indicating completion. Then the servos and motor are checked. For this, we designed simple test stands that are staked into the ground. The plane is secured in the test stand, and then the flight tech once again sends a command from the FTI to the payload. First, they check the servos: the elevons execute a left roll, right roll, up pitch, down pitch, and return to level. Then, the technician arms the throttle and commands a run-up of the motor, which lasts about two seconds. Finally, a second tech picks the plane up and rotates it manually while the first tech verifies on the screen that the autopilot correctly reports roll and pitch.
Once done with the checklist, the flight tech clicks one last button, telling the plane to set a “flight ready” status bit so the plane reports as ready for flight. The plane is then placed in a rack built from two-by-fours and PVC pipe, which is moved 100 feet out to the launch site when full. Each rack carries ten planes.
Launch Sequence
It was getting close to 1 p.m., and some of the aircraft had been powered for multiple hours. Though the amount of power they consume on the ground is small, it isn’t nothing. It was hot, so we were also worried about electronics starting to fail. From our previous field test experiences, we had begun to suspect that heat was an issue for some onboard components including the wireless network devices, so we were continually edging racks of planes under shade structures to track the sun’s movement.
As the last planes were carted out and a few last-minute recalibrations were performed, we prepared our automated launcher for its big test run. We had previously used a bungee launch system. It was simple and reliable, though required a person to sprint after the bungee each time it was released, and pull some 25 pounds of force back to the aluminum rails from which the aircraft launched. Between the number of planes to launch and the heat, we were glad to see the new launcher ready for action.
The brainchild of two Masters students at the school, the Automated Multi-Plane Propulsion System (A.M.P.P.S.) comprises a chain wrapped around two cogs spaced eight feet apart, set between two parallel rails from which the aircraft launches. On the leading edge of the launcher is an electric motor that draws an impressive 200 amps when spun up at maximum rate. It is capable of accelerating a plane from rest to roughly 16 meters per second before it leaves the rails. Technically, it can do more, though we ran into problems with shearing launch hooks from the airframe.
With a few modifications to the motor controller’s acceleration profile and a redesigned, 3D printed adapter attaching the plane to the launch chain, we were reasonably confident that A.M.P.P.S. would launch all 50 planes successfully. Nonetheless, we kept the old launcher ready with a bungee pre-tensioned just in case of a failure.
Timothy Chung, lead faculty for the project, called over the radio, “All right team ARSENL, this is it. Everyone have fun, stay safe. Mission 36... commence.” This was the thirty-sixth mission of the week. Most of those missions had been single-plane checkout flights; every brand-new and newly-repaired plane goes through a short "mechanical checkout flight" before being reintroduced into the fleet. Our only multiple-plane missions during the week had been two ten-plane flights the day before, as practice runs for the 50-plane test. The largest number of planes we had flown previously was 30 in July; we had been increasing our numbers exponentially since our first multi-plane flight with the Zephyr platform last year.
Time to hustle. We loaded the first plane onto the launcher, and our launch operator used a variant of the FTI to command final configuration parameters. Since any plane might fail to launch, we needed to tell the plane which “slot” in the flight configuration it would fill (more on this later). She then commanded the throttle to arm, and signaled both the safety pilot - who had selected that aircraft on an RC transmitter - and another operator standing next to the launcher, that the aircraft was ready for launch. The second operator pressed two buttons on a gamepad controller wirelessly connected to the launcher, commanding it to launch the plane. Three safety beeps from the launcher, then a loud whirring as the chain suddenly accelerated, kicking the plane off into the air, and that long-seeming moment before the plane detected launch and started the throttle. The plane took off steeply toward its initial waypoint.
As the safety pilot confirmed stable flight, an intern carried the next plane over to the launcher. This time, there was some confusion between operators, and it took about a minute to get the second plane airborne. At a minute per plane, we would be landing the first planes as the final ones were being launched.
Fortunately, the next few planes launched quickly, at an average of just over 30 seconds per launch. The tenth plane aloft experienced some flight instabilities and the pilot had to pause the launch sequence so he could take manual control and land that plane. That cost us an extra minute. After that, the team found a good rhythm for the rest of the launch sequence. After the 50th launch, the launch operator - who had been keeping track of launches on paper - paused and asked, “is that all of them?” We all looked at each other, then quickly remembered that we had to launch a fifty-first plane to make up for the one that was landed earlier. It’s amazing how difficult it is to keep track of so many planes.
In all, we performed 51 launches, all using A.M.P.P.S., in 27 minutes 39 seconds, at an average of 33 seconds between launches. High-fives and handshakes commenced at the launch site, while operations were just ramping up inside a trailer parked 100 feet away.
The Flight
Our flight configuration consisted of two “stacks” of 25 planes, with each “slot” in the stack separated 15 meters apart vertically. The two stacks were separated laterally so that the orbits and racetracks of one stack would not overlap with the other. Though we launched in altitude order, we designed the waypoint sequence with special “ingress” and “egress” points where planes could ascend and descend without entering the airspace of other planes. This altitude deconfliction meant that we could execute cooperative behaviors within a single stack in two dimensions, without having to worry about implementing collision avoidance due to separation in the third dimension. The launch operator sent the stack and slot configuration to each plane just prior to launch.
Example of a "stack" forming with 12 planes visible.
Each time another five or so planes entered the stack, a single operator sitting in a trailer placed away from the launch site would command them into groups and initiate cooperative behaviors. In fact, there were only two operators in the trailer: one was commanding the fleet as a whole, and the other was monitoring planes’ health.
We executed our leader-follower algorithm during this flight, due to its heavy vetting in previous field tests. This algorithm determines a single leader and commands all other planes in the group to follow it directly, while maintaining the altitudes set by the launch operator. The highest altitude plane in the group becomes the leader and enters a prescribed racetrack, while all other planes listen to its broadcast pose and compute following positions.
Author’s comment: Watching 24 planes stacked almost perfectly on top of one another, following a common lead plane, is pretty amazing. Each plane broadcasts its pose at roughly 8 Hz, but considering wireless contention and the relative orientation of antennas, follower planes may sometimes receive as little as 1-2 updates per second. The payload software extrapolates positions using the current pose and linear velocities only, so the fact that the stack performed so well under lossy network conditions continues to impress me.
Visualization in Gazebo using data captured in ROS on each payload. Aircraft are continuing to join the left stack while the rest of the stack are independently loitering at deconflicted altitudes. Those in the right stack are performing a leader-follower algorithm, with the highest aircraft acting as leader.
We had 9 minutes 45 seconds with all 50 planes airborne, performing cooperative behaviors in various groups. Due to our long preflight and launch sequence, batteries were quickly depleting in the air. We had to start bringing down planes.
With the safety pilot on standby, focusing on individual planes as quickly as possible, we commanded groups within a stack to execute a sequenced landing. Using this algorithm, planes order their landings according to altitude, lowest first, and each plane in sequence begins an autonomous landing. Our landing has each plane perform a descending loiter to 40 meters above the runway, then break into a glide slope until it finally skids to a halt on the asphalt. Each plane bears 3D printed skids that can easily be replaced when they ablate.
Landing 50 planes did not happen not without minor glitches. Because we ran close to the limit of the batteries getting all aircraft aloft and executing cooperative behaviors, a few planes triggered their low-battery failsafe and automatically executed a controlled autonomous landing alongside those performing the sequenced landing. This made for great video, though the safety pilot had to pick one to monitor and one to let land on its own. Scrolling through models on his transmitter took most of his attention, otherwise. A couple planes' ESCs also triggered their own low-voltage failsafe and cut the throttle, causing a controlled descent into a field adjacent to the runway. Thankfully, all but two of these planes had a soft landing, and all batteries were intact after landing - though some were retired due to their low remaining voltages.
Example of sequenced landing showing three aircraft inbound.
Planes able to make it to the runway faced another challenge. All planes were programmed with the same landing point, with the assumption that GPS and barometer inaccuracy would provide sufficient spread on landing. But with nearly 50 planes on the runway, on-deck collisions were unavoidable. Some of the video captured by GoPro cameras mounted to the nose of each plane show skidding into other planes, or coming down directly on top of other planes on the ground. Lesson learned for next time: add a bit more variation to the landing coordinates.
Conclusion
On August 27, 2015, at roughly 1:20 p.m., the ARSENL team achieved a major milestone in our research: we launched and flew 50 planes together, executing cooperative behaviors in flight, operated by only two humans during flight and a total team of 16 people.
The team that performed the flight (from left to right): Randy Bice, Kevin Jones, Marianna Jones, Ray Davis, Tyler Lippert, Michael Day, Evan Maki, Brian Le, Aaron Ray, Bob Been, Paul Epperson, Timothy Chung, Mike Clement, David Harach, Duane Davis, and EJ Montoya. This flight would not have been possible without the contributions of countless additional faculty, staff, students, and interns; ONR reservists; and the ongoing work of the DIY Drones community.
Photos courtesy of the ARSENL team and the Naval Postgraduate School Public Affairs Office. Find out more at arsenl.nps.edu.
tl;dr: I'm building an open-source, 3D printable rover.
