The development team is always trying to find ways to improve our testing and the quality of our code. We've recently add a new automatic testing system that should help a lot with finding bugs and ensuring that the huge range of features that we have added (and plan to add!) to the code keep working.

If you have a look at http://autotest.diydrones.com you'll see the results from 'Tools/autotest', a test system that runs on every change to the ArduPilot and ArduCopter code. The test script currently checks the following:

  • that both ArduPilot and ArduCopter build on the 1280 and 2560 based APM boards
  • that both ArduPilot and ArduCopter can build with the 'desktop' build system
  • that ArduCopter can fly in both manual and automatic (mission) mode
  • that we can download the flight logs

It also keeps detailed logs, and produces a KML file which you can load in googleearth to look at how the flight went.

The flight itself is controlled using a python scripting system that flies ArduCopter using MAVLInk commands. Currently that script does the following:

  • starts up the simulator at my local flying field
  • arms the motors (using virtual transmitter sticks to control the rudder and throttle)
  • switches the quadcopter to STABILIZE mode
  • does a manual takeoff using throttle, until the altitude reaches 30 meters
  • uses the rudder to turn the quad until it is facing north (it starts up facing west)
  • uses the elevator and aileron sticks to manually fly a square circuit about 50m on a side
  • switches to LOITER mode, and checks that it holds altitude for 10 seconds
  • switches back to STABILIZE and decreases throttle, to bring the quad in for a gentle landing
  • loads a 9 waypoint mission into EEPROM
  • switches to AUTO mode and flies the mission, which involves a zig-zag route over the field
  • the last mission step is a RTL, which brings the quad home
  • then it switches to STABILIZE, brings the quad in for a landing, and disarms the motors

You can see a track of the test flight in the above googleearth screenshot.

Running these test steps every time we make a change to the code is already ensuring that a lot of our code works, but we have plans to greatly expand this test suite. What we'd like to do is have every ArduPlane and ArduCopter feature tested, with code that validates that it actually does the right thing. At the moment the test suite only has very rudimentary testing of whether the flight was actually correct, whereas we should be checking lots of parameters (roll, pitch, altitude, speed etc) throughout the flight.

The other obvious missing piece is ArduPlane testing. The test suite checks that ArduPlane builds, but doesn't yet check that it flies correctly. I hope to fix that this weekend, with ArduPlane flying some simple missions plus some 'manual' (ie. python scripted) flying. I'll probably do this by interfacing with FlightGear, but jsbsim linkage of even a new physics model is a possibility.

Under the covers

This new test system involves quite a few new bits of code, some of which we hope will be used for things other than automatic testing. The quadcopter simulator itself is a simple physics simulator written in python, which Michael has since translated into C# for use in the planner (thats how you can fly quadcopter HIL in the planner now). The communication between the various pieces uses pymavlink, a MAVLink implementation in python. That uses a bunch of helper scripts that make it fairly easy to script a ArduPilot vehicle. I'm hoping this system will be used for scripting real copters and planes in the future, along the lines of the acrobatic scripting system we discussed earlier in the year.

The other big part of this autotest system is the 'desktop' (also known as SIL for "software in the loop") build of ArduPlane and ArduCopter. This builds the code using a normal C++ compiler, and creates executables which run directly on your desktop computer, rather than on an APM board. The advantage of this is that we can use all the debug tools (such as gdb and valgrind) which are normally used for C++ development, but also it runs much faster and doesn't rely on hardware being plugged in. We've already taken advantage of the ability to run ArduCopter under valgrind to find some bugs, and it really increases our confidence in the code to be able to run that sort of sophisticated memory checking on every commit.

If you want to have a look at autotest, then start by grabbing the latest APM code and having a look in Tools/autotest. You might like to start by looking at arducopter.py and seeing how it controls the quad. We're hoping that we will get contributions of new tests. We also plan on putting up a wiki page describing how you can run the autotest system on your own computers (right now it only runs on Linux and MacOS, although Micheal has had some limited success on Windows using cygwin).

How you can help

The autotest system is already quite useful and it has already helped us find some subtle bugs in the QuadCopter code, but we'd love to expand it to cover more aircraft and features. Here are a few things that it would be great to get contributions on if someone has the time:

  • we need physics simulations for more multi-copter frame types (start with quadcopter.py, and create ones for hexa, octa etc)
  • I'd like to produce a nice picture of the flight path from a KML file using a script, so devs don't have to load googleearth to quickly check it looks OK
  • we need some interesting test missions to look at. I hope to support loading missions in planner format soon to make this easier
  • we need python code to more carefully check the flight is OK. It may be easier to wait on this until we develop some better helper functions, but if you are a python coder then feel free to jump in now

Hopefully this will be the beginning of a much more rigorous testing effort for ArduPilot. We'll post more information as the system develops!

 

Views: 4463


Developer
Comment by Michael Oborne on November 11, 2011 at 1:57am

Good work Tridge!!!


Developer
Comment by Andrew Tridgell on November 11, 2011 at 2:20am

thanks to a suggestion from Michael we now produce a much smaller KMZ file instead of the larger KML. Please use the KMZ link if possible, it will save a lot of bandwidth on my server (the autotest system runs on my home Linux box, on an ADSL link)

 


Developer
Comment by Randy on November 11, 2011 at 7:27am

Yeah, very impressive.  I hope to write some test cases and get it working for the trad heli too although the physics engine would be a little more complex.


Moderator
Comment by Alex on November 11, 2011 at 8:16am

Great stuff!


3D Robotics
Comment by Chris Anderson on November 11, 2011 at 8:32am

This has been amazingly useful already. Just in the few weeks that we've been using it, Jason has rubbed out a dozen sneaky little bugs in ArduCopter 2.0.50. They would have taken hours each to replicate in the field, if we even knew where to look, and they just pop right out of the test suite. 

 

I don't know how we were able to work without this before! It's a huge time saver and should really accelerate the progress of ArduCopter/ArduPlane in features, performance and reliability. It allows us to focus on adding cool stuff, not trying to track down hard-to-replicate reports of issues.


Distributor
Comment by Dany Thivierge on November 11, 2011 at 8:42am

Great work (again) this just shows how mature the code and general environment is becoming. 

This is for sure no more a basement/garage project.  We have great facilities producing quality components. We have great developers and now they are given that amazing tool that will iron out bugs without needing us to crash to find! 

 

hehe it's a great day!   Keep it up boys, we make a super team when we all work in the same direction. 


Developer
Comment by Jason Short on November 11, 2011 at 9:38am

I can't tell you all how big of a deal this is. When we got HIL for Ardupilot, the code was far better and more reliable. With this new system, we can fly acrobatics, run multiple test scripts for each mode, read the logs, even peek into the EEPROM. All without any hardware connected whatsoever. It's amazing.

Jason


Admin
Comment by Gary Mortimer on November 11, 2011 at 11:08am

Excellent work, I have test missions for you Tridge, all the T3 rounds.

 

 


Developer
Comment by Andrew Tridgell on November 11, 2011 at 1:30pm

@Randy, Don't worry about getting the physics perfect for the trad-heli, just some very simple physics will be enough to test the code. I hope also to be able to hook other flight sim systems into it, starting first with flightgear for ArduPlane.

Comment by Kirill on November 11, 2011 at 2:04pm

This is really cool!

Comment

You need to be a member of DIY Drones to add comments!

Join DIY Drones

© 2014   Created by Chris Anderson.

Badges  |  Report an Issue  |  Terms of Service