Developer

ArduPilot automatic testing system

3689432467?profile=originalThe development team is always trying to find ways to improve our testing and the quality of our code. We've recently add a new automatic testing system that should help a lot with finding bugs and ensuring that the huge range of features that we have added (and plan to add!) to the code keep working.

If you have a look at http://autotest.diydrones.com you'll see the results from 'Tools/autotest', a test system that runs on every change to the ArduPilot and ArduCopter code. The test script currently checks the following:

  • that both ArduPilot and ArduCopter build on the 1280 and 2560 based APM boards
  • that both ArduPilot and ArduCopter can build with the 'desktop' build system
  • that ArduCopter can fly in both manual and automatic (mission) mode
  • that we can download the flight logs

It also keeps detailed logs, and produces a KML file which you can load in googleearth to look at how the flight went.

The flight itself is controlled using a python scripting system that flies ArduCopter using MAVLInk commands. Currently that script does the following:

  • starts up the simulator at my local flying field
  • arms the motors (using virtual transmitter sticks to control the rudder and throttle)
  • switches the quadcopter to STABILIZE mode
  • does a manual takeoff using throttle, until the altitude reaches 30 meters
  • uses the rudder to turn the quad until it is facing north (it starts up facing west)
  • uses the elevator and aileron sticks to manually fly a square circuit about 50m on a side
  • switches to LOITER mode, and checks that it holds altitude for 10 seconds
  • switches back to STABILIZE and decreases throttle, to bring the quad in for a gentle landing
  • loads a 9 waypoint mission into EEPROM
  • switches to AUTO mode and flies the mission, which involves a zig-zag route over the field
  • the last mission step is a RTL, which brings the quad home
  • then it switches to STABILIZE, brings the quad in for a landing, and disarms the motors

3689432443?profile=originalYou can see a track of the test flight in the above googleearth screenshot.

Running these test steps every time we make a change to the code is already ensuring that a lot of our code works, but we have plans to greatly expand this test suite. What we'd like to do is have every ArduPlane and ArduCopter feature tested, with code that validates that it actually does the right thing. At the moment the test suite only has very rudimentary testing of whether the flight was actually correct, whereas we should be checking lots of parameters (roll, pitch, altitude, speed etc) throughout the flight.

The other obvious missing piece is ArduPlane testing. The test suite checks that ArduPlane builds, but doesn't yet check that it flies correctly. I hope to fix that this weekend, with ArduPlane flying some simple missions plus some 'manual' (ie. python scripted) flying. I'll probably do this by interfacing with FlightGear, but jsbsim linkage of even a new physics model is a possibility.

Under the covers

This new test system involves quite a few new bits of code, some of which we hope will be used for things other than automatic testing. The quadcopter simulator itself is a simple physics simulator written in python, which Michael has since translated into C# for use in the planner (thats how you can fly quadcopter HIL in the planner now). The communication between the various pieces uses pymavlink, a MAVLink implementation in python. That uses a bunch of helper scripts that make it fairly easy to script a ArduPilot vehicle. I'm hoping this system will be used for scripting real copters and planes in the future, along the lines of the acrobatic scripting system we discussed earlier in the year.

The other big part of this autotest system is the 'desktop' (also known as SIL for "software in the loop") build of ArduPlane and ArduCopter. This builds the code using a normal C++ compiler, and creates executables which run directly on your desktop computer, rather than on an APM board. The advantage of this is that we can use all the debug tools (such as gdb and valgrind) which are normally used for C++ development, but also it runs much faster and doesn't rely on hardware being plugged in. We've already taken advantage of the ability to run ArduCopter under valgrind to find some bugs, and it really increases our confidence in the code to be able to run that sort of sophisticated memory checking on every commit.

If you want to have a look at autotest, then start by grabbing the latest APM code and having a look in Tools/autotest. You might like to start by looking at arducopter.py and seeing how it controls the quad. We're hoping that we will get contributions of new tests. We also plan on putting up a wiki page describing how you can run the autotest system on your own computers (right now it only runs on Linux and MacOS, although Micheal has had some limited success on Windows using cygwin).

How you can help

The autotest system is already quite useful and it has already helped us find some subtle bugs in the QuadCopter code, but we'd love to expand it to cover more aircraft and features. Here are a few things that it would be great to get contributions on if someone has the time:

  • we need physics simulations for more multi-copter frame types (start with quadcopter.py, and create ones for hexa, octa etc)
  • I'd like to produce a nice picture of the flight path from a KML file using a script, so devs don't have to load googleearth to quickly check it looks OK
  • we need some interesting test missions to look at. I hope to support loading missions in planner format soon to make this easier
  • we need python code to more carefully check the flight is OK. It may be easier to wait on this until we develop some better helper functions, but if you are a python coder then feel free to jump in now

Hopefully this will be the beginning of a much more rigorous testing effort for ArduPilot. We'll post more information as the system develops!

 

E-mail me when people leave their comments –

You need to be a member of diydrones to add comments!

Join diydrones

Comments

  • Hi. Cool. Will it run under Cygwin or do I need my Linux box?
  • Developer

    The SITL system has had some major updates recently, and is now autotesting ArduPlane, as well as doing much better register level umulation of the hardware. I've also added some documentation in the wiki on the SITL system and how to use it.

    @david, thanks for the patches! I'll get these fixed soon

  • Developer

    I've also found one last minor tweak to get it to run on OSX  ( which does not have strnlen) :

    diff --git a/libraries/Desktop/support/Arduino.cpp b/libraries/Desktop/support/Arduino.cpp
    index 1726a0e..e365b98 100644
    --- a/libraries/Desktop/support/Arduino.cpp
    +++ b/libraries/Desktop/support/Arduino.cpp
    @@ -64,7 +64,8 @@ size_t strlcat_P(char *d, PGM_P s, size_t bufsize)
     
     size_t strnlen_P(PGM_P str, size_t size)
     {
    -       return strnlen(str, size);
    +       //return strnlen(str, size);
    +       return strlen (str) < size ? strlen (str) : size;
     }
     
     int strcasecmp_P(PGM_P str1, PGM_P str2)

  • Developer

    Hi guys!

    I have a minor bug report for SIL on OSX: 

    -Wno-reorder  is not a valid flag for c++ , only for c.    I made a simple change to fix it:

    /libraries/Desktop/Desktop.mk

    -OPTFLAGS               =       -g -Wformat -Wall -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wformat=2 -Wno-reorder
    +OPTFLAGSC              =       -g -Wformat -Wall -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wformat=2 -Wno-reorder
    +OPTFLAGSCXX            =       -g -Wformat -Wall -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wformat=2


    -CXXFLAGS               =       -g $(DEFINES) $(OPTFLAGS) $(DEPFLAGS) $(CXXOPTS)
    -CFLAGS                 =       -g $(DEFINES) $(OPTFLAGS) $(DEPFLAGS) $(COPTS)
    +CXXFLAGS               =       -g $(DEFINES) $(OPTFLAGSCXX) $(DEPFLAGS) $(CXXOPTS)
    +CFLAGS                 =       -g $(DEFINES) $(OPTFLAGSC) $(DEPFLAGS) $(COPTS)


    -LDFLAGS                        =       -g $(OPTFLAGS)
    +LDFLAGS                        =       -g $(OPTFLAGSC)

    Hope this helps.   Oh, and I had to explicitly set this too:

    export TMPDIR=/tmp

     ( as it defaults to something strange like:  /private/var/folders/5B/5BgQfjAAH+qwAZpwxLNUoU+++TI/-Tmp-/ )

  • Sounds good.  I have some ideas for SITL-specific logging that I may try once I get the sim up and running.  I'm interested in the Wiki how-to you mentioned, because I still seem to have some dependencies to track down.  FWIW I'm building on OS X right now.

  • Developer

    @Alex, yes, we already have the ability to log internal variables. We can also attach a debugger and put in conditional breakpoints.

    The logging we have at the moment comes in several styles:

    • DataFlash logging, in Log.pde. That is what is used for real hardware, and it also works in the SIL sim. You can see those logs for our quad sim here
    • MAVLink logging, which is our telemetry logs. Those are available here. You need to use mavlogdump or mavgraph from pymavlink to view those.
    • Console serial logging. Those are printf() messages made in the APM code, and are logged with a "APM:" prefix in the main sim log
    • SIL specific logging. You can just do fprintf(stdout, "some variable=%f\n", some_variable); in the code, protected by a #ifdef DESKTOP_BUILD. I use that sometimes for private debugging, though I more commonly use gdb breakpoints

    We may well add new logging systems as the need arises, but this is already a pretty rich logging system that solves most problems that come up.

     

  • Having a SITL sim is huge.  Is there a way, or are you planning a capability, to add logging of arbitrary internal variables when running in SITL mode?  It's usually much easier to debug dynamic phenomena by plotting time histories than to look at values in a watch window.

  • This is really cool!

  • Developer

    @Randy, Don't worry about getting the physics perfect for the trad-heli, just some very simple physics will be enough to test the code. I hope also to be able to hook other flight sim systems into it, starting first with flightgear for ArduPlane.

  • Moderator

    Excellent work, I have test missions for you Tridge, all the T3 rounds.

     

     

This reply was deleted.