Arducopter Erroneous Autopilot Causes Crash

All,

https://youtu.be/w5_E3-6p4s4

Had a flight yesterday with virtually no changes from the previous day (with successful flight) on the same flying field here in San Diego.  Notably this time I started receiving erroneous data on my OSD (mavlink messages); it was clearly causing some control issues.  I'm wondering if anyone has experienced this?  I'd really like to not hit the ground going 17mph again :).

Most of the reason for the crash is my fault in that I was flying LOS and didn't look at the FPV monitor and notice the bad data coming down until watching the video later.  I could definitely tell something was wrong while maneuvering around.  Here's the video (having a DVR is an AMAZING troubleshooting tool).  I've also included the log file, showing that at one point I try to activate an RTL, which essentially triggers a flyaway situation, at which point I switch back to STAB and try to recover.  Then erroneous data comes down (clearly visible 10000ft/min VS!! and crazy horizon line) and its too hard to recover, so I set it down.  Lucky I was flying in a big empty field!  

I've also got the logs: 2016-01-24%2012-49-42.zip.  I did an analysis but not really sure what to look for.  I'm thinking the Kalmann filter went nuts... any help?  Anyone seen this kind of behavior before?  Here's my build info:

Frame: QAV500 Frame w/540mm arms
Props: 12x5.5 Carbon Props
Motors: Tiger Motor MN3110-17 (700kv), 4mm shaft
Autopilot: Raspberry Pi 2 w/ Navio+ hat
ArduCopter Version: Copter V3.3
External Compass: HMC5883L https://www.adafruit.com/products/1746
GPS: Ublox M8N + Garmin GA25 MCX GPS Antenna
Battery: Turnigy MultiStar 4S 14.8V 16Ah
Internet Cam: Raspberry Pi Cam
FPV Cam: 1/3 Inch SONY CCD Board Camera 700TVL
ESCs: Emax 4 in 1 30A w/SimonK Firmware
Telemetry: RFD900+ Long Range Radio Modem
FPV Tx/Rx: Boscam Thunderbolt 2000mW 5.8GHz
FPV Antenna: Boscam 5.8GHz Cloud Spirit Circular Polarized
RC Tx: FrSky Taranis 9XD
RC Rx: Dragon Link V2, UHF

You need to be a member of diydrones to add comments!

Join diydrones

Email me when people reply –

Replies

  • All,

    So I took comments into consideration and did a couple tests.  I took the laptop out to the field and hooked up to telemetry via my RFD900+ serial modem.  First I tried the compass configuration noted above.  The main screen I was looking at was the EKF status screen (opens when you are in Mission Planner and you click on "EKF" on the HUD.  First thing I noticed was that when you rev the motors from 0 to 50% or so (while on the ground) you get EKF very Red for Compass.  So this definitely points to the compass as being part of the issue.  If anyone has had luck using this compass for APM, let me know.  Maybe its just too close to the battery wires (at least 200mm+ away though?).  It is not currently on a mast, but I felt that it was far enough away from DC noise that interference there wouldn't be an issue, while clearly it is if the EKF status is directly proportional to the throttle level!  

    The easiest thing to do was disconnect the external compass, use the internal compass, and calibrate.  So that's what I did.  I performed the same test, and while the EKF definitely was poorer the faster I increased the throttle, it was noticeably less than using the external compass.  So the sun setting, I take off and perform what I would consider a perfectly normal flight just using the internal compass.  However, upon post flight analysis I get:

    Log File C:/Program Files (x86)/Mission Planner/logs/QUADROTOR/1/2016-01-25 16-59-17.log
    Size (kb) 9707.171875
    No of lines 129429
    Duration 0:09:36
    Vehicletype ArduCopter
    Firmware Version V3.3
    Firmware Hash 340970fc
    Hardware Type
    Free Mem 0
    Skipped Lines 0

    Test: Autotune = NA -
    Test: Balance/Twist = NA -
    Test: Brownout = GOOD -
    Test: Compass = FAIL - Large change in mag_field (191.71%)
    Max mag field length (814.09) > recommended (550.00)

    Test: Dupe Log Data = GOOD -
    Test: Empty = GOOD -
    Test: Event/Failsafe = GOOD -
    Test: GPS = GOOD -
    Test: IMU Mismatch = NA -
    Test: Parameters = GOOD -
    Test: PM = NA -
    Test: Pitch/Roll = NA -
    Test: Thrust = NA -
    Test: VCC = FAIL - VCC below minimum of 4.6v (0.0v)

    The compass failed WORSE than before... but it always pointed in the correct direction and the video clearly shows no erroneous altitude measurements nor AHI issues like before.

    So the questions arise: 

    1) Has anyone had luck with the HMC5883L https://www.adafruit.com/products/1746 compass running APM Copter v3.3+?

    2) If so, where did you mount the compass?  Was there a certain orientation you used?  Did you experience any bad mag/ekf data at first and need to adjust it?

    I think that with the right placement and tuning I'll get my external compass to work properly and wont have a "Compass Fail" for variance post flight, but for now the Navio+ internal compass is doing the job until I can figure out what to do with the above external or replace it entirely.

    Also here are the logs from my "good" flight.  Also thanks again Craig/Kev for the log analyzer.  Here are its output:

    APM Log File Analiser v2.2.0.8

    Log FileName: A:\guest\Videos\GoPro\QuadCopter\Fiesta Island\20160125\Logs\2016-01-25 16-59-17.log
    Ardu Version: V3.3 Build: (340970fc)
    Ardu Type: APM:Copter
    Hardware: 259
    Frame Type: X
    No. Motors: 4

    Data Found in APM Log File:-
    IMU, GPS, CTUN, PM, CURR, MSG, MAG, CMD, ATT, MODE, STRT, EV, DU32, AHR2, EKF1, EKF2, EKF3, EKF4, UBX1, UBX2, RCIN, RCOU, BARO, RAD

    Data NOT Found in APM Log File:-
    NTUN, ATUN, ATDE, MOT, OF, INAV, D16, DU16, D32, DFLT, PID, CAM, ERR, GPS2, IMU2, IMU3, MAG2, MAG3, TERR, POWR, SIM

    FAILSAFE SETTINGS:
    Main Battery = Not Activated
    Ground Control = Activated, with always RTL
    Receiver = Activated at 975 PWM with always RTL
    GEO Fence = Not Activated


    PARAMETER ISSUES:-
    ARMING_CHECK = 0
    Warning: Pre-Arming Check is not Active (0=Off & 1=Fully On)


    TUNE = 0
    Information: CH6 Tuning is Disabled

    06/01/1980 - 000560: Mode Changed to Stabilize
    06/01/1980 - 000560: Mode Initialised at 0 0 Alt: 0 Spd:0
    06/01/1980 - 000561: Armed
    Limited Data Available - Mode Ignored

    06/01/1980 - 000562: Mode Changed to Stabilize
    06/01/1980 - 000562: Mode Initialised at 0 0 Alt: 0 Spd:0
    24/01/2016 00:00:17 - 000579: Take Off
    24/01/2016 00:00:17 - 000579: Mode Time Changed to 24/01/2016 00:00:17
    24/01/2016 00:00:17 - 000588: Simple Mode is Disabled
    24/01/2016 00:00:17 - 000588: APM Information: RC Pre-Arm checks have been completed.
    24/01/2016 00:00:17 - 000588: APM Information: All Pre-Arm checks have been completed.
    24/01/2016 00:00:17 - 000588: APM Information: USB is NOT Connected.
    24/01/2016 00:00:17 - 000588: APM Information: System Time has been set from the GPS.
    24/01/2016 00:00:17 - 000588: APM Information: Home is NOT Set
    24/01/2016 00:00:18 - 000821: Super Simple Mode is On
    24/01/2016 00:00:18 - 000821: APM Information: Home is Locked
    24/01/2016 00:00:19 - 000978: *** Update Program to include this Event *** : 28
    24/01/2016 00:00:19 - 001048: Auto Mode is waiting for Throttle.
    24/01/2016 00:00:29 - 003332: LAND_COMPLETE_MAYBE
    24/01/2016 00:00:30 - 003509: Landed
    Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop Eff(mA/min)
    Max 0 ~ 0 1 ~ 2 0.00 ~ 0.00 4.19 ~ 13.73 8 2.22 N/A
    Avg -2 ~ -7 0 ~ 0 N/A ~ N/A N/A ~ N/A 7 1.70 ~~~
    Min -6 ~ -20 0 ~ 0 N/A ~ N/A 0.00 ~ 0.00 6 1.23 N/A
    Stabilize Flight Time (Session)= 13 seconds, 0:13
    Stabilize Flight Time (Total)= 13 seconds, 0:13
    WARNING: Less than 9 Satellites was detected during this mode.

    24/01/2016 00:00:36 - 004853: Take Off
    24/01/2016 00:00:36 - 004860: Auto Mode is waiting for Throttle.
    24/01/2016 00:00:39 - 005396: SET_HOME
    24/01/2016 00:00:39 - 005397: Set Home as 32.7774900 -117.2217000 Alt:3.85
    24/01/2016 00:00:40 - 005732: *** Update Program to include this Event *** : 28
    24/01/2016 00:09:52 - 128890: LAND_COMPLETE_MAYBE
    24/01/2016 00:09:52 - 129070: Landed
    Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop Eff(mA/min)
    Max 91 ~ 298 17 ~ 38 3.04 ~ 1.89 202.73 ~ 665.11 18 1.8 N/A
    Avg 11 ~ 36 5 ~ 11 N/A ~ N/A N/A ~ N/A 16 0.81 554.57
    Min -6 ~ -20 0 ~ 0 N/A ~ N/A 2.70 ~ 8.85 9 0.71 N/A
    Stabilize Flight Time (Session)= 556 seconds, 9:16
    Stabilize Flight Time (Total)= 569 seconds, 9:29

    24/01/2016 00:09:54 - 129429: DisArmed

    Overall Flight Summary:
    Alt(m ~ ft) Spd(m/s ~ mph) Dist(km ~ mi) Launch(m ~ ft) GPS-Sats GPS-Hdop
    Max 91 ~ 298 17 ~ 38 3.04 ~ 1.89 202.73 ~ 665.11 18 2.22
    Avg 0 ~ 0 0 ~ 0 N/A ~ N/A N/A ~ N/A 15 0.83
    Min 0 ~ 0 0 ~ 0 N/A ~ N/A N/A ~ N/A 6 0.72
    WARNING: Less than 9 Satellites was detected during this mode.

    Power Summary:
    Battery(V) Vcc(V) Current(A) Used Cap(mAh) Eff(mA/mim)
    Max 16.15 0.00 56.86 5176 N/A
    Avg 0.00 0.00 32.71 N/A 545.80
    Min 13.60 0.00 0.62 N/A N/A
    Overall Flight Time = 569 seconds, 9:29

    *** Vibration Analysis has been removed from v2.2 onwards, you should now
    *** review the "Vibration Graph" instead to understand your levels of vibration.

    • have you run compass motor  calibration?

      http://copter.ardupilot.com/wiki/common-compass-setup-advanced/#com...

      • Ivan,

        I haven't tried that yet, but definitely sounds like I need to.  Have you had success using it?  If I have significant motor interference I would think it would be more necessary to move the compass away from the motors than to solve the problem by throwing a software setting fix at it, but maybe others have used it with good results?

  • you had tons of EKF variance errors.

    The barometer is most likely garbage, it jumps from 470 to 30m , and from 2m to 262m within one reading.

    Also, whenever you fly with an AHI , and see it's way off (like 50sec in the video), just land - it cannot go well.

    vibrations seem fine - I don't know what the cause to this problem is, but if barometer is behaving like that, there can be all kinds of problems with the IMU too (check for proper power to the sensor board/Raspberry, I don't mean voltage only check ripple too)

  • took a look at your file, you got some serious altimeter issues, also there seems to be a mechanical issue as roll/pitch dont follow the desired roll/pitch right before the crash

    I am not too knowledgeable with the Navio+, you might want to post on that thread thats around here somewhere

    2016-01-25_12-16-19.png

    2016-01-25_12-16-42.png

    2016-01-25_12-17-21.png

This reply was deleted.

Activity

DIY Robocars via Twitter
RT @chr1sa: Just a week to go before our next @DIYRobocars race at @circuitlaunch, complete with famous Brazilian BBQ. It's free, fun for k…
Saturday
DIY Robocars via Twitter
How to use the new @donkey_car graphical UI to edit driving data for better training https://www.youtube.com/watch?v=J5-zHNeNebQ
Nov 28
DIY Robocars via Twitter
RT @SmallpixelCar: Wrote a program to find the light positions at @circuitlaunch. Here is the hypothesis of the light locations updating ba…
Nov 26
DIY Robocars via Twitter
RT @SmallpixelCar: Broke my @HokuyoUsa Lidar today. Luckily the non-cone localization, based on @a1k0n LightSLAM idea, works. It will help…
Nov 25
DIY Robocars via Twitter
@gclue_akira CC @NVIDIAEmbedded
Nov 23
DIY Robocars via Twitter
RT @luxonis: OAK-D PoE Autonomous Vehicle (Courtesy of zonyl in our Discord: https://discord.gg/EPsZHkg9Nx) https://t.co/PNDewvJdrb
Nov 23
DIY Robocars via Twitter
RT @f1tenth: It is getting dark and rainy on the F1TENTH racetrack in the @LGSVLSimulator. Testing out the new flood lights for the racetra…
Nov 23
DIY Robocars via Twitter
RT @JoeSpeeds: Live Now! Alex of @IndyAChallenge winning @TU_Muenchen team talking about their racing strategy and open source @OpenRobotic…
Nov 20
DIY Robocars via Twitter
RT @DAVGtech: Live NOW! Alexander Wischnewski of Indy Autonomous Challenge winning TUM team talking racing @diyrobocars @Heavy02011 @Ottawa…
Nov 20
DIY Robocars via Twitter
Incredible training performance with Donkeycar https://www.youtube.com/watch?v=9yy7ASttw04
Nov 9
DIY Robocars via Twitter
RT @JoeSpeeds: Sat Nov 6 Virtual DonkeyCar (and other cars, too) Race. So bring any car? @diyrobocars @IndyAChallenge https://t.co/nZQTff5…
Oct 31
DIY Robocars via Twitter
RT @JoeSpeeds: @chr1sa awesomely scary to see in person as our $1M robot almost clipped the walls as it spun at 140mph. But it was also awe…
Oct 29
DIY Robocars via Twitter
RT @chr1sa: Hey, @a1k0n's amazing "localize by the ceiling lights" @diyrobocars made @hackaday! It's consistently been the fastest in our…
Oct 25
DIY Robocars via Twitter
RT @IMS: It’s only fitting that @BostonDynamics Spot is waving the green flag for today’s @IndyAChallenge! Watch LIVE 👉 https://t.co/NtKnO…
Oct 23
DIY Robocars via Twitter
RT @IndyAChallenge: Congratulations to @TU_Muenchen the winners of the historic @IndyAChallenge and $1M. The first autonomous racecar comp…
Oct 23
DIY Robocars via Twitter
RT @JoeSpeeds: 🏎@TU_Muenchen #ROS 2 @EclipseCyclone #DDS #Zenoh 137mph. Saturday 10am EDT @IndyAChallenge @Twitch http://indyautonomouschallenge.com/stream
Oct 23
More…