Hi All,
Today's flight felt very unusual. No crashes, but I frequently felt my quad was behaving strangely.
I checked the log file and note that over my 20 minute flight, it threw the error EKF_CHECK-2 (subsystem 16, error code 2, bad variance) 20 times, or about once a minute. It would persist for a few seconds to 30 seconds, then go back to normal. I've never seen this error on previous flights.
Also important, every time I would enter Loiter mode, my quad would start flying in an unexpected direction, so I would need to switch back again.
I'm flying a BBBMINI, with the code pulled off the master branch back in Nov, 2015. The commit is 802ced20. I've often thought I should move back to a stable release at some point, but I've never seen any problems with this build, until today.
The onboard log file is 9 MB, exceeding the 7 MB limit, so I've attached a link to a copy in my drobox folder:
Any troubleshooting tips would be greatly appreciated!
Replies
I never did get much response on this thread (except for you Micha, thanks!). I'm guessing that is because EKF errors can be difficult to troubleshoot.I did finally find out what was wrong with my quad. I'm updating this thread so that anyone in the future who has similar symptoms might get a head start.
It turned out to be a compass problem. I have two compasses on board. One came with the GPS module I put on my quad, and one is on the MPU9250 gyro/accelerometer I use with my Beaglebone Black. I had set up the GPS module compass to be my primary compass since it is furthest away from my electronics. However, unbeknownst to me, that compass had stopped working, and my computer switched to the MPU9250 compass.
Here is the problem... my GPS compass required that I set COMPASS_ORIENT = 12 (pitch 180 deg), whereas the MPU9250 was in the default orientation (COMPASS_ORIENT2 = 0). For some reason, the computer decided to move the MPU9250 compass over to the first slot slot, meaning it was configured for the wrong orientation.
The bottom line is that with the wrong orientation, the compass was showing the yaw rate going the opposite direction compared to the gyro, which really confused EKF. This lead to some really erratic flight, especially in loiter mode.Now I have completely disconnected my GPS compass, and changed COMPASS_ORIENT = 0. I flew this morning, and the flight was BEAUTIFUL. Easy to fly, and no issues seen in the log file.
Anyway, I hope this rambling diagnosis helps someone facing a similar issue in the future. One thing I am curious about... how do I designate a specific compass to go in a particular compass slot. It seems to me that if the MPU9250 had stayed in the second slot, everything would remain okay.
Ok Larry, i ran your log through Apm log analyzer and you get a lot of EKF/ Inertial Nav Check ERROR: Bad Variance errors.
Read this post: http://diydrones.com/forum/topics/arducopter-erroneous-autopilot-ca...
According to Bill you should do a compassmot and take a look at the results.
Still looking for some tips on how to go about diagnosing the EKF errors. I'm suspecting a bad sensor, and am nervous about taking my quad out again before I have a good idea on what might be going on. Here is a screenshot of the EKF estimated North/East position. Notice how the filter believes the quad has a south acceleration. The position seems to snap back to its correct position every 10-15 seconds or so. If the units are meters, that's a pretty big jump. When flying, I did notice the quad was trying to pull one direction, and wind didn't seem to be enough to explain it.
EKF position and velocity innovations, below.
I'm going to bump this thread. I posted a week ago, still stuck and looking for answers! I'll bet there are a few folks out there with some familiarity with EKF who can tell me what might be going on. Note: I've had many hours of successful flight time on this quad without this issue. Now I'm a little nervous about taking it out again.
Incidentally, see plot below for GPS latitude. The trajectory below is fairly smooth, not much noise. The delta over the 2 minute time interval was 0.0004 deg, or about 7e-6 radians. This corresponds to about 44 meters over the two minutes, which is consistent with my flying. Bottom line is I don't think the GPS is driving these EKF errors, but I'm open to someone convincing me otherwise.