Hi all,
I'm attaching my log from our last flight as our little quad went completly bonkers before it crashed.
Short after takeoff it went completly out of control, no response at all from RC or Telemetry and then crashed near a pond (thank god).
But the mistery was about to begin when we started reviewing the log and found out that the trigger was the failsafe error Err: EKF_CHECK-2 / Err: FAILSAFE_ EKF-1.
In understand these two were recently added to APM, but this ain't the first time I see this sort of thing happening.
I'm attaching my log in hopes for some guidance.
Thank you all!
Replies
This string has been quiet for some time, so we'll see if anyone is watching.
I experienced the problem being discussed, just last Friday. I have been flying a F450 frame with Pixhawk for some time now. It has been flawless till now. I was flying in a field I fly in most often. Its a soccer field and farm field without a power line or concrete in sight. I was flying a mission that I have flown many times, perfectly. So, during a circle at about 60 meters I realized something wasn't right. I took it out of auto and flipped it into RTL - It did not! At that moment my tablet (running Tower) indicated LAND. I don't think so! I switched into stabilize and luckily got it home.
I am going to attempt to attach my log (haven't done this before) and see if anyone has any ideas. Prop balance is as perfect as humanly possible. A hindsight, prior to this flight I had noticed that PosHold was not as solid as I am accustom to.
Thoughts, suggestions or advise is welcome. I can use MP and review logs but admit I don't understand all the data. Thanks, Dave
2015-09-25 10-34-40.log
I have just had a similar issue. At a certain point the log is full of EKF_CHECK-2 and FAILSAFE_EKF-1. I am running an APM2.5 with onboard magnetometer and external GPS. I have never had this error before I changed gimbal mount and used a mroe "solid" connection to the F450 frame. I must anticipate that the gimbal's batteries were nearly dead so it was "glitching" every now and then trying to keep the camera stabilized. Maybe the glitch (a "twitch" in pitch, not roll) influenced the EKF error or maybe not. I must also say I had disabled the EKF check completely and used a "relaxed" setting for the Yaw/magnetometer variance check (never logged any bad DCM failsafe alarms). The quad was on a mission and at the 3rd waypoint it failsafed to LAND so I triggered it back to stabilize and Auto again to resume. After the 3rd attempt I couldn't get it back to RTL or Stabilize (RTL was sent via Tower App) from radio or PAD. It just drifted off and set to LAND, unfortunately on some trees. I could not find it as it became dark and the GPS had lost fix (telemetry was lost at a certain point butwalking towards it I got a signal) but recovered it this morning with daylight. I downloaded the logs and attached them FYI. I also have the footage from my gopro if anyone is interested in the mission "FAIL" :)
https://www.youtube.com/watch?v=jlvdIQP82gg
From 0:47 the mission starts. At 1:12 waypoint 2 is reached and Waypoint 3 is targeted. You will notice the gimbal twitch the pitch now and then as the batteries are dieing but when it does FAILSAFE is not triggered apparently so maybe the 2 things are unrelated. At 1:40 LAND is triggered so I switch to Stabilize and Auto again for it to resume the mission which it does (1:54). The gimbal battery is now practically too weak to keep the gimbal stable and the twitching increases. @2:27 the waypoint is reached but LAND is triggered again due to EKF Failsafe. I switch to stabilize and auto again but it fails to LAND again. A@2:52 my friend (from the Samsung Note) triggers RTL but the drone flies off in the wrong direction. I switch to Stabilize again and try to bring it back but have a hard time figuring out where the front is (it was dark and grey). I see it fly away behind a tree and try to give throttle to bring it up but nothing happens (3:40). I switch to Auto and stabilize but I get not response. I see it fly further away (4:00) and lose altitude. Telemetry is then lost behind the trees (it was about 800meters from us by now). The drone continues to drift (why?) and just "lands" on a tree.
I have just had a similar issue. At a certain point the log is full of EKF_CHECK-2 and FAILSAFE_EKF-1. I am running an APM2.5 with onboard magnetometer and external GPS. I have never had this error before I changed gimbal mount and used a mroe "solid" connection to the F450 frame. I must anticipate that the gimbal's batteries were nearly dead so it was "glitching" every now and then trying to keep the camera stabilized. Maybe the glitch (a "twitch" in pitch, not roll) influenced the EKF error or maybe not. I must also say I had disabled the EKF check completely and used a "relaxed" setting for the Yaw/magnetometer variance check (never logged any bad DCM failsafe alarms). The quad was on a mission and at the 3rd waypoint it failsafed to LAND so I triggered it back to stabilize and Auto again to resume. After the 3rd attempt I couldn't get it back to RTL or Stabilize (RTL was sent via Tower App) from radio or PAD. It just drifted off and set to LAND, unfortunately on some trees. I could not find it as it became dark and the GPS had lost fix (telemetry was lost at a certain point butwalking towards it I got a signal) but recovered it this morning with daylight. I downloaded the logs and attached them FYI. I also have the footage from my gopro if anyone is interested in the mission "FAIL" :)
2015-11-03 14-02-10.zip
Forgot the log
2015-05-10 14-16-23.bin
Here's a flight with 3.3rc3++ built from source and manual declination. This looks a lot better to me in EKF and bad but not awful in DCM. Would you agree?
Have a look at EKF4->SMX at 12mins. There is a huge jump and it is again at the point at which MAG->MagY goes sharply down and crosses from +ve to -ve. I have seen this pattern before - downward transition crossing zero causes big error. Does this not look a bit suspicious?
Question about the code - this generally occurs when very severe yaw values are requested - i.e. sudden change in yaw. This seems worse in auto as in auto the mission will turn as quickly as possible (relative to my thumbs). But you can also see that the yaw error shoots up to one almost immediately in that log, but not elsewhere. Why would this be? Is the IMU struggling to keep up with severe yaw? Will the increase in sensitivity in 3.3 help? I have had similar experiences all along - yaw error will be fine and then suddenly go ballistic, there doesn't seem any consistency to it other than the severity of the yaw that is initiated.
Thanks Paul, this is really helpful!
I had picked up (4) as a possible cause as I had experienced toilet bowling later on in the day and google indicated that auto dec could be an issue. Could (4) explain everything? Why is auto dec wrong, could it be something to do with my fairly northern latitude? I have manually entered it now and will see what the results are like.
Because I've had such bad compass problems (faulty 3DR compass) I do the compass dance all the time inside and outside and compare the offsets. They are always pretty consistent. How close would you expect them to be to be right? e.g. (-102, 7, -28) vs (-121, 1 -21). Is it worth switching on in-flight compass learning?
It was fairly windy that day and issues (TBE in particular) showed up hedaing into the wind. Does that have any bearing?
Thanks! This is really helpful but now I am really confused. I am trying to interpret the attached log. You can see that ErrYaw spikes to 1 and likewise EKF4->SMX/SMY but I am really struggling to understand why. On the face of it it is the compass. You can see that MagY jumps at the times the error jumps and similarly the yaw jumps, but there is no noticeable interference against the throttle and the compass is mounted 20cm above the PDB. I have also seen similar effects with a different compass. I'm guessing you would expect some yaw error when the yaw changes quickly (this is an auto mission), but the big spike occurs sometime after a big yaw change, pretty much coincident with the MAG values going negative.
Are spikes in SMX/SMY expected as yaw changes quickly or should they not spike this much? I suppose you could argue that EKF is doing a lot better here than DCM. The EKF->SMY spike only goes above 0.5 briefly and only to 0.56. The DCM error takes a lot longer to settle down.
Should I be worried?
Files attached
2015-05-04 14-51-36.bin
EKF.jpg
Thanks Paul, for the insightful analysis! Is there a write-up anywhere of how to analyze EKF log data?
BTW this illustrates what I have been thinking the past week - namely that ErrYaw is misnamed and misleading.
I dunno what would be better - ErrNav? ErrVel?
I have just checked my logs and up until the point of the error (at which point they skyrocket) I have very little vibration all of which is well within the bands that the wiki says is acceptable. The peak in vibrations seems to coincide with the loss of gps and after the crash all levels revert to normal apart from acc z which has flipped from -10 to 10. Weird.
77.BIN
-
1
-
2
-
3
-
4
-
5
of 7 Next