Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
I can understand not looking at older code - I don't like to do it either. However, keep in mind the possibility that RTL may trigger instability on climbs - the problem may still be hidden in later versions. If later I get the drone to fly right in the current version of the software I'll try again and post it.
Best regards,
Mihai
AC3.1.3-rc1 is now available through the mission planner's Beta Firmware's link. The only change between this version and AC3.1.2 is the stability patch bug fix. The fix is from Jonathan and Leonard and I'll post some more details on when the issue can occur once I've done some more testing.
If the patch looks good, we will do the official release on Friday so all feedback welcome!
Sincere apologies for this bug. We tested AC3.1.2 for months but this bug still got through.
This may be related. Also had a very odd problem, similar to this. I was in stable mode both times. Once leading to a near miss, then a few flights later It happened again and this time it crashed, really good onto a rock. Thinking a glitch in the APM or some issue with an ESC but all ESC's and motors work fine when tested later.
First log was the crash, radio got disconnected when it crashed and kicked in failsafe before I could disarm. Very strange, one minute I was buzzing around then, suddenly it started flipping uncontrollably.
Second log was the near miss, buzzing around then out of nowhere a hard roll right and right before I hit the ground I pulled out of it. The throttle dropped to near zero and I upped the throttle to pull out of it. here is a video also :33 http://youtu.be/ZkNNp7gq-gg
Log files
Crash log.log
ghost hard roll, Near miss.log
With no motor logs, it's really impossible to tell what went down there.
A very important note about this bug we fixed: it is not the primary cause of ANY glitch. It is only a potential *secondary* cause of crashes. So, you have an intermittent motor issue.
Thanks for the reply. I wish I had the motors and IMU logging on, but It's been flying great and I haven't had to "Tune" anything for awhile, so I had it off.
Should I leave it on for the future, even if it's flying fine? Thought I read someone that it eats CPU and should be left off if necessary.
Yes, logging eats CPU time. We're out of CPU time, so something else will fall off the back of the scheduler (e.g. telemetry.) It also eats log space, which we only have 2 to 4 MB of.
On PIXHAWK, this is a non-issue. We log a lot of stuff on PIXHAWK.
I would suggest turning on motors, leaving IMU off if you know your vibrations and such are good. I occasionally use IMU for log analysis, but not terribly often. I would log MAG, too, on the off chance that you will have compass issues that you need my help with (most of the issues I help with seem to be compass-related).
Hello
the bugfix 3.1.3 ArduCopter 7-Apr-2014 is in ReleaseNotes.txt. but not in the Mission Planner.
@ Leonard: Thanks a bunch, it's really helpful to get the straight story. I've also been reassured in the same way privately by someone who has been watching a lot of this closely. My understanding now is that the major risk factors include hauling heavy weight (relative to the motors) and high-throttle maneuvering, which are things not a lot of us are doing (though I maybe got lucky with my overweight, now retired Hex). It will nevertheless feel better to have it not be an issue at all! And I know I'm not the only one who puckered up over this, LOL.
@ Mike: Like many pilots I have two three-way switches set up to engage various flight modes. But I have the switch I always keep a finger on set so that its end position engages Stabilize no matter what position the second switch is in. Thus I always have my finger on a trigger that doesn't require any thinking or finesse. So, with just a little practice the switch into Stabilize can be nearly instantaneous, even when chaos strikes. It has saved me a couple of times.
-
1
-
2
-
3
-
4
-
5
of 12 Next