Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
Yes, by now I'm convinced it's a navigation code bug, so no related to the frame. I guess most people fly without navigation (RC control)? That code runs really nice - I was impressed today on how well a hex was doing in 30mph+ wind gusts. Alternatively, judging by the number of bug fixes in navigation code from 3.0.1 to 3.1.2 they know they have a problem but couldn't just put their finger on it. I looked at the code again today and it's... tough. It's not properly commented or documented (I guess with a proper documentation the comments would be sufficient).
Mihai
I have noticed quite a bit of unexplained LOITER and RTL issues when i was trying to solve a plane 2.78 RTL / LOITER issue.
I have no idea if they are related to the bug you describe, but if it helps the developers, here are some unexplained events tied to LOITER..
My plane issue...
http://diydrones.com/forum/topics/apm-2-5-channel-1-servo-twitching...
Another plane issue
http://diydrones.com/forum/topics/apm-2-5-locks-servos-and-spirals-...
Unanswered Copter issues (seem to be loiter related)
http://ardupilot.com/forum/viewtopic.php?f=25&t=7037
http://ardupilot.com/forum/viewtopic.php?f=25&t=6546
http://ardupilot.com/forum/viewtopic.php?f=25&t=6455
Yesterday it happened on an auto mission between two way points. Usually it was happening on an auto mission after a waypoint and before RTL, but it turns out that even between points it can happen.
It's not happening every loiter: I can have 10 minutes worth of loiter at a time without a problem, then a second battery good, then the third one crash. I think it's dependent on that is the current position and the target - in some conditions there is a calculation that blows the internal navigation state of the system and leaves everything in disarray. I just was unable to figure out what are the exact condition (or even better, what is the code responsible for it) - a couple of days ago I thought it only occurs when the navigation origin and destination are on top of each other, but yesterday I was shown that they can be some distance apart and it may still bite.
You may not have the issue - nobody else seems to have it. It may be something with my setup that I don't see, but it certainly looks like a navigation firmware problem.
Mihai, post an issue here: https://github.com/diydrones/ardupilot/issues?state=open if you feel there is one.
I don't think it's the right place to put an issue. I've seen many issue's there that are waiting for an answer.
Well, technically that it supposed to be the best place - the developers are certainly watching that list of bugs probably more than they watch the forums, but they may have bigger fish to fry and not have the time to chase smaller ones.
Best,
Mihai
Ah... that's a good spot to post - thanks for the heads up. I'm heading that way now - I didn't know they have a place for this kind of problems. I am certain that it's a firmware problem - I don't understand why not more people have the issue - can it be that there is not a big community using auto with loiter at intermediate points and using 3.x firmware?
Best,
Mihai
Posted - hope it will help.
M.
New hot data right in from an RTL crash this morning:
- this crash was also on the HEX X frame
- this new crash was on an old firmware version (3.0.1 rc 2)
- the crash does *not* occur every time. See the attached log. First try I tried auto it went well, second time it crashed
- I found that every time the RTL with positive change in altitude happens the desired roll becomes -45 degrees (which is the maximum angle limit) - that's why the first motors go down - it tries to get to -45 degree roll. Not sure why - I checked the code and the code explicitly tries to avoid that by trying to set the desired pitch and roll at the current pitch and roll (limited by +/- max angle). For some reason the result is -MAX_ANGLE_LIMIT (-45 degrees).
- sometimes, (first RTL) the drone just jitters, then recovers, but
- sometimes the drone does not recover (despite the fact that the desired roll recovers and the throttle out is maxed out, the motors go all the way down.
- it seems that RTL takes different paths in the code if it needs to climb or not, and if current speed is low or not. I have a 5 second loiter before RTL, so my speed at the time of RTL is really low (probably zero) and I need to climb 2 m.
BTW, I like the code structure otherwise.
rtlCrash.log
@Mihai: please post here the mission file, thanks!
-
4
-
5
-
6
-
7
-
8
of 8 Next