Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
They don't stop completely - I miss-spoke, but they are nowhere close to being able to sustain flight - since they are all equal, they don't provide stability either.
Motors are fine.
M.
More plane issues
http://ardupilot.com/forum/viewtopic.php?f=96&t=6899
http://ardupilot.com/forum/viewtopic.php?f=96&t=6850
Jonathan,
this is the signature of this bug: command followed shortly (100-400ms) by all motors going down - sometimes, 1 goes down first, sometimes not. Sometimes the craft recovers in time before crash, most of the times not. I had a case where the motors went all down only for 100ms, then recovered - the drone only went down 20cm or so in that 100ms.
Thanks for looking into it.
Mihai
One more thing Jonathan - what would help in debugging it is if we have visibility into the inav variables (beyond what INAV logs). How hard would it be to add an INAV2 that would log things like current bearing, stopping point, target point, and other variables in the AC_WPNav? I'm telling you that the stopping point calculation (and rtl skipping the climbing phase, but I think that's irrelevant) fixes the problem, at least for 3.0.1 rc1, but probably for more versions. Tomorrow we may try another version, as it turns out that 3.0.1 rc1 for some reason does not want to give its datalogs to mission planner.
Best,
Mihai
Actually I isolated the bug fairly well today. The most likely place for the bug is in library/AC_WPNav/AC_WPNav.cpp, in particular in get_stopping_point. Not sure where (as the comments of that function are horrendous), but somewhere it's blowing up at times. I changed the firmware today and it stopped falling from the sky.
In particular I did two changes:
1 - in RTL (commands_logic.cpp) I skipped the climb phase - asking it to directly return to home, and
2 - in get_stopping_point I just return the current position - not ideal, but better than crashing
These two alone do stop the carnage - I also lowered the max lean angle (45 is too much for my drone), but that is likely drone specific.
If you're looking for a mission, yes, I do have one that has 100% failure rate:
go waypoint 1, waypoint 2, then RTL at a higher altitude than waypoint 2. It will crash before it starts to climb. I'm not sure if it crashes on the SITL.
Thanks for looking into it.
Thank you for tracking it down Jonathan.
Unfortunately tomorrow the forecast shows 90% chance of rain. I'll certainly try the patch as soon as the weather permits! Was it in the navigation code? I know I can check, but I'm dying to hear :-)
Best,
Mihai
Hi Mihai,
The bug was actually in the stability patch. This is the last bit of code that makes sure all motor outputs are in the right range and ensures that yaw and throttle doesn't stop roll and pitch from leveling the copter.
So it the copter got a very large commands from the roll and pitch it could push the motor output to zero because we let a variable go negative. We were missing 7 characters, max(XXX,0). Where XXX was there.
So thank you very much for helping us find this problem!!
Mission accomplished!
OK, this makes sense, thanks for letting us know. I'll confirm when I can fly if it works.
Best,
Mihai
Hello
so some seems a little too drastic told what I read here.
The problem is not solved when the Qud on day 5 x crashes.
it would be very nice if one of the developers something to say about this problem-
-
3
-
4
-
5
-
6
-
7
of 8 Next