Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
Marco - the mission file is readily available from the log file - if you're on linux or mac go
grep CMD filename.log
This is what comes from the log above:
CMD, 6, 0, 16, 0, 0, 0.00, 35.7712980, -78.6745357
CMD, 6, 1, 22, 1, 0, 3.00, 0.0000000, 0.0000000
CMD, 6, 2, 19, 1, 5, 3.00, 35.7711040, -78.6746624
CMD, 6, 3, 19, 1, 3, 7.00, 35.7710560, -78.6744256
CMD, 6, 4, 16, 1, 0, 7.00, 35.7711552, -78.6744832
CMD, 6, 5, 20, 1, 0, 0.00, 0.0000000, 0.0000000
CMD, 6, 0, 16, 0, 0, 0.00, 35.7712385, -78.6744334
CMD, 6, 0, 16, 0, 0, 0.00, 35.7712395, -78.6744446
CMD, 6, 1, 22, 1, 0, 3.00, 0.0000000, 0.0000000
CMD, 6, 2, 19, 1, 5, 3.00, 35.7711040, -78.6746624
CMD, 6, 3, 19, 1, 3, 3.00, 35.7710560, -78.6744256
CMD, 6, 4, 16, 1, 0, 3.00, 35.7711552, -78.6744832
There were two missions back to back - the last one ended in a crash. It occurred to me that there might be something special with that initial mission, so after a while I started to change them - to no avail.
Two more crashes (from this evening) attached. One with firmware 3.1.2 (didn't compiled it this time - straight from the web), and one with 3.0.1 rc2. Both do the same thing. It seems to be an error in the navigation resulting in a huge desired acceleration (both x and y) and then corresponding huge desired roll and pitch (really saturated in the limits) - BTW, for 3.0.1 we changed the max and min to 25 degrees - it did limit the desired roll and pitch, but did not prevent the crash.
M.
2014-04-01 20-38 1.log
2014-04-01 19-51.log
Mihai, do you want make a new world record for number of crash in a month? :-)
When you see your quad ungovernable have you tried to switch in Stabilize/Acro for recovery?
I've done a lot of mission with my quad and never have these problems but of course, without high positive altitude variation, need investigation.
What I was trying was to figure out what happens and to see if I can rule out other things - I did rule out a lot. Last crash (yesterday afternoon - not shown yet, I'm attaching it here) shows a crash without RTL - just a waypoint after another, not big height variation - it was supposed to do the yaw change in place before going to the waypoint, but instead it crashed like all the others (with all the motors down).
Initially I didn't try to recover it, but lately I started to flip the switch back to stabilize (I think last 3-4 crashes)- visually it seemed to have helped (it looked to me that the drone re-flattened itself), but logs show that the flattening had nothing to do with flipping back to STABILIZE: the motors continue to stay down (to the exact same value as before the switch 1098 in this attached log) - so the recovery was most likely due to passive aerodynamic effects.
I'm truly desperate - last night I spent quite a bit looking through the auto-navigation code, but I don't know enough to debug it.
2014-04-02 16-07.log
LOL @ Marco's comment!
Is this something related to pendulum effect?
Sorry - what is the pendulum effect? Without know that I'll try an answer (I know, it sounds stupid): the drone does not oscillate at all - just leans over, stops the motors and falls like a brick. If it's about weight distribution having weight too heigh - again not - my drones are flat like a pancake everything is within 2 inches of the surface except a WiFi antenna and the GPS which weigh almost nothing (compared to the drone which is a beefy 2Kg). In fact stability is exceptional - we had it yesterday fly in ~15 mph wind and was doing great - not a single moment of doubt. Until motors shut off.
Oh that's serious problem right there! Maybe another motor sync problem, by the way what esc's you used?
Hi Marco
with Pixhawk
Yesterday I had such a problem, RTL after 20 seconds the copter lost orientation, and went with 45 degree inclination to the ground. switch to Stabilize, no control.
I am grateful for any idea
please check the logs
https://www.dropbox.com/sh/qimdohk3oflvh5y/QPYjD1o4gb
Rainer,
I looked at your log briefly and I didn't see any crash - maybe it's the wrong log? I see a lot of Stabilize, Alt-Hold, Loiter, come down to land, and repeat, but barometric altitude does not seem to indicate a crash - it does end somewhere in the air (30 meters up), which is weird, but this is not what I'm seeing - what I see is the whole crash all the way down (and after) - in your case it just stopped logging - maybe a power outage, or some other crash of the software - in my case only the variables blow up, not the entire software - the logging continues in my logs.
Thanks for the heads up to the problem, i am on the last steps of setting up my hexacopter, so now i will be careful and will not use loiter nether auto
-
8
-
9
-
10
-
11
-
12
of 12 Next