Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
Leonard,
Lots of trials today and here's where we're standing:
- first, when I tried the patched 3.1.2 for some reason I didn't get stable performance on my drone. Even stabilize was a bit wobbly, but in loiter was quite unstable to the point that after a bit destabilized itself and crashed.
- half an hour later I regressed to 3.1.0 rc_8 which was very stable for me before, after applying the same patch (the 7 missing characters). A few observations:
- 3.1.0 rc_8 with the patch was much more stable than 3.1.2 - probably unrelated to the patch. Navigation worked fine. Reduced WPNAV_Accel (50) and reduce HLDLat_P and HLDLON_P did miracles to calm things down.
- However, as soon as I tried RTL I saw a wobble the first try and then it came down the second time. However, it was not nearly as bad as it used to be: instead of shutting down motors, when it went down it went unstable. Super stable until then, then RTL then went unstable and lost the 4m altitude it had at the time. So... there are two problems: Yes, the patch solved the biggest problem that was leading to shutting down the motors, but RTL still has a problem. Small, but existent. I'm not sure if you have time to chase it down.
- One more test after that: I made a mission where two consecutive waypoints were perfectly on top of each other (to simulate an RTL) and that went well without a single wobble. So the problem is only with RTL, and only with the code that executes when the current altitude is smaller than RTL_alt.
I'd guess that either set_yaw_mode(YAW_HOLD); or in setting the origin and destination perfectly on top of each other.
Either way, I just disabled that part of the code (i.e., RTL returns are diagonal regardless of the current altitude and rtl_return_altitude) and everything works great. I know that it's not ideal, but it certainly beats crashing.
Everything else seems to work fine and this evening on this configuration we had some great flying tests with super-locked-in loiter and very calm navigation - BTW, any idea why we're doing much better satellite-wise during the night? During the day we have 8-9, during the evening we have 11-12, sometimes 14.
Best,
Mihai
Thanks - it was heart-breaking. I have a big event this coming Saturday (4/12) involving programming drones and this was a show stopper. We've lost more than three weeks tracking this down and now we're scrambling to catch up. That's why I hesitated to take John's offer to skype - if he still wants to hear from me, I'll be available next week.
Let's hope that this solved the problem.
Best,
Mihai
thanks mihai
Sorry to hear about your crash Mihai!
From what I can see your throttle did exactly what it was supposed to do. The angle boost function drops the throttle to zero as the copter roll's or pitches between 60 and 90 degrees.
It looks like your number 6 motor (left rear) was dying the whole flight being near maximum throttle the whole flight (and number 5 was low to compensate for it).
This motor finally died and the copter dropped on that side until the angle dropped the throttle.
Now the problem is that the copter should have kept trying to right the copter even though the throttle was at minimum. And in this case the average motor output should have been somewhere near the middle of the range because it was so bad.
So your crash was caused by a faulty motor, however, it should not have been this bad because the motors should have been able to stop it going all the way over to inverted.
Sorry again for your crash and I hope we can report back with more info!!!
Leonard,
thank you for taking the time to post in this thread. However, I believe that you're not quite correct in your assumptions. It's a big thread and you may have missed a few details:
- first, it's not *a* crash. There are more than 10 (after about 10 I stopped posting them - they don't add anything).
- second, it's not a drone - there were three different drones that crashed repeatedly - always with the same signature - a command followed by one or two motors stopping, followed by all motors stopping. Sometimes starting again (and saving the drone), and sometimes not starting.
Furthermore, at some point in the thread I pointed out that I believe I isolated the bug - it's probably in the "get_stopping_point()" function in the waypoint navigation file - if I change that function to return the current position the bug does not occur. However, I admit that the function is needed and should not be disabled the way I did it - it just needs fixing.
Finally, regarding additional info: I posted 10 or so crashing logs and pointing at the likely source of the bug. I can also post tlogs (which are mostly redundant in my experience) and I can post the video if you'd like to see it. If it helps.
Best,
Mihai
Hi Mihai,
Yeh, I came into this thread a little late when John asked me to have a look. This particular bug is a hard to find one because it is disguised behind other problems (or simply large command inputs) that you may not ever notice otherwise.
There are three factors that must come together to cause this bug to be seen.
1. High throttle command
2. High roll output to motors
3. High pitch output to motors
and it helps if yaw is high too.
The problem is all these things tend to happen if you are experiencing any sort of motor problem or cg offset. The final problem looks very similar to a lost prop or motor unless you can see the motor logging and generally that is turned off.
So thanks for being persistent with your posts, you have made a major contribution to the code with your find!!!
Leonard, can we add another safety feature then? If it detect a faulty motor it force to land instead. What do you think?
@Jay - there was no motor failure: all my motors (on many drones) run fine - the posts above come from 3 different drones, all of which flew again and crashed again and flew some more. If a motor would fail, you'd see the APM try to rev it up - instead the APM commands them to stop - first a few then all at the same low value. Not idle, not stopped, but low enough to fall (e.g., 1231 or 1241).
Best,
M.
I had a crash yesterday. The copter was out about 100 yards and I hit the RTL on the transmitter right after it hit the 1st of 2 waypoints. The copter tilted to the left hard and went down. Please if someone could take a look at my attached long and see if I may have been the cause of this or is it something in the software.
Thanks
2014-04-05 20-15-13_62774.log
2014-04-05 20-15-13_62773.log
Regards, Soren
-
2
-
3
-
4
-
5
-
6
of 8 Next