Hi there,
I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms. The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:
RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767
Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.
The RTL example is similar - the RTL command is executed right after the first RCOU:
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.
In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.
In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.
If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.
Best,
Mihai
Replies
Is having the APM centred important? I thought a bit about it and I thought that at least the accelerometers ought to show about the same values, and if not calibration would take care of it. I have to say that stability is usually just fine. The hover is usually around 50%, so there ought to be enough thrust for stability. Flying it in 30mph gusty wind also convinced me that stability is great. I was honestly impressed by it.
M.
If the APM is not on the CG, the accelerometers will measure movement when the copter rotates. In position control modes, APM will try to counteract this movement by tilting the copter, causing potential coupling, and by applying a bit of throttle to keep the APM at the same height, which shouldn't be too bad.
There's another factor, and that is that the GPS is compared to the accelerometers for inertial navigation. If the GPS and the accelerometers are in different physical locations, and the copter is rotating, then you get errors in the inertial navigation. Shouldn't be a big deal, though.
Ultimately I wouldn't have APM more than probably 20cm from the CG, but I wouldn't worry about it too much.
Ah, thanks - I'm not that far from CG - probably 10-15cm. And the GPS is exactly on top of the APM (as far up as the cable allows it).
Thanks,
Mihai
This is an image of the before and after the stability patch bug fix. The critical issue is that the overall throttle is being reduced below mid throttle (aka THR_MID parameter) when the combined throttle+roll+pitch+yaw is very high. In the image below you'll see an orange box with "PROBLEM!" written near it.
As mentioned in some other comments in this thread, this is a bug in the stability patch. So the stability patch's purpose in life is to deal with these situations where the requests on the motors are simply impossible. It's suppose to make the best decision to keep the copter in the air while pointing in the correct direction. Unfortunately because of a small bug (fixed here) it was allowing large attitude inputs to overwhelm the throttle request and it would shut down the motors.
In my testing on a copter with default parameters and running at full throttle, the problem starts appearing when the rotation error is about 333 deg/sec. This would come from a 74deg attitude error. The motors are being set to minimum at about twice that. High stabilize and rate P and D terms (like those calculated by autotune) make this more likely to occur.
If you're wondering if this has happened to your copter, it's only visible if you enable the RCOUT (aka MOTORS) dataflash logging. With that on you can see the CTUN's ThrOut will be much higher than the average motor level (i.e. average the pwm value for all motors on the vehicle).
The best solution is to load AC3.1.3-rc1 onto your copter. This version is still in testing however so if you're uncomfortable not using an official release just wait until the weekend and AC3.1.3 should be out.
Again, I'm really sorry about this issue.
Randy, I tried this AC3.1.3-rc1 fw this morning but I notice twitching on RTL. Here's my logs.
4-11-2014.zip
Randy,
Even with testing and doing everything you could to ensure no bugs, they will persist. No software company ever releases a 100% bug free product. There are just too many variables and unforeseen circumstances out there. But the difference here is it was fixed, AND it was documented and acknowledged. Other companies would have just patched it and not said anything. Labeled it "stability and flight enhancements" to hide the true nature of what was happening. This is a perfect example of why open source can lead to amazing results, and beat the best closed source software in every regard. My trigger was the RTL climb command to from breaching the GEO fence too. Thank you for your work on this, and continuing to improve the code for us that are not code savvy!
jg, very well said!
+100
I reported a very similar incident in the support forums not long ago. The analysis was that the copter was under powered and flying with too high of a hover rate. Is there a possibility that the high hover rate pushed the copter in to a situation that tripped this bug? The motors didn't totally shut down though, they just went to idle.
In this bug the motors don't shut completely down - only very nearly so (~1200 in my case), but at that value nothing flies - just falls down, and does not attempt to recover stability (with no motor input the drone keeps doing what it was doing (e.g., rolling) + gravity). May very well be. As far as I can tell from testing yesterday the secondary problem has been taken care of (i.e., motors do not come down anymore). A climbing RTL still causes instability, but the drone may recover from that (or not): yesterday in two trials, it recovered once and failed once. Both runs are visible in the attached log - in both cases the drone gets a very large input and in the first case it recovers and in the second it doesn't (but does not bring the motors to idle).
Best,
Mihai
stillRTLCrash.log
Mihai,
The fact you are running your own version of 3.1_rc8 makes it hard to support you because it isn't the usual code and to be honest I have forgotten what has been changed and fixed in that time.
If might want to reduce Angle_Rate_Max to 9000 and see how you go with that. What can happen is with some copters (large ones especially) is the rotation rate can build up to a point the copter can't stop in time. This parameter stops that build up past the specified rate.
At this point it looks like a tuning problem though. Hopefully the suggestion above fixes it. In the next version Autotune should set this for you.
-
1
-
2
-
3
-
4
-
5
of 12 Next