LOITER and RTL crashing bug

Hi there,

I'm writing to report a very likely bug affecting at least firmwares 3.1 rc8 to 3.1.2 (), at least for hex X frames. The bug manifests when RTL is the last step of an AUTO mission and RTL return altitude is different from the current altitude. The drone crashes consistently in these conditions. Additionally, the drone crashes from LOITER after a non-deterministic time - sometimes is quick (tens of seconds), sometimes it can loiter for minutes. In both cases the symptoms are the same: the drone goes from hover holding output (motors ~ 1600) to shutting down 1-2 motors in 100-200ms, and then shutting down the others in another 100-200ms.  The way it shots down the motors is very specific: it reduces throttle to 12xy where xy is the same for all motors. In the attached logs for example, after LOITER the motor commands are as follows:

RCOU, 487396, 1695, 1645, 1672, 1667, 1443, 1874, 32767, 32767
RCOU, 487495, 1523, 1597, 1517, 1602, 1231, 1824, 32767, 32767
RCOU, 487595, 1398, 1421, 1368, 1451, 1231, 1581, 32767, 32767
RCOU, 487696, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487796, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487897, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 487996, 1231, 1231, 1231, 1231, 1231, 1231, 32767, 32767
RCOU, 488096, 1329, 1444, 1401, 1372, 1231, 1545, 32767, 32767
RCOU, 488195, 1392, 1380, 1545, 1231, 1299, 1474, 32767, 32767
RCOU, 488297, 1404, 1369, 1501, 1271, 1231, 1545, 32767, 32767

Obviously 1231 is not sufficient to sustain flight - the fact that all motors have the exact same value (1231) reinforces the likelihood of this being the output of a bug.

The RTL example is similar - the RTL command is executed right after the first RCOU:

RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767
RCOU, 742799, 1697, 1603, 1542, 1756, 1571, 1728, 32767, 32767
RCOU, 742898, 1241, 1241, 1241, 1241, 1241, 1241, 32767, 32767

This time the motors are all 1241 - interestingly different from 1231, but still very close, not nearly enough for flight and all equal in all motors.

In case you wonder if there is something with the drone, both bugs have been reproduced in two different drones several times: the loiter crash three times and the RTL crash twice.

In case it helps, in addition to the dataflash logs I have telemetry logs from mission planner as well as videos, but I assume that the dataflash logs are sufficient, as they had enabled practically everything relevant (I think) - certainly they had ATT, CTUN, NTUN, IMU, RCOU, GPS, and RCIN.

If this is a known bug, please drop a line here, if I can help in tracking it down, let me know.

Best,

Mihai

loiterCrash2.log

rtlFromAutoCrash.log

You need to be a member of diydrones to add comments!

Join diydrones

Email me when people reply –

Replies

      • Marco - the mission file is readily available from the log file - if you're on linux or mac go

        grep CMD filename.log

        This is what comes from the log above:

        CMD, 6, 0, 16, 0, 0, 0.00, 35.7712980, -78.6745357
        CMD, 6, 1, 22, 1, 0, 3.00, 0.0000000, 0.0000000
        CMD, 6, 2, 19, 1, 5, 3.00, 35.7711040, -78.6746624
        CMD, 6, 3, 19, 1, 3, 7.00, 35.7710560, -78.6744256
        CMD, 6, 4, 16, 1, 0, 7.00, 35.7711552, -78.6744832
        CMD, 6, 5, 20, 1, 0, 0.00, 0.0000000, 0.0000000
        CMD, 6, 0, 16, 0, 0, 0.00, 35.7712385, -78.6744334
        CMD, 6, 0, 16, 0, 0, 0.00, 35.7712395, -78.6744446
        CMD, 6, 1, 22, 1, 0, 3.00, 0.0000000, 0.0000000
        CMD, 6, 2, 19, 1, 5, 3.00, 35.7711040, -78.6746624
        CMD, 6, 3, 19, 1, 3, 3.00, 35.7710560, -78.6744256
        CMD, 6, 4, 16, 1, 0, 3.00, 35.7711552, -78.6744832

        There were two missions back to back - the last one ended in a crash. It occurred to me that there might be something special with that initial mission, so after a while I started to change them - to no avail.

    • Two more crashes (from this evening) attached. One with firmware 3.1.2 (didn't compiled it this time - straight from the web), and one with 3.0.1 rc2. Both do the same thing. It seems to be an error in the navigation resulting in a huge desired acceleration (both x and y) and then corresponding huge desired roll and pitch (really saturated in the limits) - BTW, for 3.0.1 we changed the max and min to 25 degrees - it did limit the desired roll and pitch, but did not prevent the crash.

      M.

      2014-04-01 20-38 1.log

      2014-04-01 19-51.log

      • Developer

        Mihai, do you want make a new world record for number of crash in a month? :-)
        When you see your quad ungovernable have you tried to switch in Stabilize/Acro for recovery?
        I've done a lot of mission with my quad and never have these problems but of course, without high positive altitude variation, need investigation.

        • What I was trying was to figure out what happens and to see if I can rule out other things - I did rule out a lot. Last crash (yesterday afternoon - not shown yet, I'm attaching it here) shows a crash without RTL - just a waypoint after another, not big height variation - it was supposed to do the yaw change in place before going to the waypoint, but instead it crashed like all the others (with all the motors down).

          Initially I didn't try to recover it, but lately I started to flip the switch back to stabilize (I think last 3-4 crashes)- visually it seemed to have helped (it looked to me that the drone re-flattened itself), but logs show that the flattening had nothing to do with flipping back to STABILIZE: the motors continue to stay down (to the exact same value as before the switch 1098 in this attached log) - so the recovery was most likely due to passive aerodynamic effects.

          I'm truly desperate - last night I spent quite a bit looking through the auto-navigation code, but I don't know enough to debug it.

          2014-04-02 16-07.log

          https://storage.ning.com/topology/rest/1.0/file/get/3702753395?profile=original
        • LOL @ Marco's comment!

          Is this something related to pendulum effect?

          • Sorry - what is the pendulum effect? Without know that I'll try an answer (I know, it sounds stupid): the drone does not oscillate at all - just leans over, stops the motors and falls like a brick. If it's about weight distribution having weight too heigh - again not - my drones are flat like a pancake everything is within 2 inches of the surface except a WiFi antenna and the GPS which weigh almost nothing (compared to the drone which is a beefy 2Kg). In fact stability is exceptional - we had it yesterday fly in ~15 mph wind and was doing great - not a single moment of doubt. Until motors shut off.

            • Oh that's serious problem right there! Maybe another motor sync problem, by the way what esc's you used?

        • Hi Marco
          with Pixhawk
          Yesterday I had such a problem, RTL after 20 seconds the copter lost orientation, and went with 45 degree inclination to the ground. switch to Stabilize, no control.

          I am grateful for any idea

          please check the logs

          https://www.dropbox.com/sh/qimdohk3oflvh5y/QPYjD1o4gb

          • Rainer,

            I looked at your log briefly and I didn't see any crash - maybe it's the wrong log? I see a lot of Stabilize, Alt-Hold, Loiter, come down to land, and repeat, but barometric altitude does not seem to indicate a crash - it does end somewhere in the air (30 meters up), which is weird, but this is not what I'm seeing - what I see is the whole crash all the way down (and after) - in your case it just stopped logging - maybe a power outage, or some other crash of the software - in my case only the variables blow up, not the entire software - the logging continues in my logs.

  • Thanks for the heads up to the problem, i am on the last steps of setting up my hexacopter, so now i will be careful and will not use loiter nether auto 

This reply was deleted.

Activity

DIY Robocars via Twitter
yesterday
DIY Robocars via Twitter
Saturday
DIY Robocars via Twitter
Saturday
DIY Robocars via Twitter
RT @f1tenth: Say hi to our newest #F1TENTH creation for @ieee_ras_icra next week in Philly. It’s going to be huge! 😎 🔥 @AutowareFdn @PennEn…
Saturday
DIY Robocars via Twitter
May 11
DIY Robocars via Twitter
May 8
DIY Robocars via Twitter
RT @SmallpixelCar: Noticed my car zigzagged in last run. It turned out to be the grass stuck in the wheel and made the odometry less accura…
May 8
DIY Robocars via Twitter
RT @SmallpixelCar: Test my car. RTK GPS worked great. Thanks @emlid for their support. https://t.co/EkQ6qmjmWR
May 8
DIY Drones via Twitter
RT @chr1sa: @kane That's @diydrones circa 2009. Still have a box of those Canon cameras that we used to strap into planes, just like this.…
May 3
DIY Robocars via Twitter
RT @chr1sa: Our next @diyrobocars race is going to be outside at a real RC racetrack in Fremont on May 28. Fully autonomous racing, head-to…
Apr 30
DIY Robocars via Twitter
RT @f1tenth: Our Spring 2022 F1TENTH course @PennEngineers is coming to an end with a head-to-head race as a big finale. So proud of our st…
Apr 26
DIY Robocars via Twitter
RT @DanielChiaJH: I wrote a thing! Throughout the development of my @diyrobocars car I've been using @foxglovedev Studio to visualize and d…
Apr 23
DIY Robocars via Twitter
RT @SmallpixelCar: My new car for high speed. Low body, everything ( @NVIDIAEmbedded Jetson Xavier NX, @emlid RTK GPS, IMC) under the deck…
Apr 23
DIY Robocars via Twitter
Apr 21
DIY Robocars via Twitter
RT @f1tenth: F1TENTH Race training setup @PennEngineers for our upcoming ICRA2022 @ieee_ras_icra competition. @OpenRoboticsOrg @IndyAChalle…
Apr 21
DIY Robocars via Twitter
RT @fatcatFABLAB: Proud to be hosting a restarted DIY Robocars NYC Meetup April 26. Come by if you want to talk about and race self-driving…
Apr 17
More…