Linux autopilot users -- Please help with postmortem

Hi all,

I recently converted my quadcopter from APM over to a Linux autopilot, running a recent version off the dev branch (802ced2).  I'm running a Beaglebone Black using Mirko Denecke's PCB design, and followed his software setup instructions here, which should tell you everything needed about my Linux OS configuration.  It is running a kernel with PREEMPT RT. 

Anyway, my flight this morning seemed to be going well until a few minutes into it, when the copter started spinning madly out of control, ending in a spectacular crash. 

Reviewing the log data, I found that 2 seconds of data were missing, which likely means that ArduCopter stopped running for a few seconds.  That would explain the crash.  But it doesn't explain why.  I'm looking for suggestions on what could have caused this failure.

Here is a screenshot out of the log file, which shows data missing from 10:53:55.024 through 10:53:56.943, almost a complete loss of 2 seconds.  Also below there, I've shown a screenshot plotting the altimeter and gyro data, where you can see the disastrous result of this outage. 

Any suggestions on what to look for?  I really appreciate any help I could get!

3691258608?profile=original

3691258235?profile=original

You need to be a member of diydrones to add comments!

Join diydrones

Email me when people reply –

Replies

  • Mirko, Patrick, Bill,

    You guys are awesome!  I just had a textbook 19 minute flight with my Beaglebone Black.  I reviewed the log file and unlike before, not a single missed dataflash log entry.  MaxT was between 5-7 msec the entire flight (except during ARM as expected).

    I also couldn't help but notice that my quad flies much better than it ever did with my APM 2.6.  Very crisp.  Not sure what changed from Copter 3.2.1 to 3.3.1, but it is extremely responsive, even though I'm flying the exact same gains which I had tuned up before.  (Maybe EKF enables better performance?  Who knows...)  Also, mechanical vibrations are much quieter than they have ever been before (way below the line!), so maybe that extra BBB mass is helping.  Anyway, I couldn't be more delighted with how well it is flying now.

    I only tested Stabilize and Alt Hold today...  On to loiter with my next flight.  I expect it should be good since HDop was locked between 0.6 and 0.7 for the entire flight, with 15-17 satellites for almost the entire 19 minute flight. 

    • Good News !

      • Great to see Larry getting satisfied with a Linux autopilot and happy to see more and more people supporting Linux (really a huge difference from what happened about a year ago). A minor comment with regard the CPU frequency (and the governor) on the BBB (@Miko you may want to include it in your images if you see this issue happening over and over):

        When creating BBB images I used to restrict the governor to a single option (performance). This way, no matter what's requested through the userspace interfaces, the processor will always use that governor. For anyone interested, have a look at this kernel config.

        • @Victor

          I am not providing custom Kernels / Debian images anymore, because BBBMINI is supported by the RobertCNelson RT Kernel already.  If you follow the instructions on the BBBMINI Githug Page, nothing should go wrong. But I will write some "before your first flight" checks soon.

          Best regards

          Mirko

          mirkix/BBBMINI
          BeagleBone ArduPilot DIY Cape . Contribute to mirkix/BBBMINI development by creating an account on GitHub.
        • @ Mirko & @Victor

          You are both the main developers and leaders on the BBB project. Would it be possible that you work  on an update of the Building ArduPilot on BBB page ?

          http://dev.ardupilot.com/wiki/building-for-beaglebone-black-on-linux/

          It is the main entrance to anyone interesting in developing with the BBB and at the present state it is a little bit confusing.

          Thanks

          • Getting back to the Governor Performance issue.

            Here a simple one liner that you can use on launch to get a visual feedback of the cpu speed from the status LED:

            awk '{if($1==1000000){print "heartbeat"} else {print"none"}}' /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq > /sys/devices/platform/leds/leds/beaglebone:green:usr0/trigger

            If current frequency = 1 Ghz then the LED 0 shows a heartbeat otherwise it is off

            • @Patrick, thanks for that, I will try and add this to the startup script.

              Best regards

              Mirko

  • Just for laughs, here is the before and after photo.  The third photo is after this morning's rebuild.  This is where having a 3D printed frame really pays off.  Just print out a replacement part, and good to go...

    And yes, that is an FM antenna.  I'm just too cheap to replace my mid-90's era transmitter.  Maybe Santa could surprise me this year...

    3702134311?profile=original3702134222?profile=original3702134299?profile=original

  • Mirko, Patrick, Bill,

    I just did a 20 minute ground test following Mirko's advice to switch the CPU to stay fixed at 1 GHz.  The data looks great!  All loops appear to complete in under 10 msec, and no lost logging data.  And the application certainly didn't hang up for 2 seconds...  I think I've got enough courage to give another flight test a shot.

  • Developer

    In the plot below you can see several drops in packets. i.e. the time from the GPS isn't consistent ramp up in many places (small steps). I don't think that relates to the cause of crash. the dataflash log would help a lot here.  There a very big step near the beginning of the flight and then you go and fly successfully for a period after that.

    One point to note is that GPS alt jumps after the first very large drop out. not sure how far above MSL you are , but it's notable in the log.

    Jumps in the time usually means (especially in DF logs) that brownout have occurred, but you don't have any monitoring of Vcc in the log , inconclusive)

    3702133603?profile=original

This reply was deleted.

Activity