Hi all,

I recently converted my quadcopter from APM over to a Linux autopilot, running a recent version off the dev branch (802ced2).  I'm running a Beaglebone Black using Mirko Denecke's PCB design, and followed his software setup instructions here, which should tell you everything needed about my Linux OS configuration.  It is running a kernel with PREEMPT RT. 

Anyway, my flight this morning seemed to be going well until a few minutes into it, when the copter started spinning madly out of control, ending in a spectacular crash. 

Reviewing the log data, I found that 2 seconds of data were missing, which likely means that ArduCopter stopped running for a few seconds.  That would explain the crash.  But it doesn't explain why.  I'm looking for suggestions on what could have caused this failure.

Here is a screenshot out of the log file, which shows data missing from 10:53:55.024 through 10:53:56.943, almost a complete loss of 2 seconds.  Also below there, I've shown a screenshot plotting the altimeter and gyro data, where you can see the disastrous result of this outage. 

Any suggestions on what to look for?  I really appreciate any help I could get!

Views: 1016

Reply to This

Replies to This Discussion

Attached the log file, in case anyone wants to take a look. 

Attachments:

Hi Larry,

could you also provide the BIN file which should be under /var/APM/logs ?

Which kind of microSD do you use?

Regards Mirko

Hello Larry,

Hope that your quad is recoverable.

Question: Where's this log comes from : Onboard file on the BBB or telemetry on groundstation ?

And if it is telemetry, what kind of downlink are you using ?

Did you get a chance to look into /var/log/ files for any error ?

Regards

In the plot below you can see several drops in packets. i.e. the time from the GPS isn't consistent ramp up in many places (small steps). I don't think that relates to the cause of crash. the dataflash log would help a lot here.  There a very big step near the beginning of the flight and then you go and fly successfully for a period after that.

One point to note is that GPS alt jumps after the first very large drop out. not sure how far above MSL you are , but it's notable in the log.

Jumps in the time usually means (especially in DF logs) that brownout have occurred, but you don't have any monitoring of Vcc in the log , inconclusive)

Hi Mirko,

Thanks for the quick response.  Here is the bin file, attached. 

I don't use a microSD card.  The OS is flashed directly to the onboard eMMC.

Larry

Attachments:

Hi Larry,

using the eMMC is a good choice, we had some of these time glitches when the BBB changed the CPU frequency when Ardupilot in running. Do you follow step 28 here? That fix the CPU frequency to 1GHz.

Please test on the BBB the command: cpufreq-info, the result should look like this.

debian@beaglebone:~$ cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 300 us.
  hardware limits: 300 MHz - 1000 MHz
  available frequency steps: 300 MHz, 600 MHz, 800 MHz, 1000 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 300 MHz and 1000 MHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 1000 MHz.
  cpufreq stats: 300 MHz:0.00%, 600 MHz:0.00%, 800 MHz:0.00%, 1000 MHz:100.00%
debian@beaglebone:~$

The BBB should run 100% with 1000 MHz.

I will take a look to the BIN Log also.

Regards Mirko

Hi Patrick,

Well, the good news is that having a 3D printed quad, I can easily make new parts.  I just fired up the 3D printer.  So yes, everything is recoverable. 

I am collecting log data with onboard storage, no ground link.  Though your comment just sparked a thought...  My BBB autostarts ArduCopter with the "-A udp:192.168.1.160:14550" option for a MavLink connection over Ethernet.  However, when flying, I obviously don't have an Ethernet cable attached.  Is there any reason the -A option would cause funny behavior when the BBB isn't attached to a network? 

Only 2 errors show up in the log file, and they both show up after the crash.  Lots of cables became detached after the crash, so not a surprise.  Here is a screenshot of the errors, just for completeness:

Hi Mirko,

No I did not follow step 28.  That must be a relatively new addition.  I want to say it was early October when I followed your steps and I don't recall seeing it at that time.

Here is what I get when I run cpufreq-info:

root@beaglebone:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: cpufreq-dt
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 300 us.
  hardware limits: 300 MHz - 1000 MHz
  available frequency steps: 300 MHz, 600 MHz, 800 MHz, 1000 MHz
  available cpufreq governors: conservative, ondemand, userspace, powersave, performance
  current policy: frequency should be within 300 MHz and 1000 MHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 600 MHz (asserted by call to hardware).
  cpufreq stats: 300 MHz:1.23%, 600 MHz:78.67%, 800 MHz:4.49%, 1000 MHz:15.61%  (148)

It looks like I'm running at 600 MHz, not 1000 MHz.

Also, ArduCopter is only running at about 20%, not 100%.  See below:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
483 root -13 0 44708 44308 2780 S 19.7 8.7 2:08.94 ArduCopter.elf

Sounds like you have some ideas on what may have happened.

Hi Larry,

please follow step 28, otherwise the bbb switch the CPU frequency in flight which causes your behavior. Before next flight remove props, arm the copter raise throttle and let the copter "fly" for some minutes. After that, check the dataflash log, the timeline must be smooth, steps should only be there when the copter is disarmed and armed again (when disarmed, the Ardupilot does not write logs (you can enable that as well)).

I had also these glitches, which causes short wobble, but no crash. This was gone with Step 28.

Regards

Mirko

Thanks Bill, that is a useful plot! Note that the data you are looking at is from the onboard dataflash log, so I assume that if no data is being recorded, that the Arducopter program isn't processing any data.  Please correct me if this is a bad assumption.

I think what you are showing is that these outages were occurring very frequently, not just at the very end of the flight.  The first one (where the GPS alt jumps) happened at 10:52:10, and lasted for 8.5 seconds.  I was on the ground at that time.  The next one is at 10:53:04 and lasts for 1.3 seconds.  I don't see any significant transients at that time, so I was probably flying fairly steady. 

The very last one happens at the same time as the crash, and I would assume that there is a cause and effect here.  Am I wrong about this?  The way I read it, the ArduCopter application was momentarily stopped (perhaps preempted by a different application?) and so unable to keep the feedback loop closed.  I was able to ride through several probably because I was flying level at the time.  During the last outage I wasn't so lucky. 

Mirko,

Seem that sometimes the governor performance commande does not kick in in at 1Ghz on Debian. Might help to add cpufreq-set -f 1000Mhz  on the Ardupilot launch file.

Great, thank you so much!  I will give this a shot and report back.  I can't imagine the hours you put into debugging all of this and getting ArduCopter running on your BBB.  It's nice I can just follow in your footsteps.  Crossing my fingers that this works, but it definitely has promise.  I was momentarily considering just putting the APM back into my copter, but that would be giving up and I can't do that!

Reply to Discussion

RSS

Groups

Season Two of the Trust Time Trial (T3) Contest 
A list of all T3 contests is here. The current round, the Vertical Horizontal one, is here

© 2018   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service