After more than two hundred flights with my APM2 equipped quad, today, in the remotest of possible locations my quad failed to respond and attempted to fly away by itself! All I can say is thank goodness for Knobthorn trees.
I wanted to get some video of the area but didn't have my small cam with me so I attached my Galaxy Smartphone to the front of the quad, I have done this before and know that the video won't be that great (lots of jello) but will at least give an idea of the scenery. I waited for satellite lock and then hand launched (too much dust on the ground) just in front of where I was standing, I went slowly up to about 20m, hoverered then did a 180deg pirouette and slowly started descending back towards my position.
When the quad was about 3m from me and 2m off the ground I throttled up to stop the descent which was fine but now the quad was drifting forward slowly so I tried to tilt it backwards... suddenly I had that sickening realization that it wasn't responding, I wiggled the sticks...NOTHING!. I ran towards it but it was still drifting forward and climbing very slowly, now at about 2.5m. It continued very calmly and smoothly and I briefly considered throwing the only thing in hand (my transmitter) at it to try bring it down, but it flew on until it collided with a Knobthron tree at about 4m high.
3 motors were jammed by branches but one was still spinning away calmly. I wasn't... I was able to run to the nearby parked pickup and drive under the tree, reach up to unplug the battery and then get the quad down.
I have read the horror stories of fly-aways but one naturally never expects it to 'happen to me'.
- My radio is a tried and tested two year old Hitec Aurora 9 which has one of the most robust AFHSS 2.4GHz systems. The quad was 3m from me when it happened!
- My failsafe is setup correctly and double-tested, with the throttle channel going to 900ms on event of signal loss.
- All four my SS 30A ESC's (2A BEC) power the APM2 (each tested 0.09v of each other) effectively quadrupling the current capacity. I've never had a suggestion of a brownout.
So what happened? I don't know, I can't see anything definative in the logs (attached). If someone has any ideas I'd be very grateful!
As for now my faith in APM2 has been shattered, like an old friend you can no longer trust.
Replies
I've been brave enough to now fly untethered a few times again and have had no sign of a re-occurance of this issue. I was never able to recreate the same Rx>APM freeze in my testing that was experienced in this attempted fly away. I have put a brownout capacitor on the APM though just for a little extra peace of mind.
I've also asked Michael if it's possible for a visual and audio notification of failsafe condition here: http://diydrones.com/forum/topics/request-for-audio-and-visual-noti... but this will obviously only notify you if you have the MP connected. In my case however the APM never recognised any problem and thus never went into failsafe
Here is the onboard video of the fly-away and crash, excuse the extreme jello from the phones CMOS sensor at times, there was no attempt to insulate the phone from the frame:
Randy, I have done some extensive testing and checking. I cannot reproduce the PWM flat lines that are seen in my crash log, removing any of the Rx>APM wires causes just that channel to flat line not all four. Turning the radio off triggers failsafe.
The ONLY way I can reproduce what was in that log is to pull Ail, Ele, Thr and Rud plugs out of the Rx simultaneously which obviously can't normally happen in flight.
Have reloaded v2.8.1, did a reset and will try again with the default PID's, noticed yesterday that the power fade problem I had goes away as soon as the GPS has locked onto more than 6 sats
Graham, please could you give us you Hitec transmitter and receiver firmware version ? Did you update them ?
Thanks
Craig, John Arne Birkeland (developer of the PPM software) and I spent a couple of hours going over the logs and brain storming ideas that could lead to the loss of radio signal.
It's clear from the logs that at line 10151 all radio updates stop. The throttle pwm does not drop below 975 so the APM doesn't think that anything is wrong. You were in stabilize mode as you said and besides the radio input freezing there are almost no other signs of troubles....except we do see some increase in the gps delays around the time of the problem which adds some evidence that the phone was outputting a strong enough signal to disrupt the gps at least (and you could extrapolate that maybe it also affected the radio).
We performed various tests of the ppm software on an APM2 and APM2.5 and confirmed that:
1. if you disconnect power or ground to the radio the ppm enters failsafe (throttle drops <975)
2. if you disconnect a single channel that channel simply stops updating except for the throttle channel on the latest ppm software which ships with the APM2.5. On 2.5 boards it goes into failsafe.
3. if you disconnect all signal wires between the apm and the radio the apm enters failsafe.
...so basically we didn't have any luck reproducing the problem. The ideas that we came up with were:
a) your radio failsafe isn't set-up and your rx/tx lost contact (unlikely because you were only 3m away and you said you've tested this)
b) the wires for channels 1 to 4 (roll, pitch, yaw, throttle) somehow became disconnected (physically unlikely)
c) the phone caused some low-level interference to either the radio or the APM.
d) you're somehow running a very old version of the ppm encoder software and had a brownout (seems unlikely that you would have downgraded your ppm software)
e) a hardware fault that breaks the connection between the ppm encoder chip and the main Atmel 2560
..these are all very unlikely so we'd really like your help in trying to reproduce the events that led to the failure.
1. turn off your radio and confirm that your throttle drops to below 975
2. try disconnecting some of the signal wires and confirm that only the disconnected wire stops updating
3. put your phone on top of your APM & receiver and switch in and out of airplane mode and see if your rx/tx stops updating
Also could you tell us if your receiver is connected with separate wires for each channel to the APM2? You're not using ppm-sum or anything like that?
There's some additional failsafes that, based upon this event, we're thinking of adding to the code:
1. check in the main code that the ppm encoder is still providing values to the main chip
2. check that values from the rx have changed at least a little bit within the last two seconds
Your first 'GPS' logged data is halfway through the log at line 7021/22...
this implies the GPS lock did not occur until this line. How to sync up the GPS stamp with the flight events??
Causality is not assured though GPS lock discussions abound.
The 'NTUN' tagged lines also start showing up at this time. Again, causality??
-=Doug
Maybe your attached Galaxy Smartphone interfered with the receiver, GPS, or both.
I attached a Flip Video to my T-Rex 450 and it interfered very much with the GPS....
Just a thought.....
Graham, please check the voltage coming out of each of the BEC's individually. This is normally a really bad idea. I had two ESC's connected in parallel once, and after a few minutes, one of the BECs failed and it sent 12V down the positive wire, and destroyed the servo tester. That one ESC now permanently outputs 12V.
Just test to see if maybe that happened, and the Rx was just the first to fail?
-
1
-
2
of 2 Next