After more than two hundred flights with my APM2 equipped quad, today, in the remotest of possible locations my quad failed to respond and attempted to fly away by itself! All I can say is thank goodness for Knobthorn trees.
I wanted to get some video of the area but didn't have my small cam with me so I attached my Galaxy Smartphone to the front of the quad, I have done this before and know that the video won't be that great (lots of jello) but will at least give an idea of the scenery. I waited for satellite lock and then hand launched (too much dust on the ground) just in front of where I was standing, I went slowly up to about 20m, hoverered then did a 180deg pirouette and slowly started descending back towards my position.
When the quad was about 3m from me and 2m off the ground I throttled up to stop the descent which was fine but now the quad was drifting forward slowly so I tried to tilt it backwards... suddenly I had that sickening realization that it wasn't responding, I wiggled the sticks...NOTHING!. I ran towards it but it was still drifting forward and climbing very slowly, now at about 2.5m. It continued very calmly and smoothly and I briefly considered throwing the only thing in hand (my transmitter) at it to try bring it down, but it flew on until it collided with a Knobthron tree at about 4m high.
3 motors were jammed by branches but one was still spinning away calmly. I wasn't... I was able to run to the nearby parked pickup and drive under the tree, reach up to unplug the battery and then get the quad down.
I have read the horror stories of fly-aways but one naturally never expects it to 'happen to me'.
- My radio is a tried and tested two year old Hitec Aurora 9 which has one of the most robust AFHSS 2.4GHz systems. The quad was 3m from me when it happened!
- My failsafe is setup correctly and double-tested, with the throttle channel going to 900ms on event of signal loss.
- All four my SS 30A ESC's (2A BEC) power the APM2 (each tested 0.09v of each other) effectively quadrupling the current capacity. I've never had a suggestion of a brownout.
So what happened? I don't know, I can't see anything definative in the logs (attached). If someone has any ideas I'd be very grateful!
As for now my faith in APM2 has been shattered, like an old friend you can no longer trust.
Replies
I've been brave enough to now fly untethered a few times again and have had no sign of a re-occurance of this issue. I was never able to recreate the same Rx>APM freeze in my testing that was experienced in this attempted fly away. I have put a brownout capacitor on the APM though just for a little extra peace of mind.
I've also asked Michael if it's possible for a visual and audio notification of failsafe condition here: http://diydrones.com/forum/topics/request-for-audio-and-visual-noti... but this will obviously only notify you if you have the MP connected. In my case however the APM never recognised any problem and thus never went into failsafe
It might be useful if we could come up with a rudimentary test procedure for RF interference rejection ability of the APM from the most common sources that the APM could plausibly have to endure in common use cases. Things like WiFi tranceivers, cell phones (or the DroneCell and the like?), some of the most popular telemetry transmitters people might have and so on.
Test interference from such radio sources in a homebrew anechoic setup and see if and when the chips on the APM freak out. A location in the middle of nowhere, with no trees or buildings, is plenty anechoic enough. Start varying the setup, with different wiring for example - can the wiring from the RC Rx to the APM make it worse and how to minimize it? How about RF shielding cans over the chips on the APM board? Mu-metal enclosures and shielding for the parts that aren't magnetometers or radios?
I think we have some people with experience in software defined radios in this community; it should be plausible to create something that chirps a blast of RF energy at the board. No need for excessive power levels either; if a highly directional horn antenna close by sending a few milliwatts of power doesn't make the PPM encoder go crazy like sending frozen channel values, it would be a data point that suggests the cell phone in this fly-away might not have been the root cause after all. Or the opposite could happen and we could look into how to harden the APM to make it less likely to happen in the future...
Here is the onboard video of the fly-away and crash, excuse the extreme jello from the phones CMOS sensor at times, there was no attempt to insulate the phone from the frame:
Randy, I have done some extensive testing and checking. I cannot reproduce the PWM flat lines that are seen in my crash log, removing any of the Rx>APM wires causes just that channel to flat line not all four. Turning the radio off triggers failsafe.
The ONLY way I can reproduce what was in that log is to pull Ail, Ele, Thr and Rud plugs out of the Rx simultaneously which obviously can't normally happen in flight.
Have reloaded v2.8.1, did a reset and will try again with the default PID's, noticed yesterday that the power fade problem I had goes away as soon as the GPS has locked onto more than 6 sats
Graham, please could you give us you Hitec transmitter and receiver firmware version ? Did you update them ?
Thanks
Craig, John Arne Birkeland (developer of the PPM software) and I spent a couple of hours going over the logs and brain storming ideas that could lead to the loss of radio signal.
It's clear from the logs that at line 10151 all radio updates stop. The throttle pwm does not drop below 975 so the APM doesn't think that anything is wrong. You were in stabilize mode as you said and besides the radio input freezing there are almost no other signs of troubles....except we do see some increase in the gps delays around the time of the problem which adds some evidence that the phone was outputting a strong enough signal to disrupt the gps at least (and you could extrapolate that maybe it also affected the radio).
We performed various tests of the ppm software on an APM2 and APM2.5 and confirmed that:
1. if you disconnect power or ground to the radio the ppm enters failsafe (throttle drops <975)
2. if you disconnect a single channel that channel simply stops updating except for the throttle channel on the latest ppm software which ships with the APM2.5. On 2.5 boards it goes into failsafe.
3. if you disconnect all signal wires between the apm and the radio the apm enters failsafe.
...so basically we didn't have any luck reproducing the problem. The ideas that we came up with were:
a) your radio failsafe isn't set-up and your rx/tx lost contact (unlikely because you were only 3m away and you said you've tested this)
b) the wires for channels 1 to 4 (roll, pitch, yaw, throttle) somehow became disconnected (physically unlikely)
c) the phone caused some low-level interference to either the radio or the APM.
d) you're somehow running a very old version of the ppm encoder software and had a brownout (seems unlikely that you would have downgraded your ppm software)
e) a hardware fault that breaks the connection between the ppm encoder chip and the main Atmel 2560
..these are all very unlikely so we'd really like your help in trying to reproduce the events that led to the failure.
1. turn off your radio and confirm that your throttle drops to below 975
2. try disconnecting some of the signal wires and confirm that only the disconnected wire stops updating
3. put your phone on top of your APM & receiver and switch in and out of airplane mode and see if your rx/tx stops updating
Also could you tell us if your receiver is connected with separate wires for each channel to the APM2? You're not using ppm-sum or anything like that?
There's some additional failsafes that, based upon this event, we're thinking of adding to the code:
1. check in the main code that the ppm encoder is still providing values to the main chip
2. check that values from the rx have changed at least a little bit within the last two seconds
Your first 'GPS' logged data is halfway through the log at line 7021/22...
this implies the GPS lock did not occur until this line. How to sync up the GPS stamp with the flight events??
Causality is not assured though GPS lock discussions abound.
The 'NTUN' tagged lines also start showing up at this time. Again, causality??
-=Doug
I've use my old Nexus One in the same way. I never experienced an issue, airplane mode on or off.
-Cad
-
1
-
2
of 2 Next