OK, THIS IS A RANT!!
I am done with Arducopter! I have been trying to get this platform to be reliable for way too long.
I don't even want to add up all the damage, time and money I have wasted trying to get this flight
control system to work as advertised.
I actually thought with the new firmware and M.P., that the developers had finally got out all of the
As I write this, I notice that the top discussion is unexpected scary start-up.
This is what happened to me:
I dumped all programs from my computer, and reset my APM 2.5, I started over.
I downloaded the new versions clean, without any of the zillions of updates.
I was so excited to see that my project was finally working very well, a doing what I told it to, and it
worked very well for two days. On the third day, without any changes at all to anything, it did a full
throttle cut. I turned it off, rebooted my computer, and started over. It behaved normally for about two
minutes, then two motors cut out. I was only up about six feet, and over a lawn, so no damage.
As I approached my machine, to unplug the battery, two motors started to spin at different speeds.
Then they all spun up to full throttle. I always carry my Aurora 9 with my left thumb on the throttle
stick, so there can not be an accident.
OFF IT WENT, into the sun. Hit return to home, no joy. I said goodbye to it, as it went to an unknown
altitude, and into the sun, (downwind, it was just trying to go straight up).
I found it the next day, see the picture. Two miles of walking and searching and cussing.
I went over the logs, and determined that there were error codes all over the list, and it breached the
geofence, but kept going anyway. It had a mind of it's own, TOTALLY NOT COOL, and VERY, VERY
If this had happened in a more densely populated area, I might be in jail right now, or being sued, or
worse. "Drone kills baby", on the eleven o'clock news. My machine is big, heavy and damn-near
indestructible. I do not know how far it fell, I going to guess at somewhere between 700 and 1000 feet,
(angle of sun, do the trig.) Now it's broken.
I want the FAA to let us do this, but if these kind of failures keep happening, someone's going to get
hurt, and then the Government will make it illegal! Game Over!
What is up with all the flaws in this platform? Ardu may kill any chance for guys like us to go out and
make money with this tech, before the FAA and Congress even make their decision next year.
Oh, and by the way, I was to show and demonstrate my machine to a government contractor, with a
C.O.E., who shall remain nameless, the very next day. I got to show them a hulk. The only plus was
they were impressed by my build, because it's still repairable, and the expensive stuff lived.
My suggestion to ARDU is to get your shit together. Something very bad is going to happen, it already
has to me, and if someone had gotten hurt the other day, and I was being sued, I would sue ARDU.
Then it's all over the news, and all over for us. This "mishap" was of no fault of mine. Does the
software rewrite itself? Did someone embed malware into it? Did the hardware just "decide" to melt
You guys need to perfect this, or it's not going to go well with the FAA.
If they were to ask me if it is a safe and reliable platform, I would have to answer "HELL NO!, take it of
the market, before somebody gets killed." I am not giving up on this technology, just Arducopter,
and SO SHOULD YOU ALL!
Replies are closed for this discussion.
Looks like Rob's pretty much got this fully analysed. I've had a look but it all backs up what Rob's said.
Attached is a graph of the inertial nav alt estimate and the barometer pressure. There's a very large pressure change (blue line) which pulls the barometer altitude down and the inertial nav estimate gets pulled down with it. So it thinks it's 200m below it's actual altitude. It then recovers and drags inertial nav with it. It's unclear if the inertial nav estimate then overshoots the baro alt or not (it might have). In any case this triggers the altitude fence and the copter goes into RTL. I think the pilot momentarily gets it back into alt hold but the altitude is still going up very fast (because of the baro glitches) and it breaches the fence again and goes into LAND mode and then comes down.
On the software side there are two things we should do:
- barometer reading sanity checking. similar to the recently added GPS glitch protection. it's on the to-do list.
- after a fence breach, if the pilot re-takes control (using hte flight mode switch) give him/her at least 10seconds to control the copter. The current fence breach code isn't time limited. Instead it gives the pilot 20meters so if the copter continues to move more than that beyond the fence, the autopilot will retake control again. The problem is that if the copter is moving quickly (or if the sensors are going crazy like this time) then that 20m my be passed in a very short period of time. This item is also on the to-do list.
Steven, it seems like a good idea to not let it go negative, but every time we consider something like that, somebody has a case for why we wouldn't want to do that. Obvious case is flying in mountains where you might descend into a valley. This is the battle the team faces constantly.
This one is particularly difficult, because we're trying to protect for negative... negative what, exactly? The pressure is still positive. It's only negative relative to the pressure we declared was zero last time we armed. You know, it was 1.0158 Bar 15 minutes ago, so we decided to call that "zero". Now it's 1.0163 Bar, so what? How do we know that's an error condition?
If anybody has an easy answer to this... we're all ears.
Darrell- I totally believe you about minimal support from the orient.
I appreciate very much everyone looking into this, it's impressive. I am going to try and get the flash logs, but it looks like you guys nailed the anomaly. It never did that before, and I've flown at sunrise, and at times when the sun is at the same angle. I am wondering if the APM cooling off from room temp could have also contributed to this, it was about 45 out that morning. I've flown it at freezing temps as well.
Now, In my defense, I am not a kid and not a dummy with too much time and $. Quite the opposite, actually. I had not programmed any waypoints, I never have. I do not take unnecessary risks, and only try something new when I am sure that everything else is working.
I did do a "sanity check" on the MP, and it was correct, but I did not look at it again after the disarm/arm. This is not the first "crazy" thing it's done, just the first time I have pitched a bitch about it. The idea of one of these things going out of control over a neighborhood, which is where it was headed, will not go over well with the powers that be.
I asked the guy, (Pres/CEO/co-owner of AirCover), OK, not nameless, if I could try out their flight control system? We both laughed. Full-on encrypted military freqs, etc, etc. He didn't say that their system is perfected, though, and first thing he asked me, after, "how the hell did you find us", was if I could write code. Go figure. I was there trying to get a job, and I still may, as a builder, wish me luck. They do not do anything weaponized, or I would not have looked them up. He was genuinely impressed that after what it went through, it was still repairable, and so am I.
Maybe there needs to be a kill switch on a spare channel, a panic button. If that had been available, this fly-away could have been shut down at only a few feet, and it would have only suffered minimal damage, if any, maybe a prop or two.
I'm going to connect the APM by USB, see if it lived, and if I can get the flash logs.
Gary, I don't get the voltage problem as no servos were connected, and my secondary bank of VR's was disconnected, they are for video transmitters, cam power, etc. The dome is from some curio, nic-nac shop supply co. Googled 6" plastic sphere, and found these, already cut in have, for $5. I repurpose stuff.
"how the hell did you find us"
well...10 minutes later i have his name, address, a possible home address, and a possible phone number. For trying to be the next DOD big thing, he sure has a lot of internet footprints. Mind asking if I can call him to see if he still wants that coder?
MBerry, Dude, the guy's regular army. If I ever see him again, I'll mention it.
Darrell, that elephant won't rear it's ugly head until there IS a bad accident, then it could all unravel. We could all join the AMA, pay our dues, stay members in good standing, which provides a $million bond. I do not know if they would, or do include autonomous aircraft. I see an internet search soon, wait......
OK done. AMA provides 2.5 million in insurance, BUT, only to members who are flying at an AMA sanctioned field.
That's not gonna help us. No mention of UAV's, but they are giving away a Phantom on their home page, so I'm gonna say yes, but what good are these machines for, going around and around in circles at the same place, every Sunday? I think not. I want to fly in front of waterfalls, and over boats, and off cliffs, and over ridge tops, etc. They will not insure us. Maybe we should call Lloyds.........
Well of course they are all selling model aircraft operated under AIC 91-57 and AMA field rules so that insurance covers them. The crash at the raceway last year is going to expose several liability issues. The NTSB were very interested in that one and have leaned very hard on the FAA according to sources who were utterly embarrassed by the incident. At the time the NTSB was unaware of commercial civil operations by sUAS. If you remember an operator hired out a machine to somebody else to fly commercially at an event and the platform ran out of battery and fell into the crowd. To make things worse the chap then changed batteries and flew again.
Hang on guys, I need to get my popcorn..............
All I can add, my standard 3DR quad with APM 2.5 and latest code flies Auto missions like it is on rails.
So, some builds do seem to work :-)
First of all every barometric sensor that has holes to let sunlight pass (that directly hits the silicon) is affected severely by sunlight. Some claim that it is the "pressure of the light" but I did the math from wiki and found that this pressure is at least one magnitude smaller than the sensor noise/sensitivity - that's probably why barosensors are not used to measure the radiation/light pressure - so much for that. But the main problem is: User is taken out of control by the flightcontrol. And that seems to be a general problem of the softwareconcept. I wrote earlier some unheard infos how to prevent that. Even if the Baro or Mag or GPS goes berserk and just the gyro + RC contact is still working the user has to have the possibility to get back control by a flip of a switch (even if that is a killswitch). The platform will be always high risk (Russian Roulette with 6 bullets) for the 2km(?) surrounding population until that is ensured - otherwise fly in some desert.
I glad this rant turned into a safety meeting. My "desert' was a huge cow field, many, many acres, and it did fall in it, lucky for me. My understanding was that if your APM was covered by a dome, to keep prop wash and wind off of the sensor, then foam was not needed. Guess that was wrong. I'll put it in a case, or make something.
I thought of putting parachutes on these a long time ago, and Photo Higher in New Zealand makes a ballistic chute for multi-copters, but it is big, heavy, and expensive, made to go on the German Microcopter. One of these days, I am going to try a create one from model rocket parts. I could, no WOULD, have used it the other morning.
For me, now, it's back to the drawing board, and the shop. All I need to repair this is some more fiberglass plate, I used prototyping board for batt holder and APM stand. Motor mounts and frame plates are CF.
My APM lived, amazingly, but there was nothing in the flash log files. The dates are there, but the file is empty, unless I looking in the wrong place, but I don't think so.
So, this is a learning experience, about barosensors. Still, I'm not too happy.
Thanks for all the input, and BE SAFE!
We sort of already do something like that, but it's actually targeting accelerometer errors. This is from back in the day when we had all these vibration induced flyaways. If the accelerometer drives the altitude estimate too far from the baro, we only allow it to get so far.
That's the problem, we have two sensors, either one of which can give bad data (usually due to user error), and we can never be sure which one to trust.
Now, the guys working on the EKF actually commented on this, and they want to bring in a third datapoint: The GPS altitude estimate. This generally is not good for much, but it can help us decide which of the two sensors has failed. If the baro is no good, then it would use the GPS estimate for long-term, and the accels for short-term control. If the accels are bad, then we would probably use the baro exclusively, though there's a question in my mind about how we would do accel-based control just using the baro. We'd almost need a fail-over alt-hold control loop that can run just on baro.
The baro on the Pixhawk is exactly the same unit, I believe. Though it has a massive advantage because it's actually mounted on the bottom of the board.
@R_Lefebvre Awesome job on such a thorough analysis and getting to the cause of the crash. Once bad things happen that's the most productive to move forward. It also highlights the point that these things live and breath (well fly) by their sensors and if that starts producing nonsense then all bets are off.
In terms of a safety I agree Crashpilot1000. The approach we've been taking at Tau Labs is that anything that isn't expressly allowed should just turn off the motors (or for a plane enter a crash sequence). From there we can try to expand the set of safe fancy things. For example with the outer geofence, if you breach it then something is already horribly wrong and you should give up. This made for some funny testing of geofence:
We spent quite a while scrutinizing the code to make sure the user is ultimately always in control. Then again sometimes when weird things happen if you don't think to hit the disarm position or whatever you are screwed. I've dumb thumbed enough planes into the ground to know "having control" isn't always sufficient.
Running in autonomous mode and you hit the disarm position? Motors turn off
Turn off transmitter while in autonomous mode? Motors turn off
I think people get really excited about failsafe modes and such without really considering what it means when those go wrong and you have no control. Crashing safely is so much better. Although of course the goal is always perfectly say returns.
I personally have to take that perspective since I live in a city. There is nothing I put in the air that I wouldn't drop in a second for the slightest possible reason. After one terrifying flyaway with OpenPilot that mentality became deeply entrenched.