This is a discussion re the bad Accel, Gyro and Baro values that we're seeing with ArduCopter-3.1. The increase in the SPI bus speed from 500khz to 8Mhz has exposed a hardware problem on some boards. That hardware problem is that the 3.3V regulator has been blown so all sensors are running at 5V instead of the intended 3.3V.
How have these regulators been burnt out?
- Attaching a radio receiver or MinimOSD to the APM while the APM is only powered through the USB (see video below)
- Some clone boards seem to come from the factory with blown regulators. 3DR boards might also come with blown regulators although they do a specific check of the regulator as part of the regular QA process.
- It is not (as far as we know) actually caused by the AC3.1 software itself, it just exposes the problem. You could prove this to yourself by checking the 3.3V regulator (see video above) before and after the upgrade.
How can we fix the regulator?
Option #1: If it's a new board (so that it's less likely you burned it out yourself) you could report the problem to the retailer that sold you the board and ask for an replacement. If it's 3DR it's called an "RMA".
Option #2: if you're handy with a soldering iron you can replace the regulator yourself. On the APM2.5.2 (and higher) boards it's not that difficult. On the APM2.5 it's far more difficult.
For APM2.5.2 : TPS79133DBVR
For APM 2.5: MIC5219-3.3YML TR
How can I stop it from happening again?
Do not connect any devices such as a radio receiver, MinimOSD, GPS, etc while the APM is powered especially while powered only through the USB cable.
Attaching a 100uF capacitor across any of the APM's radio input's 5V and GND pins will stop the regulator from being blown by plugging in a receiver. video here!
There are very few reports of regulators being blown twice and no reports of it ever failing in flight.
Below are some graphs of the types of values that we are seeing on these boards.
Replies
Looks as though the list keeps growing, I had plugged my Apm 2.5.2 into the original version of mission planner many times and did not have a problem with the unit, after I updated the mission planner and uploaded 3.1 firmware the unit supposedly blew the reg i went back to the original mission planner where the unit was working and uploaded the older firm ware the unit still would not work or arm the motors as it did originaly it did work and fly.
Dennis
Just wanted to add our blown 3DR APM 2.6 to the list.
Our APM 2.6 has never been flown, it was only ever powered via USB for sim, we tested in HIL simulation many times with prior firmware versions successfully, but recently with the new firmware it was only hooked up and tested a few times before the BAD GYRO HEALTH message appeared and the HIL simulation would no longer fly correctly
Hope the actual source of this issue is found as it will be very interesting to learn from.
Sam, have your checked your 3.3V voltage level? I believe the HIL code does not even compile in the gyro driver, so this failure might not be a blown regulator at all, but something completely different.
Hi Sam,
HIL does not even use the onboard Gyro, so the 3.3v issue would have no effect on the HIL
Phil
It is weird that I see more APM2.6s having this issue lately. At first I was thinking that the primary target was the clone 2.5.2 modles
Re-posting to fix a photo hosting issue.
I've thought the same thing too. On RCG, there are an awful lot of clones with problems, seems like 10:1. Impossible to say what the true statistic is, but do clones really outsell genuine by 10:1?
There's another thing about the clones that makes these failures much more likely. Namely, they do not include a power module. Way back in the day, people were blowing APM's a lot due to reverse hook-up of the power supply. That is what led to the creation of the Power Module with a polarized power plug. With the PM, you basically cannot screw it up even if you tried.
However, clones generally are not including a power supply. They leave it up to the user to supply power. I suspect this is the cause of many of the failures on clones. If you use a noisy UBEC that spikes over 6V, <poof> 3.3V reg is gone. Plug the supply in backwards, very easy to do, <poof> the reg is gone.
These failures are really insidious, because there's absolutely no indication you just did it. No magic smoke. The board just doesn't boot. Then after your heart rate comes down, elation that fixing the reversed polarity causes the board to boot! Yay! But then you connect MP, and you get the warning. I think some people honestly don't realize what they just did.
When 3.1 rolled out, all these people who forgot that they did this once, got hit with the results of this mistake for the first time.
And there's another problem. Along the left side of the APM there's a 3Xwhatever pin header with all the analog pins on it. You could power the Vcc with your UBEC through here. In fact, Joe Abram even recommends it. But it's risky. If the entire rail is filled, that means you also have the SPI pins filled. And unfortunately, plugging power into the SPI port in what you think is the correct way, actually puts reverse voltage on the Vcc! This is an unfortunate situation.
The first row is the worst, and definitely will blow the regulator, I actually tested it. The second row, I'm not sure yet, but it's probably not good either.
Here you can see what I'm talking about. Joe has actually applied power in an OK spot, but you can see the risk!
And just to prove the point, here's a great example... Joe has hooked a power line up to the SPI port. It's not clear if this is the Cap he has, or the power supply. But it shows that even somebody who some consider to be an APM expert, can make this mistake. This is actually from an instructional blog post, so there's some risk this mistake will spread.
In fact, I'm virtually certain that this is the cause of the single reg failure I have. I use an external ULDO linear regulator to power the APM on my helis, and I plug it in the side port of my 450 heli, because it's convenient. A while back I was installing a new board, and found the dataflash wasn't working. This was just about the time that this whole problem with the regulator was being discovered. I figured it was just another blown regulator and forgot about it.
It was only when Jason Short accidentally discovered this SPI thing on Saturday, that I jogged my memory that I had done this exact thing. I plugged the power supply into the first row, the board didn't boot. I realized I had plugged it into the SPI port. I moved it over two rows, and the board was fine... until I discovered the dataflash problem a week later but by then I didn't make the connection.
But can someone from 3DR or you RobertL explain to me how to use power module with anything but light quad???on APM it goes to 90A(I could live with that on octa maybe) but only 60A on Pixhawk???!!!!maximum for 8 motors????its impossible,so if I have power hexa or octo I just cant use power module....also what will happened if I use it anyway and exceed 60A,will power module burn or just wont be able to mesure that?i can live with 4s too....and I can tell you from my experience Attopilot sensor will newer work as original Power module...voltage of batt is ok but measuring current newer work good enough for me.
Soo...this means Pixhawk will newer fly on any heavy multirotors????!!!!!
And it is like that for 2 years! Plenty of times to solve such a simple thing.
Multiple times on this forum I suggested some other similar solutions
http://www.sm-modellbau.de/shop/product_info.php?products_id=143&am...
but no one ever give me any answer except good old Randy who at least wrote he is not sure if it work with APM and Pixhawk...I would realy like to find solution for this but my knowledge is not enough for that
same goes about my questions about ublox 7 series gps
I really do not understand that because if you cant manufacture it, offer us other solutions witch can...
That is my opinion how things should work with company who have word Open source in every announcement...
Emin, these are my personal thoughts on this issue, and actually go against what 3DR is doing, but that's OK as we don't always have to agree on everything. You asked so I'll tell you, and I've said this in several other places as well.
I don't use the PM, or any current monitoring at all on any large/expensive machines. Never have. Reason is because when I build something, I try to reduce points of failure. If I'm adding something that is supposed to make operation safer, I try to make sure that it really is making the overall system safer, instead of adding one protection point, but also adding a bunch more failure points.
So in this case, the PM adds a bunch of potential failure points in the form of solder joints that your primary power has to flow through. Especially on larger machines, this can be significant. And what does it give you in return? Well, you can find out how many mAH you have drawn. So what? A battery can dump early, way before you've exhausted the mAH. Monitoring voltage, and tracking the healthy of your batteries is CRITICAL. Monitoring how many mAH you have drawn... I really just don't see the point. I've had plenty of helis come down early because the battery dumped early. Which I could see on the voltage. I've never had a case where I said "gee, if I had current monitoring, I wouldn't have had that problem."
In fact, I think current monitoring can encourage you to push the battery. "Voltage is 3.7V/cell, but I've only drawn half the battery amps, so I'll keep going." That's not a good thing to do.
Combine that with the fact that all of my helis use a 2S battery for the servos, and the PM won't operate that low anyway, and I always have to build my own APM power supply anyway. And voltage divider circuits are easy to build. So I just do that, and forget about current monitoring.
I do use PM's on small machines because it is quick and easy, and the risk is low enough.
So, what can you do if you don't want to build your own power supply from scratch? One simple thing, is to wire the battery connector to the PDB with wire directly. Do not run it through the PM. Then, connect the input of the PM to the input the PDB. Now the PM will be able to power the Pixhawk, and monitor the voltage. So it's operating as intended, but you don't have current flowing through it so that risk is gone. You don't have current data, but you've eliminated the problem of putting too much power through the PM.
One option I have been playing with to measure current, is these things:
http://amploc.com/amploc.com/AMPLOC.html
They slip over the power wire, and present zero extra points of failure. I picked one up 6 months ago, but just haven't got around to testing it yet.
I agree with you, I had a problem when PM is on a big load, voltage drops at the out put side due to resistant on the power sensor. I have separate voltage sensor/alarm plugged into my battery and im monitoring though the mission planner. deference of the voltage can go up to .6v
Could we simply mod the alarm to output a signal when the beeper sounds, input that signal into the APM an Analog port, and get a warning on MP when any cell starts to dip below your min?
Sounds easy. What do you think?