This is a discussion re the bad Accel, Gyro and Baro values that we're seeing with ArduCopter-3.1. The increase in the SPI bus speed from 500khz to 8Mhz has exposed a hardware problem on some boards. That hardware problem is that the 3.3V regulator has been blown so all sensors are running at 5V instead of the intended 3.3V.
How have these regulators been burnt out?
- Attaching a radio receiver or MinimOSD to the APM while the APM is only powered through the USB (see video below)
- Some clone boards seem to come from the factory with blown regulators. 3DR boards might also come with blown regulators although they do a specific check of the regulator as part of the regular QA process.
- It is not (as far as we know) actually caused by the AC3.1 software itself, it just exposes the problem. You could prove this to yourself by checking the 3.3V regulator (see video above) before and after the upgrade.
How can we fix the regulator?
Option #1: If it's a new board (so that it's less likely you burned it out yourself) you could report the problem to the retailer that sold you the board and ask for an replacement. If it's 3DR it's called an "RMA".
Option #2: if you're handy with a soldering iron you can replace the regulator yourself. On the APM2.5.2 (and higher) boards it's not that difficult. On the APM2.5 it's far more difficult.
For APM2.5.2 : TPS79133DBVR
For APM 2.5: MIC5219-3.3YML TR
How can I stop it from happening again?
Do not connect any devices such as a radio receiver, MinimOSD, GPS, etc while the APM is powered especially while powered only through the USB cable.
Attaching a 100uF capacitor across any of the APM's radio input's 5V and GND pins will stop the regulator from being blown by plugging in a receiver. video here!
There are very few reports of regulators being blown twice and no reports of it ever failing in flight.
Below are some graphs of the types of values that we are seeing on these boards.
Replies
For one of the tests, all buffer chips among others were physically removed from the board. He was still able to trigger the failure mode.
Philip -
It does appear that you've isolated the cause to a surge current draw overwhelming the regulator, as I suspected. However, I suspect that your test capacitor is inadequate for the same reason you chose a 100uf ceramic brute-force filter across the 5V rail: they are generally quite a bit lower ESR than axial-lead electrolytic capacitors. A little brute-force filtering at the rails is ALWAYS a good idea; others here have mentioned the same thing.
I've taken a look at several RX I have lying around; they all have between 220uF and 470uf of ceramic capacitors at the 5V rail, and another 100uf-220uF AFTER their 3.3V regulator. Measured ESR for the 220uf caps is in the 0.14Ω range; the 470uf measured 0.09Ω. I think you can see where I'm going with this. :D
I submit that this is really is evidence that the TPS79147/MIC5219-3.3 regulators simply aren't durable enough for this application; I know they SHOULD be, but empirical evidence shows otherwise. We should expect to severely over-engineer for interfaces which have end-user's equipment plugged into them, as we KNOW those interfaces are going to be abused. ;)
Don't get me wrong... as a fellow engineer, I understand the drive to know EXACTLY what is causing these failures... I really do. You have no idea how many 4am nights I've spent doing just that. :rolleyes: But maybe we need to use the empirical data and move forward with a little empirical engineering, and try something different to see if it resolves the issue.
My solution was to try a regulator already known to be VERY durable in the face of such abuse; the LM/LT1117 family. I know the filtering in place for the original regulator is a bit weak for both the VCC and VOUT rails; but it IS within spec for the LM1117, and definitely within the ranges I've used successfully elsewhere. If it holds with the existing filtering, it should be grenade-proof with the addition of a 100uf cap at the rail as suggested. If it fails, then we know that the issue is elsewhere... maybe something as simple as inadequate brute-force filtration at the 5V rail, which your capacitor mod addresses.
I've started that here: http://www.rcgroups.com/forums/showpost.php?p=27772534&postcoun...
Preliminary testing is very positive; however, I lack facilities to test this simple mod in the same manner as you are testing; primarily, I have insufficient experience USING the APM flight controller to evaluate properly. I do have quite a bit of experience designing and redesigning electronics from R&D right up through production, however.
I know I'm jumping in feet first where there has clearly been a lot of heated discussion; my intent is NOT to stir that up again. It is simply to offer a different viewpoint, from the outside, based on what I've been able to read over the last week or so. I'm prepared for the backlash; while I know that trying to apply Occam's razor to the problem can be a useful tool, I also know that it cuts both ways.
In short, PLEASE tell me why I'm wrong; I want to know. I know I'm not privy to all the reasoning behind the original choice to use these regulators; maybe I'm missing something here.
Peace,
mnem
Paul, there's absolutely nothing wrong with what you've written here. This is the essence of open source. Getting different perspectives on these issues. And we don't always have to agree, as long as we can disagree nicely.
BTW, your link is broken?
Rob -
My link was to a pretty detailed build post in the RCG APM 2.5 thread started by jabrams, where I ask for further help evaluating my LT1117-3.3 mod. He evidently felt my input wasn't of any value and decided to delete it with extreme prejudice, along with another post where I suggest that buffer caps across the power rails at any plug-in header is commonly accepted sound engineering practice. This even after admitting he doesn't fully understand the engineering practices I refer to. :rolleyes:
As you may imagine, I'm more than slightly annoyed. ;) However it is his thread and he's allowed to commit as much revisionist history on it as he sees fit. This is the weakness of the open forum environment; folks can start a thread ostensibly as a knowledge base, then delete relevant material on a whim. I'm much more annoyed at myself though; for not keeping a copy of my hard work elsewhere. Lessons learned.
In all honesty, I don't feel we are disagreeing that much in general; both our work generally leads to two important points, even though they come at it from different angles.
(A) HOT-PLUGGING DEVICES IS BAD MOJO!
(B) Buffer caps at the power rails of any plug-in header is a good idea.
(3) Any questions, See (A) above.
Simply adding those buffer caps may be all that is needed; with them in place, the additional clamping zener may not even be necessary. But I feel the buffer caps need to be soldered in place to be properly effective; when you are dealing with ESR this low, plugged-in just doesn't cut it, especially over the long haul.
I understand needing to make a Plug & Play solution for those who don't want to solder to their expensive APM board; but we should offer a solder-in recommendation for those who are willing to do so for more reliable results.
The only point we really disagree on is the choice of regulators in the design; my perspective is that "If the old tried and true will work, why replace it with something much more expensive? Who cares if it is 10x as much as we need; it's 10x cheaper and known to be nearly grenade-proof." and that this issue with the TPS79147 supports that.
The devs felt that the TPS79147 was a better choice as it is intended for the expected operating loading, voltage range, and sensitive devices attached to it. As I said, I'm not privy to all the reasons for the choice; I gladly concede that. I just wanted to know that reasoning so I can evaluate whether the LT1117 is in fact an acceptable break-fix repair substitute given the unforeseen failures of the TPS79147 in the wild. I'm most concerned with issues of whether the power it provides is stable and clean enough for the sensors it is attached to; I've tested that as far as I know how to, and really need the help of those who have more experience with this product and have better test equipment than I have to answer that.
I promise; what I'm interested in here is NOT placing blame, as seems to be the nature of much of the discourse around this issue. I just want to come up with a simple, durable fix for those oodles of us stuck with the resultant broken boards.
We can't RetCon the design; these boards have been deployed. We CAN come up with a revision that makes it so we only have to fix it once; I'd like to parallel the existing development re: buffer cap and clamping diode with testing of the LT1117-3.3 as a substitute, but that requires handing it off to those who know the product better than I. In that, I can only hope I've offered sufficient evidence that it is worth their effort.
I'll recreate the lost work and post links here; the mod is REALLY easy to implement and easily reversible. This was why I felt it was worth the time if I can get the interest of someone able to do further testing; not a lot of time or effort is required to set up the test mule.
Paul
Okay... I've recreated the lost work and added it to my Blog at RCG, where I'm reasonably sure it will remain intact.
http://www.rcgroups.com/forums/showthread.php?t=2124570
Any suggestions?
Any volunteers to carry on my research?
[EDIT]
Rob - I agree... and in retrospect, if you look at it from a different perspective, my work actually tends to support his assertion that more is required to fix the board than just a buffer cap at the power rails. What the duck?!? As you say... "No mas..."
[/EDIT]
mnem
:popcorn:
Unbelievable. That's actually what I thought might have happened. I won't say anymore to avoid igniting another war. I feel bad about what happening at RCG. But there's nothing I can do to stop it. I tried, but had the exact same results as you.
As I mentioned, I've been soldering caps to the APM power rails for a long time. Just seemed like a good thing to do. As for changing the regulator, I don't know the answer. My guess is that it might work just fine.
I read your post - I don't know whats not to like about that write-up you did other than that it was not jabram's idea... I guess you can't make everyone happy :)
I thought it would be a nice alternative fix for a broken board for someone who wants to go a more sturdy route!?
Hello Paul,
Isn't it the surge current or voltage sag on the 5V side (input voltage) and the weak USB power that creates the hang up of the 3.3V regulator? Ok, its bad luck that the 3.3V regulator reacts that fatal to it and basically kills itself but its not burning up because its overloaded on the 3.3V output. Its because of what happens on the 5V input that is killing it.
I assume if there were an identical regulator in larger size of the same design family that it would act the same way!?
Size wise I would say you can take any other regulator of same size (current) that is more forgiving to those voltage variations on the input side but it doesn't need to be 10 times larger because the 3.3V current draw doesn't need it.
-
19
-
20
-
21
-
22
-
23
of 56 Next