This is a discussion re the bad Accel, Gyro and Baro values that we're seeing with ArduCopter-3.1.  The increase in the SPI bus speed from 500khz to 8Mhz has exposed a hardware problem on some boards.  That hardware problem is that the 3.3V regulator has been blown so all sensors are running at 5V instead of the intended 3.3V.

 

How have these regulators been burnt out?

  • Attaching a radio receiver or MinimOSD to the APM while the APM is only powered through the USB (see video below)

  • Some clone boards seem to come from the factory with blown regulators.  3DR boards might also come with blown regulators although they do a specific check of the regulator as part of the regular QA process.
  • It is not (as far as we know) actually caused by the AC3.1 software itself, it just exposes the problem.  You could prove this to yourself by checking the 3.3V regulator (see video above) before and after the upgrade.

 

How can we fix the regulator?

Option #1:  If it's a new board (so that it's less likely you burned it out yourself) you could report the problem to the retailer that sold you the board and ask for an replacement.  If it's 3DR it's called an "RMA".

Option #2: if you're handy with a soldering iron you can replace the regulator yourself.  On the APM2.5.2 (and higher) boards it's not that difficult.  On the APM2.5 it's far more difficult.

For APM2.5.2 : TPS79133DBVR

For APM 2.5: MIC5219-3.3YML TR

How can I stop it from happening again?

Do not connect any devices such as a radio receiver, MinimOSD, GPS, etc while the APM is powered especially while powered only through the USB cable.

Attaching a 100uF capacitor across any of the APM's radio input's 5V and GND pins will stop the regulator from being blown by plugging in a receiver.  video here!

There are very few reports of regulators being blown twice and no reports of it ever failing in flight.

 

Below are some graphs of the types of values that we are seeing on these boards.

Views: 104662

Replies are closed for this discussion.

Replies to This Discussion

For one of the tests, all buffer chips among others were physically removed from the board.  He was still able to trigger the failure mode.

Can you try shorting 5v to ground through an inductor?

One more thing for any 3DR guys listening, this may be a good time for a PSA announcement warning against plugging in peripherals while powered up.

Detlef -

I agree in principle. However, we routinely use transistors capable of switching 100 ma or more for logic applications where they're handling less than 1ma. Why? Because they're cheap, readily available, and they come in a package that's convenient to handle.

The LM1117 family (which I believe the MIC5219-3.3 is evolved from but on a thinner die) is like the MPS2222 transistor in that respect; it gets used for everything because it just plain works and is readily and cheaply available. Bottom line is the LM1117 is actually cheaper than the original part (0.12-0.20 ea/Qty 10, incl shipping), it's available everywhere, and if it works better, why not use it? I feel we should at least TRY it and see if it is as rugged in this application as its history has shown it to be in other similar applications.

I believe that adding a stiffening cap at the 5V rail to protect that smaller regulator is a band-aid fix at best; and we still don't know that this is the ONLY scenario in which it is dying. We only know that we can consistently reproduce the failure in this fashion SO FAR. On the OTHER hand, I also feel that both the 5V rails and the 3.3V rail should ALREADY have around 220uf or so of buffer caps as well; so... drop back and punt, says I.

I'm not sure I agree with the engineering choice to use a much more expensive (and now discovered to have an idiosyncratic failure) part to save what, 0.10 gram and 0.10 sq CM real estate on a board that's at approx 60% density on one side only? Now that we have to add 1 or 2 100uf caps to protect that part, we've lost even THAT advantage; it's just more expensive and more prone to failure.

That was my aim; to get some empirical evidence of a better choice of regulator before we repeat this in a 4th iteration, and to come up with a grenade-proof fix for already failed boards.

The ONLY real drawback I see so far to the LM1117 is that it has approx twice the footprint; and maybe a slightly higher noise threshold. I feel that is a reasonable tradeoff, especially in a post-failure repair scenario. I'd be inclined to make a small PCB with the LM1117-3.3 and a couple 100uf ceramic caps as a "repair kit" and call it a day, or just take the mod I've already done and add the 100uf ceramic caps at the 5V and 3.3V rails; this would stiffen the rails as well as filter any additional noise from the LM1117. One could pour a bit of epoxy over it all if one were afraid of it coming apart (Which I'm not).

It feels to me we've been trying to use a scalpel to carve away at this problem carefully and it's not working for almost 6 months and we're now into 3 iterations of the APM board with the problem.

Now I wanna say "Please, PLEASE can we break out he machete and get some work done?"

Notice I'm NOT asking to use the chainsaw which I ALSO have in the trunk. ;)

mnem

With your new regulator installed, what happens when you power it up via USB and hot plug in a receiver?

To fix your board I think any choice is fine. I choose the original part to fix my board because it fits right in place and if I don't abuse in the now known ways it stays alive. I haven't had a regulator fail without my fingers on the board...

With the current APM there are now two known issues - the voltage level translator and the regulator. While the voltage level translator doesn't seem to cause any issues other than that it is not translating voltages the regulator would need immediate attention asking for a board update in the near future in order to correct those issues - my opinion!

For such APM board revision I didn't say to stick with the self destructing regulator I am just wondering there should be something that is in the right size that is not as sensitive. But I am no electronic developer and I don't know whats on the market available and proven to work.

On the other hand which electronic device have you seen where the manufacturer does not warn you to plug or unplug devices while under power? That should be common sense even though it would be much nicer if the APM survives such things.

iskess,

    Updated!  look ok?

As bblatter said :)
Ok, a few things.... I will be doing further tests before calling this one.... I will be trying some other regulators as well. The reason this reg was chosen to start with, is it's stability under a VERY light load. So if we were to choose a new reg, we would need a few thousand flight hours to confirm it plays well with the sensors.

Keep in mind, we have no evidence that this issue has ever happens anywhere other than the bench. With that in mind, I look forward to testing any further theories people have. If I can still kill it with the cap in place, then we will figure out a better solution.

In regards to cap values. We want a reasonably large value on the 5v rail, and small on the 3.3v rail. So that when power is removed their is no additional Bias from 3.3v rail to the 5v rail.... See Joe's comments at RCGroups on that. It is not nice for the reg to suffer that :). With the current cap value (100uF) this is not a risk. But if we were to put a large cap on the 3.3v output, we could induce this.

So, lets put this fix through the wringer... And if it comes out good, we can get it put in the 2.6.2

In the meantime I have asked 3DR to make some simple Cap+Zener leads to fix this as well as the Nasty spikes from digital servos, that can be plugged into the A2(or similar spots)

Yes, I agree that we SHOULD not be hot-plugging devices; but I have a fair amount of knowledge. As designers, we have to assume our users, at least a certain percentage of them, are as dumb as a bag of hammers and WILL do something daft, like a dead short or reverse polarity or hot-plugging a device.

As you say; the latter is easy to forget not to do, even for those of us with knowledge. ;)

As important as making sure that we design things like user interfaces to be as durable and id10t-proof as possible, we are just as responsible for making sure that the end-user knows, in no uncertain terms, that if s/he does some of the more egregiously dumb things, it WILL let out the magic smoke and they'll be left with a doorstop.

This is great for the devices supplied by 3DR and approved by the APM DEVs; we can make sure the destruction manual says "DON'T DO THIS!!!" in bright red letters just like (And for the same reasons as) The HHGTTG has "DON'T PANIC!" in big red letters on the front.

Now we have folks like me and my board which hails from the Clone Planets (ShenZhen, GuangDong, Hong Kong) and don't come with ANY destructions; we're lucky if we get a ziplock baggie wrapped around 'em. We can only hope those poor souls come here and read the "DON'T DO THIS!!!" threads before they do those things... and when they don't, well... we try and help them save their gear as best we can.

bblatter -

My testing shows similar results as with the bare board; 5V rail right at 5.05V, 3.3V right at 3.31V.

All RX connected ONLY to BATT Port (no signal) on RX; APM INPUT 1. Each test condition repeated as noted.

Testing with my RX results in the following:

Old Futaba 72mhz RX (Filter capacitor 47uf radial electrolytic) Shorted Red /Black to discharge for 5 seconds, then plugged into APM: No reaction from board or scope. Board maintains connection with APM Planner in Live Data Mode. (20x iterations)

Turnigy 9X8CV2 2.4GHZ RX (100uf ceramic filter cap ahead and 47uf behind 3.3V internal reg)

If I short RED/BLK for approx 10 sec (Long enough to discharge both capacitors, I think) then plug it in, it will cause a momentary flicker on scope (MAYBE 100mv drop on 5V ONLY, but I'm guessing; I don't have a DSO to test with) and the APM will reset. I can repeat this every time. APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (20x iterations)

ORX 3XS w/DSM2 (220uf and 100uf behind 3.3V internal reg) I decided to try this one as a torture test; it is a 3 AXIS Gyro so has its own secondary CPU and I2C bus. It adds both large ceramic capacitors and a large (50ma vs 15-20ma typical non-telemetry RX) load for an RX. I just have to wait 5 seconds for this one to self-discharge; when plugged in it will cause 5mV drop on 5V ONLY, (but again I'm guessing without a DSO to test with) and the APM will reset. I can repeat this every time. APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (20x iterations)

Finally: 2 paralleled 10uf/0.01Ω ESR measured Ceramic Capacitors: Discharges instantly when shorted; when plugged in, it will cause a momentary flicker on scope and the APM will reset. I can repeat this every time. APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (20x iterations)

Here's where these caps get fun: If I leave it plugged in, I can plug in the Turnigy 9X8CV2 with no discernible reaction; no flicker on scope, no APM reset, no loss of communication to APM Planner. If I plug in the ORX 3XS w/DSM2 instead it will still cause a reset though. (5x iterations)

Out of curiosity, I decided to add 2 more 10uf Ceramic caps to my test bank. Measured Capacitance is now 40.69uf, 0.00Ω ESR.

Discharges instantly when shorted; when plugged in, it will cause a momentary flicker on scope and the APM will reset. I can repeat this every time. APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (5x iterations)

If I leave it plugged in and plug in the ORX 3XS w/DSM2 now, the APM does NOT appear to reset (no change in status LEDs as when the RESET button is pressed) but the TX/RX LEDs stop flickering and APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (5x iterations)

Fine... let's add 2 MORE 10uf Ceramic Caps to my array. Now it measures 60.99uf, 0.00Ω ESR.

Discharges instantly when shorted; when plugged in, it will cause a momentary flicker on scope and the APM will reset. I can repeat this every time. APM loses contact with APM Planner and will not reconnect until USB Unplugged/Replugged. (5x iterations)

If I leave it plugged in and plug in the ORX 3XS w/DSM2 now, the APM does NOT reset and no loss of communication with APM Planner. (20x iterations)

At no time during any of these tests was I able to measure any variation in 3.3V output at the I2C connector (aside from during the USB Unplug cycle) and my PC never gave any USB over-current warning.

Allright; it's late and I REALLY need to get started on my Taxes tomorrow. I hope my crude testing yields some useful data.

Peace out!

PK

Thanks Philip, for this insight into the thought process behind choosing this regulator.

I'm sorry if I seemed to harp on the old LM1117; if there were reasons it was passed up when it's such a well-known old standby, I'd like to know so I understand. My apologies.

I guess I'll have to get a bit more experience with the APM platform to understand fully the intricacies of the sensors and their interaction with the APM board; I look forward to learning from you all.

Good night!

*Toddles off to dead*

Using large capacitors can be a double edged sword. They take a lot of current to charge, putting strain on the system when you connect power.

I actually think there is some relationship to the USB source, so it may be possible that this does not happen on all PCs.
It used to be just general good practice to not hot plug or unplug anything.... I think we may all be in bad habits :)

I will emphasise from what I see, this is NOT a design fault. This is a good lesson NOT TO HOT PLUG :)
However, the cap has been added to the list of things to add for the next revision.

RSS

© 2019   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service