Hello,
Here are key points in a few words:
- The "BAD ACCELL HEALTH" error may be induced by two different causes:
- IMU1 error - MPU6000 soldering issue while production in some early batches of PixHawk. Currently is fixed by 3DR.
- IMU2 error - LSM303D improper discharge process during power off.
- This solution is only for the LSM303D case.
- Needed hardware parts always were there. They can be controlled by FMU firmware.
- For some reason this feature seems has never been used in FMU initialisation. But it was always accessible via nsh CLI as "fmu sensor_reset" command.
- It seems the reason why it isn't used in FMU initialisation is a bug in sensor_reset function. It breaks SPI functionality in firmware.
- The fix for broken SPI after "fmu sensor_reset" is here.
- It permits to use sensor_reset function in FMU initialisation to properly discharge IMU2 to avoid IMU2-related BAD ACCELL HEALTH.
- It has been implemented as firmware fix and available here.
- Pre-built 3.2.1 image with mentioned fixes are available here for testing purposes.
- Currently I testing v3.2.1 fixed firmware. You can do the same on your own risk.
- There is a chance it will not resolve the issue. Then hardware fix will be needed. It is proposed below also.
- It could be implemented with a minor design change (Q601 replacement).
- Also it may be implemented as post-production hardware workaround (SMD resistor soldered on top of C506).
UPDATE 1: 20ms is not enough
Tests were performed with some number of flights. This fix don't breaks anything. But it doesn't work with 20ms discharge time. Going to increase it up to 1s, but I have not too much hope it will do. It seems pure software solution will not fix it w/o hardware fixes.
UPDATE 2: It depends on temperature
I have two items of HKPilot32. I flashed item #2 with original ArduCopter-3.2.1 Item #1 already flashed with 1s power reset fix. I put item #1 in refrigerator and started with item #2.
Powering attempts numbered as they were performed in common sequence. Powering cycles were performed by battery disconnection and connection back. To reproduce issue a disconnection time should be less than 1 second. My results are below:
Item #2, original 3.2.1, room temp about 22C
--------------------------------------------
bad: 2 4 5 6 7 8 9 10 11 12 14 15 16 17 18 19
good: 1 3 13 20Item #2, 1s reset fix for 3.2.1, room temp about 22C
------------------------------------------------------
bad: -
good: all from 21 to 40
WOW! Doesn't it work?! Let's check it with cooled item #1.
I took item #1 out of the refrigerator and put there an item #2.
Item #1, 1s reset fix for 3.2.1, from refrigerator
--------------------------------------------------
bad: 43 44 45 46 47 48 49 50 51 52
good: 41 42 53 54 55 56 57 58 59 60No luck. What about to warm it a bit?
Item #1, 1s reset fix for 3.2.1, after 30 minutes in room temperature
---------------------------------------------------------------------
bad: -
good: all from 61 to 80It looks like it works for room temperature! But what about cooled item #2?
Item #2, 1s reset fix for 3.2.1, from refrigerator
--------------------------------------------------
bad: 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
good: 81Being cooled it doesn't work again.
All logs are here. Next step I going to play with 5sec power reset on cooled FMUs..
UPDATE 3: It mostly works for 5 sec power reset time
Item #1, 5s reset fix for 3.2.1, from refrigerator
--------------------------------------------------
bad: 107
good: 101 102 103 104 105 106 108 109 110 111 112 113 114 115 116 117 118 119 120With 5s reset time it looks like it mostly works for the same item and same temperature.
Also it seems for the attempt #107 I've had really small (about 0.2s) disconnection time. It is an unlikely scenario of actual copter use.
Link to test image is updated, now it have 5sec reset time fix. It would be nice to have test results from people suffering from this problem.
UPDATE 4: It seems temperature dependency is not about capacitors
I measured discharge process on C506 of the same FMU item for two cases: a) FMU have room temperature and b) FMU has been cooled 30 minutes in refrigerator. It appears that measured values difference is really subtle. It is about 20mV difference for discharge level on 5 sec period. So it seems the temperature-capacity dependence is not a factor for this issue.
UPDATE 5: MOSFETs test results
Performed tests with SPICE models and in hardware (a number of MOSFETs were used) demonstrated that issue cannot be resolved reliable w/o hardware fix due to the fact Rds goes high when Vds and Vgs going down. VDD_3V3_SENSORS rail needs to be connected to GND via 220 Ohm resistor and it still need a software fix. As result the EN1 input of mic5332 doing its job perfectly w/o any MOSFETs. But it will not work reliable for the case with resistor connected between any other power line and GND, this way the issue still possible to happen for cases of short power disconnection.
UPDATE 6: hardware fix is enough for most cases
I must admit that it seems working ok even without a software fix. Just performed a number of tests with original ArduCopter-3.2.1 firmware on cooled (all night in refridgerator) board with 220 Ohm resistor which is installed in parallel to C506. As result: by disconnecting and connecting battery back with my hands I was unable do it in enough short time to reproduce issue. All logs from attempt 121 through attempt 140 are showing IMU2 works just fine. Software fix still needed for non-fixed boards only. In case you are able to (de)solder SMD parts the better place for resistor is instead of Q601, you need to desolder Q601 and solder resistor on emitter and collector pads.
Owners of non-modified boards are have to wait for a software fix. It have a chance to be included in ArduCopter-3.3.3.
Here is a long story:
Some time ago I decided to switch from APM to PixHawk, so I bought two items of HKPilot32 FMU from HobbyKing. So far I can see they are clones of PX4 FMU v2.4.5. Boards are labeled as "px4fmu-2.4".
Those days I had not too much experience with PX4 platform, so all intermittent issues and misbehaving were considered as a lack of my skills. When I first stuck with BAD ACCEL HEALTH issue I googled for the issue and found some mentions of power filtering and so on. So I decided to put some ferrite rings on power lines to FMU and FPV, to decouple possible noise and spikes on power lines. It seemed as it helped me for some time and I've seen no more BAD ACCEL HEALTH for a long time.
But later it happened again. So I decided to investigate it in a more details and googled this thread.
While checking my DataFlash logs I realised it wasn't MPU6K issue. It was definitely LSM303D issue. I read through all in the mentioned thread and I collected all important points together in a single document. Also I collected there some important points drom other forums like HobbyKing and Plololu. (All 'hello' and 'thanks' were skipped.)
I discovered that possible key to the problem is in the message by Artem on February 19, 2015 at 11:48am: "first hit is pololu community forum thread. Apparently lsm303d is very sensitive to how it needs to be powered down". So I googled once more and found original message on Pololu forum: "When we were testing the accelerometer on the LSM303, we noticed we could get a similar behavior where the accelerometer constantly reported a single value for all axes. It seems that if you interrupt power to the accelerometer in a certain way (like disconnecting or turning off/on power) so that the voltage falls below a certain amount but not all the way to 0, then it can brown out and get stuck in a bad state".
So I disassembled my HKPilot32 and downloaded all pix4fmu-2.4.5 schemas and pcb layouts. As you can see there are capacitors (C506, C507) on VDD_3V3_SENSORS net close to LSM303D, also there are some capacitors on the same net close to LDO and some capacitors should be close to other sensors:
Having a hope that I would be able to solder SMD resistor on top of C506 I found exact location of C506 and started to measure LSM303D power voltage on C506 pins:
Here is how LSM303D power drops down once battery get disconnected. There are no any other modules connected to FMU or PM (Power Module). As you can see it takes about 50ms to drop on level 0.8V and it takes about 0.1 sec to drop on level 0.6V. And it doesn't fall below 0.344V even after 1 minute since battery get disconnected from PM. It's definitely not good. And it can be even worse in case other modules are connected to FMU or PM.
I was ready to solder SMD 1k resistor on top of C506 when I suddenly discovered Q601. Then I traced VDD_3V3_SENSORS_EN net and..
WOW! It is controlled by PE3 pin of STM32F4 and it is there definitely to discharge sensors power line and to shutdown LDO output. By design it looks like it should do sensors power line shutdown in a proper way. It was designed to do exactly that we need to avoid BAD ACCEL HEALTH.
So why it doesn't work, what I missed? Broken Q601 transistor? PCB issue? Anything else? Well, let's measure it on R620.
To shutdown sensors power line the base of Q601 should go down with logical zero on PE3. And I believe it should happen on FMU initialisation. Ok, FMU is powered, pushing reset button..
During FMU initialisation it goes up for some (really long 5.4sec) time, until PE3 will be configured as an output. Ok, no problem. Then I setup trigger for my oscilloscope on 3V level to catch voltage drop and no luck. It means Q601 doesn't receive logical zero from PE3 on FMU initialisation. At all.
NOTE: Hereafter I will refer to source code of version 3.2.1.
Ok, it's to time to dig the source code. What about to grep PX4Firmware sources for "SENSORS_EN" word? Here it is. It is used in "sensor_reset" method to do exactly what we needed. Let's see where "sensor_reset" methods is used. Nowhere else, for the moment of v3.2.1 released it is used just in the same file "PX4Firmware/src/drivers/px4fmu/fmu.cpp". What about v3.3.1? Checked, the same.
After browsing fmu.cpp file I can state definitely: for versions 3.2.1 and 3.3.1 it is used just to provide ability to run nsh command "fmu sensor_reset" with an optional parameter. Nothing more. And it definitely doesn't used by FMU initialisation process. Ok, let's try to execute "fmu sensor_reset 20" to shutdown power line for 20ms.
It works! As you can see it falls down to 0.8V in 1.8ms and to 0.6V in 10ms.
The bad side is that it doesn't falls down to zero due to Vce(sat) parameter value for Q601. Q601 is BJT transistor of type MMBT3906 and it have Vce(sat) from 0.25V to 0.4V. So VDD_3V3_SENSORS net cannot be discharged below Vce(sat). In case a pure software solution with PE3 and Q601 will not do for BAD ACCELL HEALTH, then design for Q601 should be changed to use some MOSFET instead. It permits to discharge sensors power line much closer to zero level. Also there are low Vce(sat) BJT switching transistors on the market.
The good side is that level of power line discharge isn't an only important factor of issue. The rate of discharge may also affect LSM303D behaviour and we able to fix it right now. Anyway, here is chance it will do so let's try it.
I implemented it this way and performed FMU reset by pressing reset button. Next moment my board is turn on multicolour LED as red and it means a error while initialisation. I connected to PX4 terminal and found following startup error messages:
Starting APM sensors [MS5611_SPI] on SPI bus 1 at 3ms5611: interface init failed
Error in startup
Then I turned on debug messages and realised that SPI is affected:
Starting APM sensors [MS5611_SPI] on SPI bus 1 at 3ms5611: prom all zero
ms5611: prom readout failed
ms5611: interface init failed
Error in startup
After some reading of "sensor_reset" source code I discovered the cause of SPI issue. It doesn't enable back SCK/MOSI/MISO pins after reset. So I fixed it and then FMU loads ok.
As a bottom line:
- I unable to see why mentioned hardware feature (Q601) being known for a long time is never been used in FMU initialisation.
- It may fix IMU2-related BAD ACCEL HEALTH issue. Needs to be tested.
- In case it will not, then really minor design changes are needed to fix it in reliable way.
Comments and corrections are appreciated!
Regards, Dmitry Prokhorov
Replies
helo all,
I recently bumped into this issue when I was going over the schematics 2.4.5 and saw that the LSM303D circuit didn't correspond to the one suggested by ST itself. On the ST datasheet the lsm303D uses total different cap values and allot more parts than to be found on the pixhawk pcb. Could this be the issue or have at least something to do with it ??
Then again I got a response that the datasheet was updated by ST and that before that they had an old one, however I find this very unlikely and HARD to believe . I will ask ST about this if this is a lie or not, if it is .. it's a very poor one.
![3702281333?profile=original](https://storage.ning.com/topology/rest/1.0/file/get/3702281333?profile=original)
I added picture in attachment so you can notice the difference , there are many things that are not on the schematic not just the one I circled around to show the difference in value.
greets
You are looking for 220
You circled 122 or 1200 ohms... or 1.2K...
In a pinch the 101 below that would work but add load to the 3.3v supply
Like I said, there are LOTS of 220 ohm on old APM boards... none of those retired in your box of bits?
No old APM boards laying around here :( - I had a hope these were in fact 221 due to that small tip on top of the 1, indicating it is 221 and not 122. Well gotta keep searching for the right one. And if I find a 201 (200ohm) I guess I am better off than with a 101.
Dmitry, thank you for your awesome work and effort on this issue.
I have a pixhawk derivate called pixraptor. I am experiencing the exact same issues (inconsistent accelerometers/bad accel health etc. in cold temps/and when switching battery) as others with pixhawk clones.
I tried installing your 3.2.1 fixed firmware, and the bad.accel.health actually seem to be gone, which confirms I have the same issue. But I am not receiving the blue/green flashing lights and the ready melody is not playing from the FMU. This could perhaps be caused by something else. Is your modded version of the 3.2.1 firmware flyable? Or only for testing that the resetting of IMU2 is working?
The fix has not officialy been implemented in 3.3.3, or 3.4 beta. So I am suspecting we might not see an official software fix for this. Are you able to implement the fix in a 3.3.3 build?
Anyway, the layout of the Pixraptor is different than the Pixhawk, se attached macro photo of the board. (Full high res, and a picture of other side/top of board is attached as download) (The VDDA_3V3 is located on top of this board) The Pixraptor has the IMU separated from the main board, to decrease vibrations and increase stability. Well. Are you, or anyone else, able to see where the hardware mod should be done on the Pixraptor? And can anyone give a tips on where to find the needed resistor to solder?
Trying to decide if I should pull the plug and just buy a genuine Pixhawk, as I need my X8 copter to be working 100% in cold temperatures, at all times, since I live in the arctic. Short circuiting the power module does not help me flushing the capacitor and getting rid of the problem.
That new 2.4.6 clone version is of course also an option. But I need to know I never get these issues when out in the cold.
Again thank you for your effort on this!
Ole
DSC04776-2.jpg
DSC04778-1_web.jpg
Not seeing the LSM303 chip on either of your pics so I am no use to you on this one. Don't know what to suggest. Sorry.
Hey Terry, thanks for your reply. Silly me, the IMU is on a separate board on the Pixraptor. Here is a picture of that board. Hope we can find what we are looking for here. I cannot see LSM303 written on any of the components, but it may be the larger silver one?
Oh it is definitely there, down to the left. Now where to solder?
This pic might be slightly better.
As best as I can tell, this should be the location. or I should say this is ONE location... there are a few but this looks to me to be the easiest location. You don't replace that component, you solder the 220 ohm resistor on top of (i.e. in parallel) that component. While it is really tiny, it is not impossible with a super fine tip soldering iron. I have soldered a 64 pin QFP SMD chip with a soldering iron successfully in a camera under a microscope. You have to clean the tip EVERY connection and use 1mm or thinner solder (I actually STRETCHED my solder to make it thinner and I cleaned it with steel wool to make sure there was zero dirt and oxide to make it take better).
Terrry, I did the mod with the help of a friend. After letting the controller rest on the copter out in the cold (7-8C) for hours, cutting power and reconnecting 15 times, I have only been able to provoke the problem with "inconsistent accelerometers" once, and then I just reconnected power and we were good again.
So it seem like this mod actually worked.
(Perhaps one of the other locations will work better?)
I am far from an expert in soldering, and the second attempt done by my friend actually looked much better than the picture here from my attempt. But I don´t feel like opening again just for the purpose of documentation. My friend did say though that the components inside the Pixraptor was smaller than on the Pixhawk, so the soldering job was harder on the Pixraptor. The 220 ohm resistors he had been using on his Pixhawk, was in fact to large for the soldering mod for the Pixraptor.
Anyways, thanks for your valuable input. It does seem it is working, at least so far in my testing. Will test more, and update if any change. - Thanks! - Ole
Good to hear your testing went well so far Ole. It was for sure a bit cramped on that imu board with small smd components but with the help of magnifier glasses and an electronic microscope the soldering went quite OK.