A few users, myself included, appear to be having issues calibrating accels and/or arming when using APM 3.2 on a Pixhawk. I see posts here and there, but thought it may be best if those having the issue could all 'check in' to one post so that the developers can see how widespread the issue is and gather data.
My Pixhawk will arm occasionally, and will even fly successfully. Occasionally however it will not arm and I will receive the pre-arm error "Accels not healthy". A few days ago I was able to arm and fly thorugh one lipo, then after landing and swapping batteries I could not arm due to this reason. I have attached the logs from the successful flight, as well as the failed pre-arm logs, to this post.
As you can see from these images there is something amiss with IMU2. This is a snippit from the good flight, showing values for AccX on IMU1 and IMU2:
and here are the same values on the second attempt, pre-arm.
In the 3.2 release thread a few users had mentioned the issue and Randy advised to set the log_bitmask to "131070" so that it will log everything including the pre-arm checks. I encourage others having the issue to do the same and share the logs and experiences here so that we can find out what is going on here - is there a bad batch of Pixhawks in the wild that only now show the hardware errors due to something new in 3.2, or is there an issue in the APM software?
Replies
Rob,
First, I fly fixed wing but am commenting and seeking help in a copter thread. There may be differences by virtue of the FW.
That said, my specific experience wrt "bad accel health" does not lead me to conclude there is a solder joint fracture. Why? Typically poor solder joint integrity usually manifests itself either as open or intermittent. If intermittent, vibration and/or thermal excursion will reliably reproduce the failure signature. In my case plugging and unplugging the battery randomly produces the bad accel health warning. Granted, powering up the board could produce sufficient heat to close the solder joint enough to conduct again but doing this 10 to 12 times results in hits and misses of the bad accel warning.
In essence, it's entirely plausible that a solder joint failure was causing some of the anomalies but for me, I don't think it explains all of them. Sometimes there are two failures causes occurring simultaneously. The smoking gun may have had blanks.
As for my logs, I erased and started fresh on both my Pixhawks. I erased everything! Installed a Copter FW and then erased again and installed Plane FW. I now have two flights on one of the Hawks with no warnings. It's on the SAME plane, SAME environment, GPS sats and HDOP were very close as well. I also examined the solder joints on the IMUs at 20X and can see no evidence of a bad solder joint. (could still be a micro-fracture) Btw, I also GENTLY flexed and twisted the Hawks in their housings without creating a bad accel warning.
I did review logs with warnings and those without. On the ones with warnings, the signatures looked like Trdge's case study. BUT the ones without warnings looked fine and would have passed scrutiny. So, if one has a "good" log, it doesn't mean they necessarily don't have the issue.
If I get more anomalies, I will certainly post, in hopes of getting some assistance. In the mean time, it would be really helpful to a lot of us if someone could make a tutorial or YouTube on reading logs for the most frequently encountered issues.
Thanks!
Steven G
My oppologies if I was out of line in some way. Certainly the exchange of ideas and data here is what has helped the devs figure out what the issue is. Perhaps there is more yet to figure out. I was just trying to express my thoughts on the best way to help them out in that regard. Not that that was my place to say. In any event hope you get your intermittent problem figured out.
re- cold weather effecting the controller
I live in Michigan. It's been VERY cold. I tried many times to see if temps had any effect on it. I brought it outside and started it right up in 10 degrees no issues. Next I let it sit outside off for 45 minutes in 10 degree temps then I started it up and flew. No issues. About the only thing I was really worried about was my props being really brittle. I've had several dozen flights in cold weather never had an issue.
Thanks for the input. Are you stating that the IMU failures have not been intermittent? When they fail, they fail hard and not soft? I guess I didn't understand it that way.
Further, by deduction, the failures or should I say warnings, that can be cleared by unplugging and plugging the battery are due to vehicle environment?
Are there trouble shooting guides? Or a fault tree logic diagram?
Is there another thread with more on trouble shooting? I don't to want side track this thread if there's anither more appropriate spot to discuss.
Thanks!
I say that I am no expert in this field, but for what it's worth. Walkera insist on using two battery connectors, with the neg. connected first and then the positive. The reverse for disconnection. Some healthy Cap. sparks to be seen....
There's always this solution:
http://scriptasylum.com/rc_speed/nospark.html
I've also seen XT-90 connectors with a spark eliminator built into the connector itself. As you plug the unit together, there is a pre-shorting resistor that charges the caps prior to fully seating the connector.
I don't believe this is the cause of random boot-up errors. I've been watching this thread and have been in contact with 3DR and have had conversations with @Craig Elder. Craig has pointed out some additional pre-check functions that occur in AC3.2 (different from AC3.1.5), but I believe there are additional issues. I'm an Electronic Engineer (by BS degree), and something just doesn't make sense when the errors are cleared by simply re-booting the F/C. Temperature change is not an issue on my F/C, I live in Vegas and the temps have been in the mid 70's for the past several weeks and I still experience the errors. For me, taking the F/C from indoors to outdoors is not the problem. Power supply is not the problem, I run a very stout UBEC, and have been very careful to not move the F/C on boot-up (gently plug the battery connectors). I've also upgraded my MinimOSD firmware to the r800 version, and 50% of the time I boot-up, I get a "No Mavlink Data" error, then I power-cycle, and get Mavlink data. My OSD worked perfect everytime prior to the F/W upgrade.
I know I have described a few different errors and conditions which I am not asking anyone to troubleshoot, I'm just stating that only about 25% of the time do I have a stable boot-up. No changes what-so-ever in my hardware, or wiring, prior to the AC3.2 upgrade, it's just not reliable any more...
@Craig has invited me to look at the firmware coding, I just need to be pointed in the right direction to get started. I did a lot of programming 10 and 15 years ago, so learning one language vs. another shouldn't be a struggle.
Sorry, just frustrated, had only one good flight out of two this past weekend with the Pix-Hex, the first of which resulted in another broken landing gear (no thrust with stick full up on a gentle decent, resulted in a "bump and go" with the ground) was on a 10,000mAh battery only two minutes in on the flight. Was able to land, re-boot and fly another 15 minutes on the same battery, so charge and/or weight was not an issue.
On a side-note, got six great flights with my NAZA M V2 on my F450. I've asked 3DR for an RMA / exchange on my Pix, because if AC3.2 really is reporting additional errors on the F/C and they were always there before AC3.2 (just not reported), then there is something wrong with the F/C in my opinion. 50% success rate for flying, and 25% success rate on boot-up is not leaving any "warm-fuzzies". I may be on the short list for pulling the Pix because wrecking my $1800 multi-rotor is not an option.
I'm also a EE. I've worked on many detailed Failure Analyses, as well. A cracked solder joint on a SMT device could show up as intermittent with vibration or thermal excursion. In fact, I have seen this before. One could stretch and say that plugging in the battery heated the PWB and thus caused the connection to close. But as in your case, I could plug and unplug numerous times with random results. This signature leads one away from the cracked solder joint theory and more towards a timing or sequencing problem in the design. It could also be a self calibration problem at the device level. One thing is for sure, there are enough occurrences that this is not a very isolated, random anomaly. There is a root cause somewhere.
This is just my two cents and not meant to step on the efforts of 3DR and others to isolate the failure cause.
Steven, you and others are confusing when the Pixhawk reports problems with the setup vs reporting problems with the sensors. The cause of the sensor failures is well understood. Issues that create problems with a specific vehicle setup are also understood and the autopilot is dutifully reporting a noisy position / attitude solution, but those issue are really up to the designer of that particular vehicle to resolve.
Craig, et al,
I sounds like 3DR has determined the root cause of this issue and contained it. Can we assume that all Pixhawks that are currently shipping have been screened for the Accel anomaly?
Thanks!
Yes, that is correct. Every function and every device on every Pixhawk is tested and demonstrated to be working before it leaves the factory.