Hi All,
Rz_Ten1 was having some very strange crashes, and a few other users chimed in too once they heard his symptoms. He was able to reproduce it, then narrow it down to a hardware failure. His compass did not have the ground pin soldered leading to issues during flight and heavy vibration.
Even though HW was the issue, ultimately a software limitation of Arduino caused the APM to lock up. Most SW issues in APM can be quickly reproduced since we literally run the same code 200+ times a second. That's why I assumed HW first.
The library at fault is a poorly designed I2C driver. When it performs reads it "blocks" execution of the code. What we need is someone to write an alternative with the same functionality, with the addition of a timeout and error reporting.
We can use this new code in the hex generation and possibly in our own distribution of Arduino, until the main Arduino branch adopts it.
Rz_Ten1 has a head start looking at the code here:
After a quick skimming, these lines in twi.c really stand out:
// wait until twi is ready, become master receiver
while(TWI_READY != twi_state){
continue;
}
// wait for read operation to complete
while(TWI_MRX == twi_state){
continue;
}
Both of those look pretty bad. A simple loop counter (x++;) with a
maximum value would work, I think, ie:
int x = 0;
while(TWI_MRX == twi_state && x < 1000){
x++;
continue;
}
or this might be better, since the above will cause the program to fall
though:
int x = 0;
while(TWI_MRX == twi_state){
x++;
if (x > 1000) return 0;
continue;
}
What we really need in the code is to know that the compass did not respond so we can do the right thing up higher in the code.
Rz_Ten1 has offered to look into it more, but I thought we could coordinate the effort here. I would like to see this fixed ASAP so we can eliminate this safety concern.
Jason
Replies
Just FYI,
Tridge pushed out a new I2C Library that we're now integrating. It seems to fix I2C lockups caused by the compass.
Great news!!!
I can suggest this. Someone who had experienced failure could look into the datasheet for that particular compass module to find out if that model uses clock stretching. This is probably the only way an i2c bus can "hang up". This is when slave keeps signal (?) line high to buy some time for its internal operations of data processing, or when it can't respond to a master's request right away. So, when something happens to a unit at that moment the the line remains high and master will wait for a response indefinitely.
So the question is "does compass module use clock stretching?"
Any update on this?
I hope to fly this weekend, but I guess it'll only be in stabilize and Alt-Hold, I won't plug the compass in.
I am curious why this potentialy recuring error didn't apear during all the previous development and flying? It makes me feel that this I2C compass problem is not yet the problem that stops the code.
I did find freezing problems in my codes when SRAM was full. Sorry for not helping at that.
Perhaps it's a bit extreme but there is this i2cdev library which looks like a replacement of the standard arduino Wire library which seems to have non-blocking calls and a timeout. I should add that I've never used it.
Let me hop on the bandwagon while this thread is still active :)
The lines found by Rz_Ten1 are there for a great reason to be a guard or a last resort. I've put almost same lines in my interrupt-driven code for the pic24 (mini-bully) based ahrs. Usually the code expects certain state of the I2C when a read is requested. For instance, before reading I always check is a byte(s) are waiting in the buffer - then there is no delay and I can do other things.
The state of the i2c driver should be also checked either by looking at the state register or comparing software state variable to the expected state. Then I can always distinguish 3 cases - hardware failure, i2c protocol failure or software protocol failure. In the first two cases i2c is taken to a known state by resetting i2c, in the second case no reset required. At least this is my experience working with i2c.
In arduino wire implementation everything is done correctly as far as I remember and one can always distinguish between those failures. At least I can confirm that when I physically unplug power or i2c connection between my flight controller and ahrs they are both responsive and restore communication after I plug in the wires. My flight controller is based on arduino pro mini and uses wire library.
Well, I'll be anxiously following this situation because my heli is grounded until this is resolved. :(
Hi Jason,
I just realized that the MatrixPilot I2C driver might be a little hard to understand if you are not used to the dsPIC processors and/or interrupt driven programming, so I thought I should probably explain a couple of things about the driver:
1. The routine rxMagnetometer() is called on a regular basis, each time that it is desired to get a magnetometer reading.
2. Setting _MI2CIF = 1 causes an I2C interrupt to be generated, which causes the _MI2CInterrupt() ISR to be called.
3. The driver uses a state machine concept. (* I2C_state) () causes a handler for the present state of the I2C driver to be executed. Each handler determines what the next state should be, and sets I2C_state accordingly.
Let me know if you have any questions about how the MatrixPilot I2C driver works.
Best regards,
Bill
Hi Jason,
You have my sympathies. I2C drivers are not as easy to implement as the I2C marketing literature would have you believe...I have the scars to prove it. ;-)
Here is link to the MatrixPilot I2C magnetometer driver. You are welcome to take a look at it for ideas.
The MatrixPilot team was lucky enough to run into interference problems with our I2C driver early on. I still use a 72 Mhz Tx, and when the Tx is brought close enough to the magnetometer, it causes it to go into an unknown state, and lock up the I2C bus. Along the way, I discovered that the Honeywell magnetometer can get into some states that are forbidden by the I2C protocol!
In the end we were able to make our driver "bullet-proof". You can bring a 72 Mhz Tx right up to the magnetometer and/or disconnect and reconnect the magnetometer, and the I2C driver will recover.
One advantage we have in implementing our I2C driver is that everything in MatrixPilot is interrupt driven. There are no places in the code in which the code is blocked waiting for something.
Best regards,
Bill
Thanks for this guys.
I'm not sure if it's just coincidence that my first symptom was loss of tail control? Would that indicate a problem with the Mag? Probably not.
Anyway, just wondering if the problem was with the soldering of the ground within the board, or the wires *to* the board? I understand that a SW fix is needed, but I just want to know if I should have a closer look at my Mag as well. The wires were torn from the mag on impact, so that's one less thing for me to inspect.
Now, I'm just wondering, does the theory that this is the cause agree with my symptoms? You say the code stops executing. It appeared to me that it stopped, but then restarted just before or just after the impact. What I mean is, the log was running, then seems to have stopped or been unaware of the state of the aircraft, but appears to have started again and recorded the crash location. Does that make sense for this problem? I wonder if what happened was that something vibrated loose in the mag, the program hung, then after impact whatever was loose made contact again and the code picked up executing right were it left off?
That's not a bad theory actually, as that's kind of what it looked like. The log shows time passing, but no changes in data occur, then the heli is suddenly at the crash site and I think it even recorded some of the crash forces.
How does the I2C driver react if the Mag is enabled, but completely not present? Because currenty I'm rebuilding my heli, the mag is completely detached, but the program still runs.
-
1
-
2
of 2 Next