I've got a fair sized quad, 6 Kg and 4 Turnigy G46 motors an 15x8 props.
I've been several months more than 3, (just weekends) tryung to get loiter working properly.
I THINK I have position hold OK now, but altitude hold just isn't doing it for me.
I use the CRIUS V1.0 all in one and megapirate 2.7.4
At one point it seemed to be working, but put it in a slight wind and forget it. Just talking P gains, with the hold gain at 2.4 and the alt rate gain at 0.11, it would slowly drop down about 4 meters, and then near the bottom, zoom up to the altitude where it should be, but there it cuts power and starts dropping down again.
I decided to put it back to where it seemed to work better before, alt hold gain, 0.43 and alt rate gain at 0.33. Seems better, but I have to have it perfectly stabalized before puting in to loiter. If it's off by a bit, it drops like before, and zooms back up again after about 3 meters or so.
I the last test, I had done a first test which did as I described, I flipped it into stab before it fell too far. Got it back to a good position and stable again, then flipped to loiter again, Again it dropped a little and then shot of, as though it was in RTL mode. The cycling mentioned before doesn't very throttle all that much, but this last move cranked the throttle up to about 60%, and it headed of to a very definite direction, like it thought it knew where it was going.... in any case it wasn't heading for home.
Flipped to stab and got control back.
From time to time, it has beetled off before, but usually early in a test. I found that you need to give a good three minutes for the GPS to stabalize on a home position. So now I wait for a good lock before I start any tests, and also, on this occasion, the first test of loiter didn't beetle off.
No-one at "another forum" could help.
A big part of my problem is having time to work and test. At best I get two 15 min tests per weekend. And since I don't have telemetry yet, test cycles are more complex.
I'm getting the alt hold gains adjusted. Still doesn't hold better than +/- 2 m. and for the same reason, it's still not very stable in wind. So I'll see if I can find INS_MPU6K_FILTER to help solidify there, and go to a 2.9.5 where I can get more out of the accelerometers for stability in alt. and loiter.
During tests, I find a strange thing every once in a while, and it's sort of like what happens in the RTL Starnge Circles post in Top Content lately.
I put it in Loiter, and it B-Lines off as though Home where somewhere else. Checking Mission PLanner, all looks well. This last time, it did it twice in a row, so I landed, and reset the processor. Waited about a minute, saw that the GPS had it positioned well, and when I took off and flipped to loiter, it worked fine.
Other times, it flies OK in loiter (despite the alt hold problem), but on a next attempt to flip to loiter it beetles off. It's like the home position randomly gets set to some other place. Wouldn't want that to happen on a Failsafe RTL!!
I run a new small business (www.droneresearch.co.uk) and had exactly the same problem with one of our large X4 quads using the crius v2 board and megapirate 2.9
The problem was solved by reducing this setting to 20hz (INS_MPU6K_FILTER) and waiting a full three minutes after connecting power for a good Sat Fix and Baro to settle. Also we found the best altitude lock at no less than 10 meters.
Now we have no more bouncing and good altitude hold to around half a meter. Just though this would be of help as its taken us three weeks or testing to solve the problem.
Interestingly this bouncing and climbing would not happen on 3S or 4S lipos, but on 5S its was impossibe to get a stable altitude/loiter situation.
If you want to see some of the strange craft we are building for customers etc, take a look at our facebook page https://www.facebook.com/advanceddrones
I have a similar sized quad that I fly with an AUW up to 6.5Kg (depending on what I'm carrying) using Turnigy SK3 4240 motors with 16x8" props. I am sorry to say that I have no experience with your controller hardware/software - I currently fly the version 3 Arducopter code with the APM2.5 and although I have tweaked my settings a little, I have found that the defaults settings provide very good results. In fact, I have to depart from the defaults quite considerably before I get any sort of adverse behaviour. A large part of getting my system to behave nicely came down to vibration isolation. Could this possibly be a factor in your setup?