I've read what I could find on the topic and most of them point to it being a compass or vibration issue. Compass drifts slightly but stays in the correct direction +/- 5 degrees, so I don't think it's that. compassmot was run which significantly improved compass. Vibration looks acceptable to me, not the best but it should be low enough to hold position with toilette bowling or some drift.
Essentially what happens is as soon as it goes into loiter it starts drifting. The most consistent drift is it pitching back and flying towards me. Occasionally it drifts wherever I feel like. I'm at a total loss of what to look at or try to tune. If anyone could take a look at the logs and see if there is something I missed I'd appreciate it.
EDIT: Using SimonK ESC's with 750kv motors, 9 inch props, on an HT-FPV frame with APM 2.5 and ArduCopter 3.0.1 loaded with default settings.
(ESC calibrated, compass/accel calibrated, compassmot run)