Mick pleas add your logs so the "experts" can have a look.
I would lean more to the PID's for you Alt Hold/ Loiter. I had the same issue until i adjusted the PID's and the jumping bean is no more. How big (KV rating) are your motors? If they are like mine (1000Kv) then you will need to really reduce your PID's from the stock setting. I would start by reducing the P term by 1.0 (ex. stock: 2.5 new: 1.5) and see if that helps your jumping.
At the very least DO NOT try any other settings/ flt modes until you get Alt Hold or Loiter set! This WILL end in a crash/ fly away almost every time!
Then you might need to Add some P gain.
But like Dany said, to better help you we would like to see the logs from you copter to REALLY be able to help
In Alt Hold and RTL your throttle proportional (P) gains or perhaps though less likely your derivative (D) gains are drastically off (way too high)
In your simple mode your correctional proportional gains are too low (once again could be derivative, but this is rarely the case)
Your Integral (I) gains may be wayyyyyyyyy to high in the simple mode, which can cause long and under-responsive correctional tendencies.
I highly doubt it is a hardware issue.
More info: Hexa; motors 880 KV; APM 2.5; v2.9.1
Logs attached (sorry, no Wireless Telem).
I agree usually too high PIDs for a high power of copter will give a up and down movement when attempting to hover.It seems you can never find the right throttle to hold a given altitude. Same thing with other control boards such as KK2.
As far as the RTL, If you've been hovering, loitering or generally staying in one spot and switch to RTL the aircraft will almost always turn 90 degrees or more head off in the new direction until it can get a compass heading cross referenced with the GPS. THEN it will actually return to launch more directly. I've done the RTL test about half dozen times from a loiter and it has done this each time.
Mick asked me to upload his log files here... he has issues with the forum... let's see if it works ...