The solution is a bit of an anti-climax so in short:
I got Loiter to work by re-flashing the firmware.
The longer version:
I built the quad which was a 3DR ArduCopter Quad-B. Everything seemed to work like a charm. It flew like a dream in stabilize and the GPS got a really good signal (over 10 satellites). I had read about how people got at least poor loitering with the B model 3RD quad but every time I turned on any of the auto modes the quad would sit for a while and then shoot of to a random direction. Altitude hold would however work.
I fiddled with the PIDs and examined logs for hours. Thanks you guys here who tried to help me. Nothing worked. Nothing actually made any difference to the auto modes.
After 3 weeks of this I emptied the EEPROM, re installed the firmware, did the config exactly as before and tried it. It loitered, it returned to launch and flew missions straight away! Not very well but at least it tried! :-) And now after some PID tuning it works significantly better... But the lesson being. If you cant fix it, try re installing it... :-)