This is just an idea, and not a well formed one, but would it be possible to send all the IMU sensor data from a quadcopter UAV to a remote ground station or laptop that's running an autopilot program, then send motor commands back? Its the same idea as a remote terminal PC, if you do the processing somewhere else, the remote terminal (quadcopter UAV) doesn't need to be powerful or expensive.
A PC could compute an extended kalman filter for the UAV a lot faster than a microcontroller. A setup like this could potentially coordinate multiple UAV's at the same time, much like those videos of motion tracked quadcopters in netted rooms but out in the field.
I've heard Xbee has some latency issues. Is there any wireless serial link that would be fast enough to do this? A regular PPM radio transmitter obviously works to transmit motor information. Could you map accelerometers/gyros/magnetometers/pressure sensors to radio channels transmitted from the quadcopter UAV to be interpreted by the base station PC?