Don't know if this has been thought of and dismissed or not. How about using the IMU as an INS.
I think most of us are expecting higher accuracy than is possible while keeping cost down. We also have a problem with perception of error due to scale. Baro and GPS errors are acceptable in full scale but seem huge for our small toys.
My proposal is to use baro and gps only for waypoints and then switch over to INS to maintain position. This would also allow the copter to fight the wind better. The stick would then become purely rate based for speed unless in acro mode, just like FBW in APM. The IMU already has the accuracy as is evidenced by the multitude of indoor videos show a perfectly stationary copter. Sonar is still the best choice for low level ground avoidance.
A possible addition some time in the distant future (AC4 or5) would be to add INS based missions to allow circus tricks like flying through hoops. The board could possibly even be placed in learn mode and a short indoor mission walked through by hand so it can learn to put on a show.
Is any of this possible? Don't know. Just my thoughts.