Is that the older APM 2.0 or a 2.5?
Figured it out.
I swapped the APM 2 for a 2.5 I have and got exactly the same symptoms - which was doubly weird because even the power connection was different.
I unplugged everything and added things in one at a time.
Turns out something is wrong with my telemetry radio (3DR 915) - when it is plugged in the board will not initialize unless the board is powered by the USB port. If I unplug it, the board works great. When I swapped the radio for a different one, it worked again.
Not sure how to go about trying to debug the radio, but at least now I know what happened.