Figured out how to close the loop!
Programs can interface with FlightGear through FlightGear's property tree. Very powerful.
If you run FlightGear with the --httpd= option, it will run a webserver on http://localhost: where you can browse the property tree and refresh the current values. Alternatively, in FlightGear you can type "/", and the property tree viewer will pop up and you can view the values changing in real time. I did this to figure out the sign convention for different properties.
The property tree can be read from and written to using a "Protocol". The protocol basically defines the format in which data is exchanged. You identify the nodes in the property tree which you want to interact with, the data types, the string formatting, and voila! Attached are the input and output .xml protocol files which I used (copy to FlightGear's data/Protocol directory).
The data can be passed through several means, serial port, tcp, udp, etc... I chose UDP for ease.
Here is how I ran FlightGear:
fgfs --httpd=5500 --generic=socket,out,40,localhost,5000,udp,output_protocol --generic=socket,in,45,localhost,5010,udp,input_protocol
I made the input data rate faster than the output data rate, because I would rather have FlightGear waiting for a new input value (when it doesn't get a new input, the last one persists) than the input being buffered.
The --generic option allows one to specify a generic protocol, which was defined in the .xml file (attached).
Attached is a .c program which I wrote to interact with the property tree and provide feedback controls to FlightGear. I've implemented simple proportional control for roll and pitch to stabilize the aircraft. Timing is driven by FlightGear. The code makes a blocking call to read UDP data. It waits until it receives a new packet, processes it, and sends a packet with control inputs back to FlightGear. Then it loops again, waiting for another packet from FlightGear. Currently, everything is hardcoded (port numbers, etc ..), but it should be trivial to make it more general purpose.
This should provide a complete framework with which to write an autopilot controller in C.