Creating a dedicated camera photo logger with IMU, GPS, and trigger sensor.

Problem 1:

The APM .log files contain CAM messages based on when the autopilot commanded the camera to be triggered. It doesn't know when or if the camera actually triggered. This can result in CAM records that don't have corresponding photos or latency issues, both can result in difficulty in the Geotag process.

Problem 2:

I'm using a Sony NEX in a roll gimbal. The .log file contains the IMU attitude data for the aircraft, not for the camera. I would like to be able to feed the camera's IMU data into the image processing software.

The Solution:

These problems were solved by the Field Of View GeoSnap http://fieldofviewllc.com/true-color-imaging-payloads/

However they want $10,000 for it, and it is big and heavy. How can we DIY this?

We could use a separate APM mounted directly on the back of the camera. The new breed of mini APMs would fit nicely and can be found for $40 online (no 3DR mini option).We would need to feed this with the GPS data which could be simply tapped with a splice off the aircraft's GPS receiver. The same could be done with the magnometer.

This APM could be used as a stand alone triggering device if desired. If triggering by distance, it wouldn't even need a flight plan, unless you wanted a couple waypoints to start/stop the triggering with DO_SET_ CAM_TRIGG_DIST. You could use a very large WP Radius to make sure they are not missed.

Here is the part that gets more difficult and requires some code modification...

How do we tell this APM that the camera has actually fired so it can record that event along with the GPS and IMU data at that moment?  For the Sony NEX the iISO Hot Shoe could provide the signal that the camera has fired. This is the solution that the GeoSnap uses. There are a number of cheap eBay hot shoe adapters for around $8 that could be hacked to provide an easy slide in connection without soldering to the contacts. Cameras without a hotshoe could be hacked to provide a trigger signal from other sources, like perhaps the speaker. We may need a voltage divider or multiplier.

So now the question is, how will the APM read this trigger signal, and how can we have each event recorded in the logs? This is where I need help from the community as it exceeds my knowledge base. Could we simply use the voltage pin to detect the spike from the hot shoe, or would it be better to write new code for a new sensor pin, or I2C perhaps? Is there already any code in place that can provide a sort of reverse relay? Like for a proximity switch or a another sensor that provides a simple on/off switch?

Then the next issue is getting the trigger signals recorded in the DF log. Should it replace the CAM message, or be recorded in addition to it? If we had a CAM2 log that recorded the actual trigger moment, and the old CAM log that measured when the trigger was commanded by the APM, we would be able to measure the latency and troubleshoot issues.

So what do you guys think? Would others be interested in this? Who could help write the code modifications?

Looking forward to some brainstorming.

Views: 6389

Reply to This

Replies to This Discussion

For problem 1, you can use the time taken of the photo in the EXIF data together with the cam timestamps to figure out which goes where. The delay between the photo and the CAM message really shouldn't be more than 200ms, or the camera hasn't been calibrated correctly (focus still set to auto, etc.). Basically, you can figure out from logs, timestamps and the timing on the camera where you may have photos missing and skip to the next CAM message.

For problem 2, you don't need the attitude information for these softwares anymore. The gimbal is there to ensure you get as close to nadir view as possible.

Better measures of the GPS may slightly improve the positional accuracy, but I'm not sure whether this is worth the effort, because you need to look at the application area for each method. Since GPS is 5Hz usually, there will always be some travel between the actual shutter position and what you logged, plus the inaccuracy of 1m of the GPS on a mobile device that is variant in actual speed. The only way to reliably improve accuracy is to use ground control points.

We are using an IR trigger that has an inconsistent delay, sometimes resulting in missing images. Finding out where the missing images occur is problematic and time consuming since often work with large datasets. For example, to map 1 sq km at 1.5 cm GSD takes over 2800 images with our Sony NEX-7. Moreover, the EXIF time stamps are only to the nearest second, but depending on our flying speed, the camera may be triggered in sub-second intervals. 

Of course, we could do image processing without any data on camera positions, but that adds a significant amount of hours on top of an already lengthy computational process. A hardware solution that can reliably produce GPS + IMU data to match the images would be ideal. Redundancy is helpful for saving time and could even make a difference by not requiring additional fieldwork to make up for lost data. However, if there are software solutions that could overcome these issues, that would be even better.

Here's a script that goes through the APM log file and extracts GPS + CAM information and uses the EXIF time of the photo with second resolution.The way it works is that it assumes the first photo taken succeeded. It uses that first photo EXIF time to calculate the delta between the camera time and gps time (the first CAM message). Then it rereads the photos, it rereads the log and tries to find the closest position. In my case I didn't bother to interpolate the GPS position, but you could attempt that as an extra.

This script was applied to a survey where the vehicle traveled 3 m/s, so I'd be max 3m off.

In your case, with sub-second intervals you'd get the same position for a couple of photos.

If you have extra knowledge on the spacing or minimum delay between photos, you can exploit that to refine your timings. for example, if 3 photos were taken and you know there was a 250ms delay minimum, you'd apply 0.75, 1.0 and 1.25 as times instead. When you reapply the GPS data, you'd get different positions for them again.

This sensor data however is not the ground truth in orthophoto/model construction, the actual pixel data is. The values that you insert eventually determine the orientation and placement (with some margin of error) and help to establish the parameters of the camera (radial / tangential distortion ). The good news is that if you take many measurements, the error is much less than that for a single photo. Plus, I don't think the current softwares pay that much attention to attitude data anyway. Not sure you can even import it in some to begin with, because at a certain distance a minute change in degrees has enormous consequences for matching, so from that perspective it's practically useless (whereas "slight" positional errors have less consequences).

Attachments:

I see there hasn't been any discussions for a while.  How is this project going?  I would love to see this.  Did the script from Gerard solve the problem? 

I also have another questions.  I'm running a Flytron sLED v2 IR trigger from the PIXHAWK RC10 to trigger a sony nex 7.  I get it to trigger, but it wont' stop triggering.  Anyone have any ideas?

I'm actually working on something new, but it's in research stage.

That geosnap solution is criminally expensive. 

Here's what you could look at:

- the flash or "strobe" signal is the best way to do this. That's probably a signal that fires halfway into the "shutter-open" time. However, it's possible that when the shutter time is really low (fast shutter), the signal is not triggered. Or it's possible that the camera only sends the trigger signal in specific lighting conditions, so keep that in mind when testing.

- The P&S cameras probably don't have an external flash connection like the above, but may have a "strobe" pin somewhere inside, hidden... like many canon cams also have a hidden uart on their board.

- CHDK has a special mode where you can sync with multiple cameras. Here you send a signal once to start the picture taking process (exposure, gains, etc), then send another signal to take the actual picture. Unfortunately, this again is control without feedback, so you can't tell whether it actually did it:

http://chdk.wikia.com/wiki/USB_Remote

The weirdest way to do this?  It may be possible to force the flash on, put together a simple schematic with an arduino, a light sensor and then hook up the light sensor to when the on-board flash fires. This may not work if the camera calculates the shuttter speed differently, assuming the flash actually floods the environment (you get dark photos), but it may be possible to compensate for those light losses.

Maybe the P&S camera has an internal pin that goes high when the flash fires. 

Hooking it up to the flash directly is not recommended :). Flashes are designed to store high charges and release a very short, high voltage to a lamp to make it flash. So you don't want to send that voltage signal to any sensitive electornics.

I've bought a NEX-5R a week ago and use the gentLED-SHUTTER cable.
Not yet in the air with this cam but interested in the issue as well.

Just did a quick test on the ground.

- missing shots
Triggering via the mission planner manually shows that the cam
is not awaiting the disk write but engaging the somehow 10 pictures
deep buffer. That means missed pictures indicates to run out of
buffer. Is this correct? If yes then it indicates that the scenario
mentioned above is beyond the capacity of the NEX-7.

- latency
I've used my video cam to record a stop watch on the monitor and the manual trigger
seen in the mission planner. The stop watch is taken in the NEX pictures as well.
Not a very precise measurement but to the single frame (25fps) of the video cam.
After 10 pictures taken I've got
average delay: 360ms
max deviation: 31ms

These values are in a range where my test setup is insufficient. May be
in reality even better figures.

Compared to the GPS precision mentioned above these values seems quite good
to me. No need to have a backward channel for better values.

regards
Wolfgang R.

I found the delay when using the IR trigger inconsistent. So I did this mod and now the camera fires much faster and it's reliable: http://diydrones.com/profiles/blogs/remote-shutter-for-sony-nex-5n-cam

Then a friend built this circuit which we found on the forum to trigger the camera

There is already a solution on the works for this issue, is probably coming out soon. 

https://github.com/diydrones/ardupilot/issues/2289

As far as the tagging aspect goes and without modifying the autopilot, we have released a free standalone photo tagger that works with the autopilots log, and uses a script with Mission Planner. Using this method we can handle missing arbitrary photos within, and have shown good results with tagging photos that were taken more often then once a second. (Although our typical times between photos are usually roughly 1-2 seconds rather then sub one second). It's available here: http://swiftradioplanes.com/swifttag/

It primarily been tested with APM:Plane with great success, but we have also used it with copter as well. Feedback will be a great refinement to camera accuracy.

For Canon Powershots...

I've been working with working with the KAPUAV script guy. http://chdk.setepontos.com/index.php?topic=12398.0

He has modified the script so it flashes the Assist focus LED on on shutter open. I'm in the process of modifying my (Tuffwing) trigger cable with a phototransistor so it will create a feedback loop to the autopilot - precisely when and if the picture was taken.

Anxiously waiting on this update: https://github.com/diydrones/ardupilot/issues/2289

Hi Brian!

I tested the script kap_uav3.6 beta and works perfectly on my Canon SX260. Solid flashes every shot. My camera can take a maximum of one photo every 3 seconds. What is the minimum time between shots in the Canon Power Shot elph 115? I also manufacture my triggers cables. I will start working on the phototransistor cable tomorrow.  I will share with you the cable development, so we can compare.

Regards,

Plínio 

"Anxiously waiting on this update: https://github.com/diydrones/ardupilot/issues/2289"

Do you have a date for when this will be implemented in official code?

Reply to Discussion

RSS

Groups

Season Two of the Trust Time Trial (T3) Contest 
A list of all T3 contests is here. The current round, the Vertical Horizontal one, is here

© 2018   Created by Chris Anderson.   Powered by

Badges  |  Report an Issue  |  Terms of Service