For a long time I've been wanting to make an ultra minimalist vision / optical flow sensor for the hobbyist and experimentalist community. I've been pursuing this as a small IR&D pet project at Centeye. We're almost there.
The above photo shows one of these sensors next to a millimeter scale. The part count is small- One of our 64x64 custom image sensors, an Atmel ATmega644 processor, several resistors and capacitors, and some lightweight flat optics we developed. Two complete sensors are shown, including with mounted optics (yes it's that thin!). Total mass is about 440mg. The primary interface is via I2C/TWI, which will allow many sensors to be hooked up to a common bus. A secondary connector includes the interface with the ISP for uploading firmware.
We chose to use an ATmega processor since they are loved by hardware hackers and are easy to use. Ideally for a single sensor, one can upload any number of different "application firmwares" to the sensor to make it whatever one wants, limited by just the processor and the base resolution. One firmware will turn it into an optical flow sensor . Another firmware will let it track bright lights. Yet another firmware could turn it into something else. Or someone could write their own firmware, whether by tweaking existing source code (yes I plan to share it) or writing something completely new.
An ATmega644 may not sound like much for image processing- 64kB flash, 4k SRAM, 2k EEPROM, 20MHz max. Neither does a 64x64 array. But the reality is if you are witty you really don't need at lot of resolution or processing power to get some nice results. (We once did an altitude hold demo with just 16 pixels an 1MIPS back in 2001.)
We've already made our first batch of these (about 20) and handed them out to a few close collaborators. Based on feedback we are preparing our second run. The new sensors will be slightly larger and heavier (thicker PCB) but more rigid, and use strictly 0.1" headers for all IO and power (including programming). Mass should still be under a gram.
We also have an even smaller version in the works, shown below with a chip mounted and wire bonded (sorry about the mess). This board uses ATtiny and the 7mm x 8mm board alone weighs about 95mg. I think we can get a whole sensor made for about 120mg, if only I had the time! (Maybe some brave person here would like to take a stab at programming it???)
Comments
We haven't tried this particular sensor over grass yet. We have tried it over asphalt (exploring tire slip measurement) and it sensed that texture. However we did fly earlier sensors over grass and they worked fine. Obviously the sensors could not pick up every blade of grass, but generally there is enough lower spatial frequency texture variations for the sensor to pick up motion- no lawn is perfectly uniform. We've never flown over water- that would be a worthwhile challenge! We did fly an earlier sensor over snow though and that worked.
One consideration is that the "earlier" sensor we flew had a different focal plane architecture and, more important, an optical flow algorithm that more gracefully handled low contrast environments. Those were insect vision inspired algorithms, as opposed to more traditional algorithms like Lucas Kanade or Image Interpolation (both variants of gradient methods).
We can actually make an application firmware version that uses these insect inspired methods. On the plus side they are quite sensitive and fast. On the negative side they are a bit noisy and not appropriate for, say, odometry.
Geoffrey, how well does the OF algorith cope with flying above water or grass? I could imagine that it's a way too uniform surface to detect optical flow. Then again, I can't imagine it working at all with that kind of resolution and computing power ;)
Xander- the earlier altitude hold demos were on a fixed-wing aircraft. Essentially we assumed a (somewhat) constant ground speed and thus OF is inversely proportional to height. The aircraft tended to fly lower into the wing and higher with it, but at least it kept off the ground which was most important.
How were you able to measure altitude while looking down? Does that require the sensor to notice a sort of zooming in and out effect of the features it sees? Seems like that would be tough while also moving.
I look forward to hearing about pricing.
Regarding landing: Back in 2001 we tried just that- we put a sensor on a fixed-wing foam Wingo (remember those?) and aimed it down. We could control altitude by applying a proportional control rule to the elevator or throttle based on the downward measured optical flow. It actually worked pretty well. Then we tried very simple take-off and landing. For take-off, we would push the throttle up, then the Wingo would take off and stop ascending when it reached the set altitude (more or less). For landing, we had the elevator cause the Wingo to flare up a bit before landing. These of course were simple toy tests and not what you need. But to answer your question- yes I think one of these sensors could do this. One thing about landing is that the optical flow will generally be forward-backwards, so that one can use a one-dimensional algorithm and fewer pixels, thus giving us greater frame rate for a given processor.
Regarding data stream: Right now we have just a single firmware prototype with two operating modes. The first mode outputs six bytes: X OF, Y OF, X integrated OF, Y integrated OF, X low-passed OF, Y low-passed OF. The second mode outputs just two bytes: X integrated OF and Y integrated OF, but with the integral computed in a different manner. So basically once you've set up the sensor (sending it a few commands by I2C) you just do I2C reads from it and read out those bytes. Of course, if you hack the firmware you can provide other options.
I have not really stumbled upon a cheap way for smooth autonomous landings. Pressure sensors seem to drift, and sonars or anything of the sort is too expensive, isn't it?
The pixel circuits are actually continuous-time with a logarithmic response so there really is no shutter. Each pixel produces a voltage and that voltage is read out when needed. The log-response allows it to operate over a large range of light levels.
As for frame rate- that depends on the algorithm and how many pixels you wish to process. It can vary from a few Hz to 100s of Hz, maybe even a kHz in some cases though I haven't yet obtained the latter with this particular hardware. (An older 10MIPS PIC-driven sensor, our original Ladybug, got 1.4kHz with an 88 pixel image sensor.)
-
1
-
2
-
3
of 3 Next