New night-vision technology developed at Purdue creating a buzz

A new, AI-assisted form of thermal vision under development at Purdue University could one day revolutionize everything from sensors in self-driving cars to remote patient monitoring in medical facilities.

The system is called heat-assisted detection and ranging, or HADAR, and it’s basically a method for using thermal vision (which generates visual images from the heat put off by people and objects) to create precision views in the dark that include fine detail and even color.

Though it’s still a few years from practical use, the night-vision technology shows great potential in significantly boosting the way computers see the world around them.

“We anticipate that more breakthrough discoveries can happen in the following three to five years to significantly improve the hyperspectral thermal imager,” said Zubin Jacob, associate professor of electrical and computer engineering in Purdue’s Elmore Family School of Electoral and Computer Engineering. “The biggest hurdle is the size of the infrared thermal camera.”

The work is being done by Jacob and Purdue research scientist Fanglin Bao, and it’s important enough to have recently been featured on the cover of the peer-reviewec journal Nature.

If a practical system can be produced, it could be a boon to robots and self-driving vehicles, many of which use conventional video cameras, radar and LIDAR (light detection and imaging) to find their way around. The problem is that each of these systems has weaknesses. Video cameras are, for obvious reasons, not great at night, and LIDAR gathers data by painting its environment with low-power lasers. Which could be a potential danger—or at the very least, an annoyance—to humans close to lots of laser-firing cars.

“Cameras can’t capture images well in the dark, in which there is no reflected light, and radar and LIDAR are prone to interference,” wrote the journal Science.org in a recent article about HADAR’s potential. “This poses a challenge for high-risk applications such as self-driving cars, where errors can lead to potentially fatal failures to brake or to recognize pedestrians.”

Thermal sensors, on the other hand, can see in pitch darkness, because they detect the heat signatures of everything from people to inanimate objects. Plus, the technique is entirely passive. Nothing is fired out of the thermal sensor in the direction of approaching objects.

The problem is that today’s state-of-the-art thermal systems just don’t see well. Every object, from blades of grass to the road itself, emits its own heat signature, creating an effect called “ghosting,” in which everything in the frame mushes together into one low-resolution image. The disadvantages of thermal imaging have, until now, made it of little use for such high-value functions as vehicle navigation.

About four years ago, Jacob and Bao began wondering what could be done to remedy that problem.

Bao said he asked himself, “‘What is the mechanism causing [ghosting], and can we overcome it?’ Once we noticed that the bottleneck was the ghosting effect, then we started digging and finally found something interesting.”

To produce crisper images, the two scientists trained a neural network program to process and “clean up” signals from a conventional thermal camera, filtering out background chaff and focusing on each object’s specific heat signature.

They did this by teaching the system to recognize the emission spectra of numerous materials, including fabric and glass. AI uses this information to identify various objects, strain out background interference, and meld everything into an extremely high-quality image. So high-quality, in fact, that things like the bark patterns on individual trees can be accurately perceived.

While the system, for now, relies on traditional machine artificial intelligence, the researchers hope to add generative AI in the future.

HADAR can also provide “ranging” information about the distances to objects. It can even accurately render images in color, by assessing the particular wavelengths of the thermal radiation from various objects in its view.

“The most striking feature is that HADAR shows accuracy comparable to daytime detection and ranging, even though HADAR can work in pitch darkness,” Jacob said. “This makes it useful for a variety of applications.”

Which is putting it mildly. HADAR could provide vision to the aforementioned cars and robots, plus find any number of other applications, including monitoring wildlife at night, or checking remotely on hospital patients in dark rooms.

The Defense Advanced Research Projects Agency—or DARPA—has kicked an undisclosed amount of money into the work. Certainly, the military could think of many applications for a system that can see as clearly at night as during the day.

But several software and hardware kinks must be worked out before HADAR can make its real-world debut.

Perhaps most important, the thermal sensor used right now is about the size of one of those mini-refrigerators college kids keep in their dorm rooms—though considerably more expensive. Too expensive, in fact, to be applicable for most commercial products.

Also, the speed of processing sensor readings must be considerably increased. The current version needs about one second to create one image, whereas a system for autonomous cars requires 30 to 60 frames per second.

“The prototype HADAR is based on hyperspectral thermal imagers, which are currently large in size, slow in imaging speed, heavy and extremely expensive,” Jacob said. “Those practical challenges need to be overcome for HADAR to have widespread applications.”

As has become something of a tradition at Purdue, the scientists might see themselves at some point creating a startup to market HADAR. The duo recently disclosed the technology to the Purdue Innovates Office of Technology Commercialization, which has applied for a patent on it. In addition to ongoing funding from DARPA, Jacob and Bao also received $50,000 from the university’s Office of Technology Commercialization through its Trask Innovation Fund.

However, there are still years of work ahead before it’s time to pursue seed money for a startup or approach companies about marketing the researchers’ brainchild. Though that hasn’t stopped a few firms from making tentative approaches to them.

“We do get some contact from companies,” Bao said. “But this is a very early phase of research. We still need three to five years to push it further. I think we need to verify the theory first and maybe we need to make some breakthroughs in the hardware as well before we can talk with companies and industries.”

For now, they’re not feeling the heat of competition. Not much, anyway. But that could change as their concept receives more attention.

“Right now, we are ahead, but definitely many industry leaders in thermal imaging will be working on HADAR,” Jacob said. “The fact that it can make AI see through the darkness like broad daylight has a deep and far-reaching importance. Animals and human beings are subject to the dichotomy between day and night, due to evolution. Now, we have proven that AI equipped with HADAR can actually overcome the long-standing dichotomy and see through the night.”•