Cornell University creates 'memory' for self-driving cars, making future driving safer

Self-driving cars use artificial intelligence to identify pedestrians, other vehicles, and potential obstacles, allowing them to navigate city streets and less-traffic environments.All of this is achieved with the help of artificial neural networks, which, after training, can mimic the human visual perception system and "see" the environment around the vehicle.

However, unlike humans, vehicles using artificial neural networks have no memory of the past. No matter how many times they have driven on a certain road before, they are always in the state of seeing the world for the first time.This can be a big problem in bad weather, as the car won't be able to rely on sensors to drive safely.

According to foreign media reports, researchers at Cornell Ann S. Bowers College of Computing and Information Science and the College of Engineering published three papers simultaneously. It explains how to overcome these limitations by creating a "memory" for the car based on previous experience, allowing it to use that memory for future driving navigation.

The researcher drives the data collection vehicle (Credit: Cornell University)

"The fundamental question is, can we learn from repeated experiences? For example, a car's laser scanner perceives the shape of a tree from a distance for the first time," said Kilian Weinberger, a professor in the Ann S. Powers School of Computer Science. Strange tree, the tree may be mistaken for a pedestrian. However, once the distance is close enough, the kind of object is clear. Therefore, when driving past the same tree a second time, even in fog or snow, The car will hopefully learn to correctly identify the tree."

Katie Luo, a doctoral student in the research team, said: "In fact, it is very rare that people drive on a road for the first time. You or others have driven the road recently, so it is natural to collect and use these experiences. ."

Led by doctoral student Carlos Diaz-Ruiz, the research team drove a vehicle equipped with lidar (light detection and ranging) on ​​a 15-kilometer road in and around Ithaca over an 18-month period. A data set was compiled by repeatedly driving the loop 40 times.The tour captures a variety of different situations: (different roads: highway, city, campus), different weather (sunny, rainy, snowy), and different times of day.The research team named the dataset Ithaca365, which has over 600,000 scenes.

Diaz-Ruiz said: “It deliberately exposes self-driving cars to one of the key challenges: bad weather conditions. If the streets are covered with snow, humans can rely on memory, but without memory, the neural network encounters a large trouble."

HINDSIGHT is a method that uses a neural network to compute object descriptors as vehicles pass by them, which can then compress such descriptions, named SQuaSH (Spatial Quantized Short History) features by the group, and stored in a On a virtual map, similar to the "memory" stored in the human brain.

"Such information can be used as features to add to any lidar-based 3D object detector," the researchers said. Afterwards, the detector and SQuaSH features can be jointly trained without any additional supervision or human annotation, the latter being both Time consuming and laborious.”

While HINDSIGHT still believes that artificial neural networks have been trained to detect objects and have been enhanced with the ability to create memories, MODEST (Transient Moving Object Detection and Self-Training) goes further.

In the study, the researchers had the vehicle learn the entire perception process from scratch.In the beginning, the artificial neural network of the vehicle never touched any object or street.By driving the same path multiple times, the car can learn which parts of the environment are stationary and which are moving objects.Slowly, it is able to teach itself what other traffic participants have and what is safe to ignore.

The algorithm can then reliably detect such objects, even on roads that have not been driven repeatedly before.

The researchers hope that both approaches could significantly reduce the cost of developing self-driving cars, which are still heavily reliant on expensive human-annotated data, and that they could enable autonomous driving by learning the most frequently traveled paths before. Cars are more efficient.