Now you can command the car with just one command. For example, if you say “I'm late for the meeting”, “I don't want my friend to wait too long”, etc., the car will understand and automatically accelerate.
If you are not satisfied with this trip, you only need to provide feedback and suggestions to the car: “Under the premise of fully ensuring safety,” the car's autonomous driving behavior will correct itself during the next trip.
Advertisement
This is the latest research from Purdue University's Digital Twin Laboratory – allowing large language models to be deployed on real autonomous vehicles to understand various personalized instructions from passengers in multiple traffic scenarios, such as parking lots, intersections and highways. Build their digital twin.
In the end, GPT-4 stood out among many experiments on large models.
LLM-based autonomous driving system
The implementation behind this mainly comes from a framework Talk2Drive.
Advertisement
This framework has three main features:
1. It converts human verbal commands into text instructions, which are then processed by large models in the cloud. In this process, some open source APIs for speech recognition, such as Whisper, will be used to accurately capture passwords and translate them into text. Large models on the cloud combine information such as weather, traffic conditions and local traffic rules to generate contextual driving data.
2. The large model generates specific autonomous driving code, which is then sent back to the vehicle’s electronic control unit (ECU) and executed there.
Code execution involves adjusting basic driving behavior as well as various parameters in vehicle planning and control systems.
The vehicle's actuators then control the throttle, brakes, gear selection and steering through the CAN bus and drive-by-wire system.
3. The vehicle's storage module adheres to the concept of “thousands of people, thousands of faces” and records everyone's vehicle interaction data to ensure that each driving experience is customized based on passengers' historical preferences and real-time commands, achieving a true digital twin personality. experience.
After comparing a large number of large models, they finally chose GPT-4 because of its relatively small delay and stronger reasoning capabilities.
In the Talk2Drive system, three types of passengers with different driving preferences interact with the large model through verbal instructions to prompt the system to make corresponding adjustments. Autopilot mode will be turned off when passengers are dissatisfied with the adjusted autopilot behavior, and the system will also record the “takeover” in this case.
The results show that Talk2Drive can significantly reduce takeover rates regardless of driving preference.
The heavily armed self-driving car collaborated with a large language model API to complete the research.
From Purdue University Digital Twin Lab
The research comes from Purdue University’s Digital Twins Lab.
Judging from the research team, most of them have Chinese faces. One of them, Can Cui, is currently a first-year doctoral student at Purdue University. degree in electrical and computer engineering from the University of Michigan before joining the Purdue University School of Engineering. Graduated from Wuhan University of Technology with a bachelor's degree.
His mentor, Dr. Wang Ziran, worked at Toyota Silicon Valley R&D Center for four years before joining Purdue University in 2022, leading digital twin-related research as a chief researcher.
According to reports, the Purdue Digital Twin Laboratory began to delve into the intersection of large language models and autonomous driving in June 2023, and conducted a series of work including literature review, creation of public data sets, simulation environment testing, and hosting seminars.
Paper link:
https://arxiv.org/abs/2312.09397
Project website:
https://purduedigitaltwin.github.io/llm4ad