Columbia Engineers Create Self-Learning Robotic Face for Natural Lip Movements
Researchers at Columbia University have developed a robotic face capable of learning to synchronise its lip movements with speech and singing by observing itself and humans. The project aims to make humanoid robots appear less “uncanny” and more natural in face-to-face interactions.
Teaching Robots Through Observation
The study, published in Science Robotics, introduces a two-stage “observational learning” method that replaces traditional programming with adaptive learning.
Hod Lipson, James and Sally Scapa Professor of Innovation in the Department of Mechanical Engineering and director of Columbia’s Creative Machines Lab, explained, “We used AI in this project to train the robot, so that it learned how to use its lips correctly.”
In the first phase, the robotic face, powered by 26 motors, generated thousands of random facial expressions while observing itself in a mirror. Through this process, it learned how specific motor commands affected its visible mouth shapes.
During the second phase, the system watched online videos of people speaking and singing. It analysed how human mouth movements corresponded with specific sounds, building a model that linked audio patterns to facial motion.
From Sounds to Realistic Movements
By combining both models, the robot could interpret audio input and produce corresponding mouth movements without understanding the meaning of the sounds. According to the researchers, this allowed it to lip-sync accurately across multiple languages and contexts.
The team showcased the robot’s skills by having it perform “metalman,” a song from its AI-generated debut album hello world_. Although its articulation was not flawless, the robot successfully demonstrated expressive, coordinated movements. Lipson noted that the system struggled with certain sounds such as “B” and “W,” but performance is expected to improve with more training data.
Toward More Natural Human–Robot Interaction
Lipson emphasised that the research is part of a broader effort to make robots communicate in a more lifelike and emotionally resonant manner. The technology could enhance human–robot interaction in entertainment, education and healthcare settings.
“I guarantee you, before long, these robots are going to look so human,” he said. “People will start connecting with them, and it’s going to be an incredibly powerful and disruptive technology.”
with inputs from Reuters