The Robot Brains Podcast• Aug 16, 2023• 1:31:32Interview

S3 E20 Jitendra Malik on Building AI from the ground up: Sensorimotor learning before language

Jitendra Malik•Professor, UC Berkeley & Research Scientist Director, Meta

Executive Summary

Professor Jitendra Malik argues that true AI requires embodied, sensory-motor intelligence, viewing current large language models like GPT-4 as impressive but incomplete without a foundation in physical interaction.
Malik's research has made significant strides in robotics, particularly with the Rapid Motor Adaptation (RMA) technique, which has nearly solved quadrupedal (four-legged) locomotion across diverse terrains.
He posits that for AI to advance, it must follow a path similar to human evolution and child development, where physical interaction and grounded learning precede and inform abstract reasoning and language.
In computer vision, while problems like object recognition and segmentation are largely solved, 3D reconstruction from a single image remains a major unsolved challenge.

8 quotes

Concerns Raised

Current AI development is overly focused on disembodied language models, neglecting the foundational need for sensory-motor intelligence.
Bipedal locomotion for humanoid robots remains a significant, unsolved challenge.
3D reconstruction from a single image is still far from human-level capability.
The resource gap makes it impossible for academic labs to compete with industry on 'big science' projects like training foundational models.

Opportunities Identified

Grounding language models in physical experience could make them more robust and less brittle.
The near-solved problem of quadrupedal locomotion opens the door for practical, real-world robot applications.
Using simulation in a tight feedback loop with real-world data can dramatically accelerate progress in robotics.
Combining vision with proprioceptive feedback allows robots to navigate extremely challenging terrains like stairs and uneven ground.

Key Themes

Embodied Intelligence vs. Disembodied Language

Malik contrasts the abstract, text-based intelligence of LLMs with the grounded, sensory-motor intelligence of animals and humans. He argues that the latter, developed over millions of years of evolution, is a necessary foundation for robust, general intelligence, citing how a child's early language is tied to physical objects and actions.

This challenges the dominant narrative that scaling language models alone will lead to AGI, suggesting that robotics and interaction with the physical world are critical, non-negotiable components for future progress.

Solving Robotic Locomotion

The discussion highlights the success of the Rapid Motor Adaptation (RMA) technique, which enables robots to adapt to changing terrains in real-time without prior knowledge. While quadrupedal locomotion is considered nearly solved, bipedal locomotion for humanoid robots remains a significantly harder, open problem.

This marks a major milestone in robotics, moving robust mobility from a research challenge to an engineering one, which is a key enabler for deploying robots in unstructured, real-world environments.

The State of Computer Vision

Malik outlines the 'three R's' of vision: recognition, reconstruction, and reorganization (segmentation). He states that recognition and segmentation have seen remarkable progress and are largely solved problems, thanks to deep learning and large-scale data. However, 3D reconstruction from a single image is still far from human-level performance.

This provides a clear status report on a core AI field, indicating where commercial applications are mature (e.g., image tagging) and where significant research opportunities still exist (e.g., creating 3D models for AR/VR and robotics).

The Power of Simulation

Malik is bullish on using simulation for training robots, viewing it as a way to leverage Moore's Law. He addresses the 'sim-to-real' gap by proposing a continuous feedback loop where failures in the real world are used to improve the simulation, treating the simulator as a scientific theory to be refined by experiments.

This insight reframes simulation not as a flawed shortcut but as a core part of the scientific process for robotics, enabling faster, safer, and more scalable development of robotic skills.

Big Science vs. Small Science in AI

The conversation touches on the two modes of AI research: 'big science' conducted in large industrial labs with massive computational resources (e.g., training GPT-4), and 'small science' driven by individual curiosity in academia. Malik argues that both are essential and complementary for the health and progress of the field.

This addresses the growing concern about an innovation gap between industry and academia, advocating for a balanced ecosystem where both large-scale engineering feats and foundational conceptual breakthroughs can flourish.

Get started free

Topics

Embodied Intelligence Robotics Computer Vision Jitendra Malik Rapid Motor Adaptation (RMA)Locomotion Quadrupedal Robots Bipedal Robots Humanoid Robots Grounded Learning Large Language Models (LLMs)GPT-4 Moravec's Paradox Simulation Sim-to-Real Transfer 3D Reconstruction History of Science

Processed May 4, 2026 yt-dlp + mlx-whisper + Gemini

You're reading a preview

Get started free →