A self-driving car runs a pipeline of AI models that repeats many times a second: turn raw sensor data into a picture of the world, guess what everyone else on the road is about to do, then decide — and carry out — the car’s own next move. The three stages are usually called perception, prediction, and planning, and together they replace the judgment a human driver makes instinctively.

Reading the road: perception

Cameras, radar, ultrasonic sensors, and — on many systems — lidar feed a constant stream of raw data into onboard neural networks trained to make sense of it. Lidar bounces laser pulses off nearby objects to build a precise 3D map of distances; radar tracks speed and range in poor visibility; cameras supply the color, text, and fine detail that only computer vision models can extract. The perception stack fuses these inputs into a single live model of the scene: where every car, cyclist, and pedestrian is, and what the lane markings, signs, and traffic lights say.

Guessing what happens next, then acting

Prediction models take that scene and forecast a short list of likely paths for every other road user — will the cyclist keep straight, or turn? The planning module scores candidate routes for the car itself against those predictions, weighing safety, comfort, and traffic rules, and picks one. A control layer then converts the chosen path into steering, braking, and acceleration commands sent to the vehicle’s motors, dozens of times per second. Developers train the underlying models mostly through imitation — showing the system millions of miles of recorded human driving — then refine it in simulation and, increasingly, with reinforcement learning that rewards safe, smooth outcomes.

Two different bets on how to get there

Not every company builds this pipeline the same way. Waymo runs a modular system stacked with cameras, radar, and multiple lidar units, and only turns its robotaxi service on in cities it has mapped and validated in detail — that’s SAE Level 4, meaning no human needs to pay attention at all within that area. It now operates driverless rides in more than a dozen US cities. Tesla has taken the opposite bet: a single end-to-end neural network trained on camera video alone, with no lidar, running what the company calls Full Self-Driving. For the vast majority of Tesla owners it remains SAE Level 2 — “supervised” — because a licensed driver must stay ready to take over at any time. Since January 2026, Tesla has also run a small, separate pilot of unsupervised robotaxi rides in Austin alongside a much larger monitored fleet, testing whether the same camera-only approach can eventually match Waymo’s independence.

Why it needs its own chips

Training these models takes the same kind of large-scale computing that powers other physical AI systems — racks of data-center accelerators running for weeks. But running the trained model inside a moving car is a different engineering problem: every decision has to happen in real time, on a computer that fits under the trunk and draws only as much power and heat as the car’s cooling system can handle, with no cloud round-trip to fall back on. That mismatch — between how much compute self-driving AI needs and how little power a car can spare — is why automakers increasingly design their own inference chips rather than buy general-purpose hardware, and, in some cases, why they invest directly in the factories that make them.

Where the limits are

Even the most capable systems still struggle with situations their training data rarely covered: unusual road debris, an unmarked construction detour, extreme weather, or a traffic officer waving contradictory hand signals. Regulators respond by approving driverless service one city, or even one route, at a time rather than nationwide, and most manufacturers still require a human to stay alert. Self-driving AI has gotten dramatically better at the common case; the remaining work is mostly about the rare one.

In the news

Two recent stories show both sides of this build-out. Some of the chips that make onboard self-driving compute possible are now fabricated closer to home: see our report on Tesla and SpaceX’s $16.8 billion “Terafab” chip plant in Texas. And on the robotaxi side, Uber and China’s Pony.ai are expanding driverless rides to more cities across Europe and the Middle East.