At Udacity, where I work, we have a self-driving car. Her name is Carla.

Carla’s technology is divided into four subsystems: sensors, perception, planning, and control.

Sensors

Carla’s sensor subsystem encompasses the physical hardware that gathers data about the environment.

For example, Carla has cameras mounted behind the top of the windshield. There are usually between one and three cameras lined up in a row, although we can add or remove cameras as our needs change.

Carla also has a single front-facing radar, embedded in the bumper, and one 360-degree lidar, mounted on the roof.

Sometimes Carla will utilize other sensors, too, like GPS, IMU, and ultrasonic sensors.

Data from these sensors flows into various components of the perception subsystem.

Perception

Carla’s perception subsystem translates raw sensor data into meaningful intelligence about her environment. The components of the perception subsystem can be grouped into two blocks: detection and localization.

The detection block uses sensor information to detect objects outside the vehicle. These detection components include traffic light detection and classification, object detection and tracking, and free space detection.

The localization block determines where the vehicle is in the world. This is harder than it sounds. GPS can help, but GPS is only accurate to within 1–2 meters. For a car, a 1–2 meter error range is unacceptably large. A car that thinks it’s in the center of a lane could be off by 1–2 meters and really be on the sidewalk, running into things. We need to do a lot better than the 1–2 meters of accuracy that GPS provides.

Fortunately, we can localize Carla to within 10 centimeters or less, using a combination of high-definition maps, Carla’s own lidar sensor, and sophisticated mathematical algorithms. Carla’s lidar scans the environment, compares what it sees to a high-definition map, and then determines a precise location.

The components of the perception subsystem route their output to the planning subsystem.

Planning

Carla has a straightforward planning subsystem. The planner builds a series of waypoints for Carla to follow. These waypoints are just spots on the road that Carla needs to drive over.

Each waypoint has a specific location and associated target velocity that Carla should match when she passes through that waypoint.

Carla’s planner uses the perception data to predict the movements of other vehicles on the road and update the waypoints accordingly.

For example, if the planning subsystem were to predict that the vehicle in front of Carla would be slowing down, then Carla’s own planner would likely decide to decelerate.

The final step in the planning process would be for the trajectory generation component to build new waypoints that have slower target velocities, since in this example Carla would be slowing down as she passes through the upcoming waypoints.

Similar calculations affect how the planning subsystem treats traffic lights and traffic signs.

Once the planner has generated a trajectory of new waypoints, this trajectory is passed to the final subsystem, the control subsystem.

Control

The control subsystem actuates the vehicle by sending acceleration, brake, and steering messages. Some of these messages are purely electronic, and others have a physical manifestation. For example, if you ride in Carla, you will actually see the steering wheel turn itself.

The control subsystem takes as input the list of waypoints and target velocities generated by the planning subsystem. Then the control subsystem passes these waypoints and velocities to an algorithm, which calculates just how much to steer, accelerate, or brake, in order to hit the target trajectory.

There are many different algorithms that the control subsystem can use to map waypoints to steering and throttle commands. These different algorithms are called, appropriately enough, controllers.

Carla uses a fairly simple proportional-integral-derivative (PID) controller, but more sophisticated controllers are possible.

A Self-Driving Car

That’s how Carla works!

First, the sensor subsystem collects data from Carla’s cameras, radar, and lidar. The perception subsystem uses that sensor data to detect objects in the world and localize the vehicle within its environment. Next, the planning subsystem uses that environmental data to build a trajectory for Carla to follow. Finally, the control system turns the steering wheel and fires the accelerator and brake in order to move the car along that trajectory.

We’re very proud of Carla. She’s driven from Mountain View to San Francisco, and done lots of driving on our test track.

The most exciting thing about Carla is that every Udacity student gets to load their code onto her computer at the end of the Nanodegree Program and see how well she drives on our test track.

If you’re interested in learning about how to program self-driving cars, and you want to try out driving Carla yourself, you should sign up for the Udacity Self-Driving Car Engineer Nanodegree Program!