Udacity Self-Driving Car Engineer Nanodegree Projects

Enjoy a look at some of the projects our students are building, including Finding Lane Lines, Traffic Sign Classifier, Behavioral Cloning, and more!

Students in our Self-Driving Car Engineer Nanodegree program engage in a project-based curriculum, and from the moment they enroll, they begin addressing key challenges and topics through building specialized projects. Here are all of the projects they build!

Finding Lane Lines

This is the first project students complete, one week into the program.

They learn to work with images, color spaces, thresholds, and gradients, in order to find lane lines on the road.
Stack: Python, NumPy, OpenCV

Traffic Sign Classifier

In this project, students train a convolutional neural network to classify traffic signs.

To do so, they use the German Traffic Sign Recognition Benchmark dataset. This particular student went above and beyond to train his network to not only classify signs, but also localize them within the image, and applied his classifier to a video.
Stack: Python, NumPy, TensorFlow

Behavioral Cloning

Here, students record training data by manually driving a car around a track in a simulator.

Then they use this camera, steering, and throttle data to train an end-to-end neural network for driving the vehicle, based on NVIDIA’s famous research paper.
Stack: Python, NumPy, Keras

Advanced Lane Finding

By applying advanced computer vision techniques, such as sliding window tracking, to a dashcam video, students are able to track lane lines on the road under a variety of challenging conditions.
Stack: Python, NumPy, OpenCV

Vehicle Detection and Tracking

Students use machine learning techniques and feature extraction to identify and track vehicles on a highway.
Stack: Python, NumPy, scikit-learn, OpenCV

Extended Kalman Filter

An extended Kalman filter merges noisy simulated radar and lidar data to track a vehicle.
Stack: C++, Eigen

Unscented Kalman Filter

An unscented Kalman filter merges noisy, highly non-linear simulated radar and lidar data to track a vehicle.
Stack: C++, Eigen

Kidnapped Vehicle

Students develop a particle filter in C++ to probabilistically determine a vehicles location relative to a sparse landmark map.
Stack: C++

PID Controller

Students build and tune a proportional-integral-derivative controller to steer a vehicle around a test track, following a target trajectory.
Stack: C++

Model Predictive Control

Students build and optimize a model predictive controller to steer a vehicle around a test track, following a target trajectory.
Stack: C++, ipopt

Path Planning

In this project, students construct a path planner for highway driving based on a finite state machine.

The planner has three components: environmental prediction, maneuver selection, and trajectory generation.
Stack: C++

Semantic Segmentation

Students train a pixel-wise segmentation network that identifies and colors road pixels to identify free space for driving.
Stack: Python, TensorFlow

Safety Case

Students build a prototype of a safety case for a lane-keeping assistance ADAS feature, including the safety plan, hazard analysis and risk assessment, functional safety concept, technical safety concept, and software requirements.

Programming a Real Self-Driving Car

For this project, students form teams to drive a real self-driving car around the Udacity test track.

The car is required to negotiate a traffic light and follow a waypoint trajectory. Code is built first in the simulator, and then deployed to Udacity’s self-driving car in California.
Stack: Python, ROS, Autoware, TensorFlow

Would you like to be building these kinds of projects yourself? Then you should apply to the Udacity Self-Driving Car Engineer Nanodegree Program!

The “Deep Neural Networks” Lesson

Lesson 7 of the Udacity Self-Driving Car Engineer Nanodegree Program is “Deep Neural Networks”

I am continuing on my quest to write a post detailing every one of the 67 projects that currently comprise our Self-Driving Car Engineer Nanodegree program curriculum, and today, we look at the “Deep Neural Networks” lesson!

Students actually start learning about deep neural networks prior to this lesson, but this is the lesson where students begin to implement deep neural networks in TensorFlow, Google’s deep learning framework.

In the previous lesson, “Introduction to TensorFlow,” students learned to use TensorFlow to build linear models, like linear or logistic regression. In the “Deep Neural Networks” lesson, students learn new techniques in TensorFlow, to build up these models into neural networks.

Some of the most important foundational blocks of neural networks are demonstrated in TensorFlow.

Activation functions help neural networks represent non-linear models
Backpropagation trains neural networks from real data quickly and efficiently
Dropout removes neurons randomly during training to prevent overfitting the training data, which makes the model more accurate on new data

Students also learn some practical skills, like how to save and restore models in TensorFlow.

Future lessons take these basic skills and help students apply them to important problems for autonomous vehicles, like how to recognize traffic signs.

The “Introduction to TensorFlow” Lesson

The sixth lesson of the Udacity Self-Driving Car Engineer Nanodegree Program is “Introduction to TensorFlow.”

TensorFlow is Google’s library for deep learning, and one of the most popular tools for building and training deep neural networks. In the previous lesson, MiniFlow, students build their own miniature versions of a deep learning library. But for real deep learning work, an industry-standard library like TensorFlow is essential.

This lesson combines videos from Vincent Vanhoucke’s free Udacity Deep Learning course with new material we have added to support installing and working with TensorFlow.

Students learn the differences between regression and classification problems. Then they to build a logistic classifier in TensorFlow. Finally, students use fundamental techniques like activation functions, one-hot encoding, and cross-entropy loss to train feedforward networks.

Most of these topics are already familiar to students from the previous “Introduction to Neural Networks” and “MiniFlow” lessons, but implementing them in TensorFlow is a whole new animal. This lesson provides lots of quizzes and solutions demonstrating how to do that.

Towards the end of the lesson, students walk through a quick tutorial on using GPU-enabled AWS EC2 instances to train deep neural networks. Thank you to our friends at AWS Educate for providing free credits to Udacity students to use for training neural networks!

Deep learning has been around for a long time, but it has only really taken off in the last five years because of the ability to use GPUs to dramatically accelerate the training of neural networks. Students who have their own high-performance GPUs are able to experience this acceleration locally. But many students do not own their own GPUs, and AWS EC2 instances are a cloud tool for achieving the same results from anywhere.

The lesson closes with a lab in which students use TensorFlow to perform the classic deep learning exercise of classifying characters: ‘A’, ‘B’, ‘C’ and so on.

TensorFlow on Windows

TensorFlow is the main deep learning library we are using in the Udacity Self-Driving Car Engineer Nanodegree Program, and it’s been a little bit painful because of the lack of Windows support.

We’ve had to work with our Windows users to set up Docker containers in which to run TensorFlow, and frankly we haven’t done as good a job with that as we should.

So thank goodness Google announced yesterday that they’re releasing Windows support for TensorFlow.

It looks to be early stages and I’m not sure if we can safely point our students there yet, but hopefully it means we can get there soon.

And it’s also another step toward TensorFlow becoming the library of choice for deep learning.

TensorFlow vs. TF Learn vs. Keras vs. TF-Slim

One module in Udacity’s Self-Driving Car Nanodegree program will cover deep learning, with a focus on automotive applications.

We’ve decided to use the TensorFlow library that Google has built as the main tool for this module.

Caffe, an alternative framework, has lots of great research behind it, but TensorFlow uses Python, and our hope is that this will make learning it a lot easier for students.

Even with TensorFlow, however, we face a choice of which “front-end” framework to use. Should we use straight TensorFlow, or TF Learn, or Keras, or the new TF-Slim library that Google released within TensorFlow.

Right now we’re learning toward TF Learn, almost by default. Straight TensorFlow is really verbose, TF-Slim seems new and under-documented. Keras and TF Learn both seem solid, but the TF Learn syntax seems a little cleaner.

One big drawback to TF Learn, though, is the lack of easily integrated pre-trained models. I spent a while today trying to figure out how to migrate pre-trained AlexNet weights from Caffe to TF Learn.

So far, no one solution is jumping out at me as perfect. Let me know in the comments if you’ve got a suggestion.