Amazing Self-Driving Car Projects

Udacity Self-Driving Car Engineer Nanodegree program students are taking their newly-mastered skills into the broader world, and their projects are incredible!

The talent and passion of students in Udacity’s Self-Driving Car Engineer Nanodeegree Program regularly astounds me. Here are five independent projects that students did outside of the program to build their skills as autonomous vehicle engineers.

OpenCV Python Neural Network Autonomous RC Car

Wang Zheng

Check out the autonomous hardware package strapped to the top of this tiny red range rover! And the various test track configurations it navigates. Super cool.

ConvNets Series. Spatial Transformer Networks

Kirill Danilyuk

Spatial Transformers are modules that can be inserted into convolutional neural network architectures to focus the network on the most important object in the image. This is helpful because scale and rotation make object localization (finding an object within an image) a complex problem.

“The STN is a differentiable module which can be injected in a convolutional neural network. The default choice is to place it right “after” the input layer to make it learn the best transformation matrix theta which minimizes the loss function of the main classifier (in our case, this is IDSIA).”

Building an Autonomous Car using a 1/10th Scale RC Car — Part 1

Eric Maggard

This is an awesome four-part series on building a miniature self-driving car from scratch, with a big emphasis on hardware and electrical engineering. Part 1 is ROS setup, Part 2 is the sensor suite, Part 3 is the microcontroller, and Part 4 is working with the NVIDIA Jetson TX1. This is quite the hacker project.

“The goal of this project is to build an autonomous base that can navigate the sidewalks of my subdivision. It will use GPS, LIDAR, and other sensors to navigate to GPS way points, avoid obstacles, and return to the start position.”

4K SSD Object Detection #1

Karol Majek

I love watching videos that students shoot themselves. Here Karol is applying a Single-Shot Detector (SSD) network to identify other vehicles on the road.

Building an affordable self-driving car development platform

Bogdan Djukic

Scaling up from miniature self-driving cars to human-sized self-driving cars, Bogdan outlines a self-driving car development platform accessible for under US$10,000. This does not include the sensor suite — just the drive-by-wire platform. He settled on the Renault Twizy and is looking for partners to work on this with him 🙂

“One of the main challenges in the self-driving car industry (among other things like technology itself, policy updates, ethical issues, etc.) is the barrier of entry. If you are a small start up building a local autonomous delivery service or a single engineer trying out latest deep learning approaches for car/traffic sign detection, it is incredible hard (sometimes even impossible) to get things off the ground and test your solution in the real world setting.”

These amazing engineers are skilled, passionate, and actively building. If you like the idea of engaging with innovators like these, you should enroll in the Udacity Self-Driving Car Engineer Nanodegree Program!

The “Introduction to TensorFlow” Lesson

The sixth lesson of the Udacity Self-Driving Car Engineer Nanodegree Program is “Introduction to TensorFlow.”

TensorFlow is Google’s library for deep learning, and one of the most popular tools for building and training deep neural networks. In the previous lesson, MiniFlow, students build their own miniature versions of a deep learning library. But for real deep learning work, an industry-standard library like TensorFlow is essential.

This lesson combines videos from Vincent Vanhoucke’s free Udacity Deep Learning course with new material we have added to support installing and working with TensorFlow.

Students learn the differences between regression and classification problems. Then they to build a logistic classifier in TensorFlow. Finally, students use fundamental techniques like activation functions, one-hot encoding, and cross-entropy loss to train feedforward networks.

Most of these topics are already familiar to students from the previous “Introduction to Neural Networks” and “MiniFlow” lessons, but implementing them in TensorFlow is a whole new animal. This lesson provides lots of quizzes and solutions demonstrating how to do that.

Towards the end of the lesson, students walk through a quick tutorial on using GPU-enabled AWS EC2 instances to train deep neural networks. Thank you to our friends at AWS Educate for providing free credits to Udacity students to use for training neural networks!

Deep learning has been around for a long time, but it has only really taken off in the last five years because of the ability to use GPUs to dramatically accelerate the training of neural networks. Students who have their own high-performance GPUs are able to experience this acceleration locally. But many students do not own their own GPUs, and AWS EC2 instances are a cloud tool for achieving the same results from anywhere.

The lesson closes with a lab in which students use TensorFlow to perform the classic deep learning exercise of classifying characters: ‘A’, ‘B’, ‘C’ and so on.

Self-Driving Path Planning, Brought to You by Udacity Students

Term 3 of the Udacity Self-Driving Car Engineer Nanodegree Program starts with path planning. This is one of the deepest and hardest problems for a self-driving car.

Here are three Udacity student approaches that show the complexity and beauty of path planning.

Reflections on Designing a Virtual Highway Path Planner (Part 1/3)


Mithi published a three-part series about what she calls “the most difficult project yet” of the Nanodegree Program. In Part 1, she outlines the goals and constraints of the project, and decides on how to approach the solution. Part 2 covers the architecture of the solution, including the classes Mithi developed and the math for trajectory generation. Part 3 covers implementation, behavior planning, cost functions, and some extra considerations that could be added to improve the planner. This is a great series to review if you’re just starting the project.

“I decided that I should start with a simple model with many simple assumptions and work from there. If the assumption does not work then I will then make my model more complex. I should keep it simple (stupid!).

A programmer should not add functionality until deemed necessary. Always implement things when you actually need them, never when you just foresee that you need them. A famous programmer said that somewhere.

My design principle is, make everything simple if you can get away with it.”

Path Planning in Highways for an Autonomous Vehicle

Mohan Karthik

Mohan takes a different approach to path planning, in which he combines a cost function with a feasibility checklist. He builds a cost function and then ranks each lane by how it does on a cost function. Then he decides whether to move to a lane based on the feasibility checklist.

“This comes down to two things (and I’m going to be specific to highway scenario).

Estimating a score for each lane, to determine the best lane for us to be in (efficiency)

Evaluating the feasibility of moving to that lane in the immediate future (safety & comfort)”

Self-Driving Car Engineer Diary — 11

Andrew Wilkie

The 11th post in Andrew’s series on the Nanodegree Program covers Term 3 broadly and path planning specifically. In particular, Andrew lays out where this path planning project falls in the taxonomy of autonomous driving, and the high-level inputs and outputs of a path planner. This is a great post to review if you’re interested in what a path planner does.

“I found the path planning project challenging, in large part due to fact that we are implementing SAE Level 4 functionality in C++ and the complexity that comes with the interactions required between the various modules.”

These examples make clear the vision, skill, and tenacity our students are applying to even the most difficult challenges, and it’s a real pleasure to share their incredible work. It won’t be long before these talented individuals graduate the program, and begin making significant, real-world contributions to the future of self-driving cars. I know I speak for everyone at Udacity when I say that I’m very excited for the future they’re going to help build!

Which Udacity Nanodegree Program Is Right For You?

Are you trying to decide which Udacity Nanodegree Program you should enroll in? Here’s an all-in-one guide to help you determine which program is best for you.

Android Basics

Partner: Google
Lead Instructors: Katherine Kuan, Chris Lei
Difficulty: Beginner
Time: 6 months
Syllabus: User Interface + User Input + Multiscreen Apps + Networking + Data Storage
Prerequisites: None!
Cost: $199 / month
Best For: Aspiring Android Developers with no programming experience.

Android Developer

Partner: Google
Lead Instructor: James Williams, Reto Meier
Difficulty: Intermediate
Time: 8 months
Syllabus: Developing Android Apps + Advanced Android App Development + Gradle for Android and Java + Material Design for Android Developers + Capstone Project
Prerequisites: Java, git, GitHub
Cost: $999 upfront OR $199/month
Best For: Intermediate programmers who want to become Android specialists.

Artificial Intelligence

Partners: IBM Watson, Amazon Alexa, DiDi Chuxing, Affectiva
Lead Instructor: Sebastian Thrun, Peter Norvig
Difficulty: Advanced
Time: 6 months
Syllabus: Foundations of AI + Deep Learning and Applications + Computer Vision + Natural Language Processing + Voice User Interfaces
Prerequisites: Python, basic linear algebra, calculus, and probability
Cost: $1600
Best For: Engineers who want to apply AI tools across an array of domains, from computer vision to natural language processing to voice interfaces.

Become an iOS Developer

Partners: AT&T, Lyft, Google
Difficulty: Intermediate
Time: 6 months
Syllabus: UIKit Fundamentals + iOS Networking with Swift + iOS Persistence and Core Data + How to Make an iOS App
Prerequisites: macOS 10.12 or OS X 10.11.5
Cost: $199 / month
Best For:
Beginners who want to launch their iOS developer careers.

Business Analyst

Partners: Alteryx, Tableau
Lead Instructor: Patrick Nussbaumer
Difficulty: Intermediate
Time: 160 hours
Syllabus: Problem Solving with Advanced Analytics + Creating an Analytical Dataset + Segmentation and Clustering + Data Visualization in Tableau + Classification Models + A/B Testing for Business Analysts + Time Series Forecasting
Prerequisites: Basic statistics and spreadsheet skills, a Windows computer
Cost: $199 / month
Best For:
Aspiring data analysts who want to launch a career in data-driven decision-making and visualization, as opposed to programming.

Data Analyst

Partners: Facebook, Tableau
Lead Instructor: Caroline Buckey
Difficulty: Intermediate
Time: 260 hours
Syllabus: Descriptive Statistics + Intro to Data Analysis + Git and GitHub + Data Wrangling + MongoDB + Exploratory Data Analysis + Inferential Statistics + Intro to Machine Learning + Data Visualization in Tableau + Introduction to Python Programming
Prerequisites: None!
Cost: $199 / month
Best For:
Aspiring data scientists who want to launch a career in developing software to extract meaning from data.

Deep Learning Foundations

Lead Instructors: Ian Goodfellow, Andrew Trask, Mat Leonard
Difficulty: Intermediate
Time: 6 months
Syllabus: Introduction + Neural Networks + Convolutional Neural Networks + Recurrent Neural Networks + Generative Adversarial Networks
Prerequisites: Python, basic linear algebra and calculus
Best For:
Students excited by the potential for deep learning to change the world, and who additionally wish to earn guaranteed entry into Udacity’s Artificial Intelligence, Robotics, or Self-Driving Car Engineer Nanodegree Programs (a special “perk” of the program for graduates!).

Digital Marketing

Partners: Facebook, Google, Hootsuite, HubSpot, MailChimp, Moz
Lead Instructor: Anke Audenaert
Time: 3 months
Syllabus: Marketing Fundamentals + Content Strategy + Social Media Marketing + Social Media Advertising through Facebook + Search Engine Optimization (SEO) + Search Engine Marketing with AdWords + Display Advertising + Email Marketing + Measure and Optimize with Google Analytics
Prerequisites: None!
Best For:
Hard workers seeking to launch or advance their digital marketing careers through real-world experience and multi-platform fluency.

Front-End Web Developer

Partners: AT&T, Google, GitHub, HackReactor
Lead Instructors: Mike Wales, Cameron Pittman
Difficulty: Intermediate
Time: 6 months
Syllabus: Intro to HTML and CSS + Responsive Web Design Fundamentals + Responsive Images + JavaScript Basics + Intro to jQuery + Object-Oriented JavaScript + HTML5 Canvas + Browser Rendering Optimization + Website Performance Optimization + Intro to AJAX + JavaScript Design Patterns + JavaScript Testing
Prerequisites: Basic computer programming
Cost: $199 / month
Best For:
New web developers who want to build a portfolio and get a job!

Full Stack Web Developer

Partners: Amazon Web Services, GitHub, AT&T, Google
Lead Instructors: Mike Wales, Karl Krueger
Difficulty: Intermediate
Time: 6 months
Syllabus: Programming Foundations with Python + Responsive Web Design Fundamentals + Intro to HTML and CSS + Responsive Images + Intro to Relational Databases + Authentication & Authorization: OAuth + Full Stack Foundations + Intro to AJAX + JavaScript Design Patterns + Configuring Linux Web Servers + Linux Command Line Basics
Prerequisites: Python and git
Cost: $199 / month
Best For:
Developers who want to learn to build web applications from end-to-end.

Intro to Programming

Lead Instructor: Andy Brown
Time: 5 months
Syllabus: Learn to Code + Make a Stylish Webpage + Python Programming Foundations + Object-Oriented Programming with Python + Explore Programming Career Options + Experience a Career Path
Prerequisites: None!
Cost: $399
Best For:
Beginners looking for an accessible approach to coding.

Machine Learning Engineer

Partner: Kaggle
Lead Instructors:
Apran Chakraborty, David Joyner, Luis Serrano
Time: 6 months
Syllabus: Machine Learning Foundations + Supervised Learning + Unsupervised Learning + Reinforcement Learning + Deep Learning + Capstone
Prerequisites: Intermediate Python, statistics, calculus, and linear algebra
Cost: $199 / month
Best For:
Engineers who want to build applications that learn from data.


Lead Instructors: Michael Jackson, Ryan Florence, Tyler McGinnis
Time: 4 months
Syllabus: React Fundamentals + React & Redux + React Native
Prerequisites: HTML, JavaScript, Git
Cost: $499
Best For:
Front-end engineers who want to master the web’s hottest framework. React is the highest-paid sub-field of web development!


Partners: Bosch, Electric Movement, iRobot, Kuka, Lockheed Martin, MegaBots, Uber ATG, X
Lead Instructor:
Ryan Keenan
Time: 6 months
Syllabus: ROS Essentials, Kinematics, Perception, Controls, Deep Learning for Robotics
Prerequisites: Intermediate Python, calculus, linear algebra, and statistics
Cost: $2400
Best For:
Makers who dream of building machines that impact everything from agriculture to manufacturing to security to healthcare.

Self-Driving Car Engineer

Partners: Mercedes-Benz, NVIDIA, Uber ATG
Lead Instructor:
David Silver (that’s me!)
Time: 9 months
Syllabus: Deep Learning + Computer Vision + Sensor Fusion + Localization + Path Planning + Control + System Integration
Prerequisites: Intermediate Python, calculus, linear algebra, and statistics
Cost: $2400
Best For:
Engineers who want to join technology’s hottest field and revolutionize how we live.

VR Developer

Partners: Google VR, Vive, Upload, Unity, Samsung
Lead Instructor:
Christian Plagemann
Time: 6 months
Syllabus: Unity + C# + Google Cardboard + Ergonomics + User Testing + Interface Design + Mobile Performance + High-Immersion Unity + High-Immersion Unreal
Prerequisites: None!
Cost: $1200
Best For:
People who want to build new worlds. VR is the most in-demand skill for freelance developers!