Visiting Apollo

Talking Self-Driving Cars with Baidu’s Apollo team in China, and visiting Udacity’s offices in Shanghai and Beijing!

This past week I had the pleasure of visiting China for the first time in 20 years! I spent a few days working with colleagues at Udacity’s office in Shanghai, and followed that with several days at Udacity’s Beijing office. I was also able to take in some additional Beijing-based events. It was a whirlwind tour, and I loved it!

My traveling group was hosted by the terrific Apollo team at Baidu. Baidu is China’s largest search engine company, and one of the largest Internet companies in the world. And Udacity is building a free self-driving car course with them!

“Together with Baidu, we look forward to popularizing the Apollo system, and to giving everyone the opportunity to become a self-driving car engineer.” — Sebastian Thrun

This course will provide a conceptual overview of self-driving car technology, illustrated with the Apollo open-source self-driving car stack that Baidu is building.

Baidu has invested heavily in self-driving cars and has rapidly become an important player in the ecosystem. They are already testing vehicles on their Beijing campus.

Their vehicles come in all shapes and sizes. Some of Baidu’s vehicles look like Carla, Udacity’s very own self-driving car.

Carla!

But Baidu has 13 different types of self-driving vehicles, ranging from small cars to big trucks!

Beyond autonomous vehicles, Baidu has a world-leading artificial intelligence group. In their lobby I got to play with one of their robots, which talked, snapped my photo, and walked around with me.

The Baidu team was also kind enough to arrange and host an on-camera interview for me, with CSDN, a Chinese software developer network.

The most important part of the visit, however, were the ping-pong matches. I played two matches and went 1–1 in my first international ping-pong competition. No photos, you’ll have to take my word for it 🙂

Udacity’s Self-Driving European Tour

Taking Udacity’s Self-Driving Car Engineer Nanodegree program on the road in Europe to meet with students and partners!

Michael Ikemann (Udacity’s first Intro to Self-Driving Car Nanodegree graduate) and me

A few weeks ago, I had the delight of visiting Europe with Udacity’s Berlin-based European team, meeting both automotive partners and Udacity students. The trip was so much fun!

Stuttgart

We started in Stuttgart, where we met with our partners at Bosch and toured their Abstatt campus. Their campus reminds me of a plush Silicon Valley office, except instead of overlooking Highway 101, they overlook vineyards and a European castle.

Thanks to Udacity student Tolga Mert for organizing!

In the evening, Bosch’s Mirko Franke joined us at the Connected Autonomous Driving Meetup, organized by Udacity student Rainer Bariess.

We discussed the self-driving ecosystem and, of course, how to get a job working on self-driving cars at Bosch.

Berlin

The next day we headed to Berlin to prepare for our deep learning workshop at Automotive Tech.AD. What a great collection of autonomous vehicle engineers from companies across Europe!

In the evening we hosted a Meetup for current and prospective Udacity students at our Berlin office. It is always a delight to meet students and hear firsthand what they love about Udacity, and how they feel we can improve the student experience.

London

Our final stop was London, for an interview with Alan Martin at Alphr.

That evening, Udacity alumnus Brian Holt, Head of Autonomous Driving at Parkopedia, hosted us for at the London Self-Driving/Autonomous Car Technology Meetup. We had a blast talking about the future of self-driving (and even flying!) cars.

It’s a lot of fun to fly across the globe and see different places, but the best experience of all is getting to meet students from all different parts of the world.

We learn what our students are working on, what excites them about self-driving cars, and about the difference Udacity has made in their lives. It’s wonderful!

If you’re interested in becoming a part of our global Self-Driving Car community, consider enrolling in one of our Nanodegree programs. No matter your skills and experience, we’ve got a program for you!

https://www.udacity.com/course/intro-to-self-driving-cars–nd113https://www.udacity.com/course/intro-to-self-driving-cars–nd113

Next Week in Berlin!

I am excited to be heading to Germany next week!

Stuttgart

On Monday I will be in Stuttgart. Join my Udacity colleagues and me at the Connected Autonomous Vehicle Meetup on March 5, organized by Udacity student Rainer BareiĂź. RSVP here.

Berlin

On Wednesday I will be in Berlin. Join us on March 7 at Udacity’s Berlin office. Free pizza and drinks! I’ll show off some self-driving car videos and share the latest news about what Udacity is building for students. 7–9pm. RSVP here.

Bis bald in Deutschland 🙂

Udacity’s Intro to Self-Driving Cars Nanodegree Program

Experience a free preview of our program today, and discover how YOU can enter the amazing world of self-driving cars!

Sebastian Thrun, Udacity founder, and the “Godfather” of Self-Driving Cars

Udacity’s Intro to Self-Driving Cars Nanodegree Program is for anyone in the world who ever dreamed of working on self-driving cars, but thought they never could.

This program covers the prerequisite skills necessary to advance to our career-ready Self-Driving Car Engineer Nanodegree Program, including:

  • Intermediate programming ability in C++ or Python (the languages of the autonomous vehicle industry)
  • Basic linear algebra
  • Basic calculus
  • Basic statistics
  • Basic physics

As a student in the Intro to Self-Driving Cars Nanodegree program, you’ll build your skills up over the course of a four-month curriculum path that tackles each of these areas at a pace that is both manageable and rewarding. Best of all, you’ll practice putting these skills to work on the types of projects that real self-driving car engineers work on every day.

If you love self-driving cars, but thought you’d never get the chance to work on them, then this is the program for you.

Check out a free preview of the program right now!

Amazing Self-Driving Car Projects

Udacity Self-Driving Car Engineer Nanodegree program students are taking their newly-mastered skills into the broader world, and their projects are incredible!

The talent and passion of students in Udacity’s Self-Driving Car Engineer Nanodeegree Program regularly astounds me. Here are five independent projects that students did outside of the program to build their skills as autonomous vehicle engineers.

OpenCV Python Neural Network Autonomous RC Car

Wang Zheng

Check out the autonomous hardware package strapped to the top of this tiny red range rover! And the various test track configurations it navigates. Super cool.

ConvNets Series. Spatial Transformer Networks

Kirill Danilyuk

Spatial Transformers are modules that can be inserted into convolutional neural network architectures to focus the network on the most important object in the image. This is helpful because scale and rotation make object localization (finding an object within an image) a complex problem.

“The STN is a differentiable module which can be injected in a convolutional neural network. The default choice is to place it right “after” the input layer to make it learn the best transformation matrix theta which minimizes the loss function of the main classifier (in our case, this is IDSIA).”

Building an Autonomous Car using a 1/10th Scale RC Car — Part 1

Eric Maggard

This is an awesome four-part series on building a miniature self-driving car from scratch, with a big emphasis on hardware and electrical engineering. Part 1 is ROS setup, Part 2 is the sensor suite, Part 3 is the microcontroller, and Part 4 is working with the NVIDIA Jetson TX1. This is quite the hacker project.

“The goal of this project is to build an autonomous base that can navigate the sidewalks of my subdivision. It will use GPS, LIDAR, and other sensors to navigate to GPS way points, avoid obstacles, and return to the start position.”

4K SSD Object Detection #1

Karol Majek

I love watching videos that students shoot themselves. Here Karol is applying a Single-Shot Detector (SSD) network to identify other vehicles on the road.

Building an affordable self-driving car development platform

Bogdan Djukic

Scaling up from miniature self-driving cars to human-sized self-driving cars, Bogdan outlines a self-driving car development platform accessible for under US$10,000. This does not include the sensor suite — just the drive-by-wire platform. He settled on the Renault Twizy and is looking for partners to work on this with him 🙂

“One of the main challenges in the self-driving car industry (among other things like technology itself, policy updates, ethical issues, etc.) is the barrier of entry. If you are a small start up building a local autonomous delivery service or a single engineer trying out latest deep learning approaches for car/traffic sign detection, it is incredible hard (sometimes even impossible) to get things off the ground and test your solution in the real world setting.”


These amazing engineers are skilled, passionate, and actively building. If you like the idea of engaging with innovators like these, you should enroll in the Udacity Self-Driving Car Engineer Nanodegree Program!

Udacity Self-Driving Car Engineer Nanodegree Projects

Enjoy a look at some of the projects our students are building, including Finding Lane Lines, Traffic Sign Classifier, Behavioral Cloning, and more!

Students in our Self-Driving Car Engineer Nanodegree program engage in a project-based curriculum, and from the moment they enroll, they begin addressing key challenges and topics through building specialized projects. Here are all of the projects they build!

Finding Lane Lines

This is the first project students complete, one week into the program.

They learn to work with images, color spaces, thresholds, and gradients, in order to find lane lines on the road.
Stack: Python, NumPy, OpenCV

Traffic Sign Classifier

In this project, students train a convolutional neural network to classify traffic signs.

To do so, they use the German Traffic Sign Recognition Benchmark dataset. This particular student went above and beyond to train his network to not only classify signs, but also localize them within the image, and applied his classifier to a video.
Stack: Python, NumPy, TensorFlow

Behavioral Cloning

Here, students record training data by manually driving a car around a track in a simulator.

Then they use this camera, steering, and throttle data to train an end-to-end neural network for driving the vehicle, based on NVIDIA’s famous research paper.
Stack: Python, NumPy, Keras

Advanced Lane Finding

By applying advanced computer vision techniques, such as sliding window tracking, to a dashcam video, students are able to track lane lines on the road under a variety of challenging conditions.
Stack: Python, NumPy, OpenCV

Vehicle Detection and Tracking

Students use machine learning techniques and feature extraction to identify and track vehicles on a highway.
Stack: Python, NumPy, scikit-learn, OpenCV

Extended Kalman Filter

An extended Kalman filter merges noisy simulated radar and lidar data to track a vehicle.
Stack: C++, Eigen

Unscented Kalman Filter

An unscented Kalman filter merges noisy, highly non-linear simulated radar and lidar data to track a vehicle.
Stack: C++, Eigen

Kidnapped Vehicle

Students develop a particle filter in C++ to probabilistically determine a vehicles location relative to a sparse landmark map.
Stack: C++

PID Controller

Students build and tune a proportional-integral-derivative controller to steer a vehicle around a test track, following a target trajectory.
Stack: C++

Model Predictive Control

Students build and optimize a model predictive controller to steer a vehicle around a test track, following a target trajectory.
Stack: C++, ipopt

Path Planning

In this project, students construct a path planner for highway driving based on a finite state machine.

The planner has three components: environmental prediction, maneuver selection, and trajectory generation.
Stack: C++

Semantic Segmentation

Students train a pixel-wise segmentation network that identifies and colors road pixels to identify free space for driving.
Stack: Python, TensorFlow

Safety Case

Students build a prototype of a safety case for a lane-keeping assistance ADAS feature, including the safety plan, hazard analysis and risk assessment, functional safety concept, technical safety concept, and software requirements.

Programming a Real Self-Driving Car

For this project, students form teams to drive a real self-driving car around the Udacity test track.

The car is required to negotiate a traffic light and follow a waypoint trajectory. Code is built first in the simulator, and then deployed to Udacity’s self-driving car in California.
Stack: Python, ROS, Autoware, TensorFlow


Would you like to be building these kinds of projects yourself? Then you should apply to the Udacity Self-Driving Car Engineer Nanodegree Program!

The “Traffic Sign Classifier” Project

Traffic Sign Classifier is the second project, and the ninth lesson, in the Udacity Self-Driving Car Engineer Nanodegree Program.

In this project, students build and train a deep neural network to classify images from the German Traffic Sign Recognition Benchmark dataset. There are about 40 different types of German traffic signs in the dataset, each 32×32 pixels big. That’s not very big!

Nonetheless, each image is big enough for students to train a convolutional neural network to recognize what type of sign it is, with 95%+ accuracy. That’s close to, or even better than, the accuracy that humans like you and I reach when we classify images by sight.

The lesson starts out with a tour of LeNet, one of the canonical network architectures for image classification. We step through how to implement LeNet in TensorFlow, highlighting data preparation, training and testing, and configuring convolutional, pooling, and fully-connected layers.

We also show students how to spin-up a GPU-enabled EC2 instance from our partners at Amazon Web Services. Thank you to AWS Educate for providing free AWS credits to Udacity students!

At the end of the lesson, students get to apply, tweak, or completely revamp LeNet to train their own classifier. If you want to compare yourself to Yann LeCun, here’s how he did with the same dataset:

Ready to start learning how to build self-driving cars yourself? Great! If you have some experience already, you can apply to our Self-Driving Car Engineer Nanodegree program here, and if you’re just getting started, then we encourage you to enroll in our Intro to Self-Driving Cars Nanodegree program here!

Udacity goes to the North American International Auto Show in Detroit

Udacity students in attendance on January 17th, you’re invited to lunch!

I am excited to be visiting Detroit next week for the North American International Auto Show. Please come say hello if you’re attending, or send me an email at david.silver@udacity.com.

Here’s the schedule:

Wednesday, January 17, 12:30pm: I’ll be at Automobili-D, speaking on a panel about the autonomous driving industry. Come watch! If you’re a Udacity student, we’ll take you to lunch afterward.

Thursday, January 18, 6pm: RSVP for our Career Workshop and Meetup! We’ll teach you how to present yourself to recruiters in the autonomous vehicle industry, and you’ll practice pitching yourself to other attendees. Also, free food!

Saturday, January 20, 10:30am: I’ll be speaking at the Future Automotive Career Exposition at NAIAS. Introduce yourself to me, then introduce yourself to the many recruiters who will be hiring autonomous vehicle engineers!

See you in Detroit!

Udacity and Baidu’s New Course: Introduction to Apollo

At CES, Udacity and Baidu jointly announced a free intro course for the Apollo self-driving platform!

Big news from CES! Udacity is going to produce a one-month free course on developing self-driving car software with Baidu Apollo! Sebastian Thrun, Udacity’s founder (and the father of the self-driving car), announced this together with Baidu COO Qi Lu at CES today.

Apollo

Baidu has open-sourced their self-driving car software stack, Apollo, with the goal of creating the “Android of the autonomous driving industry”.

Apollo has four layers:

  • Cloud Service
  • Apollo Open Software Stack
  • Reference Hardware Platform
  • Reference Vehicle Platform

Udacity’s upcoming “Intro to Apollo” course will focus on the top two layers: Cloud Service and Apollo Open Software Stack.

The Course

Apollo is an incredibly exciting platform in the autonomous vehicle industry. We are thrilled to work with the Apollo team to teach students and engineers around the world how to build self-driving car software quickly using the Apollo stack.

I am especially delighted that this will be a free course, open to anyone with the desire to enter this amazing field. There is a huge demand for knowledge about how self-driving cars work, and this course will help educate the world on this topic. Our Self-Driving Car Engineer Nanodegree Program is an intense nine-month journey to becoming a self-driving car engineer, and it offers an amazing learning experience, but it is for advanced engineers. And while our Intro to Self-Driving Cars Nanodegree program is an excellent point-of-entry for aspiring learners newer to the field, it offers an equally immersive experience. This course offers adds something new and important to the range of learning options.

China

This is a special opportunity for us to collaborate with Baidu, one of the leading companies in China. China is a leader in the autonomous vehicle industry. And Chinese students currently make up 5% of enrollment in Udacity’s Self-Driving Car Engineer Nanodegree Program, and 20% of enrollments in all Udacity programs. A major focus for our Self-Driving Car Program in 2018 is to reach even more students in China.

The course will be developed jointly by Baidu’s Apollo team, the Udacity Self-Driving Car team in Mountain View, and the Udacity China team. The course will be in English, but this is a new experiment for us in developing course material in one of our offices outside of the US. I’m excited.

Did I mention I’m excited about this course? Because I’m excited!

How Udacity Students Build Path Planners for Highway Driving

Many students describe the Path Planning Project as the most challenging project in the entire Udacity Self-Driving Car Engineer Nanodegree program. This is understandable. Path planning is hard! But it’s not too hard, and I’m going to tell you a bit about the project—and about path planning in general—in this post.

There are three core components to path planning: 1) Predicting what other vehicles on the road will do next, 2) Deciding on a maneuver to execute, in response to our own goals, and to our predictions about other vehicles, and 3) Building a trajectory to execute the maneuver we decide on.

This is a project that provides students a lot of freedom in how to implement their solution. Here are five approaches from our amazing students!

Reflections on Designing a Virtual Highway Path Planner (Part 1/3)

Mithi

I love Mithi’s series of posts on the Path Planning Project. Her first post covers the project outline and her solution design process. The second post covers her data structures and pipeline. The third and final post dives into the mechanics and math required to actually produce a path. This is a great series of posts for anybody thinking about building a path planner.

The goal is to create a path planning pipeline that would smartly, safely, and comfortably navigate a virtual car around a virtual highway with other traffic. We are given a map of the highway, as well as sensor fusion and localization data about our car and nearby cars .We are supposed to give back a set of points (x , y)in a map that a perfect controller will execute every 0.02 seconds. Navigating safely and comfortably means we don’t bump into other cars, and we don’t exceed the maximum speed, acceleration, and jerk requirements. Navigating smartly means we change lanes when the car in front of us is too slow.

Path Planning in Highways for an Autonomous Vehicle

Mohan Karthik

In contrast to Mithi’s articles, which take you through her process of building a path planner, Mohan’s writeup does a great job of describing the final result. In particular, I was interested to read about the voting system he used for deciding on lane changes.

Instead of quoting Mohan, I’ll share the flowchart he built:

Self-Driving Car Engineer Diary — 11

Andrew Wilkie

This installment of Andrew’s long-running “diary” covers the Path Planning Project at a high level, and details how it fits into the third term of the Nanodegree program. Like his fellow classmates, Andrew also found this to be a challenging project.

I found the path planning project challenging, in large part due to fact that we are implementing SAE Level 4 functionality in C++ and the complexity that comes with the interactions required between the various modules.

Udacity Self Driving Car — Project 11 Path planning

Shyam Jagannathan

Shyam’s post contains a particularly concise 6-point walkthrough of trajectory generation, which is both fundamental to building a path planner, and surprisingly challenging.

The trajectory generation part which is the most difficult is covered as part of the project walk-through by Aaron Brown and David Silver. LINK. They recommend using the open source C++ tk:spline() method to generate a 5th degree polynomial which help minimize jerk while accelerating, decelerating and changing lanes.

Highway Path Planning

Alena Kastsiukavets

Alena touches on several interesting points with her post. She focuses on cost functions, which she identifies as the most important part of the project. The post describes her finite state machine and the associated cost functions in detail, and describes how the car decides when to shift lanes. She also touches on how she merged the two branches of her path planner — one for the Nanodegree project, and one for the Bosch Challenge — to create a more generalized planner.

The most important part of the project is to define cost functions. Let’s say we are on a highway with no other cars around. In such situation, we should stay in the same lane. Constant lane changing is extremely uncomfortable, so my first cost function is change_lane_cost. We penalized our trajectory if we want to change lane. Honestly, I did a small trick for Bosch challenge. I did not penalize the trajectory if I want to move in the middle lane. It gives me more freedom with maneuvers. Otherwise, I can be stuck in the left-most and right-most lanes when my lane and middle lanes are busy.

Seeing our students working through these challenges, experiencing their solutions, and learning about their processes fills me with so much excitement about the future of this field—these students represent the next generation of self-driving car engineers, and based on the work they’re already doing, I am certain they’re going to be making incredible contributions. I am especially moved by their generosity in taking the time to share in such detail the work they’re engaged in, and it’s a real pleasure to share their articles with you.

~

Ready to start working on self-driving cars yourself? Apply for our Udacity Self-Driving Car Engineer Nanodegree program today!