When Udacity launched the Self-Driving Car Program last summer, Otto, which has since become Uber Advanced Technologies Group, was there from the beginning.
They only had 100 people in a semi-deserted warehouse, and not a lot of engineering talent to spare. But they have always been so enthusiastic to spend time teaching students about self-driving cars.
Drew Gray, in particular, is a wonder of an engineer. Whether we need help with deep learning, or control, or tying computer vision to trajectory planning, Drew knows everything and is always excited to help Udacity students.
Thereās a wrinkle at the end of one of our Computer Vision projects where we ask students to use lane lines to calculate the radius of curvature of the road. This seems like a weird thing to do, but it ultimately ties into trajectory planning. We only made the leap because Drew pointed it out to us.
Uber has received a lot of criticismrecently, and maybe deservedly so. I donāt have much insight into that and you can read better analyses elsewhere.
But I know the folks at Uber ATG, and they have been great partners and great advocates for students who want to get jobs working on autonomous vehicles. And I am grateful for that.
Now would be a good time reiterate something I havenāt posted in a while, which is that this is a personal blog. I have an agenda, which is pretty transparent in my Medium bio. But I donāt speak for Udacity, or CarND students, and certainly not for Udacity partners.
These topics are taught by expert engineers from Mercedes-Benz and Uber ATG, who teach the skills they actually use on the job. These are also the skills students need to know get jobs working on autonomous vehicles.
Although thereās a lot of math and programming involved, there are also some good stories. Watch this video of Dominik and Andrei, from the Mercedes-Benz Sensor Fusion Team, talking with Sebastian about how the unscented Kalman filter got its name.
And if youād like to understand what the unscented Kalman filter is, hereās Dominik (this one involves more math):
If you want to explore different areas of computer vision, you should check out these awesome posts by Udacity students on different ways to use OpenCV to find lane lines.
Upon completing all of his Term 1 projects, Arnaldo wrote a high-level overview of all of the projects, and reflected on what he learned:
āIt had a very practical focus: theory enough to understand the core concepts, and then, the practical application. It is a reason why it requires a lot of background. It is not a course on basic python, or basic neural networks, but how to apply it in real cases.ā
Sujay has an incredibly thorough analysis of his computer vision pipeline for lane-finding, including a great debugging tool:
āThis project involves fine tuning of lot of parameters like color thresholding, gradient thresholding values to obtain the best lane detection. This can be trickier if the pipeline fails for few video frames. To efficiently debug this I had to build a frame that captures multiple stages of the pipeline, like the color transformation, gradient thresholding, line fitting on present and averaged past frames.ā
Andrew has posts fun images from his lane-finding pipeline, but what really caught my eye was his analysis of Udacityās Career Services:
āThe SDC Engineer course emphasises job readiness and the Udacity team provides an excellent Careers Service built right in. We were asked to search for an advertised job that interested us, provide a resume tailored to either Entry Level, Prior Experience or Career Change and the associated Cover Letter. For a ācareer changerā like me, I was surprised by the amount of self-reflection this caused.ā
Morgane is completing CarND while on a nine-month world tour, starting in Ecuador!
āOur next stop was Cuenca, Ecuador, where my fianceās immediate family lives. I had an amazing time there, visiting the city and Cajas National Park. The only issue is that people did not understand why I was spending so much time on my laptop! They expected me to be free all the time since I was on holidays. I had set up a routine where Iād work for a few hours in the morning while waiting for people to get up and at night. After explaining to them what my goal was and showing them what I was doing, they were definitely more understanding. They got particularly interested when I showed them how I had trained a neural network to drive a car in a simulator, and how I used computer vision and machine learning to recognize lanes and other vehicles on the road.ā
Liam took the introductory Udacity Lane-Finding Project and optimized it:
āI created a buffer to store the slope and y-intercept values for each line detected in the last N frames. The actual line drawn on the current frame is simply the average slope/intercept of all these lines. By continuously pushing the latest detected line onto this buffer and simultaneously dropping the oldest line, I can calculate a rolling mean of the lines over time, or what I call ātemporal smoothingā.ā
The very first partner to sign up was NVIDIA. The NVIDIA team is super-excited about the Udacity Nanodegree Program and is actively interviewing students in the program, even before they graduate.
If youād like to learn more about how NVIDIA drives autonomous vehicle technology, watch the video we made with them:
Here are some great explanatory posts from Udacity Self-Driving Car students about Kalman filters, computer vision, and how to build a mini autonomous vehicle.
Vivek has transposed some of the notes from his Advanced Controls course at SUNY-Stony Brook. These notes are great for understanding the intuition of how Kalman filters reduce uncertainty:
āThis process of combining system dynamics with state measurements is the underlying principle of Kalman filters. Kalman filters provide good estimation properties and are optimal in the special case when the process and measurement follow a Gaussian distributions.ā
I like the intuitive explanation that Arnaldo provides for the histogram of oriented gradients (HOG) algorithm:
āThe HOG extractor is the heart of the method described here. It is a way to extract meaningful features of a image. It captures the āgeneral aspectā of cars, not the āspecific detailsā of it. It is the same as we, humans, do: in a first glance, we locate the car, not the make, the plate, the wheel, or other small detail.
HOG stands for āHistogram of Oriented Gradientsā. Basically, it divides an image in several pieces. For each piece, it calculates the gradient of variation in a given number of orientations.ā
I love the mini autonomous vehicle that Shrek built, and especially I love that he trained a deep neural network so that the car react to traffic signs!
ādriver.pyāāāis the heart of the project. It includes the image processing pipeline that identifies the traffic sign from the camera by using the previously trained DNN and then send appropriate driving control signals to the car.ā
Alena went beyond the minimum requirements for the Advanced Lane Finding Project, and she got her computer vision pipeline to work on the challenge video:
āWhen the first line was identified successfully, I use line equation with margin as an area to search for the next line. No need to do Sliding Window Search again.ā
Although the headline is about Uber and Google, the article is really about current Uber executive, and former Google executive, Anthony Levandowski.
The article is a kind of mini-autobiography of Levandowski, with a particular emphasis on the latest stages in his careerāāāGoogle (now Waymo), then Otto, then Uber Advanced Technologies Group.
The article doesnāt take a strong position on the merits of Googleās lawsuit against Uber and Levandowski. Itās more useful as background information on some of the most important individuals and companies in the industry.
Before I drop in some quotes, I should mention that I have a number of connections to people who appear in the article. My boss, Sebastian Thrun, launched the Google Self-Driving Car program. The article doesnāt quote him, but he does appear in a few cameos. Additionally, Otto, and now Uber ATG, has been a terrific partner of the Udacity Self-Driving Car Engineer Nanodegree Program. Iāve met Levandowski briefly, and he seems like good people.
Here are some of the quotes that stuck with me:
āAt 16 he started a web design firm that a former colleague says made him a millionaire by the end of high school.ā
āGhostrider, his self-balancing, self-driving motorcycle, was the only two-wheel vehicle in the contest [the 2004 DARPA Challenge].ā
āAnthony is a rogue force of nature,ā says a former Google self-driving car executive. āEach phase of his Google career he had a separate company doing exactly the same work.ā According to two former Google employees, founders Page and Sergey Brin tolerated Levandowskiās freelancing because they saw it as the fastest way to make progress. Googleās car team embraced Levandowskiās nature, too. The attitude, says a former colleague, was āheās an asshole, but heās our asshole.ā
Here are five Udacity Self-Driving Car students that went above and beyond the project requirements by using trigonometry, color spaces, dashcams, buffers, and teaching other students.
Andrew digs way back into high school trigonometry to investigate the relationship between the multiple cameras used in the Behavioral Cloning Project. The result is a pretty awesome run:
āAs the car is making a stronger turn from the center angle, it can be safely assumed that the distance between the center camera and its goal (point A) is shrinking in relation to when point A was a significant distance away from the carāāāwhen steering angle D was 0. This can easily be seen in how far away the most distant center point on the road is when the car is driving straight versus when it is in a 25 degree turn. After getting tired of manually calculating SOH CAH TOA/law of cosines and sines over and over again, I wrote a couple python scripts to chart out what the change in the angle would be given a center angle and a distance between each camera along with a distance to the goal at 0 degrees and the distance to the goal at the given angle.ā
Joshua is a Nigerian PhD student at a Japanese university, making him a perfect example of Udacityās goal of democratizing education. He also does a terrific job experimenting with various color spaces for isolating lane lines:
āWe know that white is represented by (255,255,255) in both RGB and HSL colour space, but we have to ascertain the values for the yellow lane colour. To do this, I pulled out colour picker from the Inkscape software, just to have a visual representation.ā
Chris bought a dashcam and super-imposed his lane-finding algorithm on one of his regular routesāāāup the 5 in San Diego:
āI found it eerily calming to safely watch a drive I do way too often. Itās like the difference between watching a horse and being dragged behind one. The video starts just after what I believe to be the most dangerous on-ramp in the world. Having survived that, we can check out the camera, my driving, and my algorithmās ability to find lane lines.ā
Kirill goes way beyond the requirements in his initial lane-finding project, and builds a computer vision pipeline that can identify lanes in all sorts of conditions. In particular, he develops a nice stability heuristic:
āLane lines stability. This is an important issue to be addressed. There are several stabilization techniques I used:
1. Buffers. My lane line objects memorize N recent states and update the buffer itself by inserting a line state from the current frame.
2. Smarter lane line state updates. If we still get noisy data after our filtering efforts, line fitting can easily go wrong. If we see that the estimated slope of the fitted line from the current frame differs too much from the bufferās average, we need to treat this line more conservatively. For this very purpose, I created DECISION_MAT, which is a simple decision matrix on how to combine current line position and bufferās average position.ā
Robin has an easy-to-follow writeup of my sequence of videos demonstrating an iterative approach to completing the Behavioral Cloning Project. Start with some basic data, normalize it, augment it, add side cameras, and keep going:
āThis is the stage at which the car began to make it around the track successfully for me. The behavioral cloning project is a great lesson in the fun and value in experimentation when working with deep learning. Altering model architecture and parameters help us build an intuition of how convolutions, drop out layers and subsamples work together to make a useful model.ā
Jonathanās post has a really nice walkthrough of his behavioral cloning project, including a visual explanation of the data pre-processing pipeline:
āOur data comes in as 160 x 320 x 3 RGB images. I used several preprocessing techniques to augment, transform, and create more data to give my network a better chance at generalizing to different track features.ā
Kaspar has a terrific comparison of vehicle detection pipelines using standard computer vision, compared with a deep learning solution using YOLO:
āA forward pass of an entire image through the network is more expensive than extracting a feature vector of an image patch and passing it through an SVM. Hoever, this operation needs to be done exactly once for an entire image, as opposed to the roughly 150 times in the SVM+HOG approach. For generating the video above I did no performance optimization, like reducing the image or defining a region of interest, or even training specifically for cars. Nevertheless, YOLO is more than 20x faster than the SVM+HOG and at least as accurate.ā
Caleb put together an awesome list of autonomous vehicle skills, and which companies are looking for them:
āThe one constant in all of the postings is that experience with programming in C++ is a highly sought attribute for self-driving companies. Since performance is so vital for any code running on a real time system, itās necessary to use a language that can be compiled to machine code for speed.ā
Machine versus human learning in traffic sign classification
Arnaldo has a fun comparison of how machine learning compares to human learning, specifically applied to the Traffic Sign Classifier Project:
āOverfit: the guy who has a perfect grade in school, in all subjects, but outside school knows nothing in real world. Or someone who has a phD in nuclear advanced theoretical gravitational quantum physics, but works as a waiter in a restaurant, because his knowledge is so specific it has no real world application.ā
Andrew has a generous assessment of Term 1 of CarND:
āāAmazing projects. Steep learning curve. Strong student community. Incredibly supportive and adaptive Udacity staff. Be prepared to commit 2ā3 times estimated 10 hours per week to complete Term 1 successfully as projects encourage experimentation. Now to catch-up on sleep before the start of Term 2 on 24/Mar/2017.ā
The automotive supply chain is highly-structured, with Tier 2 suppliers (like NVIDIA) providing specialized products to Tier 1 suppliers (like Bosch) who package them as automotive-grade components that automotive manufacturers (like Mercedes-Benz or BMW) can use.
One of the big challenges from NVIDIA and Intel and any other chipmaker breaking into the automotive space is how to take their incredible technology and fit it into the automotive supply chain, with all of the safety and reliability checks that entails.
Bosch solves that manufacturing and distribution problem for NVIDIA.
David Silver 