There is an angle here that ties into the Michigan vs. Silicon Valley competition for autonomous vehicle development, but that’s not what interests me.
What interests me is what this move says about Big Data in automotive applications. Thus far, most autonomous vehicle development work has proceeded with relatively small amounts of data, certainly compared the amount of data that companies like Google deal with.
Ford’s investment this new Flat Rock data center portends a future in which autonomous vehicle teams need to know about Hadoop and Spark, in addition to deep learning and robotics.
This is kind of last-minute (sorry!), but I have some free passes to give away for that conference, so send me an email (david.silver@udacity.com) if you want one.
One of the big challenges with working on cutting-edge technology is the lack of established tools to rely on. Sometimes you have to build your own.
Here are tools that different Udacity Self-Driving Car students built to help them solve problems related to deep learning, computer vision, and the Didi Challenge!
Alex provides great step-by-step analysis of his lane detection and vehicle tracking software. I really like his detailed explanation of the feature-tracking pipeline:
“After experimenting with various features I settled on a combination of HOG (Histogram of Oriented Gradients), spatial information and color channel histograms, all using YCbCr color space. Feature extraction is implemented as a context-preserving class (FeatureExtractor) to allow some pre-calculations for each frame. As some features take a lot of time to compute (looking at you, HOG), we only do that once for entire image and then return regions of it.”
Jonathan built a really cool independent project to estimate vehicle speed from camera images. I really enjoyed his explanation of using optical flow for velocity:
“The Farneback method computes the Dense optical flow. That means it computes the optical flow from each pixel point in the current image to each pixel point in the next image.”
Galen is particularly interested in how to deploy neural networks in industry. To that end, he ran an experiment to see how well and how quickly various neural networks converged on classifying a training set:
“These networks (especially ResNet50 in this case) required extremely little training time and were relatively easy to implement. Once there is a proof of concept, it is a lot easier to write an optimized network that suits your needs (and maybe mimics the network you transfer learned from) than it is to both write and train from scratch.”
One of the knocks on neural networks is that they’re black boxes. Figuring out what drives their decisions is hard. Param built a tool to help visualize the internals of his network:
“On the right we have our Udacity Simulator running. On the left is my little React app that is visualizing all the outputs of the convolutional layers in my neural network.”
Cherkeng is keeping a diary of his work on the Didi Challenge!
“During development, visualization is very important. It helps to ensure that the code implementation and mathematical formulation are correct. I first covert a rectangular region of lidar 3d point cloud into a multi-channel top view image. I use the kitti dataset for my initial development”
When Udacity launched the Self-Driving Car Program last summer, Otto, which has since become Uber Advanced Technologies Group, was there from the beginning.
They only had 100 people in a semi-deserted warehouse, and not a lot of engineering talent to spare. But they have always been so enthusiastic to spend time teaching students about self-driving cars.
Drew Gray, in particular, is a wonder of an engineer. Whether we need help with deep learning, or control, or tying computer vision to trajectory planning, Drew knows everything and is always excited to help Udacity students.
There’s a wrinkle at the end of one of our Computer Vision projects where we ask students to use lane lines to calculate the radius of curvature of the road. This seems like a weird thing to do, but it ultimately ties into trajectory planning. We only made the leap because Drew pointed it out to us.
Uber has received a lot of criticismrecently, and maybe deservedly so. I don’t have much insight into that and you can read better analyses elsewhere.
But I know the folks at Uber ATG, and they have been great partners and great advocates for students who want to get jobs working on autonomous vehicles. And I am grateful for that.
Now would be a good time reiterate something I haven’t posted in a while, which is that this is a personal blog. I have an agenda, which is pretty transparent in my Medium bio. But I don’t speak for Udacity, or CarND students, and certainly not for Udacity partners.
These topics are taught by expert engineers from Mercedes-Benz and Uber ATG, who teach the skills they actually use on the job. These are also the skills students need to know get jobs working on autonomous vehicles.
Although there’s a lot of math and programming involved, there are also some good stories. Watch this video of Dominik and Andrei, from the Mercedes-Benz Sensor Fusion Team, talking with Sebastian about how the unscented Kalman filter got its name.
And if you’d like to understand what the unscented Kalman filter is, here’s Dominik (this one involves more math):
If you want to explore different areas of computer vision, you should check out these awesome posts by Udacity students on different ways to use OpenCV to find lane lines.
Upon completing all of his Term 1 projects, Arnaldo wrote a high-level overview of all of the projects, and reflected on what he learned:
“It had a very practical focus: theory enough to understand the core concepts, and then, the practical application. It is a reason why it requires a lot of background. It is not a course on basic python, or basic neural networks, but how to apply it in real cases.”
Sujay has an incredibly thorough analysis of his computer vision pipeline for lane-finding, including a great debugging tool:
“This project involves fine tuning of lot of parameters like color thresholding, gradient thresholding values to obtain the best lane detection. This can be trickier if the pipeline fails for few video frames. To efficiently debug this I had to build a frame that captures multiple stages of the pipeline, like the color transformation, gradient thresholding, line fitting on present and averaged past frames.”
Andrew has posts fun images from his lane-finding pipeline, but what really caught my eye was his analysis of Udacity’s Career Services:
“The SDC Engineer course emphasises job readiness and the Udacity team provides an excellent Careers Service built right in. We were asked to search for an advertised job that interested us, provide a resume tailored to either Entry Level, Prior Experience or Career Change and the associated Cover Letter. For a ‘career changer’ like me, I was surprised by the amount of self-reflection this caused.”
Morgane is completing CarND while on a nine-month world tour, starting in Ecuador!
“Our next stop was Cuenca, Ecuador, where my fiance’s immediate family lives. I had an amazing time there, visiting the city and Cajas National Park. The only issue is that people did not understand why I was spending so much time on my laptop! They expected me to be free all the time since I was on holidays. I had set up a routine where I’d work for a few hours in the morning while waiting for people to get up and at night. After explaining to them what my goal was and showing them what I was doing, they were definitely more understanding. They got particularly interested when I showed them how I had trained a neural network to drive a car in a simulator, and how I used computer vision and machine learning to recognize lanes and other vehicles on the road.”
Liam took the introductory Udacity Lane-Finding Project and optimized it:
“I created a buffer to store the slope and y-intercept values for each line detected in the last N frames. The actual line drawn on the current frame is simply the average slope/intercept of all these lines. By continuously pushing the latest detected line onto this buffer and simultaneously dropping the oldest line, I can calculate a rolling mean of the lines over time, or what I call “temporal smoothing”.”
The very first partner to sign up was NVIDIA. The NVIDIA team is super-excited about the Udacity Nanodegree Program and is actively interviewing students in the program, even before they graduate.
If you’d like to learn more about how NVIDIA drives autonomous vehicle technology, watch the video we made with them:
Here are some great explanatory posts from Udacity Self-Driving Car students about Kalman filters, computer vision, and how to build a mini autonomous vehicle.
Vivek has transposed some of the notes from his Advanced Controls course at SUNY-Stony Brook. These notes are great for understanding the intuition of how Kalman filters reduce uncertainty:
“This process of combining system dynamics with state measurements is the underlying principle of Kalman filters. Kalman filters provide good estimation properties and are optimal in the special case when the process and measurement follow a Gaussian distributions.”
I like the intuitive explanation that Arnaldo provides for the histogram of oriented gradients (HOG) algorithm:
“The HOG extractor is the heart of the method described here. It is a way to extract meaningful features of a image. It captures the “general aspect” of cars, not the “specific details” of it. It is the same as we, humans, do: in a first glance, we locate the car, not the make, the plate, the wheel, or other small detail.
HOG stands for “Histogram of Oriented Gradients”. Basically, it divides an image in several pieces. For each piece, it calculates the gradient of variation in a given number of orientations.”
I love the mini autonomous vehicle that Shrek built, and especially I love that he trained a deep neural network so that the car react to traffic signs!
“driver.py — is the heart of the project. It includes the image processing pipeline that identifies the traffic sign from the camera by using the previously trained DNN and then send appropriate driving control signals to the car.”
Alena went beyond the minimum requirements for the Advanced Lane Finding Project, and she got her computer vision pipeline to work on the challenge video:
“When the first line was identified successfully, I use line equation with margin as an area to search for the next line. No need to do Sliding Window Search again.”
Although the headline is about Uber and Google, the article is really about current Uber executive, and former Google executive, Anthony Levandowski.
The article is a kind of mini-autobiography of Levandowski, with a particular emphasis on the latest stages in his career — Google (now Waymo), then Otto, then Uber Advanced Technologies Group.
The article doesn’t take a strong position on the merits of Google’s lawsuit against Uber and Levandowski. It’s more useful as background information on some of the most important individuals and companies in the industry.
Before I drop in some quotes, I should mention that I have a number of connections to people who appear in the article. My boss, Sebastian Thrun, launched the Google Self-Driving Car program. The article doesn’t quote him, but he does appear in a few cameos. Additionally, Otto, and now Uber ATG, has been a terrific partner of the Udacity Self-Driving Car Engineer Nanodegree Program. I’ve met Levandowski briefly, and he seems like good people.
Here are some of the quotes that stuck with me:
“At 16 he started a web design firm that a former colleague says made him a millionaire by the end of high school.”
“Ghostrider, his self-balancing, self-driving motorcycle, was the only two-wheel vehicle in the contest [the 2004 DARPA Challenge].”
“Anthony is a rogue force of nature,” says a former Google self-driving car executive. “Each phase of his Google career he had a separate company doing exactly the same work.” According to two former Google employees, founders Page and Sergey Brin tolerated Levandowski’s freelancing because they saw it as the fastest way to make progress. Google’s car team embraced Levandowski’s nature, too. The attitude, says a former colleague, was “he’s an asshole, but he’s our asshole.”
David Silver 