Udacity Students on Lane Lines, Curvature, and Cutting-Edge Network Architectures

Here is a terrific collection of blog posts from Udacity Self-Driving Car students.

They cover the waterfront — from debugging computer vision algorithms, to detecting radius of curvature on the road, to using Faster-RCNN, YOLO, and other cutting edge network architectures.

Bugger! Detecting Lane Lines

Jessica Yung

Jessica has a fun post analyzing some of the bugs she had to fix during her first project — Finding Lane Lines. Click through to see why the lines above are rotated 90 degrees!

Here I want to share what I did to investigate the bug. I printed the coordinates of the points my algorithm used to extrapolate the linens and plotted them separately. This was to check whether the problem was in the points or in the way the points were extrapolated into a line. E.g.: did I just throw away many of the useful points because they didn’t pass my test?

Towards a real-time vehicle detection: SSD multibox approach

Vivek Yadav

Vivek has gone above and beyond the minimum requirements in almost every area of the Self-Driving Car program, including helping students on the forums and in our Slack community, and in terms of his project submissions. He really outdid himself with this post, which compares using several different cutting-edge neural network architectures for vehicle detection.

The final architecture, and the title of this post is called the Single Shot Multibox Detector (SSD). SSD addresses the low resolution issue in YOLO by making predictions based on feature maps taken at different stages of the convolutional network, it is as accurate and in some cases more accurate than the state-of-the-art faster-RCNN. As the layers closer to the image have higher resolution. To keep the number of bounding boxes manageable an atrous convolutional layer was proposed. Atrous convolutional layers are inspired by “algorithme a trous” in wavelet signal processing, where blank filters are applied to subsample the data for faster calculations.

CNN Model Comparison in Udacity’s Driving Simulator

Chris Gundling

This is a fantastic post by Chris comparing and contrasting the performance of two different CNN architectures for end-to-end driving. Chris looked at an average-sized CNN architecture proposed by NVIDIA, and a huge, VGG-style architecture he built himself.

I experimented with various data pre-processing techniques, 7 different data augmentation methods and varied the dropout of each of the two models that I tested. In the end I found that while the VGG style model drove slightly smoother, it took more hyperparameter tuning to get there. NVIDIA’s architecture did a better job generalizing to the test Track (Track2) with less effort.

Hello Lane Lines

Josh Pierro

Josh Pierro took his lane-finding algorithm for a spin in his 1986 Mercedes-Benz!

From the moment I started project 1 (p1 — finding lane lines on the road) all I wanted to do was hook up a web cam and pump a live stream through my pipeline as I was driving down the road.

So, I gave it a shot and it was actually quite easy. With a PyCharm port of P1, cv2 (open cv) and a cheap web cam I was able to pipe a live stream through my model!

Udacity SDCND : Advanced Lane Finding Using OpenCV

Paul Heraty

Paul does a great job laying out his computer vision pipeline for detecting lane lines on a curving road. He even compares his findings for radius of curvature to US Department of Transportation standards!

Overall, my pipeline looks like the following:

Let’s look at each stage in some detail

CarND Students on Preparation, Generalization, and Hacking Cars

Here are five great posts from students in Udacity’s Self-Driving Car Engineer Nanodegree Program, dealing with generalizing machine learning models and hacking cars!

SDC

Daniel Stang

Daniel has devoted a section of his blog to the Self-Driving Car projects, including applying his lane-line finder to video he took himself!

The first project for the Udacity Self-Driving Car Nanodegree was to create a software pipeline capable of detecting the lane lines in video feed. The project was done using python with the bulk of work being performed using the OpenCV library. The video to the side shows the software pipeline I developed in action using video footage I took myself.

Traffic Sign Classifier: Normalising Data

Jessica Yung

Jessica’s post discusses the need to normalize image data before feeding it into a neural network, including a bonus explainer on the differences between normalization and standardization.

The same range of values for each of the inputs to the neural network can guarantee stable convergence of weights and biases. (Source: Mahmoud Omid on ResearchGate)

Suppose we have one image that’s really dark (almost all black) and one that’s really bright (almost all white). Our model has to address both cases using the same parameters (weights and biases). It’s hard for our model to be accurate and generalise well if it has to tackle both extreme cases.

Hardware, tools, and cardboard mockups.

Dylan Brown

Dylan is a student in both the Georgia Tech Online Master’s in Computer Science Program (run by Udacity) and also in CarND. He’s also turning his own Subaru into a self-driving car! (Note: We do not recommend this.)

Below I’ve put together a list of purchases needed for this project. There will definitely be more items coming soon, at least a decent power supply or UPS. Thankfully, this list covers all the big-ticket items.

Jetson TX1 Developement Kit (with .edu discount) NVIDIA $299
ZED Stereo Camera with 6-axis pose Stereolabs $449 CAN(-FD) to USB interface PEAK-System $299
Touch display, 10.1” Toguard $139
Wireless keyboard K400 Logitech $30
Total $1216

Self-Driving Car Engineer Diary — 1

Andrew Wilkie

Andrew has a running blog of his experiences in CarND, including his preparation.

I REALLY want a deep understanding of the material so followed Gilad Gressel’s recommendation (course mentor): Essence Of Linear Algebra (for linear classifiers which is step 1 towards CNNs), Gradients & Derivatives (for back propagation understanding) and CS231n: Convolutional Neural Networks for Visual Recognition lectures (for full Neural Networks and Convolutional Deep Neural Networks understanding).

Comparing model performance: Including Max Pooling and Dropout Layers

Jessica Yung

Another post by Jessica Yung! This time, she runs experiments on her model by training with and without different layers, to see which version of the model generalizes best.

Means of training accuracy - validation accuracy in epochs 80-100 (lower gap first):

Pooling and dropout (0.0009)
Dropout but no pooling (0.0061)
Pooling but no dropout (0.0069)
No pooling or dropout (0.0094)

Udacity Self-Driving Car Students on Neural Networks and Docker

I’ve spent the past several days highlighting posts from Udacity Self-Driving Car students in areas related to computer vision, neural networks, careers, and more.

I’ll be doing more of that today, and for the next several days. Then we’ll return to our regularly scheduled programming, although I expect to periodically include more student posts each week.

Teaching a car to mimic your driving behaviour

Subodh Malgonde

Subodh summarizes his approach to building an end-to-end network for driving. He was even able to get his network to drive the car on a track it had never seen before!

The true test of a neural network is how well it performs on unseen data i.e. data not used in training. To evaluate this the simulator had a second track which was very different from the one used for training. It was darker, had slopes while the first track was more or less flat, had sharper turns and more right turns as compared to the first track. The network had never seen data from this track. However some of these differences were accounted for in the network due to image augmentation techniques described above.

Udacity Self-Driving Car Nanodegree Project 2 — Traffic Sign Classifier

Jeremy Shannon

Jeremy exploited some pretty cool data augmentation techniques to help his traffic sign classifier generalize better. As a consequence, his model was able to correctly classify many different images that he found on the Internet.

Wait… a hundred percent? Seriously? I don’t know how this model can be that sure of its prediction. Granted, I gave it some real softballs, there. But still, I would have expected them all to be in the 75–90% range. Whatever — I’ll take it! Good job, model!

Self-Driving Car Engineer Diary — 3

Andrew Wilkie

Andrew has a running journal documenting his journey through CarND. In this post, he provides great notes about the deep learning content the course provides in the run-up to the Traffic Sign Classifier Project.

Hi. Dived into part 1 of Deep Learning during the last 2 weeks. The course covered a lot of material over 7 lessons, defined with succinct, well commented, executable code … just how I like it!

Neural Network Tuning with TensorFlow

Param Aggarwal

Param built a neural network to classify traffic signs and in the process, spent a lot of time thinking about how to size the network, preprocess the images, and tune hyperparameters.

Or, how I struggled with Project 2: Traffic Sign Classfication as part of the Self-driving Car Engineering Nanodegree from Udacity. In the end, the struggle taught me more than the results I got.

Docker Image for the Udacity Self-Driving Car Nanodegree (with UI)

Youcef Rahal

One of the big challenges when we launched the program was helping to get everybody’s software packages working. We finally published the Udacity CarND Starter Kit, but we also rely on students like Youcef to help their fellow students by guiding them on setup.

The image I created is at https://hub.docker.com/r/yrahal/udacity-carnd, and is built based on the install steps required for the Lane Lines Finding Project (first project of the Nanodegree). The Dockerfile I used to create the container is inspired from the Anaconda3 Dockerfile and on instructions on how to run a VNC server inside a container.

More Udacity Self-Driving Car Students, In Their Own Words

Yesterday I shared 5 amazing blog posts by students in the Udacity Self-Driving Car Nanodegree Program.

Here are 5 more!

DeepTrafficJS Solution

Anton Pechenko

Anton is a student in the Udacity Self-Driving Car Program, and also in MIT’s class on Deep Learning for Self-Driving Cars. He is currently third in MIT’s Deep Traffic competition, and he reviews his deep neural network in this video. Pay attention to his choice of activation functions!

My first self-driving car

Bill Zito

Bill has a really nice walkthrough of some of the key lessons he learned while completing the final project in the Deep Learning Module. This project, Behavioral Cloning, requires students to drive a car in a simulator, record their driving data, use that data to train a neural network, and then use that network to drive the car.

This challenge is no walk in the park, and that’s part of what makes it really fun. You’re implementing similar code to the code that drives self-driving cars in real life. And you’re required to think through lots of the steps of the process by yourself to do so. If you end up stuck, remember that the experts just figured out this stuff was even possible in the last couple years.

German Traffic Sign Classification Using Deep Learning

Muddassir Ahmed

Muddassir gives a great explanation of what a neural network is, and what a convolutional neural network is — including the history behind them! Then he explains how he implemented the Traffic Sign Classifier Project.

The CNN was inspired by the work of Hubel and Wiesel back in the 1950s and 1960s. In the study, they discovered that the mammalian brain was structured hierarchically and that objects were recognized based on the hierarchical build-up of features from small ones, such as colors, stripes, lines into bigger ones such as patterns and even larger concepts of dog, cat, human e.t.c.

Self-Driving Car Engineer Diary — 2

Andrew Wilkie

We put a lot of effort into making the first week of the program fun and rewarding, so students understand from the very beginning what they might be able to accomplish. Andrew has a great journal of how his first week in the program went.

Andrew Gray popped into our student ‘ama’ Slack channel for a 30 minute Q&A. Students were asking a lot about future job opportunities in this new Self-Driving vehicle industry. Some were concerned with the rapid rate of improvement and that by the time we graduate (Sep/2017 for my Dec/2016 co-hort) that we might have missed the best opportunities. Andrew highlighted the fact many existing companies outside of Self-Driving cars and trucks are actively pursuing combined AI / Robotics strategies and completing this intensive 9 month training readies us for this new industry.

Udacity Self-Driving Car Nanodegree Project 1 — Finding Lane Lines

Jeremy Shannon

Jeremy says some really nice things about the Udacity program in his post, and then outlines the steps he took to complete the Finding Lane Lines Project. This was my own very first project as an autonomous vehicle engineering student, so it is near and dear to my heart.

This is the best online course (or, should I say, collection of courses) I’ve taken so far. Yes, even better than Fire Safety Refresher Training. Really! The quality is top-notch (both video and written/supplemental material), the feedback is amazing, and the community they’ve built around it is incredibly helpful. (I wish that during my undergrad days I’d had an online forum I could go to and find that dozens of other students were having the same problem I was having.) It’s so easy to become completely immersed in the subject material, and I’m so thankful that this program exists. Udacity has really outdone themselves and I can’t possibly heap enough praise on them.

Self-Driving Car Student Posts

One of the most fun things about the Udacity Self-Driving Car Engineer Nanodegree Program is the student community. Thousands of students are active on Slack and in the forums, helping each other complete projects, understand concepts, and get jobs in the autonomous vehicle industry.

Students also write online about their experiences in the program and what they’ve accomplished. Here are five, in their own words.

Behavioral Cloning for Autonomous Vehicles

David Ventimiglia

It often pays to explore your data with relatively few constraints before diving in to build and train the actual model. One may gain insights that help guide you to better models and strategies, and avoid pitfalls and dead-ends.

A transforming reality — Udacity’s Self Driving Car Nano-degree

Vishal Rangras

I was so much into this program that I started talking about it with senior folks at my workplace. They encouraged me, supported me and appreciated my love for the technology and research. They got interested in knowing more about the program. They got stunned about this state-of-the-art program and they wanted to help me pursue my dream of research. They extended a helping hand to me by arranging an online fundraiser to support me.

Studying for the Udacity SDCND, or How I got my law license

Michael Toback

Udacity has a great program, but they refer you to other sources as you go, both inside and outside of Udacity. Hurrying through it won’t help you, because people need time to develop and strengthen the neural pathways that help you really learn something.

Why I enrolled in Udacity Self-Driving Car Classes

Boris Dayma

I’m currently in a very different industry than tech and automotive (I’m actually in the oil & gas industry). However I’ve always tried to apply latest innovations to assist me in my daily activities. This has helped me in automating the “boring tasks” to let me focus on more fun challenges, leading to new creative solutions.

Experiment Using Deep Learning to find Road Lane Lines

Paul Heraty

I modified a neural network that I had used in the SDCND BehavioralCloning lab (5 CNN layers followed by 3 FCNN layers), and added 5 new outputs to it. So now the network looks like 5 CNN layers with 6x 3 FCNN layers. The outputs are generating lane polynomial coefficients for both the left and right lanes, i.e. a*y² + b*y + c where I’m predicting a, b & c for each lane.

“Full Self-Driving” in 6 Months?

One of the most common questions I get from friends and relatives is, “How soon will we get self-driving cars?”

And my answer is that it’s a question of where, not when. Uber already has tightly-geofenced autonomous vehicle trials in Pittsburgh, and nuTonomy and Delphi both have limited trials in Singapore. nuTonomy has announced more trials in Boston. Uber came to San Francisco and then left within a week, due to a dispute with the California DMV.

Over the next year, we’ll see these types of public trials roll out to even more cities.

Elon Musk, though, said something last night that might blow all of that out of the water.

That’s all I’ve got, and maybe Musk is over-promising. But it sounds pretty exciting.

News Roundup

California Is Sunsetting Botts’ Dots: Standardizing lane markings helps self-driving cars. If you don’t know what Botts’ Dots are, click through to learn a little bit about transportation history!

Nissan and Volvo to Test Self-Driving Cars in London This Year: It’s unclear whether these particular programs will be open to the public, but we will see more of these single-city tests this year.

Self-Driving Cars Will Bring Jobs to Disabled People: Disabled people have been staunch advocates of self-driving cars, for obvious reasons, but I never really thought through the economics of this. One estimate is that 2 million disabled Americans will become employable simply by being able to get to work.

Is Amazon Developing a Self-Driving Car?

Sean Everett makes a strident case that Amazon is developing it’s own self-driving vehicles.

I’ve heard rumors about this for a while, including from startups that wouldn’t mind one day being acquired by Amazon.

And it makes all the sense in the world.

But I’m not sure I believe it.

The main reason it’s such an open secret that Apple is working on self-driving cars is that self-driving car experts keep joining Apple to work on some secret project.

I haven’t heard anything like that about Amazon.

So if prominent autonomous vehicle engineers aren’t building Amazon’s self-driving vehicle program, then who is?

Autonomous Vehicle Customer Power

Porter’s Five Forces is one of the classic frameworks that business schools teach for evaluating the attractiveness of a business or industry.. Michael Porter, a professor at Harvard Business School, invented it.

The framework is, at its core, a five-item checklist:

Barriers to Entry
Substitution
Supplier Power
Customer Power
Existing Competition

I’m re-wording Porter’s original definitions to the phrases I think of when I step through the checklist, but that’s the gist of it.

I thought of this checklist during a recent conversation about strategy for Waymo (Google’s Self-Driving Car division).

One school of thought is that Google should position itself as the supplier of software to the automotive industry. That’s a position that’s worked well for Android, and it plays to Google’s strengths as a software company.

An alternative school of thought is that Google should build out its own transportation-as-a-service business, because car manufacturers are too smart to get trapped with Google’s software. They’ve seen how that played out for the mobile handset manufacturers, and they don’t want to get “Samsungized”.

Rather than choose between these two approaches, I want to focus here on the narrow question of whether the car manufacturers are “too smart” to get trapped with Google’s software.

The people I worked with at Ford are very smart, and I imagine the same is true at Toyota and GM and most other manufacturers.

But this is where Porter’s five forces comes in.

As a automotive software vendor, Google is in an industry with low “customer power”. There are a lot of different car manufacturers.

Some of those manufacturers are going to be way behind the curve in terms of developing autonomous vehicle software. And those manufacturers will, rationally, decide that their best bet for catching up with the pack is to partner with Waymo.

So regardless of how smart the car manufacturers are, there will be manufacturers out there who will be interested in using Waymo’s software.

Waymo Debut

This year was the first CES appearance for Waymo, although it has appeared in previous years under the name, “Google Self-Driving Car Program”.

Waymo CEO John Krafcik used the opportunity to shed some light on a perpetual parlor game for self-driving car enthusiasts: “what is Google’s strategy for self-driving cars?”

Google has had self-driving cars scooting around Mountain View for years, but it’s always been unclear what the go-to-market strategy would be, and when it would start rolling.

Krafcik’s statements indicate that Google’s long game may be as a provider of autonomous vehicle hardware and software, rather than as an auto manufacturer or a transportation-as-a-service provider.

Krafcik specifically highlighted Waymo’s gains with its in-house LIDAR sensor technology. Along with adding short- and long-range sensing units for better detection than its previous systems, the company has managed to slash the cost of its LIDAR by more than 90 percent in two years from about $75,000 per vehicle.

…

Krafcik’s language is telling — those “firsts” imply that this is just the start of Waymo’s autonomous partnerships, with many more to come. By focusing on the production of its own hardware along with software, Waymo won’t become a major automaker, at least not yet. Instead, it clearly intends to be the go-to supplier of autonomous driving systems for automakers that don’t want the expense of developing their own.

I’m not convinced the mystery has been solved yet, but it’s an important clue in the Google guessing game.