3 Approaches to Vehicle Detection and Tracking

Three Udacity students each took different approaches to vehicle detection and tracking — some using deep learning and others using standard computer vision. Here’s what they learned!

Vehicle Detection and Tracking

Ivan Kazakov

Ivan has a terrific writeup of how to use deep learning for vehicle detection. He builds a model based on Faster-RCNN, but smaller and faster.

“The main idea is that since there is a binary classification problem (vehicle/non-vehicle), a model could be constructed in such a way that it would have an input size of a small training sample (e.g., 64×64) and a single-feature convolutional layer of 1×1 at the top, which output could be used as a probability value for classification.”

Udacity Self Driving Car Engineering Project 5 — Vehicle Detection

Martijn de Boer

https://www.youtube.com/watch?v=7h1iv-9sqys

Martijn uses a HOG and SVM approach to build a vehicle detection pipeline. He encountered some issues with noise and finds a creative solution.

“I was advised do try Hard Negative Mining to train my model more accurate, so I captured multiple images of the shadows / threes and added them to the non car image dataset. (to classify them among the non-car classes instead of the car classes)”

Automatic Vehicle Detection for Self Driving Cars

Priya Dwivedi

Priya uses a HOG and SVM approach to vehicle detection. By combining those with a threshold over time, she achieves great performance. She discusses some of the tradeoffs, however.

“Firstly, I am not sure this model would perform well when it is a heavy traffic situations when there are multiple vehicles. You need something with near perfect accuracy to avoid bumping into other cars or to ensure there are no crashes on a crossing. More importantly, the model was slow to run. It took 6–7 minutes to process 1 minute of video. I am not sure this model would work in a real life situation with cars and pedestrians on the road.”

The Apple Car (System?)

One of the big open secrets in the autonomous vehicle world is Apple’s development of a car. Apple has refused to publicly acknowledge this, however, to the point that engineers widely believed to be working on the Apple car have just removed their LinkedIn profiles.

Apple CEO Tim Cook just recently opened up about this effort, a smidge, to Bloomberg.

While maintaining a determined poker face about exactly what they’re building (is it a car? an automotive operating system?), Cook talked about the convergence of autonomy, electrification, and ride-sharing.

In a very short discussion, he seems to emphasize two points. One is the importance of electrification, which perhaps points to Apple building a physical product. The other is the application of autonomy beyond cars. Maybe Apple drones are next.

Literature Review: Fully Convolutional Networks

Here’s what I pulled out of “Fully Convolutional Networks for Semantic Segmentation”, by Long, Shelhamer, and Darrell, all at UC Berkeley. This is a pretty important research result for semantic segmentation, which we’ll be covering in the elective Advanced Deep Learning Module in the Udacity Self-Driving Car Program.

Segmentation

The ultimate goal of FCNs is to produce “semantic segmentation”. This is an output that is the same size as the original input image, and roughly resembles the original input image, but in which each pixel in the image is colored one of C colors, where C is the number of classes we are segmenting.

For a road image, this could be as simple as C=2 (“road”, or “not road”). Or C could capture a much richer class set.

An example of semantic segmentation from the KITTI dataset.

Fully Convolutional

The basic idea behind a fully convolutional network is that it is “fully convolutional”, that is, all of its layers are convolutional layers.

FCNs don’t have any of the fully-connected layers at the end, which are typically use for classification. Instead, FCNs use convolutional layers to classify each pixel in the image.

So the final output layer will be the same height and width as the input image, but the number of channels will be equal to the number of classes. If we’re classifying each pixel as one of fifteen different classes, then the final output layer will be height x width x 15 classes.

Using a softmax probability function, we can find the most likely class for each pixel.

Learnable Upsampling

A logistical hurdle to overcome in FCNs is that the intermediate layers typically get smaller and smaller (although often deeper), as striding and pooling reduce the height and width dimensions of the tensors.

FCNs use “deconvolutions”, or essentially backwards convolutions, to upsample the intermediate tensors so that they match the width and height of the original input image.

Justin Johnson has a pretty good visual explanation of deconvolutions (start at slide 46 here).

Because backward convolution layers are just convolutions, turned around, their weights are learnable, just like normal convolutional layers.

Smart.

Combining Layers

The authors had success converting canonical networks like AlexNet, VGG, and GoogLeNet into FCNs by replacing their final layers. But there was a consistent problem, which was that upsampling from the final convolutional tensor seemed to be inaccurate. Too much spatial information had been lost by all the downsampling in the network.

So they combined upsampling from that final intermediate tensor with upsampling from earlier tensors, to get more precise spatial information.

Pretty neat paper.

Washington DC DIY Robocar Meetup

Thanks very much to Juan and Antonio and Mapbox and the Washington DC DIY Robocars Meetup, who hosted me for a short presentation about the Udacity Self-Driving Car Nanodegree Program, followed by a great Q&A session.

We covered everything from deep learning, to the SAE automation levels, to safety and security, to public policy. There were lots of great questions and it was lots of fun.

Juan and Antonio rigged up a lightweight video recording from a laptop webcam, and I think it came out surprisingly well. Feel free to watch below.

Lyft’s Autonomous Ridesharing Platform

nuTonomy is partnering with Lyft to launch self-driving cars in Boston this year.

While nuTonomy has been targeting self-driving cars in Boston for a while, this is great news for Lyft. Lyft continues to expand its platform as a provider of ridesharing infrastructure, while letting other companies figure out the actual autonomous technology.

Lyft turned its much-smaller-than-Uber size to its advantage here, by credibly committing not to develop autonomous vehicles. That presumably makes it a more attractive partner than Uber, which is developing its own self-driving technology and thus might have conflicts of interest.

I am on-record as a vocal supporter of Uber ATG, whose staff have been terrific partners for the Udacity Self-Driving Car Nanodegree Program. But it also seems likely that all of the negative news coming out of Uber this year might be causing other companies to second-guess partnerships or vendor-supplier relationships with Uber. Of course, that redounds to Lyft’s benefit.

Lyft, through a combination of using a presumed weakness to their advantage, and through avoiding unforced errors, is having a pretty great 2017.

How to Guides from Udacity Self-Driving Car Students

Here are some great how-to guides from Udacity students! Everything from how to find a job to how to build a self-driving (minature) car 🙂

Becoming a Self-Driving Car & Machine Learning Engineer

George Sung

George landed a job working on deep learning with BMW’s autonomous vehicle team in Silicon Valley! His stats on the hiring funnel are instructive for anybody interviewing in software, and especially in this industry.

“I had 9 interviews out of my ~90 job applications, i.e. around 10% of applications lead to interviews. In my mind this was a pretty good conversion rate. Out of those 9 interviews, 4 of them lead to final-round interviews: 2 final-round interviews for full-time jobs, 2 final-round interviews for internships. I did well on those 4 interviews as they all lead to offers.”

How I Landed My Dream Job Working On Self-driving Cars

Galen Ballew

Galen got a job working on autonomous vehicles at HERE’s Boulder, Colorado, office! It’s also a great example of how being flexible about roles (Galen is starting on the DevOps team) can help you get a foot in the door with autonomous vehicle teams.

“Mathematics is a wonderful thing, but it’s not very career specific. Just a few months after graduating, I made two very important decisions: to enroll at Metis and to enroll in the Udacity Self-driving Car Engineer Nanodegree (SDCEND). Both of these were instrumental in my career path, but the Udacity SDCEND was critical.”

Ubuntu + Deep Learning Software Installation Guide

Nick Condo

In the Udacity Self-Driving Car Nanodegree Program, we provide an AWS AMI for utilizing NVIDIA GPUs for accelerating deep learning. We don’t, however, explain how to set up this software on your own machine. Probably we should do that. In the meantime, Nick has this terrific guide.

“There are a number of good installation guides out there — particularly this one from floydhub that much of this is based on — but I found myself having to dig through many different resources to get everything installed properly. The goal of this article is to consolidate all the necessary resources into one place.”

How I use Docker for Robotics Development

Jari Safi

“ros skillz pay Jari’s billz”, and here he walks through how to get ROS set up using the Docker virtual environment.

“Image: This is essentially the “installation” of something that you want to run using Docker. An image contains all the data necessary to run containers. Images are hierarchical and a new image that shares information with an older one will not reproduce this information and instead just re-use it (i.e. if you have two Ubuntu based images with different software installed, they will both refer to the same base Ubuntu image rather than copy its contents). This is what people mean when they say that Docker’s filesystem is layered.”

Building Self-Driving RC Car Series #1 — Equipment & Plan

Yazeed Alrubyli

This is the first part of Yazeed’s multi-part series on how to build a deep-learning powered miniature autonomous vehicle. Super cool!

“I decided to build my first self-driving car, I mean RC Car 😅 . I think I already have the knowledge and tools to start crafting my RC’s future.”

Visiting Japan

Last week I was in Japan, meeting with Udacity students and with Japanese automotive companies. It was a lot of fun, and it was exciting to see the work that Japanese automotive companies are putting into autonomous vehicles!

Japan is home to a dozen large automotive manufacturers: Toyota, Honda, Nissan, Subaru, Mazda, and more. Supporting these manufacturers are large and small suppliers, providing Japan the third-largest automotive industry in the world.

Japan’s automotive market is more dispersed than America’s, both organizationally and geographically. Whereas the US automotive industry is centered around Detroit, the Japanese automotive industry is spread all over the country. This gives the Japanese economy a little bit of a Detroit-like feel; not everybody works in the automotive industry, but a lot of people do.

Localization (in the language sense, not in the lidar sense) is a big challenge for bringing the Udacity Self-Driving Car Nanodegree Program to Japan. English is not widely spoken in the country, but it seems to be more prevalent among software engineers, who need to at least read English to participate in cutting-edge projects and research. So in that sense, Self-Driving Car has an easier time than, say, Udacity’s Introduction to Programming Nanodegree Program.

One thing that really struck me in meeting with Udacity students in Japan is how important the Udacity student network can be. We hosted about 30 Self-Driving Car students in Tokyo, some of whom already worked in the automotive industry and some of whom were trying to break into that field. The students in the field were eager to connect with newcomers, particularly in a relatively small community of Udacity students.

That’s been one of our goals for the program since the beginning, that as Udacity students get jobs working on autonomous vehicles, they’ll want to pull in other Udacity students. It was fun to see that in operation in Tokyo.

Link Roundup

I was traveling last week (more on that soon) and fell way behind on autonomous vehicle news and on my own posts.

Here are some things I missed.

Keras is broadening the deep learning frameworks that it supports. This is actually slightly old news, but pointed out to me recently. We use Keras in the Udacity Self-Driving Car Nanodegree Program.

HERE demos its next generation mapping vehicle. If you’re particularly interested in localization, the vehicle has a fancy LIDAR and DGPS setup.

Waymo is building self-driving trucks. The Google-Uber competition continues.

Ford plays catch up on self-driving car technology. No, wait, another article says Ford is ahead! The truth is Ford itself doesn’t really know for sure, because none of the car companies are releasing metrics in this area. The only group that even has a clue about this, interestingly enough, are the automotive suppliers, since they see what everyone is doing.

Uber fires Anthony Levandowski. In hindsight, this seems inevitable. Although I dislike the way the judge in the Waymo lawsuit basically ran roughshod over Levandowski’s fifth amendment rights.

Yandex is working on a self-driving taxi service. Of course they are.