Last week my colleague Yousuf and I spoke at the Open Source Software for Decision Making Conference at Stanford.

It was a lot of fun! Thanks to Mykel Kochenderfer and Tim Wheeler for inviting us.

Yousuf and I spoke about building the Udacity open source self-driving car. If you’re interested in what Udacity and our students have done, check it out:

You can find all the presentations, including some pretty impressive academic work, at the conference website.

In a few weeks, I’ll be speaking at Car HMI USA, so please say hi if you’re there.

HMI stands for Human-Machine Interaction, and while I’m at the conference, I’m really excited to hear from UX and HMI engineers about what the future holds for riders of autonomous vehicles.

The Motley Fool predicts that self-driving cars will be great for Netflix and terrible for radio companies, which seems likely, but not particularly creative.

If we spend close to an hour per day in a self-driving car, how will we use that?

Maybe we’ll use it like we use our leisure time: 55% watching TV, 14% socializing, and 8% gaming.

I like to think we can do better. We could use self-driving cars to spend more time with our families — maybe we’ll drag our kids to work with us and have the self-driving car take them home. Maybe we’ll use that time to do housework like paying the bills or online grocery shopping.

Anything but more TV.

From OnMilwaukee.com comes this important April 1st story about the dangers of autonomous snowplowing:

With the 2016–17 winter nearly behind us, the tally is in: The DPW’s fleet of 200 self-driving snowplows destroyed 3,019 parked cars, killed or injured 29 stray cats, created 17,898 potholes/sinkholes and sent one elderly South Side man to the hospital after burying him in a snow drift.

“Yeah, I guess Milwaukee isn’t quite ready for this technology,” admits Kowolski. “We are considering ‘hiring’ monkeys to drive the plows next season.”

But consider the vendor:

In its pilot program, the City had considered using well-tested self-driving plows built by Google and Tesla, but instead opted to install hardware from RadioShack in its existing trucks.

They tested the equipment in Mountain View, of course.

Our guiding star in developing the Udacity Self-Driving Car Engineer Nanodegree Program is to help students get jobs working on autonomous vehicles.

To that end, we’ve built hiring partnerships with some of the most exciting employers in the world of autonomous vehicles.

Our newest hiring partners include giants like Fiat-Chrysler and Lockheed Martin, critical suppliers like Delphi and Velodyne and Dataspeed, as well as exciting startups like Renovo.

We work directly with recruiters at these companies to identify open positions that Udacity students might be interested in. Then we announce those positions in the Udacity Career Resource Center, and encourage students to apply.

Once students apply, we connect students to employers and guide students through the interview process.

Udacity has been focused on careers for several years, but this level of support is new to the Self-Driving Car Program, and we’re really excited about it. As Sebastian said recently, you can’t talk about education today without talking about jobs.

Ford just announced a $200 MM investment in a transforming its Flat Rock, Michigan, assembly plant into a data center.

There is an angle here that ties into the Michigan vs. Silicon Valley competition for autonomous vehicle development, but that’s not what interests me.

What interests me is what this move says about Big Data in automotive applications. Thus far, most autonomous vehicle development work has proceeded with relatively small amounts of data, certainly compared the amount of data that companies like Google deal with.

Ford’s investment this new Flat Rock data center portends a future in which autonomous vehicle teams need to know about Hadoop and Spark, in addition to deep learning and robotics.

I’ll be speaking at two conferences this week! So much talking.

Today I’ll be talking at 4pm at the Global Data Science Conference about “How to Become a Self-Driving Car Engineer”.

This is kind of last-minute (sorry!), but I have some free passes to give away for that conference, so send me an email (david.silver@udacity.com) if you want one.

On Thursday, at 11:30am, I’ll be speaking at the Open-Source Software for Decision-Making Conference at Stanford University, on the Udacity Open-Source Self-Driving Car.

That conference is free to attend.

If you see me, say hello!

One of the big challenges with working on cutting-edge technology is the lack of established tools to rely on. Sometimes you have to build your own.

Here are tools that different Udacity Self-Driving Car students built to help them solve problems related to deep learning, computer vision, and the Didi Challenge!

Detecting road features

Alex Staravoitau

Alex provides great step-by-step analysis of his lane detection and vehicle tracking software. I really like his detailed explanation of the feature-tracking pipeline:

“After experimenting with various features I settled on a combination of HOG (Histogram of Oriented Gradients), spatial information and color channel histograms, all using YCbCr color space. Feature extraction is implemented as a context-preserving class (FeatureExtractor) to allow some pre-calculations for each frame. As some features take a lot of time to compute (looking at you, HOG), we only do that once for entire image and then return regions of it.”

Autonomous Vehicle Speed Estimation from dashboard cam

Jonathan Mitchell

Jonathan built a really cool independent project to estimate vehicle speed from camera images. I really enjoyed his explanation of using optical flow for velocity:

“The Farneback method computes the Dense optical flow. That means it computes the optical flow from each pixel point in the current image to each pixel point in the next image.”

Transfer Learning in Keras

Galen Ballew

Galen is particularly interested in how to deploy neural networks in industry. To that end, he ran an experiment to see how well and how quickly various neural networks converged on classifying a training set:

“These networks (especially ResNet50 in this case) required extremely little training time and were relatively easy to implement. Once there is a proof of concept, it is a lot easier to write an optimized network that suits your needs (and maybe mimics the network you transfer learned from) than it is to both write and train from scratch.”

Attempting to Visualize a Convolutional Neural Network in Realtime

Param Aggarwal

One of the knocks on neural networks is that they’re black boxes. Figuring out what drives their decisions is hard. Param built a tool to help visualize the internals of his network:

“On the right we have our Udacity Simulator running. On the left is my little React app that is visualizing all the outputs of the convolutional layers in my neural network.”

part.1: Didi Udacity Challenge 2017 — Car and pedestrian Detection using Lidar and RGB

Cherkeng Heng

Cherkeng is keeping a diary of his work on the Didi Challenge!

“During development, visualization is very important. It helps to ensure that the code implementation and mathematical formulation are correct. I first covert a rectangular region of lidar 3d point cloud into a multi-channel top view image. I use the kitti dataset for my initial development”

I am a big fan of the team at Uber ATG, and especially of our friends Drew Gray and Lior Ron.

When Udacity launched the Self-Driving Car Program last summer, Otto, which has since become Uber Advanced Technologies Group, was there from the beginning.

They only had 100 people in a semi-deserted warehouse, and not a lot of engineering talent to spare. But they have always been so enthusiastic to spend time teaching students about self-driving cars.

Drew Gray, in particular, is a wonder of an engineer. Whether we need help with deep learning, or control, or tying computer vision to trajectory planning, Drew knows everything and is always excited to help Udacity students.

There’s a wrinkle at the end of one of our Computer Vision projects where we ask students to use lane lines to calculate the radius of curvature of the road. This seems like a weird thing to do, but it ultimately ties into trajectory planning. We only made the leap because Drew pointed it out to us.

Uber has received a lot of criticism recently, and maybe deservedly so. I don’t have much insight into that and you can read better analyses elsewhere.

But I know the folks at Uber ATG, and they have been great partners and great advocates for students who want to get jobs working on autonomous vehicles. And I am grateful for that.

---

Now would be a good time reiterate something I haven’t posted in a while, which is that this is a personal blog. I have an agenda, which is pretty transparent in my Medium bio. But I don’t speak for Udacity, or CarND students, and certainly not for Udacity partners.

Term 2 of the Udacity Self-Driving Car Engineer Nanodegree Program is hardcore robotics — sensor fusion, localization, and control.

These topics are taught by expert engineers from Mercedes-Benz and Uber ATG, who teach the skills they actually use on the job. These are also the skills students need to know get jobs working on autonomous vehicles.

Although there’s a lot of math and programming involved, there are also some good stories. Watch this video of Dominik and Andrei, from the Mercedes-Benz Sensor Fusion Team, talking with Sebastian about how the unscented Kalman filter got its name.

And if you’d like to understand what the unscented Kalman filter is, here’s Dominik (this one involves more math):