TensorFlow vs. TF Learn vs. Keras vs. TF-Slim

One module in Udacity’s Self-Driving Car Nanodegree program will cover deep learning, with a focus on automotive applications.

We’ve decided to use the TensorFlow library that Google has built as the main tool for this module.

Caffe, an alternative framework, has lots of great research behind it, but TensorFlow uses Python, and our hope is that this will make learning it a lot easier for students.

Even with TensorFlow, however, we face a choice of which “front-end” framework to use. Should we use straight TensorFlow, or TF Learn, or Keras, or the new TF-Slim library that Google released within TensorFlow.

Right now we’re learning toward TF Learn, almost by default. Straight TensorFlow is really verbose, TF-Slim seems new and under-documented. Keras and TF Learn both seem solid, but the TF Learn syntax seems a little cleaner.

One big drawback to TF Learn, though, is the lack of easily integrated pre-trained models. I spent a while today trying to figure out how to migrate pre-trained AlexNet weights from Caffe to TF Learn.

So far, no one solution is jumping out at me as perfect. Let me know in the comments if you’ve got a suggestion.

More Self-Driving Car News

Ford Will Sell Self-Driving Cars to the Public by 2025

Monday’s news of an employee-only autonomous service is a middle step before the automaker’s first self-driving public implementation in 2021. And President and CEO Mark Fields said such vehicles likely won’t be ready for sale directly to the public until at least the middle of next decade.

George Hotz Will Ship by the End of the Year

Would you pay $999 to give your car self-driving chops?

George Hotz is betting the answer is yes. The 26-year-old iPhone and PlayStation hacker turned entrepreneur is behind Comma.ai, a new Bay Area company that is powered largely by his brains and chutzpah, as well as $3 million in funding fromAndreessen Horowitz.

Google X Employees (get it?) Are Forming a New Company

[Nuro.ai] wouldn’t reveal too many additional details about what exactly they’re doing, but here’s what we know:

The company’s plans involve creating a “level four,” which is geek for an entirely hands-free self-driving car.

But the car is only the first in a line of products Nuro plans to develop. We don’t know what else they’ll create, but it won’t be solely transportation-related.

That’s because Nuro’s team includes engineers with robotics, artificial intelligence and self-driving experience who had a hand in either developing or shipping an unusually wide range of products including Nexus cameras, Google Image search, the Mars Exploration and Curiosity Rovers, Google street view, Google’s self-driving cars and a number of surgical tools.

The company has raised funding, but it won’t say how much and from whom.

Nuro plans to launch its first product — a self-driving car — in two to four years.

Learn to Build Self-Driving Cars with Udacity

Build a self-driving car with Udacity!

I was at TechCrunch Disrupt yesterday, where Udacity’s founder and chairman, Sebastian Thrun, announced the opening of applications for our Self-Driving Car Nanodegree program.

He also announced that students will be able to run their code from the course on a real car.

The biggest company-wide emphasis at Udacity since I joined has been “Only at Udacity”, a focus on launching programs and courses and experiences that only Udacity provides.

A self-driving car program that is available to students everywhere in the world is, on its own, an Only at Udacity program.

Helping students around the world take their code and put it on an actual car takes us to something even beyond that.

So come build a crowd-sourced self-driving car with us. Sign up here!

Tesla Makes Radar a First-Class Citizen

Tesla has announced that Autopilot will increase its reliance on radar, promoting it to first-class status within the sensor suite of Tesla vehicles.

The radar was added to all Tesla vehicles in October 2014 as part of the Autopilot hardware suite, but was only meant to be a supplementary sensor to the primary camera and image processing system.

After careful consideration, we now believe it can be used as a primary control sensor without requiring the camera to confirm visual image recognition.

The blog post does not mention the fatal accident back in May that occurred while the car was on Autopilot, although it’s easy to speculate that the Autopilot update may be related to that specific accident.

When the car is approaching an overhead highway road sign positioned on a rise in the road or a bridge where the road dips underneath, this often looks like a collision course. The navigation data and height accuracy of the GPS are not enough to know whether the car will pass under the object or not. By the time the car is close and the road pitch changes, it is too late to brake.

That is basically the same scenario that caused the May accident.

Tesla does relay some interesting information about why they initially relied much more heavily on camera than on radar.

This is a non-trivial and counter-intuitive problem, because of how strange the world looks in radar. Photons of that wavelength travel easily through fog, dust, rain and snow, but anything metallic looks like a mirror. The radar can see people, but they appear partially translucent. Something made of wood or painted plastic, though opaque to a person, is almost as transparent as glass to radar.

The blog post also covers at least one scenario in which Tesla is uploading driving data from its users and using that to teach the fleet to drive better. And that’s notable in and of itself.

So, all around, a blog post worth reading.

Sunday Self-Driving Stories

Volvo Self-Driving Mining Vehicles: I forgot to mention this in my round-up of niche industries for self-driving cars, but Paul Lienert reminded me on Twitter. Volvo has self-driving dump trucks operating in Sweden’s Kristineberg Mine.

Denso Buys Fujitsu Ten: Denso isn’t well-known in the US, but they are Japan’s biggest automotive supplier, and the second-largest in the world, behind Bosch. They purchased a controlling stake in a subsidiary of Fujitsu, specifically the radar group, which is critical for self-driving cars. It’s a move that signals the Japanese auto industry may focusing more on autonomous vehicles.

Layoffs in Apple’s Car Group: Apple’s self-driving car group, allegedly code-named Project Titan, is a little bit like the Delta Force — everybody knows they exist but nobody will confirm it. Recent layoffs in the group were reported by just about every major news outlet, though, so I think it’s safe to say that both the group and the layoffs are real. Reading between the lines, it looks like Apple may be moving away from physical hardware and focusing increasingly on self-driving software.

Udacity at TechCrunch Disrupt

Udacity will have a big presence at TechCrunch Disrupt in San Francisco this coming week, with a big emphasis on our Self-Driving Car Nanodegree program.

I’ll be at the conference all day on Tuesday, so please come say hello if you’re there. I’ll be standing next to the car wrapped in the Udacity logo, with lidars and radars on it 🙂

Also, Sebastian Thrun goes on-stage on Tuesday at 9:25am to speak about self-driving cars and online education, and he’s more interesting than me, so be sure to catch that.

See you there!

Baidu and NVIDIA Partner Up

I have been remiss in not mentioning all the press around Baidu’s newest efforts to build a self-driving car in Silicon Valley.

The latest news is that Baidu signed a partnership with NVIDIA, although it’s not clear to what extent this differs from the vendor-customer relationship that NVIDIA has with many automotive companies.

What is less obvious to me is why Baidu is putting so much effort into building a self-driving car center in Silicon Valley, instead of China or even Detroit.

It makes sense for them to have a presence here, as does almost every automotive company, but it seems like Baidu has gone beyond that and is centering their entire AV effort here.

Does anybody know the story on that?

Niche Industries for Autonomous Vehicles

I’m coming to the end of the terrific list of questions asked by Kyle Jepson.

That’s too bad because it’s been good content for slow news days, but it will be nice to finally clear that email out of my inbox 🙂

Finally, I’m a huge fan of Otto. I think long-haul trucking is a job that autonomous vehicles are especially well-suited for because computers don’t get bored. Are there other niche industries like this that you think autonomous vehicles will take over before they become commonplace for consumers like me? And, conversely, are there vehicles that you think might never be autonomous? (Construction vehicles come to mind, as do any sort of vehicles that operate in primarily off-road environments.)

Kyle wrote this in early August (sorry for the delayed response!) and we have since learn that Travis Kalanick is also a huge fan. Kyle, I hope you got in early on that one.

More seriously, I think there are a lot of niche industries where autonomous vehicles will make an outsize impact.

Agriculture: John Deere is the world’s largest operator of autonomous vehicles, in particular tractors that can plow fields overnight.

Delivery: Cutting edge autonomous vehicles are expensive, because of all the sensors and specialized hardware they require. Delivery companies like FedEx and UPS are well-positioned to amortize this cost over many hours of delivery driving.

Logistics: Autonomous forklifts are already deployed to move heavy materials around warehouses faster, cheaper, and safer.

Trucking: Although Otto has been acquired by Uber for presumably non-trucking purposes, the trucking industry is still a huge opportunity for autonomous vehicles. Many trucks are owner-operated, and autonomous driving means more hours on the road, more cargo carried, and more money for driver.

Interestingly, the construction opportunity that Kyle proposes is a terrific opportunity for self-driving vehicles (safety), but could be hard to implement in practice.

The nice thing about warehouses or agricultural fields or highways is that they’re basically constant. Construction zones are constantly in flux and might require more vehicle intelligence than we currently have.

But I’m not an expert on this and if readers know about self-driving construction vehicles, please put them in the comments!

Volvo Forms a Software Company

Volvo and Autoliv are forming a software company. Sweden’s largest car manufacturer and one of its major auto suppliers are joining the movement of auto manufacturers diving headfirst into software:

Chinese-owned Volvo Car Group and auto safety group Autoliv said on Tuesday they would form a joint venture to develop autonomous driving software as automotive firms across the industry race to embrace the emerging technology.

The two Sweden-based companies said in separate statements the new company would have an initial work force of about 200 staff taken from both parent companies, a number that would increase to more than 600 over the medium term.

The joint venture, which is to be headquartered in Gothenburg, Sweden, and had yet to be named, will develop advanced driver assistance systems (ADAS) and autonomous drive (AD) systems for use in Volvo Cars.

Software continues to eat the world.

Self-Driving Cars and the Labor Market

Happy Labor Day!

At least if you live in the United States or Canada, where the first Monday of every September is a public holiday set aside to recognize the contributions of workers. It also marks the end of summer, but that’s neither here nor there.

I thought I’d take a few minutes on Labor Day to riff about how self-driving cars might affect the labor market and our future jobs.

According to the US Bureau of Labor Statistics, about 3.9 million Americans make their living as “motor vehicle operators”. That’s about 3% of the total labor force, which is surprisingly high.

About half of those workers are in trucking, with bus drivers and deliverymen making up most of the rest.

I’m not sure all of those jobs will go away, but eventually (40 years, say) a lot of them will.

So that’s the downside, or at least the largest part of the downside.

What’s the upside, particularly as it concerns the labor market?

I think most of the value will come from expanded opportunity. Getting to and from a job is a big cost, particularly for the poorest Americans. The New York Times goes so far as to assert (based on a Harvard study): “commuting time has emerged as the single strongest factor in the odds of escaping poverty.”

Lowering the cost of transportation, and the time it takes will help millions of Americans escape poverty.

It will also improve the lives of middle-class and affluent Americans, who will have access to a wider array of jobs, thanks to lower-cost transportation.

And companies should find hiring workers to be less difficult, as their talent pool expands.

This is a classic case of diffuse benefits and concentrated costs, and it’s important to remember that for the people bearing the cost, their whole way of life is disappearing. It’s catastrophic.

But, on net, the labor benefits to society should be tremendous.