Virtual Reality and Self-Driving Cars

For years I’ve heard whispers about virtual reality inside of self-driving cars, supposedly incubated as quiet projects inside of secretive companies. Variety just delivered a small scoop in that vein, by digging up a couple of Apple patent applications related to virtual reality inside of self-driving cars.

Variety quotes from one of the patent applications:

“For example, a virtual representation of an author or talk show host may appear to be sitting in the seat next to the passenger; the virtual author may be reading one of their books to the passenger, or the virtual talk show host may be hosting their show from the seat next to the passenger, with their voices provided through the audio system. As another example, the passenger may experience riding on a flatbed truck with a band playing a gig on the flatbed, with the band’s music provide (sic) through the audio system.”

An extra wrinkle in the article is that the name on a couple of these patent applications is Mark Rober. Rober was previously unknown to me, but he has a YouTube channel with 3.5 million subscribers and videos like Lemon Powered Supercar, and World’s LARGEST NERF GUN.

Regardless of the patents’ author, a more prosaic description comes in the first sentence of one of the patent summaries:

“A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers.”

I’ve written in the past about my difficulty doing serious cognitive work in a moving vehicle, even though I’m not driving. There are many ways to address this: improve the quality of roads, improve the quality of motion control through vehicular software, better mechanical suspension systems, possibly flying cars, and of course virtual reality.

Of these options, virtual reality seems less likely to me, because my impression is that virtual reality tends to induce motion sickness even in stationary situations. There is an argument presented in the patent that a moving vehicle in some ways mitigates the motion sickness problem, but my intuition is that this seems unlikely.

I would be excited to be proven wrong, though, and would love to try it out.

Driverless Grocery Delivery

Grocery delivery seems like one of the obvious applications of self-driving vehicles, but I wouldn’t have guessed that Kroger would be out front on it.

“To start out, Nuro will use a fleet of self-driving test vehicles with human safety drivers to make deliveries for Kroger’s grocery stores. Customers can track and interact with the vehicles via a Nuro app or Kroger’s pre-existing online delivery platform. But if Nuro’s human test drivers don’t get out to help you, don’t be mad because in our driverless future, we all need to pitch in and unload our own groceries.”

I think the tipping point with autonomous vehicles will be when boring, non-tech businesses like groceries can put them to work without having develop their own fleet. So far, most self-driving car companies have either been in the mobility space to begin with, or had to dive into the mobility space as part of developing that business line.

If Kroger can utilize self-driving cars as a service, without having to build self-driving cars themselves, that would be a big step forward.

Connected Teslas

Tesla is updating the terms of its in-car Internet service. Existing Teslas will keep free “premium” cellular connectivity indefinitely. New Teslas will receive free “standard” connectivity, and one year of “premium” connectivity, with the option of paying for ongoing premium connectivity.

That is all fine, as far as it goes, and frankly it seems like a nice benefit of owning a Tesla. It’s not too hard to switch my mobile phone into hotspot mode nowadays, but it runs down the phone battery and it’s just nice to hop in the car and have WiFi connectivity, without having to think about it.

What I really wonder, though, when Teslas will start talking to each other. As fas as I know, Teslas are not equipped with DSRC transponders, which is the communications technology that high-end Cadillacs now use to communicate amongst themselves.

There is a lively debate in the connected car community over whether the future of vehicle-to-vehicle communication is peer-to-peer networking via DSRC, or cloud connectivity via the Internet.

The main advantage of cloud connectivity is that it’s easier to bootstrap — cars can begin talking with each other via the Internet, even if they’re not very physically close. The main disadvantage is the cost of cellular data connectivity.

Tesla is already covering the cost of “standard” data connectivity for all its customers, and I hope at some point soon they start to test out how helpful that can be for vehicle-to-vehicle communication.

Small World

Our friends and former Udacity colleagues at Voyage just announced the addition of Drew Gray as their CTO. Drew is one of the key contributors to building the Udacity Self-Driving Car Engineer Nanodegree Program, and has taught both our students and myself so much. Any company would be lucky to have Drew, so this is a huge hire for Voyage.

In just a few years, the self-driving car industry has seen so much movement, and in many ways it still feels like a small community that is poised for tremendous growth.

At Udacity, we are fortunate to be right in the middle of this. We met Drew right after he’d joined Otto from Cruise, and then worked with him at Uber and now Voyage. Our partners at NVIDIA and Mercedes-Benz have hired many of our graduates, and we see them regularly at conferences and industry events.

Udacity students work at every company I can think of the in the autonomous vehicle industry, from manufacturers like BMW and Ford to suppliers like Bosch to ride-sharing companies like Lyft to startups like Parkopedia and Phantom Auto.

It’s a lot of fun to be part of this community and grow with it.

Michigan Central Station

A couple of miles from downtown Detroit, in the Corktown neighborhood, stands Michigan Central Station. At 18 stories, it is a grand building that has been vacant and deteriorating for decades.

Ford Motor Company just announced that they have purchased the building and surrounding land, with plans to turn it into a campus for their autonomous vehicle and mobility teams.

The plans call for all of this work to complete by 2022, so it will be a while coming, but the vision is big and I’m looking forward to seeing it unfold.

Berkeley Deep Drive

A few weeks ago, the University of California-Berkeley released DeepDrive, which appears to be the largest open dataset ever compiled for self-driving cars.

This is super-exciting, because annotated data is one of the major roadblocks to developing self-driving cars.

Prior to DeepDrive, the state-of-the-art dataset was KITTI, from the Karlsruhe Institute of Technology in Germany.

KITTI has been tremendously important for self-driving cars, and DeepDrive will build on that. For comparison, whereas KITTI has 7,500 images annotated with 3D bounding boxes, DeepDrive has 100,000. And while KITTI has 400 images for semantic segmentation, DeepDrive has 10,000.

It’s really astounding what Berkeley has released, and I’m excited to see the models people are able to build with this.

Full Self-Driving Tesla Features

Elon Musk broke the Internet a few days ago with a tweet promising that in August Tesla “will begin to enable full self-driving features.”

I am pretty excited about this but it is also worth noting that this verbiage is vague enough to drive a Tesla semi through.

“Full self-driving features” presumably means something beyond current Tesla Autopilot, but short of “full self-driving”. What are the features that make up full self-driving?

Some ideas include end-to-end routing, even if drivers still have to pay attention to the road. Or even just automatic lane change decision and execution on a highway.

Potentially this could mean that Tesla is enabling drivers to stop paying attention to the road under certain scenarios, although that seems unlikely, given the recent spate of crashes.

It will be exciting to see what, if anything, comes of this. But Elon Musk himself warns us that these tweets are not well-thought out strategy, but rather off-the-cuff remarks:

Interviews in India

Last month I had the pleasure of making my first trip to India for Udacity. As part of the trip, I sat for several interviews with Indian journalists. All of them had great questions about self-driving cars and how that would impact India. Here are some of those articles:

“Infosys Building Self-Driving Golf Carts To Be Future-Ready”. Bloomberg Quint.

“How much does a self-driving car engineer earn?” Business Today.

Visit

In April I had the great pleasure of visiting the Beijing campus of Baidu, China’s leading Internet company. Udacity is working together with Baidu on an upcoming free course that teaches how self-driving cars work through the lens of Baidu’s Apollo open-source software.

We got to see various Baidu self-driving cars, and we even played ping-pong. I went 1–1 in my ping-pong matches and was told I was the best American they had ever played against. They were the best Chinese opponents I had ever played against!

Baidu wrote up a blog post about the visit. It’s in Chinese, but I’m told the phrase, “self-driving car Internet celebrity” appears. My mother will be so proud 😜