GM’s management of Cruise remains one of the most underreported stories in autonomous driving. It’s not a flashy story — basically, it amounts to, “how did GM not screw this up?” — but it’s kind of amazing.

All the more so in light of today’s news that SoftBank is investing $2.25 BB for 19.6% of Cruise.

For those of us keeping score at home, GM purchased Cruise just over two years ago for between $500 MM and $1 BB (the exact price varies depending on the source, but who’s counting?). Now, two years later, GM is selling a large minority stake in Cruise to SoftBank (one of the world’s premier late stage venture investors) for an implied valuation of $11.5 BB. That is a return on investment that would make even the best venture capitalists turn backflips.

At the time of the original GM acquisition, I would’ve bet that by today all the San Francisco engineers would’ve left and GM would be quietly folding away the remnants of the Cruise unit. That story has played out in Silicon Valley a million times, and there was every reason to believe that a giant Detroit industrial behemoth would never be able to nuture a tiny, cutting-edge San Francisco startup.

The continued success of Cruise — they have logged more autonomous miles than any other company except Waymo or Uber — is a testament to the management skill of the entire GM organization, especially GM President Dan Ammann, who has been the public face of the relationship.

I would love to read more reporting on how they pulled this off.

The National Transportation Safety Board has released a preliminary report on the fatal Arizona collision between an Uber self-driving car and a pedestrian walking a bicycle.

The report is short and easy to read, so I’ll embed it here and post some follow-up thoughts below.

https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf

First and foremost, every vehicle fatality is a tragedy. Elaine Herzberg, the pedestrian killed in this collision, was a mother, a daughter, and a sister. Over 1 million automotive fatalities occur worldwide every year, 37,000 of them in the United States. In many cases, including this one, the tragedy is borne by both the deceased and the survivors.

While Uber ATG is a Udacity partner, I have not spoken with them about this incident and thus have no inside knowledge of what happened. The Uber ATG engineers are some of the best in the business. Uber has logged more self-driving miles than any company except Waymo. This is an important collision to study because it provides a window onto how top-notch engineers design an autonomous system, and what, if any, failures might still occur.

It seems that parts of the system functioned exactly as intended, and other parts seemed to fail, although the failures are not yet clear.

Classification

The report states:

“..the system first registered radar and LIDAR observations of the pedestrian about 6 seconds before impact, when the vehicle was traveling at 43 mph. As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path.”

It makes sense that the Herzberg was picked up by radar and lidar, but not by cameras. Watching video of the collision, it seems that the environment is too dark for cameras to see.

Cameras are the best sensor for classifying objects, and in the absence of camera data, it makes sense that the rest of the sensor system had difficulty classifying Herzberg. This is especially true because Herzberg was engaging in unusual behavior: jaywalking a bicycle across the street on foot. That behavior is not super-rare, but it is less usual than a bicycle riding down the street, or walking a bicycle through an intersection, or a pedestrian jaywalking. The combination of a lack of camera data plus unusual behavior probably were presumably the driving factors behind the difficulty in classifying the object.

Prediction

This classification difficulties are important because the vehicle prediction system relies on classification to predict what other objects on the road will do next. Imagine, for example, that a self-driving car classifies a vehicle in the next lane as cycling with the flow of traffic. In that case, the prediction system is likely to predict that the future behavior of the cyclist will be to continue cycling in that lane, with the flow of traffic. Thus it is safe for the self-driving vehicle to proceed ahead, and pass in the adjacent lane.

On the other hand, if the object is classified as a cyclist crossing the street, the prediction system may well predict that the future behavior of the cyclist will be to cross into the self-driving vehicle’s lane. Thus it is not safe to proceed.

The absence of an accurate classification of Herzberg as a pedestrian, while understandable, made it difficult to predict what she would do next.

Braking

The report goes on to state:

“At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver was needed to mitigate a collision. According to Uber, emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior. The vehicle operator is relied on to intervene and take action. The system is not designed to alert the operator.”

This is the crux of the collision, and the description confuses me.

The phrase, “emergency braking maneuvers are not enabled while the vehicle in under computer control”, in the context of the report, appears to refer to the Volvo factory-installed emergency braking system. The report states that Uber disabled this system while the vehicle was in self-driving mode. The disablement in and of itself strikes me as reasonable — you don’t necessarily want the brakes controlled simultaneously by two separate safety systems, designed by two different companies, independently of each other. It would be interesting to know how other third-party self-driving car developers handle this.

However, the report also states that, “the self-driving system determined that an emergency braking maneuver was needed”. This appears to tie the Uber self-driving system to the Volvo emergency braking system, which presumably it cannot control and has been disabled anyway.

The report then goes on to state: “The vehicle operator is relied on to intervene and take action. The system is not designed to alert the operator.” The juxtaposition of those two sentences is hard to comprehend, and makes me believe there must be more to the story. I am curious to read further reporting on how Uber designed and anticipated both emergency braking and intervention by vehicle operators.

Vehicle Operator

The same video, above, that shows the collision, also has footage of the vehicle interior. That footage shows the vehicle operator looking down most of the time leading to the collision. Speculation at the time of the crash was that the operator may have been distracted by a personal phone, however:

“the vehicle operator stated that she had been monitoring the self-driving system interface. The operator further stated that although her personal and business phones were in the vehicle, neither was in use until after the crash, when she called 911.”

A few items of note here. The vehicle operator appears to have been tasked with several duties while operating the vehicle:

“the operator is responsible for monitoring diagnostic messages that appear on an interface in the center stack of the vehicle dash and tagging events of interest for subsequent review.”

This could be a problem or maybe not. Most human drivers are habituated to monitoring vehicle diagnostic messages (i.e. the check engine light) while driving, although it’s not clear how frequent or cognitively taxing the diagnostic messages were in this case. Similarly, “tagging events of interest” could mean anything from simply tapping a button on the steering wheel, or it could mean looking down and typing notes on a touch-screen keypad, which would be much more distracting.

Elaine Herzberg

The ultimate tragedy in this sequence of events is the death of Elaine Herzberg. Nonetheless, her actions leading up to the collision also raise questions.

“the pedestrian was dressed in dark clothing and that the bicycle did not have any side reflectors. The bicycle had front and rear reflectors and a forward headlamp, but all were facing in directions perpendicular to the path of the oncoming vehicle. The videos show that the pedestrian crossed in a section of roadway not directly illuminated by the roadway lighting.”

Herzberg also tested positive for methamphetamine and marijuana on autopsy, which may have contributed to her decisions and behavior.

At a much higher level, it depresses me that a homeless woman collided with a self-driving car, one of the most advanced technological innovations humans have ever created. I have been involved with efforts to end homelessness in the San Francisco Bay Area for six years, and this incident in Arizona really brings home the struggle and failure to provide for the most vulnerable people in our midst.

Hope

The goal of self-driving car engineers, and the hope of self-driving cars, is to bring automotive fatalities as close to zero as possible.

The silver lining, such as it is, of this collision, is that this scenario, or close simulations of it, will go into the test suite of every self-driving car company. Every time software gets updated or improved, it will be checked against this scenario, to make sure the vehicle successfully averts this type of collision. Unlike humans, self-driving cars learn from each other, and they get better and better over time.

Hopefully this particular type of collision never happens again.

A company called AB Dynamics recently made the news by demonstrating their self-driving scooter system, featuring a BMW C1 scooter.

I have no other knowledge of AB Dynamics, and while I hope they succeed with this product, my interest was primarily piqued by safety.

In the United States, motorcycles are 27 times more dangerous than passenger vehicles. Specifically, the fatality rate per 100 million vehicle miles traveled is ~25 for motorcycles and less than 1 for passenger vehicles.

I’m confident there are complicating factors underlying those numbers, such as motorcyclists getting hit by passenger vehicles, and unsafe riders who don’t wear proper equipment (including riding without helmets!). But perception is that riding a motorcycle or scooter is risky, at least relative to cars, and statistics seem to bear that out.

Thus, the potential of self-driving motorcycles to be significantly safer than human-driven motorcycles could be an important advance for autonomous vehicles, and not just for current riders.

For one thing, motorcycle are much more fuel-efficient than passenger vehicles. Average fuel efficiency for motorcycles is 44 miles per gallon, compared to 23 for cars. Motorcycles also take up less space on the road, and presumably damage the road less than heavier passenger vehicles.

Realistically, probably 80% of my miles could happen on a motorcycle. In reality, probably 80% of my miles happen in a Ford C-MAX (Energi!), with the rest on public transit. The reason I don’t use a motorcycle is mainly safety.

Also I’m not trained or licensed. That’s another problem self-driving motorcycles would solve.

A world of super-safe self-driving vehicles might still mean single-seat four-wheelers tooling through the city. All sorts of people and clothing and cargo requires a four-wheeler. But super-safe self-driving vehicles might also mean a lot more scooters.

For the last few weeks Udacity and Lyft’s Level 5 Engineering Center have been hosting the Lyft Perception Challenge to identify the best Udacity students for interviews on Lyft’s self-driving car team.

The competition involves segmenting driving images, like the image above. If you’re in or have graduated from the Self-Driving Car Nanodegree Program and interested in working at Lyft, you should participate! And if you’re not in the Nanodegree Program, you should join 🙂

The competition closes on June 1, 2018, so there are still a few days left to enter your submission.

If you’re looking for an approach to get started, my Udacity colleague Kyle Stewart-Frantz has posted a terrific walkthrough of his deep learning approach to the challenge.

Kyle trained a fully-convolutional network using data from the CARLA simulator, and then found that the data applied pretty well to real world camera data.

https://youtu.be/K8W_MzmooI0

The latest episode of Udacity Talks is up!

This episode flips the format. This time, I interview Sebastian Thrun, the founder of Udacity and the father of the self-driving car, about his career, self-driving cars, flying cars, and the future.

This was a lot of fun to record, and I hope you enjoy it 😎

Intel-owned Mobileye has announced they have a fleet of 100 self-driving vehicles on the roads in Jerusalem. That’s a lot of cars.

Israel has been a hotbed of autonomous vehicle development, but Israeli automotive companies face the same problem that almost all Israeli companies must surmount — a small local market. Mobileye has grown into a world leader in computer vision by supplying its perception systems to large automotive manufacturers outside of Israel.

Nonetheless, Mobileye founder and CEO Amnon Shashua writes that testing in Israel is important because of the challenges of Israeli driving. The title of the post is, “If You Can Drive in Jerusalem You Can Drive (Almost) Anywhere”.

“Jerusalem is notorious for aggressive driving. There aren’t perfectly marked roads. And there are complicated merges. People don’t always use crosswalks. You can’t have an autonomous car traveling at an overly cautious speed, congesting traffic or potentially causing an accident. You must drive assertively and make quick decisions like a local driver.”

Mobileye also published a video showing its self-driving cars executing several lane changes and other maneuvers in heavy traffic. The accompanying description of how the Mobileye vehicle is signaling to other cars on the road is impressive.

I was particularly struck by the description of Mobileye’s planning function.

“The part of our driving policy system that proposes actions is trained offline to optimize an assertive, smooth and human-like driving style. This is a proprietary software developed using artificial intelligence-based reinforcement learning [DS: my emphasis] techniques. This system is the largest advancement demonstrated in the fleet, and you can see the impressive results in the event visuals.”

Reinforcement learning has led to tremendous breakthroughs in games like chess and Go. It has stalled in its ability to translate to the real world, where simulation is less high-fidelity. This marks one of the first successful deployments of reinforcement learning to physical world problems.

Lots of neat stuff here.