Paul Lienert has an interesting piece in Reuters today about self-driving car startups. The piece touches on a few things: a particular startup called Nullmax, the geography of self-driving car startups, and valuations.
Two things caught my eye in the piece, though. One is the outsized role of Israeli startups in the autonomous vehicle space. There are relatively few Israeli startups working on full, end-to-end self-driving cars, but Reuters counts more Israeli startups in the perception and automotive connectivity spaces than the US has, respectively.
It’s notoriously difficult to count startups and I’m not sure I quite believe that Israel has more startups in any sector than in the US, but it’s nonetheless worth considering Israel as one of the world’s centers for autonomous technology.
The other part of the article that caught my eye is the dichotomy between how venture capitalists view autonomous startups and how traditional automotive companies view the same startups:
“While big automotive and technology companies are pouring billions into the autonomous vehicle space, Silicon Valley investors so far have been fairly restrained in increasing their bets.”
On the one hand:
With the notable exceptions of Andreessen Horowitz and New Enterprise Associates, few of the big Valley venture capital firms are heavily invested in the sector. Overall, only seven of the top 30 self-driving startups have received later-stage funding…
On the other hand:
All told, U.S. automotive and technology firms likely have invested some $40 billion to $50 billion in self-driving technology in recent years, mainly through acquisitions and partnerships…
Among the top corporate investors in the sector are Samsung Group [SAGR.UL], Intel Corp (INTC.O), Qualcomm Inc (QCOM.O), Delphi and Robert Bosch GmbH [ROBG.UL].
At a lunch a while ago, I sat next to nuTonomy CEO and former MIT researcher Karl Iagnemma. He seemed both nice and humble and super-smart. He’s also probably the most successful person to send me a LinkedIn connection request, so obviously I’m a fan.
Among startups (e.g. not Uber), nuTonomy seems to have a big lead in terms of actual autonomous vehicles being tested out on the road.
Delphi and nuTonomy were both independently testing self-driving cars in Singapore for the last year, so presumably they got to know each other pretty well.
Delphi is one of the world’s premier automotive suppliers, but they’ve been moving into the self-driving car industry in a way that sets them up as competitive with automotive manufacturers. This purchase further complicates that industry dynamic.
Following up on #3, Delphi’s multi-domain controller was at one point positioned as the core computation platform for autonomous vehicles. It might become less attractive to automotive manufacturers, as they won’t want to purchase key components from a potential competitor.
On the other hand, Delphi’s expertise in autonomous driving sets it apart from many other suppliers, all the more so due to this acquisition. If Delphi components become much more effective than the alternatives, the competitive vendor-supplier dynamics might matter less.
nuTonomy’s $450 million acquisition tag is fantastic, but not quite as mind-boggling as $1 billion GM paid for Cruise, or the $680 million Uber paid for Otto. This is especially true given how much further along nuTonomy appears to be than Cruise or Otto were at acquisition. Maybe valuations in the self-driving car market are slowing down just a tiny bit.
Somewhat surprisingly, Axios reports that nuTonomy was on the hunt for more funding, but couldn’t raise it at a valuation they liked. So they sold the company instead. I haven’t seen this reported elsewhere, but if it’s true, that’s another indication that self-driving car valuations may be coming down just a bit.
For a while I have been telling people that there are self-driving cars already on the roads in several cities around the world — Pittsburgh, Phoenix, San Francisco, Singapore, and a few other places.
Over the next year, I expect we’ll see the number of cities on this map blow up, and at the same time we’ll see the geofences surrounding the individual test sites expand.
As of today, if you’d like to see or ride in an autonomous vehicle, here’s where they are:
Removing the safety driver is one of the next big steps forward for self-driving cars, along with:
Testing the cars in more cities, especially internationally
Expanding the geofences around existing test locations
Opening up more of the tests to the public
Here are a few steps that seem likely to happen next, although the timeframe is not obvious:
It’s safe to assume that the first versions of this test will happen on private test tracks (or maybe they already are happening — the regulations on how to operate self-driving cars on private land vary wildly from state to state).
The next step will probably be to operate self-driving cars on public roads, with no humans in them. Possibly with a trailing vehicle carrying test engineers who can remotely seize control of the driverless car in an emergency. That will be quite a site.
Presumably only after companies are super-confident about the ability of cars to operate with no humans inside will these cars be opened up for public ridership.
It’s still a ways to go, but each little step is exciting.
Sensor fusion is the task of combining data from multiple sensors to build a robust understanding of the surrounding environment. For example, we might be building software for a vehicle with multiple radar and lidar units. One unit might tell us that we have another vehicle directly ahead of us, and then a different unit might tell us that we have another vehicle ahead of us and to the right. We use sensor fusion to determine whether those are two different vehicles, or the same vehicle.
Here’s some work Udacity students have done in this domain.
Sensor fusion turns out to be a highly mathematical discipline, and Mithi uses this post to succinctly review the linear algebra behind extended Kalman filters. This is part of a series of posts Mithi wrote about how Kalman filters work.
“For the radar, I’d have to linearize the extraction matrix H as mentioned several times before. To linearize, I compute what is called a Jacobian matrix this is based on first-order partial derivatives of the function that converts cartesian to polar. You can see the derivation here. The value of this Jacobian is based on what the state is if the radar sensor measurement is correct.”
Once upon a time I tried to teach myself how to build an extended Kalman filter by reading the Wikipedia entry. It didn’t work out.
Raul starts with the Wikipedia entry and proceeds to build a coherent explanation of how Kalman filters works, incorporating both time and uncertainty.
“As reminder, F is the Transition Matrix (the one that deals with time steps and constant velocities) and Q is the Covariance Matrix (the one that deals with the uncertainty). Let’s say there is a relation between the Uncertain and the the velocity, the Covariance Q is proportional to the velocity being bigger the Uncertainty whith higher velocities and more accurate with lower ones. This Process has a noise wich notation can be written as ν∼N(0,Q) wht means zero mean and covariance Q, Gaussian Distribution is the proper name.”
A while back, Peter messaged me on Twitter, asking about some open-source lidar data that he could fiddle with. I pointed him to some data the Voyage team had open-sourced while they were still working at Udacity. My former colleague, and current Voyage CEO, perhaps knew more about that data than I did and pointed Peter to KITTI, instead, which is probably the right answer.
Nonetheless, Peter spent a while getting the point cloud files spun up and learned how to work with them.
“For my purposes, there are two datasets of interest which each contain camera, GPS, lidar, and steering/throttle/brake data. The driving data is stored in ROS bags and are downloaded via the torrent links found below. I recommend you start downloading these now; they are pretty large!”
A couple of professors from the University of Pennsylvania, John Paul MacDuffie and Rahul Kapoor, are running a “wisdom of the crowds” project about the future of the automotive industry. It’s called the “2017–2018 Vehicle Innovations Challenge”.
It’s fun and stretched my mind a little bit about where the car industry is going over the next year.
They ask nine questions, which you can see on the homepage. Anybody (including you!) can sign up and participate for free.
Waymo recently released a Safety Report, that explains how they test and validate their self-driving cars.
On the one hand, it is awesome that Waymo wrote and released this. On the other hand, it’s not obvious who the audience is.
It’s clearly a marketing document. It even feels a little bit like something you’d pick up in a new car showroom. The pages have snazzy designs and pleasing fonts and graphics. Many of the 43 pages are given over to just a handful of statistics.
It’s also definitely a one-time document, as compared to the monthly safety report cards Google used to distribute as part of the Self-Driving Car Project. There is nothing in this new Safety Report that is calendar-specific.
That said, there is a wealth of high-level information in the report. Waymo goes into some detail on its testing processes and the scenarios that it tests. Not enough detail to be useful to engineers hoping to replicate Waymo’s processes, but enough to reassure the general public that Waymo has indeed thought this through.
Several sections explain how Waymo’s self-driving cars work, one section breaks down the ways in which Waymo analyzes safety (behavioral safety, functional safety, crash safety, operational safety, and non-collision safety).
But compared to the academic paper recently published by Mobileye, “On a Formal Model of Safe and Scalable Self-driving Cars”, Waymo’s report is aimed much more toward journalists and regulators and I suppose whoever in the general public is likely to flip through 43 pages of safety reporting.
My main takeaway from this is that Waymo must be inching ever closer to a public rollout of their vehicles. This is the type of document that they can send to new users, who will feel better having 43 pages of safety text, even if most of them never actually read it.
And if that’s the case, thank goodness for small steps toward a much better future.
This map of Silicon Valley appeared in Computerworld two years ago. Now there are even more automotive companies in the Bay Area.
One topic that has come up now and again for me is the intersection of Silicon Valley and the automotive industry. There are a lot of angles to this topic, but one thing I have generally been impressed by is how traditional automotive companies run their Silicon Valley offices.
There are a lot of obstacles to overcome: cost of living, new employees, veteran employees, communication with headquarters, division of labor across teams, multi-time-zone meetings.
The companies I’ve seen that do this well — and a surprising number of them seem to do it well — have the right mix of veteran managers and younger line engineers.
It’s hard to say exactly what that mix is, and it can vary considerably depending on the purpose of the office. Small offices focused primarily on technology scouting can tilt pretty far toward veteran managers on rotation from headquarters. Larger offices that are performing significant engineering work in Silicon Valley often succeed with a larger share of actual Silicon Valley engineers on the payroll.
The veteran managers provide access to the internal corporate social networks and informal power structures that facilitate progress in any organization.
The Silicon Valley engineers provide some of the raw engineering talent for which the Valley is famous, and perhaps also access to, “how Silicon Valley works”. “This is how we solved this problem at my last startup,” for example.
Another surprise is the number of junior engineers who went to school outside of the Bay Area, were hired by a traditional automotive company, and then shipped straight to California. These engineers have some of the attributes of veteran employees — their current automotive employer is the only employer they’ve ever known — and some attributes of traditional Silicon Valley engineers — youth and migration and audacity.
If I had to stick my finger in the wind and call a number, I’d say maybe 1/3 veteran managers and 2/3 Silicon Valley engineers (also marketers, business development, etc.) is the right mix, but it would interesting to have firmer numbers on this. It also seems like a good case study for a business school.
Last spring Udacity partnered with Didi to release datasets from Udacity’s self-driving car and test how well groups of engineers from around the world could track vehicles.
This Udacity-Didi Challenge was a big effort for us at Udacity — in terms of gathering data. It was also a huge effort for teams of students worldwide, who tackled cutting-edge research challenges in an effort to win the $100,000 prize.
We pared the teams of entrants down to five, based on their accuracy a vehicle and pedestrian tracking. Those five teams presented to a panel of judges at Udacity. Their work was incredibly impressive and a real testament to the ability of people from around the world to contribute to autonomous vehicle engineering.
I have a pretty awesome backlog of blog posts from Udacity Self-Driving Car students, partly because they’re doing awesome things and partly because I fell behind on reviewing them for a bit.
Alex visualizes lidar data from the canonical KITTI dataset with just a few simple Python commands. This is a great blog post if you’re looking to get started with point cloud files.
“A lidar operates by streaming a laser beam at high frequencies, generating a 3D point cloud as an output in realtime. We are going to use a couple of dependencies to work with the point cloud presented in the KITTI dataset: apart from the familiar toolset of numpy and matplotlib we will use pykitti. In order to make tracklets parsing math easier we will use a couple of methods originally implemented by Christian Herdtweck that I have updated for Python 3, you can find them in source/parseTrackletXML.py in the project repo.”
The most popular laptop among Silicon Valley software developers is the Macbook Pro. The current version of the Macbook Pro, however, does not include an NVIDIA GPU, which restricts its ability to use CUDA and cuDNN, NVIDIA’s tools for accelerating deep learning. However, older Macbook Pro machines do have NVIDIA GPUs. Darien’s tutorial shows you how to take advantage of this, if you do have an older Macbook Pro.
“Nevertheless, I could see great improvements on performance by using GPUs in my experiments. It worth trying to have it done locally if you have the hardware already. This article will describe the process of setting up CUDA and TensorFlow with GPU support on a Conda environment. It doesn’t mean this is the only way to do it, but I just want to let it rest somewhere I could find it if I needed in the future, and also share it to help anybody else with the same objective. And the journey begins!”
Vivek is constantly posting super-cool things he’s done with deep neural networks. In this post, he applies YOLOv2 to the KITTI dataset. He does a really nice job going through the process of how he prepares the data and selects his parameters, too.
“In this post, I covered the concept of generating candidate anchor boxes from bounding box data, and then assigning them to the ground truth boxes. The anchor boxes or templates are computed using K-means clustering with intersection over union (IOU) as the distance measure. The anchors thus computed do not ignore smaller boxes, and ensure that the resulting anchors ensure high IOU between ground truth boxes. In generating the target for training, these anchor boxes are assigned or are responsible for predicting one ground truth bounding box. The anchor box that gives highest IOU with the ground truth data when located at its center is responsible for predicting that ground truth label. The location of the anchor box is the center of the grid cell within which the ground truth box falls.”
“Illustrating the difference between aleatoric and epistemic uncertainty for semantic segmentation. You can notice that aleatoric uncertainty captures object boundaries where labels are noisy. The bottom row shows a failure case of the segmentation model, when the model is unfamiliar with the footpath, and the corresponding increased epistemic uncertainty.” link
This post is kind of a tour de force in investigating the links between probability, deep learning, and epistemology. Kyle is basically replicating and summarizing the work of Cambridge researchers who are trying to merge Bayesian probability with deep learning learning. It’s long, and it will take a few passes through to grasp everything here, but I am interested in Kyle’s assertion that this is a path to merge deep learning and Kalman filters.
“Self driving cars use a powerful technique called Kalman filters to track objects. Kalman filters combine a series of measurement data containing statistical noise and produce estimates that tend to be more accurate than any single measurement. Traditional deep learning models are not able to contribute to Kalman filters because they only predict an outcome and do not include an uncertainty term. In theory, Bayesian deep learning models could contribute to Kalman filter tracking.”
Bogdon started off with the now-standard Donkey Car instructions, and actually got ROS running!
“I decided to go for Robotic Operating System (ROS) for the setup as middle-ware between Deep learning based auto-pilot and hardware. It was a steep learning curve, but it totally paid off in the end in terms of size of the complete code base for the project.”
David Silver 