As self-driving cars move closer and closer to reality, we’re seeing more and more places in the world that people are working on them.
Some of these efforts are big. Some are small but growing. More will come.
It’s an exciting time to be in the business.
Audi has announced Level 3 autonomous driving functionality in the upcoming 2018 A8 model. This would make Audi the first car manufacturer ever to release a Level 3 vehicle.
As a brief recap, the Society of Automotive Engineers publishes five autonomy levels.
Level 1 — Driver Assistance: The driving mode-specific execution by a driver assistance system of either steering or acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task
Level 2 — Partial Automation: The driving mode-specific execution by one or more driver assistance systems of both steering and acceleration/deceleration using information about the driving environment and with the expectation that the human driver performs all remaining aspects of the dynamic driving task
Level 3 — Conditional Automation: The driving mode-specific performance by an Automated Driving System of all aspects of the dynamic driving task with the expectation that the human driver will respond appropriately to a request to intervene
Level 4 — High Automation: The driving mode-specific performance by an Automated Driving System of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene
Level 5 — Full Automation: The full-time performance by an Automated Driving System of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver
The controversial phrase in the Level 3 definition is:
“with the expectation that the human driver will respond appropriately to a request to intervene”
Some companies — most notably Google and Ford — contend that it’s not realistic to tell human drivers that they can divert their attention and then expect them to intervene quickly enough to avert an accident.
Audi seems more confident about human drivers, although they are rolling their system out slowly, presumably in an effort to better test and verify the car and the drivers.
Full Level 3 autonomous driving will be limited to divided highway scenarios at under 60 kmh (~35 mph). Basically, traffic jam driving. Which is the worst type of driving, so I look forward to the day when the computer takes that over in my own car.
The 2018 Audi A8 isn’t actually on the market yet, although it should be soon, and it’s price will start at 90,600 euro (~US$103,000). Definitely a luxury vehicle, and an exciting one.
My colleague Aaron pointed me toward a YouTube video for what looks like a pretty awesome photorealisic simulator called SynCity. It’s built by the artificial intelligence company CVEDIA, in Holland.
The photorealism of the simulator takes it to a whole new level beyond comparable simulators I’ve seen previously. At least what’s shown in the YouTube video 😉
This is the dream of autonomous vehicle simulators — that we’ll be able to take data derived from the simulator and transfer it to the real world. Particularly for computer vision, the closer the simulator looks to reality, the more likely that is.
CVEDIA appears to be previewing the simulator now, and I’m not sure when it will hit production release. Keep an eye out.
According to Financial Times, the autonomous vehicle gold rush is on, or maybe it has already jumped the shark.
“Global investment in autotech start-ups in the first half was double last year’s total of $790m for the same period, CB Insights said. Deal volume also hit a record high, with 35 investments made during the second quarter, according to their data.”
In the final project of the Udacity Self-Driving Car Nanodegree Program students build code to drive Udacity’s very own self-driving car.
As with almost any type of computer programming, however, we’re not starting from scratch. There are existing operating systems and middleware and libraries that students will get to build on to drive the car.
One of these libraries is Autoware, which is an open-source self-driving car library maintained by Tier IV. We use Autoware particularly for its localization functions, which use our lidar data and a high-definition lidar map to figure out where our vehicle is in the world.
The specific localization algorithm that Autoware uses is called normal distributions transform (NDT) matching, which was originally developed by Peter Biber at the University of Tubingen. NDT is a little different than the particle filter localization we’ve worked with previously, so I’ve spent time over the last few days reviewing how it works.
In order to figure out where we are in the world, we’ll probably use a map. There’s a whole branch of localization called simultaneous localization and mapping (SLAM), where we figure out how to navigate without a map, but that’s difficult. It’s easier just to have a map and so we’ll assume we have one.
In order to figure out where we are in the world, we take our own lidar scan and compare what we see to this map. You can basically imagine that we line up points and try to figure out, given what our current laser scan shows, where are we in this map?
One problem: our points will probably be a little off from the map. Measurement errors will cause points to be slightly mis-aligned, plus the world might change a little between when we record the map and when we make our new scan.
NDT matching provides a solution for these minor errors. Instead of trying to match points from our current scan to point on the map, we try to match points from our current scan to a grid of probability functions created from the map.
We break the point cloud map into three-dimensional boxes essentially assign a probability distribution to each box. The image above is actually a 2D probability function, but we can make a 3D function following the same principles.
This way, if we detect a point a few millimeters away from where the map thinks a point should be, instead of being completely unable to match those two points, our NDT matching function connects our detected point to the probability function on the map. There’s a kind of “near match”.
For anybody who’s taken Udacity’s lessons on particle filters, or studied them elsewhere, there is a whole separate issue of monte carlo randomization that particle filters use. It seems like that could be applied to NDT matching in pretty much the same fashion, and indeed there is a paper called “Normal distributions transform Monte-Carlo localization (NDT-MCL)” by Saarinen, et al. that seems to work out the details, although I haven’t gone through that in detail.
The sixth month of the Udacity Self-Driving Car Engineer Nanodegree Program teaches students about control. Control is how we actually turn the steering wheel or press the pedals to get the car to follow a trajectory, and the algorithms that perform this work are called “controllers”.
Two of the most common controllers for automotive applications are proportional-integral-derivative (PID) controllers and model predictive controllers (MPC). These are the two controllers we teach in the Udacity program.
Here are five different approaches Udacity students have taken to build controllers that drive the Udacity self-driving car around our simulator!
Andrey’s YouTube video simply mentions that his PID controller automatically learns hyperparameters. It looks like Andrey ran his car around the track multiple times and used some version of coordinate descent (the formal name for Sebastian’s Twiddle algorithm) to automatically tune the parameters.
“Implemented PID controller with automatically learned hyperparameters in C++ which allows the car learns to drive itself from scratch in the simulator.”
Anupriya talks about a number of the complications she encountered in developing her model predictive controller, and how she overcame them. Many of these are similar to what autonomous vehicle engineers find when deploying controllers to actual vehicles.
“This project also factors in real world latency that can occur while applying actuator inputs. To simulate this the project’s main thread sleeps for 100ms before sending the actuations to simulator. To account for this while returning the actuations to simulator I use 2 set of actuations — the real actuations for next step and the next predicted actuation after dt which is 0.1 second(100 ms) in my case. Sending the sum of these 2 actuations makes the model proactively apply the next actuation and hence handles the 100ms latency.”
Priya provides a thorough walkthrough of how she built her model predictive controller. If you’re interested in a line-by-line breakdown of how MPC works, this is a great read.
“To estimate the ideal steering angle and throttle, we estimate the error of our new state from our ideal state — actual trajectory we want to follow and the velocity and orientation we want to maintain and use a Ipopt Solver to minimize this error. This helps select the steering and throttle that minimizes the error with the desired trajectory.”
Andrew compares and contrasts PID and MPC, along with a brief review of Term 2 of the Nanodegree Program. He does a nice job of summarizing the different levels of fidelity at which you can build a controller.
“PID controller enables the car (robot) to follow some trajectory (x-axis reference line) while proportionally (how hard to correct steering), differentially (how gradually to return to the reference) and integrally (allow for wheel misalignment) applying the cross track error (CTE). It is simple to code, inexpensive to run (computationally) and takes little tuning effort to get something working. The down-side is that the car moves erratically and cannot accurately handle actuation latency (delay between command send and physical activation) .”
I enjoy the musical selections Jeremy picks to underscore his project submissions, and this one is a lot of fun. The blog post also provides a nice perspective of what it’s like to complete the Udacity MPC project as a student.
“After some debugging and tuning the cost function, my car was making its way around the track. It was time to tear it all down by adding the latency — and that’s just what happened. My approach to dealing with it was twofold (not counting simply limiting the speed): the original kinematic equations depend upon the actuations from the previous time step, but with a delay of 100ms (which happened to be my time step interval) the actuations are applied another time step later, so I altered the equations to account for this.”
A company called Otsaw has signed a partnership with Dubai to deploy self-driving police robots this year.
The robots are “the size of a child’s toy car” and come equipped with a drone that can track suspects in areas where the car can’t drive.
It’s unclear how real this is, as I haven’t heard of Singapore-based Otsaw before and their website is currently down.
The technology honestly seems a little out there right now, but it might be perfectly plausible in a year or two or three. Even today, robots surveil shopping malls.
To me, the most interesting aspect to this is how self-driving cars affect the balance between citizens, criminals, and the police.
I know less about Dubai, but in the US, police often use pretextual traffic stops to investigate more serious crimes. Self-driving cars might take away the pretext for a traffic stop, however, tipping the balance of power away from the police. Self-driving police cars with self-flying drones might tip that balance of power back.
I was raised in Virginia, so I have a strong interest in anything Virginia-based.
Torc Robotics is a Virginia-based startup with extensive roots in autonomous vehicle research. The team is a spin-out from Virginia Tech that placed third in the DARPA Urban Challenge in 2006. Like many Virginia-based companies, Torc has done extensive defense contract work, but with the self-driving car boom, Torc is returning to its autonomous vehicle roots.
Udacity students often ask if it’s possible to work on self-driving cars outside of Michigan, Germany, and the Bay Area. While those are the current centers of autonomous vehicle development, there are lots of companies in unexpected places around the world working on self-driving cars.
If you’d like to work on self-driving cars and live in Blacksburg, Virginia, (which sounds pretty nice to me) Torc is there for you.
About six months ago, when we were working with Mercedes-Benz on the Sensor Fusion Module of the Udacity Self-Driving Car Nanodegree Program, I was looking online for a concise and comprehensive comparison of sensor types.
I couldn’t find what I was looking for, so I sat down and sketched out a table myself.
I came across that table recently when I was cleaning up my desk, so I threw it into Google Slides and here it is.
I’m not positive I got every cell in this table correct, as I never ran it by the Mercedes experts. So if you see something wrong here let me know.
But maybe this will be useful to somebody looking for that same comparison table that I never found. And if you know where a better version of that table is, please mention it in the comments.
One of my family traditions on the 4th of July is to read the Declaration of Independence.
It is a wonderful, imperfect document, which birthed a wonderful, imperfect country.
David Silver 