A Critique of NHTSA and SAE "Levels" of self-driving

In 2013, the National Highway Transportation Safety Agency (NHTSA) issued its policy on automated vehicles where they outlined 4 "levels" of robocar technology.

Essentially ADAS functions, like anti-lock brakes, electronic stability control, lane-keeping (on its own) and the like.
Multiple functions combined, most commonly the combination of adaptive cruise control and lane-keeping to make a car that drives itself but needs constant human monitoring and quick intervention. Also self-park monitored from outside the vehicle.
Limited automation: Can do full self-driving but need a human in various situations, and can even call upon the human while driving, but with some amount of delay allowed in the human response.
Full self-driving, including unmanned⁽¹⁾ operation.

There is also a "Level Zero" which is just an ordinary manually driven car.

Around the same time, the Society of Automotive Engineers also proposed a set of levels which were very similar. However, they split level 4 into two levels. The extra Level 5 requires driving in every situation a human can handle, while the lesser level 4 is allowed to not handle some roads or styles of driving, but is expected to be full-auto in the places it is rated to handle. Their full spec costs $70 to download, but the chart is enough.

These levels were generated because people do indeed crave a taxonomy to talk about the various forms of robocar technology. They want to talk about what's beeing made today (levels 0, 1 or 2,) though these are not really robocar technology, and we all want to talk about the future technology that's arriving soon. Having a simple hiearchy is very appealing, but problematic if it's just not possible to make one that's likely to match reality. These levels focus on the amount of human supervision needed as the key differentiator.

The Problem with Numbered Levels

Numbered levels strongly suggest an ordering or hierarchy to a technology that almost surely will not evolve in the manner laid out. The levels create an expectation of evolution in this direction and also an expectation that each level is a superset of the one below it. Regulators, press and the public are led to expect this progression by the levels, and may even write rules that demand it.

Already it's quite common in writings in the area to talk about the levels being developed in order, and in the NHTSA recommendations, even suggestions that level 3 be legalized but level 4 not be, giving that ordering the potential force of law.

The reality is that even the best engineers can not predict the actual path of the technology that far in the future, not with the high certainty you would want for drafting laws. In fact, the current progress already has gone against the levels.

In January of 2014, the French company Navia released the Induct, a "level 4" vehicle intended for the campus shuttle market. The Induct is summoned with a phone, drives on ordinary roads among pedestrians, cyclists and other vehicles, and does not even have a steering wheel. It is for sale as a commercial product (at a high price,) though Induct claims that shuttle operators can easily recuperate that price with just 2 years of operation without paying driver salaries.

While there are some basic level 2 products on the market, such as Mercedes-Benz "traffic jam assist" in the 2014 S-Class, the "levels" progression is completely disrupted by the first real product being already at level 4. The Navia pulls this off by going slowly to be safe, and advising operation only on private roads where government regulation is less of an issue. However, Induct did test demos of the vehicle in public pedestrian downtowns of European cities and there is no reason they could not go into operation in any place that could tolerate 20 km/h vehicles. The recommendations of NHTSA would ban not just unknown research but the first real product.

Levels think of "Driving" as a semi-monlithic problem, and the key issue is how much human supervision a system needs. Driving is really hundreds of problems, and the amount of supervision on each varies by situation, speed, time and even weather.

In reality, there are not levels, but a set of capabilities and features, which operate on a certain subset of the roads. Looking just at vehicles out there today:

The Navia does full unmanned operation, but only slowly, and on limited roads.
The Google car is not tested unmanned, but does full auto operation on highways and many private streets approved and mapped by Google, at full road speed. Currently this is only the highways and a few streets around Google in the San Francisco area, and a few random streets in special cities where demos were done. It may signal the driver non-urgently to take over, which is the first implementation of the Level 3 concept.
The Mercedes traffic jam assist requires supervision, and goes under 40 km/h on highways only, but works on almost all highways. Other demo products from car makers do this at full highway speed but either forbid or just discourage operation off of highways.

It's clear immediately that it's not easy to sort even these 3 first examples into levels of any kind. The Mercedes function is faster than the Induct but less automated. It's less capable and slower than Google but also works on more roads. Each vehicle is better at some things than the others, and worse at other things.

Others have also issued critiques of levels. Notably section 1.2 of the Defence Science Board's task force on Autonomy declares "[levels] are not particularly helpful to the autonomy design process." This paper from the Institute for Human and Machine Cognition which includes Maarten Sierhuis (now the director of Nissan's self-driving car research project) as an author, also issues a critque of levels.

SAE Level 5

SAE level 5's requirement of full-auto driving in every place humans can drive is an ideal goal, but unlikely to even be desired, financially, for many years to come. This would have to include include off-road driving, such as tracking a lion through the bush of Africa -- where your jeep, if on private property, is knocking down small trees to make new pathways -- or understanding addresses in the temporary city at Burning Man. Someday those capabilities might be there, but for now a more reasonable goal might be handling all marked roads and most private drives.

SAE Level 4 is the only probable level, as it will describe most vehicles for many decades. A vehicle purely at SAE 3 or 5 may not exist for a long time, if ever. They SAE spec more accurately suggests that a vehicle at levels 1-4 might also be at lower levels in some situations.

Alternatives to Levels

The above examples suggest that the key issues for a robocar are where it can travel safely, and how fast. The issue of how much human supervision is needed (which is the primary factor in the level systems) is important, but what's missing is that the amount of supervision varies not just with the technological prowess of the vehicle, but the road, speed and weather situations.

It's very likely we'll mostly see vehicles (like the Navia) that can run unmanned -- no human supervision or occasional remote supervision -- in some circumstances but not in others, in combinations of the following:

On approved mapped roads but not on others.
On roads where travel at 30 km/h is tolerable.
Only on the highways.
Only at night when traffic is very light.
Around school zones but not around 8-9am or 3-5pm when school is in session.
Only in areas with pre-tested network availability.
Avoiding certain types of intersections, such as unprotected crossing of high-speed streets.

The theme here is that you can make a vehicle that only handles a subset of road situations and simply tell it not to drive in situations it is not rated for. If the subset is large enough to be useful, you have a useful product, but it does not match the levels.

Of particular note is the idea of only driving where there is mapped network connectivity. There is much interest in the idea of a vehicle which operates unmanned (the NHTSA level 4) but may find situations where it needs a human's help in a non-urgent way (the NHTSA level 3.) If such a vehicle has connectivity, it can often get that help from a remote human over the network. This does not mean remote tele-operation in a tricky situation (though that technology is fairly far along in the military world) but does allow for the remote guide to offer advice about which lane to take, how to turn, or where to stop and wait for more help.

For example, the remote helper might examine a high-res 3D image of the scene, and plot a course which the vehicle is able to safely follow, while the remote helper watches lower resolution delayed video of the execution of the course. The vehicle still stops for any unexpected obstacle unless told to disregard it by a human, of course.

It's also possible in that situation for an unmanned vehicle to just pull over and wait for a human to come in another vehicle and solve any really hard problem that can't be handled remotely. The company that was sending the unmanned vehicle to do a task can send another while they deal with the first one's problem. As long as this is not very frequent, it can be affordably managed. This is also true for issues like mechanical breakdowns.

Before driving these roads, the cars would have mapped not just the geometry of the road, but also the network signal strengths. If a road does not have clear, strong signal on one of the available networks, the car will simply not route along it while unmanned. If the road is important, the companies will pay for WIFI points or other network improvements. They can also use more expensive but universal bandwidth in the rare situations. Network strengths change, but not suddenly. A car can notice if network strengths fade in unexpected ways before they vanish.

Nobody has done this yet, of course, but it tells us that the drafted levels don't even encapsulate the levels of human supervision that are possible.

In addition, what it means is that NHTSA 4 and SAE 5 are ideals, ideals that there is not much financial incentive to attain, because it is cheaper to solve the rare problems in other ways. They may not be the actual path of progress.

Rather, we're likely to see vehicles that can operate unmanned on a growing subset of roads and situations, and can operate with occasional human supervision on a superset of those roads.

My prediction is we'll see this mix of the levels at the very dawn of operations (I do not consider the level 2 to be part of this) until almost the very end. I expect every vehicle to be a mix of levels:

There will be some situations where the vehicle can operate entirely unmanned. The most likely early one will be short-range operations for parking, delivery and recharging. At first, this will be in private situations (like a vehicle that parks itself in a private lot) but very soon street parking and pick-up will be available.
There will also be some areas where the car requires an occupant and requires the occupant be availble on non-urgent notice (NHTSA 3).
There will be some situations where a human driver must take over, or take over using the robocar technologies for first-rate driver assist. (NHTSA 0, 1 or 2.)

Even 15 years later, the best cars will still have a few roads and a few situations they might need human assistance with. They'll know what those are, and operate in them according to rules to assure safety.

In other words, if my prediction is true, there might never be a vehicle sold which closely matches any specific NHTSA level, and all vehicles might match the SAE level 4.

My prediction is almost surely not fully correct, of course. But that uncertainty tells us it would be unwise to code these levels into any laws or regulations if there's the chance that they will never really exist.

Road classifications

If we do wish to have a taxonomy -- for discussion, not regulation -- that taxonomy should start by classifying road situations by their difficulty. Be advised there is no strict ordering, some situations are quite distinct from others.

Controlled: Specially controlled roads with limitations on activity by other road users. For example, roads closed to the public, or to certain types of vehicles. Also includes roads with special infrastructure to assist the robocars. Roads meant for super-slow traffic. In other words, roads whose rules and structure are made for the robocar's capabilities.
Limited Access: Roads, like highways which are divided, well marked and allow only motor vehicles.
Easy: Low speed roads with limited traffic, controlled intersections (stop-signs and signals,) No blind corners, fast cross-traffic, or frequent schoolchildren. Driveways (if mapped.)
Traffic Jam: Situations where speeds are low, and cars are packed densely around, so it is only necessary to travel with the flow of the cars, and notice if that flow breaks up.
Medium: Higher speed roads, non-hard intersections, modest traffic, night-time low traffic periods. Entry-exit lanes on freeways. Unmapped driveways and parking lots.
General: Higher speed roads, most intersections, heavy traffic.
Hard: Unprotected high speed turns, highly congested merges, blind corners. Urban streets over 60km/h. Roads that differ from what was previously mapped. Construction zones. Jaywalkers.
Very Hard: Recent snow and ice. High speeds with dense pedestrian and cyclist traffic. Accidents or road blockages ahead. Pedestrians darting into traffic. Police redirecting traffic.

These are not precise, and in reality the various activities may be solved sooner or later than laid out here. I built this list with today's knowledge of what things look easy and what things look hard. That list will keep changing as time goes by.

Within these sets of conditions, there will be cars that can do the following things which are roughly akin to the levels:

(Level 4) Operate unmanned.
(Level 3.5) Operate unmanned as long as network connectivity is sufficient for remote assistance. Recognize problems needing assistance and signal for it.
(Level 3) Operate unsupervised, but recognize problems needing assistance, and ask the person inside to resolve them (or a remote assist.)
(Level 0-2) Operate only with regular human supervision or just plain human driving with top notch safety systems that protect the human driver from mistakes.

About Level 2

There is additional controversy, it should be noted, about the proposed level 2. In level 2, the system drives, but it is not trusted, and so a human must always be watching and may need to intervene on very short notice -- sub-second -- to correct things if it makes a mistake.

This level was inspired by the products being made today and in the near future by car manufacturers, which combine adaptive cruise control -- which is able to follow a leading car at variable speed -- and lane-keeping, which involves cameras which find the lanemarkers on the road and simple course correction to stay between them.

Everybody knows that these systems fail from time to time. Lane markers vanish or become confusing even to humans. Unusual road geometries confuse cruise controls. Radar reflections from the front of trucks confuse them as to the distance to a vehicle ahead. So like having just cruise control, you need to be watching, ready to hit the brakes or grab the wheel at any time.

It is possible to build such systems and make them pretty reliable, though not yet the near perfection needed for unsupervised "eyes-free" operation. However, many people have suggested this is a very dangerous design, because humans will trust it too much, and not pay attention to the road. In addition, they will make mistakes about the transition from self-driving to manual driving. All of this could add up to significant risk.

As such, while some believe this level is a natural first step on the path to robocars, others believe it should not be attempted, and that only a car able to operate unsupervised without needing urgent attention should be sold.

If this is true, we see a further problem with the system of levels. Level 2 might be too dangerous, and all cars sold might be a hybrid of levels 3 and 4, or like the Navia, level 4 in a limited area.

The Alternative

The use of the levels should be discontinued. Cars should be described with some basic capabilities over sets of roads and situations. The capabilities and situations should be understood as imprecise, simply guides to understanding and tools for discussion, not well defined terms for regulation.

For the supervision level, I suggest terms of this form:

Hands Free (Cruise Control or other Level 2)
Eyes Free
Automatic (Unmanned, network connection required)
Autonomous (Empty, 100% networking not required)

This would be combined with a description of the situations in which it can safely operate with this level of superision. So a car would be "Eyes Free on the Highway" or "Automatic under 60 km/h, Autonomous in parking lots or under 20km/h." These names are longer and more complex, but more useful.

Please leave comments at this blog post.

Footnotes:

¹I am not thrilled with the archaic use of "manned" here but lack a superior term. "Vacant," "uncrewed" and "body out" don't seem to fit well.