Are Autonomous Vehicles Safe?

On February 6th we sat in on a FHWA T3 webinar entitled “Are Autonomous Vehicles Safe?” It was moderated by Dr. Francesca Favaro of San Jose State’s Mineta Transportation Institute. She runs a program known as RISA2S – “Risk & Safety Assessment of Autonomous Systems” and recently examined California DMV data from 2014 through 2016 on crashes and disengagements of automated vehicles.

Because their data is from automated vehicles on the road today, their focus has been on SAE Level 3 automation. That is, by and large, what is being tested now. Most manufacturers are not planning to offer Level 3 vehicles to the general public. But as a result their findings point out the strengths and weaknesses of Level 3 automation. So for that reason most of their presentation came to focus on disengagements – when the vehicle gives control back to the driver – and driver reaction time to those disengagements.

This is an issue of critical importance to work zones. At some point in the future autonomous vehicles will negotiate work zones without need of human input. But that is many, many years from now. In the meantime, a mixed fleet of cars and trucks with varying levels of automation will be passing through our work zones. As Paul Carlson said at the recent ATSSA Innovation Council meeting, “We have had a mixed fleet for some time now. Any discussions of a mixed fleet now are just the next iteration of that.”

So the two related issues of when AVs disengage control, and how drivers react to those disengagements will be an important point of discussion for the foreseeable future.

Their data on disengagements was very interesting. The frequency of disengagements is declining. In fact in 2016 they were one-third of what they were per mile travelled in 2014. So the technology is improving rapidly. Machine learning will continue that trend.

11% of all disengagements were due to external conditions. 49% were due to system failures of some kind. 33% were due to human factors. And 7% to “other”.

Work zones fall within external conditions. 2.22% of all disengagements were due to construction zones and 4.63% were due to poorly marked lanes. Now, we don’t know much about the conditions when and where this testing took place. Speeds were all at or below 30 MPH. But we don’t know if it was dry and sunny, or raining at night. And we don’t know if the time spent in work zones was typical of your average driver.

Chances are they were not. In fact, chances are current testing avoids work zones most of the time so disengagements in real-world work zones would likely be many times greater.

30.12% of disengagements are due to human factors – usually driver discomfort. Some of that is a trust issue with the technology. But a large part is a desire to stay safe in fast changing situations such as near incidents, work zones, or other higher volume conditions. So I think we need to include some of that data as well.

Next let’s consider driver reaction to these disengagements. This is really two issues: what the driver does, and how fast he or she does it. But the study looked only at driver reaction time. The California DMV has not yet defined what is meant by “reaction time” yet manufacturers are required to measure it. So the data presented is inconsistent in its terminology. With that in mind it is still interesting and helps point the way for future studies.

This testing was done in driving simulators. Subjects reaction times varied from 0.87 seconds to as much as 3.17 seconds. Again, that is in a simulator when the test subject was told to expect something. Other studies have shown mean reaction times of 2 seconds or more.

Let’s use the 2 second number, though it may be much higher. At 65 MPH a vehicle would travel nearly 200 feet in that time. A lot can happen in 200 feet when approaching or adjacent to a work zone. So clearly disengagements should occur in advance of the work zone giving drivers time to acclimate themselves to the situation around them. For the next 20+ years many of the vehicles passing through work zones will turn control over to the driver. So rather than try to deal with those within the confines of the work zone, lets’ just automatically turn control over to drivers before they enter it. That will provide a more predictable hand-off in a safer environment.

This study was just their first attempt. As more data is collected and as terminology becomes better defined and as testing ventures into more “real world” scenarios, we will learn much more. We look forward to future reports from RISA2S and other AV research organizations.