On a Test Track, a Self-Driving Car Explained Its Own Mistakes

On a private test track, a robotaxi kept stopping at the same spot. A traffic cone stood a few feet away, and the safety driver had the obvious explanation: the car had seen the cone. The car's own readout disagreed. Just before each stop, one label on the dashboard climbed toward certainty ("approaching stopped vehicle") while the road ahead stayed empty. So the researchers took the cone away and ran it again. The car stopped in the same place, at the same speed, with the same label lighting up.

That is one of three episodes recorded on the track for a study published Sept. 2 in Nature. Its lead author is Eoin Kenny, then a postdoc at MIT's Computer Science and Artificial Intelligence Laboratory and now at J.P. Morgan Chase, according to MIT's announcement of the work. That announcement also identifies the test car as a Motional robotaxi. The co-senior authors are Momchil Tomov of the autonomous-vehicle company Motional and Julie Shah of MIT. Their system, CW-Net, takes a driving model that has already been trained and makes it state its reasons, as it drives, in a handful of ordinary phrases.

A planner, the software that picks the car's next move, is a deep neural network, and when it does something strange, the person responsible for the car is reduced to guessing. Kenny and colleagues point to crashes, some fatal, in which the driver could not anticipate what the automation would do. Attempts to read what these networks are doing inside have multiplied, but, as the paper notes, almost all of that work stops at simulation.

The change is in the last layer

The planner they started from had been trained to imitate 80 hours of human driving: it generates a set of candidate paths, scores each for how human it looks and takes the winner. The team cut off that final scoring layer and put a classifier in its place, one that rates how well the moment matches a short list of concepts ("close to another vehicle," defined as within 3 meters; "approaching stopped vehicle"; "close to cyclist"), then trained a fresh scoring layer that sees nothing except those ratings. Because the concept scores are all the final decision has to work from, the authors call the explanations "causally faithful." That guarantee comes from the wiring, not from an experiment.

The rewiring had to cost something, and the paper's central number is that it did not. Replayed against nuPlan, a public benchmark built from real driving logs, the wrapped planner and the original differed by less than 1% on every measure. The original planner was competitive with the benchmark's best entries and weaker at pulling away from a stop, which gave the team imperfect behavior worth explaining.

On the track, the readout kept contradicting the driver

Before the cone, the same car had been coming to a halt just short of a passenger pickup zone, and the driver assumed the zone was the reason. The readout said otherwise: what was firing was "close to another vehicle," set off by the cars parked alongside. The driver started self-driving again farther from them; the concept's probability fell, and the car drove on. That episode ran a variant with the concept layer beside the planner rather than inside it, which the authors say may be less faithful.

The cone is the case worth sitting with. The explanation was accurate about the planner while the planner was wrong about the world: no stopped vehicle existed, and the car braked for one anyway. A driver who had blamed the cone would have gone on believing the car handles cones badly.

The third case ran the other way. Approaching a cyclist, the driver engaged self-driving from whatever speed felt safe, and the car stopped for the rider every time. But "close to cyclist" never rose above 1%, and the driver, watching a concept that would not fire, grew more cautious and began engaging from lower speeds. That caution turned out to be warranted. A later analysis found that the perception system had seen the cyclist while the planner was never wired to accept cyclists as an input, so the paths it chose would have collided with the rider. What stopped the car each time was a separate emergency braking system with no connection to CW-Net. The car always stopped safely, and the part of it doing the driving had not registered the cyclist.

What one test vehicle can show, and what it cannot

All of that is one vehicle on a closed track, with a safety driver aboard: not a fleet, and not a service carrying passengers. The public-road half of the study is narrower than it sounds. On Las Vegas streets, the safety driver drove manually for several hours with CW-Net running in the background, recording what it would have said rather than letting it drive. Those recordings became the material for an online study of 100 people, 99 after attention checks, half seeing the concept readouts and half only speed and steering. On clips of surprising behavior, the group with the concepts more accurately answered questions about what the car perceived and why it acted; on routine clips the two groups came out the same.

The labels themselves are coarse. On held-out data the classifier averaged 54% accuracy and 23% precision, and the cyclist concept scored close to zero, the weakness the third track episode then made visible. The authors argue that a concept the network never learned to separate is exactly what a driver needs to be told. The dashboard is not a reliable account of what the car can see. It is a reliable account of what its planner is acting on.

The work was funded by Motional, a joint venture of Hyundai Motor Group and Aptiv, which supplied the vehicles and the operations staff and took part in the decision to publish; seven of the nine authors are employees, and the company has a patent application pending on the architecture. Against that sit peer review at Nature and a benchmark nobody involved controls. The plotting code and the study videos are public; the model weights and the training code are not, on intellectual-property grounds, and what the team offers instead is a capsule reproducing the algorithm on a toy driving problem. One measurement argues quietly for the method: the pattern of concept activations barely moved between the track tests and the Las Vegas recordings, made more than a year apart, on different vehicles running different software.