After 2016’s the National Highway Traffic Safety Administration investigated the Model S’s Autopilot and Autosteer functions. It ultimately concluded the system was safe. But a new paper says we shouldn’t trust the government agency’s research.

NHTSA in 2017 and actually went so far as to state that “the data show that the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation” in on the matter. (You can read that and 13 other docs associated with the investigation on .)

But that “40 percent” claim was suspect, resulting in a Freedom of Information Act against the U.S. Department of Transportation from an organization called Quality Control Systems Corporation. QCSC identifies itself as a provider of “forensic statistical services and products to businesses and organizations.”

Its website is almost strangely aesthetically simplistic, but the outfit’s work in data analysis has been cited by quite a few reputable sources, such as the New York Times and NBC News, .

QCSC wanted to see the data in question for itself to try and corroborate or debunk the NHTSA’s statement which, months after being published, still appeared to be .

The National Highway Traffic Safety Administration can’t provide details on how it reached a…

The organization explicitly stated that its study was not funded by Tesla or its rivals, nor any other outfit involved in the autonomous automobile space. QCSC says that no “individual or entity” provided financial support to write the report or deal with the litigation necessary to get the data it’s based on. Also noteworthy is that “neither the Reviewer nor the Investigator leading NHTSA’s Preliminary Evaluation” responded to QCSC’s invitation to discuss the data.

QCSC finally did get information it was looking for through its FIOA request, and on Friday it published a report called “.”

The 24-page report says that, “The actual methodology applied by NHTSA to Tesla’s data was not adequately explained by the Agency when its claim about Autosteer was announced.”

Based on information the NHTSA had, but didn’t disclose until it was FOIA’d, QCSC says “Autosteer is actually associated with an increase in the odds ratio of airbag deployment by more than a factor of 2.4.”

From the report’s abstract:

...we discovered that the actual mileage at the time the Autosteer software was installed appears to have been reported for fewer than half the vehicles NHTSA studied. For those vehicles that do have apparently exact measurements of exposure mileage both before and after the software’s installation, the change in crash rates associated with Autosteer is the opposite of that claimed by NHTSA - if these data are to be believed.”

“For the remainder of the dataset, NHTSA ignored exposure mileage that could not be classified as either before or after the installation of Autosteer. We show that this uncounted exposure is overwhelmingly concentrated among vehicles with the least “before Autosteer” exposure. As a consequence, the overall 40 percent reduction in the crash rates reported by NHTSA following the installation of Autosteer is an artifact of the Agency’s treatment of mileage information that is actually missing in the underlying dataset.”

The report goes on to explain what QCSC identifies as failings of NHTSA’S analysis: Namely that the agency did a bad job interpreting data it had about Tesla crashes.

We reached out to Tesla and the NHTSA for comment on QCSC’s report. The car company clapped back arguing that QCSC was not seeing the whole picture either:

“QCS[C]’ analysis dismissed the data from all but 5,714 vehicles of the total 43,781 vehicles in the data set we provided to NHTSA back in 2016. And given the dramatic increase in the number of Tesla vehicles on the road, their analysis today represents about 0.5 [percent] of the total mileage that Tesla vehicles have traveled to date, and about 1 [percent] of the total mileage that Tesla vehicles have traveled to date with Autopilot engaged.

NHTSA’s did not only indicate that “Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation”, but the agency also concluded that it “did not identify any defects in the design or performance of the AEB or Autopilot systems” nor did it find “any incidents in which the systems did not perform as designed” (Page 10, last paragraph and Page 11, last paragraph, respectively). They also found that “the potential for driver misuse was evaluated as part of Tesla’s design process and solutions were tested, validated, and incorporated into the wide release of the product” (Page 10, first paragraph).

Our own vehicle safety data for , which includes data from roughly two billion miles driven in Tesla vehicles, shows that drivers using Autopilot were significantly less likely to be involved in an accident than those driving without using Autopilot.”

I guess it is neat that Tesla publishes safety data now, but I’m not convinced this response speaks to the purpose of QCSC’s report, which was to reveal the weakness of the NHTSA’s investigation following the 2016 “Autopilot” crash.

Tesla’s statement has now appeared in a few articles about this situation around the Internet, so I asked Randy Whitfield of QCSC what he thought of it. Whitfield responded to the accusation that QCSC “dismissed” data in an email reply to me:

“Between Figures 1-4 and Table 3, we carefully accounted for each and every one of the 43,781 vehicles.

By grouping the vehicles into different cohorts that shared common characteristics, we show the readers of our paper how NHTSA’s treatment of these individual groups affected NHTSA’s crash rate comparison.

We dismissed nothing.”

And we made all of the data available on the Internet for all 43,781 vehicles for anyone to check for themselves.”

The Drive’s Ed Niedermeyer, who , tweeted a simplified breakdown of all this that I’m inclined to agree with:

NHTSA, for its part, came back by saying, well, not much: “The agency is reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”

I can’t tell if that means the government agency is taking the report seriously enough as to warrant a long and comprehensive rebuttal, or if they’re just brushing it off. Time will tell.

As for the report’s intended outcome, I had earlier asked Mr. Whitfield what the ideal result was now that it’s been published. He responded with the following in an email:

[The ideal result would be] That government and industry might accept the necessity of international, comprehensive, open, and trustworthy surveillance of advanced driver-assistance systems as they operate in the field.

Such surveillance systems have been the key to advances in motor vehicle history for decades.

Why wouldn’t we want to apply those lessons to this new field?

It seems to me like the general public’s attitudes toward self-driving cars, or even semi-autonomous functions, . The fatal self-driving Uber crash had a lot to do with that, as did other high-profile incidents as this technology gestates in real time on our roads.

Drive free or die.

Overall, consumers and corporations seem to have come to grips with the notion that we won’t all have robotic chauffeurs by 2020, which is what it felt like everybody thought when I was at the Consumer Electronics Show a few years ago.

But advanced driver aids and automotive semi-autonomy isn’t really going away. It’s just taking longer to develop, and as these features are added and refined, they’re definitely something a lot of people are going to keep working on and trying to use. The agencies that govern our roads are going to have to stay sharp to keep up.