Checkable differences between
filings and independent records.
filing-check reconciles NHTSA Standing General Order incident
filings against independent public records that the reporting party does not control,
and reports the places where two documents about the same event do not line up.
It never issues a verdict about a company.
It reports differences between documents. It never states or implies that anything was concealed, left out improperly, under-reported, or done with intent, and it produces no score and no match rate for any company. A difference between two independently-maintained public records is normal, and usually has an ordinary explanation: different reporting thresholds, different timing, an error in either source, duplicate records, or the plain fact that the two sources were built for different purposes. Those explanations are attached to every finding.
What the tool actually emits
A single FIELD_DIFFERENCE from the first real run. Both documents are cited
by their public identifiers, so any reader can retrieve both and check the comparison
rather than take it on trust.
report output, fixed 74-column width
CHECKABLE DIFFERENCE -- a matched pair of records differs
Independent record: NHTSA consumer complaint ODI 11693937
Filed by: a member of the public (unverified self-report)
Manufacturer: Tesla, Inc.
Incident date: 10/14/2025 Complaint filed: 10/16/2025
VIN prefix: 5YJ3E1EA5LF
Source: https://www.nhtsa.gov/?nhtsaId=11693937
Filing: SGO report 13781-12029 version 2 (ADAS)
Reporting entity: Tesla, Inc.
Incident date: OCT-2025 Location: Hesperia, CA
Vehicle: TESLA Model 3 2020 VIN prefix: 5YJ3E1EA5LF
Record linkage: VIN_PREFIX_MONTH [confidence: LOW]
This linkage is a candidate, not an established fact that the two
records describe the same event.
Fields that differ:
Injury reported:
filing [Highest Injury Severity Alleged]: Property Damage. No Injured Reported
independent [numberOfInjuries]: 1 injuries reported
This is a demonstration of what the tool emits, not a finding about a company. It is included so the output format and its caveats can be inspected, and for no other purpose.
- The linkage is LOW confidence, as every linkage this tool produces currently is.
- The two records share a manufacturer, a vehicle configuration and a month. They do not share an identified vehicle: the VIN serial is not published by either source. The two records may well describe entirely different events.
- A difference between a filing and a consumer complaint has many ordinary explanations: different reporting thresholds, different timing, an error in either source, duplicate records, or a wrong linkage. Consumer complaints are unverified self-reports.
- Pending further validation. This is a single demonstration case from a first run. It has not been independently checked, and it is not a headline finding and should not be treated as conclusive.
In the tool's own output this block is followed by all six innocent explanations and its mandatory caveat, before a reader reaches any interpretation at all.
It runs in one direction only
Auditing distinguishes two directions, and only one of them can catch something missing from a record that is otherwise complete, well-formed and verifiable.
Vouching
Starts from the records and works backward. Catches overstatement. A report that is filed, correctly formatted and internally consistent will pass, because no amount of checking a report against itself can show what is not in it.
Tracing, the direction used here
Starts from an independent population the reporting party does not control and works forward into the records. This is the only direction that can surface an absence.
The reverse is never computed. A filing with no corresponding complaint is not a finding: consumer complaints exist for a small, non-random slice of vehicles, so "no complaint corresponds to this filing" is the normal state of almost every filing and means nothing.
Methodology
Five decisions, in the order they were made.
The framing guard, built first
Both guards were written and proven green before any matching or comparison
logic existed. Of the 132 tests in the repository, the first 74 predate the
first line of reconciliation code. framing_guard.py scans any
tool-generated prose against eleven attribution constructions and a set of hard-banned
lemmas. Text that trips the guard is discarded, not repaired: a
narrative that has drifted into attributing intent cannot be patched into safety. The
guard runs clean across roughly 90,000 lines of real rendered output.
The data model cannot represent a verdict
No model has a field for concealment, intent, negligence, wrongdoing, or fault. A
finding holds two cited records and the fields on which they differ, and a test pins
its exact field set. The finding vocabulary is deliberately small and contains no kind
that names concealment, omission, or a failure to report. The two kinds are
UNMATCHED_INDEPENDENT_RECORD and FIELD_DIFFERENCE.
Record linkage, and why nothing is high confidence
A candidate link requires the same reporting entity, resolved through a normalizing fuzzy matcher with a curated alias table, a shared 11-character VIN prefix, and incident months no more than one month apart. Entity names are scored into three bands: at or above 90 links automatically, 80 to 89 is flagged and never linked unattended, below 80 is treated as distinct. No similarity score can override the gate. No linkage is treated as high confidence. Neither source publishes VIN characters 12-17 (the serial that identifies an individual vehicle), so a shared prefix means a shared configuration, model year and plant, not a shared vehicle.
Amended filings are kept separate
Amendments are reported in their own section, excluded from the match rate, and never diffed word by word. Only which fields changed is reported. A word-level diff of an incident narrative invites exactly the inference this tool refuses to make: that what was added should have been there the first time. A filer amending its own filing is the reporting system working as intended.
The rate and its meaning are one object
The match-rate object raises on construction if its interpretation text is empty, so no code path can emit a bare percentage: not the text report, not the JSON export, not any future consumer. The rate is printed before the findings, because a reader who meets a list of differences first has already formed an impression the statistic then has to argue against. An empty population yields no rate at all, never 0% or 100%.
The headline result is a low match rate
First real run, snapshot 2026-08-09. This is the expected outcome, and it was expected before the tool was written.
This is not a finding about anyone's reporting. The two populations barely overlap by construction: consumer complaints are filed by vehicle owners, and a driverless fleet has no owner to file one. The largest fleet in this snapshot accounts for 947 filings on a single vehicle configuration, for which 18 consumer complaints exist in total, none of them a crash.
For calibration, EMIR's dual-sided trade reconciliation (mandatory, in a far more structured domain, with both sides legally required to report the same transaction) still left up to 55% of trades unreconciled after four years. Cross-source reconciliation is genuinely messy everywhere it has been attempted.
| Snapshot 2026-08-09 | ADS filings | ADAS (Level 2) |
|---|---|---|
| Independent records examined | 1,181 | 1,181 |
| Linked to a filing | 0 | 44 |
| Non-match rate | 100% | 96.3% |
The substantive methodological result is narrower and more useful than a pass or fail: with the populations that are actually public and reachable without an account, cross-source corroboration of automated-vehicle incident filings is close to unavailable in the driverless case, which is precisely where autonomy, and therefore interest in independent corroboration, is highest.
Findings about the public data itself
Three properties of the live sources were established during the build and materially shaped the design. They are reported because they are useful to anyone else working with this data.
- CA DMV OL 316
listing is gone - California DMV's public listing of autonomous-vehicle collision reports is no longer reachable, and the collision-reports link on the DMV's autonomous vehicles page now points to a dashboard built from NHTSA SGO data. That is the same population being checked, so it cannot serve as an independent source.
- Month precision only
- Filings publish incident dates to month precision (for example
APR-2026). The day is not published, so any comparison against them can only happen at month precision. - An 11-character VIN prefix
is a configuration - The prefix both sources publish identifies a vehicle configuration, not a vehicle. Characters 12-17, the serial, are not published by either source, so a shared prefix is shared by thousands of vehicles.
Known coverage gaps
Stated up front rather than discovered by a reader.
- Police-reported crash data (Berkeley TIMS and California SWITRS) is not included. It is the population academic crash-rate research uses and would be the strongest independent source available, but it requires an account.
- California DMV OL 316 collision reports are not included, for the reason above.
- Court filings and news coverage are deferred, not ruled out.
- NHTSA consumer complaints are the only independent population in this version. That single-source scope is the most important limit on everything reported here.
Running it
Findings are computed against a pinned, dated snapshot, because NHTSA republishes the filing CSVs on a rolling window: an unpinned run could not be reproduced by a reader later.
python cli.py fetch # pin a dated snapshot of the public data python cli.py reconcile --report-type ADAS python cli.py reconcile --report-type ADS --entity Waymo --limit 5 python cli.py reconcile --json # rate, findings, caveats, all sources python cli.py amendments # filings amended by their own filer python cli.py sources # sources, roles, and coverage gaps python cli.py scope # scope and the standing disclaimer
Requires Python 3.10 or newer, click and rich. Run the tests
with python -m pytest tests/ -q.