An average of 545 people killed by local and state law enforcement officers in the U.S. went uncounted in the country’s most authoritative crime statistics every year for almost a decade, according to a report released on Tuesday.

The first-ever attempt by U.S. record-keepers to estimate the number of uncounted “law enforcement homicides” exposed previous official tallies as capturing less than half of the real picture. The new estimate – an average of 928 people killed by police annually over eight recent years, compared to 383 in published FBI data – amounted to a more glaring admission than ever before of the government’s failure to track how many people police kill.

The revelation called into particular question the FBI practice of publishing annual totals of “justifiable homicides by law enforcement” – tallies that are widely cited in the media and elsewhere as the most accurate official count of police homicides.

The new estimates added crucial framing to a criminal justice crisis in the U.S. that was coming into sharp focus this week. The Department of Justice announced on Wednesday that after a six-month inquiry it has concluded no civil rights charges should be brought against Ferguson police officer Darren Wilson for killing unarmed black 18-year-old Michael Brown.

A grand jury in St. Louis decided last November not to indict Wilson on state charges. On Monday, the president’s taskforce on policing issued recommendations for better data collection as part of a call for top-to-bottom criminal justice reform.

“There was a great emphasis on the need to collect more data,” Barack Obama said after a meeting with the taskforce. “Right now, we do not have a good sense, and local communities do not have a good sense, of how frequently there may be interactions with police and community members that result in a death, result in a shooting.”

The president’s warning of a national blind spot on police killings significantly amplified growing calls for policing reforms and for a revolution in crime statistics. Yet Obama did not, perhaps, capture just how bad the information was that the country has been working with. Independent tallies had previously indicated that the FBI’s “justifiable homicide” counts were flawed. But until recently, the FBI discouraged challenges to its numbers, insisting that they were carefully audited – and pointing out that the bureau, in any case, was required by law to publish them.

Tuesday’s Bureau of Justice Statistics (BJS) report, produced in collaboration with RTI International, the research institute, explodes the notion – if its findings are accurate – that the figures the FBI publishes annually are anything other than hugely misleading.

The data underlying the FBI tally “is estimated to cover 46% of officer-involved homicides at best” for the years 2003-2009 and 2011, the BJS report concluded. But the published FBI tallies cover even fewer of the total deaths, closer to 41%, in part because the FBI publishes no data from Florida. A separate tally of “arrest-related deaths”, conducted by BJS itself, was slightly more accurate for the years in question, capturing 49% of law enforcement homicides, at best, the report found.

The report estimated “an average of 928 law enforcement homicides per year” for the years in question, suggesting that the FBI’s published count of 414 such deaths in 2009, for example, might be 124% off, while its count of 347 such deaths in 2005 might be 167% off.

“The point of this was to try to quantify the coverage error,” said Michael Planty, an author of the report.

The years under study saw several high-profile homicides by law enforcement of unarmed civilians, such as the 2009 shooting of Oscar Grant in a train station in Oakland, California – an episode that would become the subject of the award-winning film Fruitvale Station – and the 2006 killing of Sean Bell, who died in a hail of 50 bullets outside a nightclub in Queens, New York.

But the majority of victims in law enforcement homicides for those years not only went unnamed – they went uncounted in any one tally. Even the two counting systems combined, as overseen by the FBI and BJS, missed an average of 263 homicides by law enforcement each year, BJS found.

Academics and specialists have long been aware of flaws in the FBI numbers, which are based on voluntary submissions by local law enforcement agencies of paperwork known as supplementary homicide reports. No law requires local agencies to fill out the reports, and some agencies do not, especially not for officer-involved homicides, according to experts who have studied the issue.

But no accredited source had publicly ventured to claim that the numbers published by the FBI were more than 100% wrong. That’s notwithstanding an unusually public airing of doubts about the numbers by the FBI director, James Comey, in a speech at Georgetown University last month. “It’s ridiculous that I can’t tell you how many people were shot by the police in this country – last week, last year, the last decade – it’s ridiculous,” Comey said.

In an interview with the Guardian following Comey’s remarks, Stephen L Morris, assistant director of the FBI’s criminal justice information services division, acknowledged “holes and gaps” in crime data the bureau publishes.

“We caveat this data – we have been for decades, cautioning individuals and organizations from drawing conclusions from it, because we recognize it is incomplete data, it is disparate data that leaves too many holes and gaps,” Morris said. “That’s been a point of frustration for decades within the FBI.”

But it is unclear whether the public understands the impressionistic nature of the FBI’s crime data. A visitor to the FBI web page presenting the “justifiable homicides” figures, for example, encounters no caveat or disclaimer language of any kind.

Morris indicated last month that the FBI had no plans to change how it publishes its “justifiable homicides” numbers, repeating that the bureau was required by law to report the figures. “I’m not going to characterize them as reliable or unreliable,” he said. A spokesman for Morris declined on Tuesday to offer additional comments on the most recent BJS study.

The deaths referred to as “law enforcement homicides” in the BJS report and what the FBI calls “justifiable homicides by law enforcement” are roughly equivalent, Planty and a second senior BJS official told the Guardian, in part because nearly all homicides by police, if adjudicated at all, are deemed “justifiable”, meaning not criminal.

“The homicides that were not justifiable, where a law enforcement officer is found guilty of homicide, there’s no way to identify that,” said Planty. “But from what we know, that number is relatively low.”

The scathing take on the FBI numbers came about as a result of an attempt by statisticians within BJS to evaluate what was worth salvaging in the bureau’s arrest-related deaths program.

The program was mothballed in March of last year, based on BJS’s own judgment of its shortcomings, but it may be resurrected with the renewal by Congress last December of the Deaths in Custody Reporting Act of 2000, designed primarily to count deaths in prisons and jails.

Exhorted by the 2000 law to count arrest-related deaths, BJS adopted a strategy that was fundamentally different from the FBI practice of collecting data submitted voluntarily by local law enforcement agencies. Instead, BJS activated a kind of sleeper network of crime statisticians in almost every state across the country, who harvested data from police departments but also from coroners’ offices and other sources.

Years before Tuesday’s report, the statisticians knew their data was not good enough. “We no longer wanted to be the place, the repository where those data lived, because we couldn’t guarantee that they were methodologically rigorous enough for us to actually publish,” said Erica L Smith, chief of BJS’s law enforcement statistics unit.

Statisticians at BJS and elsewhere hope that a revolution in crime data collection is now at hand. A National Academy of Sciences panel is studying how to modernize crime statistics, new cooperation has been detected between FBI statisticians and BJS statisticians – and then there was that eyebrow-raising speech at Georgetown by Comey, the FBI director.

James P. Lynch, a former Justice Department statistician and current chair of the department of criminology and criminal justice at the University of Maryland, hailed Comey’s complaint about the “ridiculous” state of crime stats as a watershed moment.

“I’m very excited,” said Lynch. “The idea that someone is finally saying, ‘These aggregate statistics that we have from the FBI are not adequate’ – I think a lot of people are saying that.”

While the FBI and other government tallies have long been criticized for underreporting, an admission of the problem at the top levels of US government is swiftly emerging. Joining Comey and Obama this year has been the outgoing attorney general, Eric Holder, who in January called the government’s accounting for use of force “unacceptable”.

In a highly anticipated investigation of its own, Holder’s Justice Department was expected to report on Wednesday that African Americans were subject to a full 88% of use-of-force cases actually documented by the police in Ferguson, according to a law enforcement official familiar with the department’s findings. In a separate report also expected as soon as Wednesday, the Justice Department was expected to clear the officer, Darren Wilson, who shot and killed an unarmed teenager there seven months ago.

Originally published by The Guardian