How To Lie And Mislead With Rape Statistics: Part 1

As it turns out, only 7.8% of rape reports are true.

I know that may seem hard to believe, but I didn’t just make it up.  Technically, it is completely true.  It is also completely horse shit.  It is so misleading and built upon so many undisclosed caveats, that most people would consider it as good as lying if they knew how it was actually derived. The thing is, that “only 2-8% of rape allegations turn out to be false” figure you may have heard?  Not only is it just as misleading (if not more), it actually comes from the exact same data set.

So where did the 2-8% number come from?  If you read about it in Zerlina Maxwell’s piece in the Washington post you might be reasonably confused that the range is from the FBI

In fact (despite various popular myths), the FBI reports that only 2-8 percent of rape allegations turn out to be false, a number that is smaller than the number (10 percent) who lie about car theft.

Maxwell linked to a 2009 paper that hypothesized the 2-8% range, but has nothing to do with FBI statistics or car thefts. However, there is an earlier version of the 2009 article that was published in 2007 with the same name that included this bit of data:

For example, the Portland, Oregon police department examined 431 complaints of completed or attempted sexual assaults in 1990, and found that 1.6%¹ were determined to be false. This was in comparison with a rate of 2.6% for false reports of stolen vehicles (Oregon Attorney General’s Office, 2006).

This is from a single police department, in a single year, and is not an FBI statistic.  As it so happens, there is a relevant statistics from the FBI, but instead of 1.6%, it is 8%, which would mean it wouldn’t fit her point about stolen vehicles. On the other hand,  there is also a separate statistic² that says about 10% of stolen vehicle reports are false.  OK, so now we have a source for the 2-8% range, an FBI estimate that is in that range, and another statistic that shows a higher percentage of car thefts.  So while she is confusing her sources a little bit and that makes what she is saying somewhat misleading, overall her point is accurate, right?

That is where we run into a slight issue³ – we are sourcing all these statistics from different places, so how do we know that they are on a similar enough basis as to be comparable? Were Maxwell were to check that FBI 8% statistic, she would find that they actually have something to say on this matter – though it doesn’t fit in so well with her narrative. The 8% number comes from section II of the FBI’s Crime in the US reports from 1995 to 1997 which say:

1995

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. In 1995, 8 percent of forcible rape complaints were “unfounded,” while the average for all Index crimes was 2 percent.

1996

The “unfounded” rate, or percentage of complaints determined through investigation to be false, is higher for forcible rape than for any other Index crime. Eight percent of forcible rape complaints in 1996 were “unfounded,” while the average for all Index crimes was 2 percent.

1997

A higher percentage of complaints of forcible rape are determined “unfounded,” or found by investigation to be false, than for any other Index crime. While the average of “unfounded” rates for all Crime Index offenses was 2 percent in 1997, 8 percent of forcible rape complaints were “unfounded” for the same timeframe.

So when you look at things on a comparable basis and from crime data across the country and across multiple years, instead of the percentage of unfounded reports being lower for rape than other types of crime, forcible rape actually has the highest unfounded rates and was consistently 4x the average.

Let’s go back to that 2-8% range from 2009.  Think for a minute about what would make for a good statistical source on false rape reports.  Given how easy it can be to manipulate statistical research, you’d probably want someone unbiased if the data is to be at all useful. Now click through to the source Maxwell listed.  It is an article that appears in The Voice, the newsletter for The National Center for the Prosecution of Violence Against Women.  The article is titled False Reports: Moving Beyond the Issue to Successfully Investigate and Prosecute Non-Stranger Sexual Assault. OK, so not starting out so hot on the unbiased front, but let’s give it the benefit of doubt and dig into the article.

The article starts out with a note that estimates for sexual assault false reports vary quite a bit (1.5% to 90%). They then go on to say that they don’t find many of the prior data points to be credible, and attempt to narrow that range. The first study⁴ they attempt to discredit was done in 1994 by Professor Eugene Kanin. In it, Kanin found that 41% of the 109 sexual assault reports made to a midwestern police department over a 9 year period were determined to be false. This would seem to stand in pretty stark contrast to their assertion that the real number is 2-8%. The authors present two main reasons why the Kanin study is invalid. This first is that the determinations that the reports were false were made only by the detectives.  They work in a quote from an article written by David Lisak, one of the authors of the paper (the 2009 iteration anyway):

Kanin describes no effort to systemize his own ‘evaluation’ of the police reports—for example, by listing details or facts that he used to evaluate the criteria used by the police to draw their conclusions. Nor does Kanin describe any effort to compare his evaluation of those reports to that of a second, independent research— providing a ‘reliability’ analysis. This violates a cardinal rule of science, a rule designed to ensure that observations are not simply the reflection of the bias of the observer (p. 2).

They then go on to sum up this first point as follows:

In other words, there is no way to explore whether the classification of these cases as false was simply made as a result of the detectives’ own perceptions and biases, without any real investigation being conducted.

I’ll take this opportunity for a brief detour into irony land. Remember how I said there was an earlier version of this paper from two years prior? Though David Lisak was not an author of that paper (Dr. Alan Berkowitz took his place), the other two authors – Dr. Kimberly A. Lonsway, and Sergeant Joanne Archambault (Ret.) – were.  In that version, the “credible” range was 2-4% instead of 2-8% and they offered three pieces of evidence to support it.  The first is the Oregon Police Department data (it would seem the police are perfectly capable of an unbiased determination – so long as you like the number they come up with).  The second was this⁵:

Similarly, Sgt. Joanne Archambault of the Sex Crimes Division of the San Diego Police Department routinely evaluated the rate of false reports over several years and found them to be around 4%

Observer bias can be a real issue, but it does work both ways. Later on we’ll get to the evidence that they are using for their new estimate (thankfully they scraped the first two sources from 2007 this time around). When we do, keep in mind what kind of views they had going in.

Let’s return to the Kanin study. Their primary concern is essentially that the police are making the true/false determination on their own, potentially on a whim without much investigation, which on the surface is a pretty valid complaint.  Except, of course, for the fact that it completely misrepresents how the reports were determined to be false.  What they want you to think is that this is just a bunch of cops who don’t believe the women and so they classify it as false without digging into the details at all.  They can do that because they know that pretty much no one is actually going to go and read the underlying research.  If you did, you would find this right up front in the methods section:

This city was targeted for study because it offered an almost model laboratory for studying false rape allegations. First, its police agency is not inundated with serious felony cases and, therefore, has the freedom and the motivation to record and thoroughly pursue all rape complaints. In fact, agency policy forbids police officers to use their discretion in deciding whether to officially acknowledge a rape complaint, regardless how suspect that complaint may be. Second, the declaration of a false allegation follows a highly institutionalized procedure. The investigation of all rape complaints always involves a serious offer to polygraph the complainants and the suspects. Additionally, for a declaration of false charge to be made, the complainant must admit that no rape had occurred. She is the sole agent who can say that the rape charge is false. The police department will not declare a rape charge as false when the complainant, for whatever reason, fails to pursue the charge or cooperate on the case, regardless how much doubt the police may have regarding the validity of the charge. In short, these cases are declared false only because the complainant admitted they are false. Furthermore, only one person is then empowered to enter into the records a formal declaration that the charge is false, the officer in charge of records. Last, it should be noted that this department does not confuse reported rape attempts with completed rapes. Thus, the rape complainants referred to in this paper are for completed forcible rapes only. The foregoing leaves us with a certain confidence that cases declared false by this police agency are indeed a reasonable–if not a minimal reflection of false rape allegations made to this agency, especially when one considers that a finding of false allegation is totally dependent upon the recantation of the rape charge.

Since this is the internet and that was a long quote, allow me to sum it up

• Police officers are forbidden to use their discretion in deciding whether a complaint is false
• A complaint is determined to be false if, and only if, the complainant admits that no rape occurred

Credibility is important here.  In order for the 2-8% range to be believed, you have to accept the premise that the article’s authors reviewed the relevant research and fairly and accurately distilled the data down to that range. With what you now know about the methodology, ask yourself the following question – Does the view put forth by the authors fairly and accurately represent the facts of the study?

Let’s move on to their second problem with the Kanin study.

This concern is compounded by the fact that the practice of this particular police department was to make a “serious offer to polygraph” all rape complainants and suspects (Kanin, 1994, p. 82). In fact, this practice “has been rejected and, in many cases, outlawed because of its intimidating impact on victims” (Lisak, 2007, p. 6).

I have to take another detour here.  People use a lot of different tactics when they are trying to convince you of things and some are more effective than others.  I don’t know about you, but quoting yourself as authority sits pretty far down on the list of things that convince me. It kind of has that “My point is really great, but I seem to be the only one making it” ring to it. Sometimes it may be a good option, and I’m sure if I keep this blogging thing up long enough I’ll end up doing it a few times myself, but in disputing the Kanin study the authors use three different quotation – and they’re all the same Lisak article.  Oh and just for bonus points, that Lisak article they keep citing? In it Lisak cites Lonsway and Archambault’s 2007 version of False Reports. So I guess that makes it authority squared?

Setting aside the fact their sources for critiques of the Kanin study only seem to come from one of the authors themselves, their complaint is not completely without merit. In the underlying article Lisak notes that when investigators express doubt when interviewing potential victims it can cause feelings of confusion, shame, and self-blame.  I’m not going to dispute that or say that I think the police using a polygraph makes for good policy.  However, I’m not really sure how it disproves the fact that, in this case, 41% of the women admitted they made the allegation up. In fact, Kanin himself deals with this very point in the study

A possible objection to these recantations concerns their validity. Rape recantations could be the result of the complainants’ desire to avoid a “second assault” at the hands of the police. Rather than proceed with the real charge of rape, the argument goes, these women withdrew their accusations to avoid the trauma of police investigation.

Several responses are possible to this type of criticism. First, with very few exceptions, these complainants were suspect at the time of the complaint or within a day or two after charging. These recantations did not follow prolonged periods of investigation and interrogation that would constitute anything approximating a second assault. Second…in those cases where a suspect was identified and interrogated, the facts of the recantation dovetailed with the suspect’s own defense. Last, the policy of this police agency is to apply a statute regarding the false reporting of a felony. After the recant, the complainant is informed that she will be charged with filing a false complaint, punishable by a substantial fine and a jail sentence. In no case, has an effort been made on the part of the complainant to retract the recantation.

Don’t get me wrong, I’m not naive enough to say that there is no such thing as a false confession.  However false confessions don’t just happen – they are made for certain reasons⁶, and I’m not really sure those reasons have much to do with the situations being discussed here.  False confessions made in an attempt to end long long hours of coercive interrogations, or to avoid physical violence make sense.  What doesn’t make sense is the idea that these women admit to making up the the allegations and provide explanations that comport with other facts of the case instead of just getting up and leaving if they felt uncomfortable with the questioning.  It makes even less sense when they are then told they will be charged with false reporting and continue to stick with the the story that they made it all up.

You don’t have to just take my word for it though, we can use some simple math to help us evaluate these claims.  If the 2-8% range is correct, how many of the confessions in the Kanin study would be false? To avoid issues with rounding, let’s multiply all the reported numbers by 1,000 (so total reports goes from 109 to 109,000 and the number of women who confess to making false reports goes from 45 to 45,000).  If we assume that the true false reporting rate is 8%, there would be 8,720 reports that were actually false. That means that 80.6% of the time (36,280 cases), the women were admitting to making the report up and accepting punishment for doing so, despite there genuinely being a rape.  Actually, the number is probably even higher.  Remember that the Kanin study only counts a report as false if the woman admits that it was false.  The above numbers assume that 100% of the women who filed the false reports are caught and admit to doing so.  If only 75% of the those who file false reports confess, than it goes from an 80.6% false confession rate to 85.5%.  The following chart shows the false confession rate based on a variety of actual false reporting rates (down the left) and false reporter confession rates (across the top).

So the question you have to ask yourself is – Does this seem like a plausible range for the the percentage of false confessions in situations where the women are free to leave without confessing, and face fines and jail time if they do?

There is one final thing I find curious about their critique of the Kanin study. If they feel that the use of the polygraph is throwing off the results of the study so much, you would think they would at least mention the addenda to the study that describes a different data set that they later gained access to⁷.  In 1998 Kanin obtained access to the police records of two large Midwestern state universities.  The result?  Over a three year period, 50% of the forcible rape complaints were false.  Once again, only cases that included a recantation of the charge were classified as false, and there was no use of polygraph this time.  Here is what Kanin had to say about that data

In both police agencies, the taking of the complaint and the follow-up investigation was the exclusive responsibility of a ranking female officer. Neither agency employed the polygraph and neither declared the complaint false without a recantation of the charge. Most striking is the patterning of the reasons for the false allegations given by the complainants, a patterning similar to that found for the nonstudent city complainants. Approximately one half (53%) of the false charges were verbalized as serving an alibi function. In every case, consensual sexual involvement led to problems whose solution seemed to be found in the filing of a rape charge. The complaints motivated by revenge, about 44%, were of the same seemingly trivial and spiteful nature as those encountered by the city police agency.

Given the amount of time I have spent defending the Kanin study, you might think that I regard it as some sort of gold standard – I do not.  There are a number of valid complaints about the study.  First of all, the initial study is from a Midwestern city with a population of 70,000 and the universities were Midwestern as well, so you can’t reliably extrapolate these rates as being indicative of the entire country.  Second, the data is fairly dated at this point, with the study being published in 1994 and addenda from 1998, so it doesn’t necessarily give us a lot of information about how things are currently. However, the criticisms that Lonsway et al chose to bring up are not only weak, but significantly misleading. [Update: I have since written more about the flaws in the Kanin study here]

So if that is the data that they don’t like, what kind of data do they think we should be relying on?  Check back soon for How to Lie and Mislead With Rape Statistics: Part 2.

UPDATE: Part 2 is now available.