Want to listen to the full episode and all our other episodes?
Hearsay allows you to fulfill your legal CPD requirements every year.
Our yearly subscription is only $299/year.
With a yearly subscription, you can access all of our episodes AND every episode we release over the next year.
Statistics in Adjudicative Fact-Finding
Substantive Law| What area(s) of law does this episode consider? | This episode considers the use of statistical evidence in judicial fact-finding; it focuses on evidence law. |
| Why is this topic relevant? | Traditionally, lawyers and the judiciary have been sceptical about the use of statistical evidence to prove material facts. However, statistical evidence – most notably, DNA evidence in criminal trials, and epidemiological evidence in toxic tort cases – is becoming more and more commonplace. As we continue to generate and collect data that is ripe for statistical analysis at a dizzying speed, we can expect this trend to continue. This trend was predicted over a hundred years ago by Oliver Wendell Holmes, Jr. former Associate Justice of the Supreme Court of the United States, who once said that, “For the rational study of the law the blackletter man may be the man of the present, but the man of the future is the man of statistics and the master of economics.” This episode explores bridging the gap between how lawyers view evidence, versus how scientists and statisticians view that very same evidence. After speaking to Nicholas, John-Henry Eversgerd helps us break down and understand statistics from the perspective of an expert. |
| What legislation is considered in this episode?
| Evidence Act 1995 (Cth), specifically: Tendency evidence: the tendency rule is covered in section 97 of the Evidence Act 1995. Tendency evidence refers to evidence that is adduced for the purpose of showing that a person has a tendency to act in a certain way or have a certain state of mind. Coincidence evidence: the coincidence evidence rule is covered in section 98 of the Evidence Act 1995. Coincidence evidence refers to evidence that is adduced to show that it is unlikely that two or more events happened coincidentally having regard to their respective features. |
| Nicholas’ thesis | The study of statistics is concerned with populations, not with individuals. The challenge lies in how to understand the population’s (often referred to as ‘N’) contribution to the probability of an individual (often referred to as ‘P’) in light of all the evidence. The dilemma for courts and the legal profession, as is explored in Nicholas’ thesis, is how to take conclusions about N, the general population, and apply them to P, an individual. |
| What cases are considered in this episode? | Makita (Australia) Pty Ltd v Sprowles (2001) 52 NSWLR 705
Crown v Galli (2001) 127 A Crim R 493
They argued that such evidence was would invite the jury to engage in tendency reasoning.
ALA15 v Minister for Immigration and Border Protection [2015] FCCA 2047
Seltsam v McGuiness (2000) 49 NSWLR 262
|
| What are the practical takeaways? |
|
| David Turner:
1:00 | Hello and welcome to Hearsay, a podcast about Australian laws and lawyers for the Australian legal profession, my name is David Turner. As always, this podcast is proudly supported by Assured Legal Solutions, a boutique commercial law firm making complex simple. Just a quick note before we begin, the episode of Hearsay you’re about to listen to was recorded in the midst of the coronavirus crisis and as a result of social distancing measures we had to conduct this interview over remote technology such as Zoom or Google Meet, the audio quality might be a little different than what you were expecting. Still we think it’s pretty good in the circumstances and we hope you enjoy the episode. In this episode I speak to Nicholas Lennings, Senior Associate at Quinn Emmanuel Erkhart and Sullivan an PhD candidate at the University of Oxford about his research into statistics in adjudicative fact-finding. Nic explains quite a few different statistical concepts that are considered in litigation – to further aid in our understanding of these concepts, we also speak to John Henry Eversgerd, expert valuer and economic and damage specialist at FTI Consulting to help us understand the importance of statistical evidence. We hope you enjoy the episode. Joining me on here today to talk about the use of statistical evidence in legal proceedings is Nicholas Lennings, Senior Associate at Quinn Emmanuel. Nicholas, welcome to Hearsay! |
| Nicholas Lennings: | Thank you for having me. |
| DT: | Now what is the title of your PhD thesis? |
| NL:
2:00 | Well that was somewhat of an evolutionary process in and of itself! When it was first formed, it was ‘Statistics in fact finding’. I later changed it to ‘Statistics in adjudicative in fact finding’ mainly because whenever I spoke about it no one could tell it was a legal thesis. |
| DT: | I see that’s interesting and so you’ve added the word ‘adjudicative’ to kind of highlight that it’s really in the legal realm. |
| NL: | I’m not sure it makes any difference whatsoever, but it felt like a good idea at the time. |
| DT:
3:00
| It’s a really interesting insight that you have there because I think when we’re considering questions of evidence as lawyers, often we place a question about the utility of or admissibility of or usefulness of evidence into one or another category but only that category. So we might say ‘well this is expert evidence and the appropriate test there is Makita. TIP: That test is from Makita (Australia) Pty Ltd v Sprowles a 2001 decision of the NSW Court of Appeal which you may remember from your studies.[1] In that case, the judgment of Judge of Appeal Heydon as his honour then was, said that expert opinion evidence has to be in a field of specialised knowledge to be admissible. His Honour also said that the duty of experts was to “furnish the trier of fact with criteria that enabled evaluation of the validity of the expert’s conclusions.” And it can be used for these purposes, and this is tendency in coincidence evidence and this can be used in these circumstances, for these purposes, but what you’re identifying is that frequently experts are actually drawing conclusions from tendency and coincidence evidence, I suppose, from statistical data. |
| NL:
4:00
5:00
| So, it is a very curious way of putting it. There is actually case law in the NSW Supreme Court where a council tried to argue that statistical evidence was tendency and coincidence evidence for the purposes of the Evidence Act, and that was rejected because it doesn’t fit the form of what those sections are talking about. Tendency and coincidence evidence is talking about what one individual does at one point in time, and then how does that inform the likelihood of them doing a similar thing at a later time? TIP: Tendency and coincidence evidence are dealt with under section 97 and 98 of the Evidence Act 1995 (NSW) respectively. Tendency evidence refers to evidence that’s adduced for the purpose of showing that a person has a tendency to act in a certain or have a certain state of mind. Coincidence evidence refers to evidence that is adduced to show that it is unlikely that two or more events happened coincidentally having regard to their respective features. In each case, that is tendency and coincidence evidence, it is necessary that the person seeking to adduce the evidence give notice of their intention to do so to the other side and that the evidence has significant probative value. Whereas statistical evidence takes, and the aggregate of actions of a reference class or a group of people and tries to extrapolate onto a different person, and so whilst in my thesis I say that they are similar, they’re not the same but certainly there is a commonality of reasoning type. Where you take, what we’ll call the generic proposition, so in a sample of individuals we see a particular effect at a particular rate, what is the probability that the individual matches that probability or, if you’re talking about something that has happened in the past, how do you take that generic proposition and use it in a way that is useful to fact finding about the particular individual. |
| DT: | There’s casuistry to the use of tendency and coincidence evidence in a legal sense and the statistical evidence that an expert is relying on because they’re taking the general case and seeking to draw a conclusion or an insight into the specific case. |
| NL: 6:00
7:00
8:00 | That’s exactly right. The way I describe it is that there’s a statistical syllogism when you’re talking about statistical inference. You will firstly, you establish the general proposition from the statistical evidence about the sample. You then say well because that sample is representative of that population, you infer the existence of the effect you saw in the sample into the population and that’s purely statistical inference that scientists use, you take a sample and you try to make a general conclusion. What lawyers then do, which causes scientists quite a lot of concern, is then say well because you’ve made these conclusions about general populations, we can then say this particular individual also followed that same pathway. There is a diverse literature on how you take that inferential process from science and apply it to law. There’s been various attempts to do so, all of which are highly controversial when they come down to the actual reasoning processes that they use. It’s curious then that when we talk about tendency and coincidence evidence or similar fact evidence, as it’s known at common law, that also is a highly controversial area of evidence, it’s probably been so in the High Court even recently. The reasoning process is almost identical when you’re talking about what a fact finder does in extrapolating from the general proposition to the individual. So here, in tendency and coincidence evidence or let’s focus on tendency evidence, you say that a particular individual did a certain number of things or did one thing, that in turn is used to say that person has a character trait that predisposes them to acting in a certain way. Therefore, on this subsequent occasion, which seems to fit that same mould, we are prepared to conclude that because they did that first set, they had this character trait, which is the general proposition and then from that general proposition we extrapolate to the individual instance of that behaviour. |
| DT:
9:00 | It’s an interesting point you make about the, I suppose, the scientific quest for truth not necessarily sitting quite so well with the application of statistical evidence in a legal context. I suppose that’s because we’re not so much searching for truth in a civil or criminal hearing, as we are searching for evidence that one thing is more or less likely than another. Is that, do you think, the source of this disjunct between, I suppose what a scientist would be comfortable using statistical evidence for and what a trial lawyer would be? |
| NL:
10:00
11:00
12:00
| There is in my view, and it’s not a universally held view by any means, a fundamental disconnect between how lawyers view evidence and how scientists view evidence. I should say that the quest for truth I think between the two or between all of those disciplines is common. Everyone understands what the truth is, well most people would understand that the truth is, well at least on my thesis, an impossibility, in terms of trying to work out what actually happened beyond all doubt, but where the law and the science really differ is the evidence that they say is the best evidence in proving the truth. When you look at the medical profession, there’s a very hierarchical approach to what is good evidence and what he’s bad evidence. It’s referred to as the Evidence Pyramid. TIP: Nick is about to explain the evidence-based medicine pyramid in some more detail in a moment. While he does that – try to imagine a pyramid in your head – with the least reliable evidence being on the bottom, becoming more and more credible and reliable further up the pyramid you go. Medical practitioners in particular will place the least weight on case studies, so individual instances of an event occurring. Those often appear in medical journals when they are novel or new or someone is identifying something for the first time, as well this really interesting thing happened, but in terms of trying to extrapolate that to a population it’s almost impossible statistically because there’s no control. There’s no way of saying that person is representative, we don’t know what the cause of that condition would be, so you would never rely, or it would be very unlikely, to rely on a single case study as being a basis for a particular causal event in a state of nature. As you go up the pyramid, what you see is a more and more controlled environment. So, you’ve got things like, almost the very top, the randomised control trial where you take the thing that you’re trying to test, let’s say it’s a drug. So you take a drug and you give it to a population of people and take a placebo, and you give it to a population of people. If the people with the drug start to respond in the way that you hypothesised they should, and the placebo don’t, then you can say well the drug is doing something over the top of the placebo. In other words, it’s doing something that you wouldn’t otherwise expect a normal reaction and therefore the drug is efficacious, if it’s having the response that you wanted to have. And at the very top, you have the meta-analysis, so you take an aggregate of those sorts of studies and say these studies all point in this one direction and you can do that that statistically, so at that point your sample is actually the studies of this effect, and you can come to a justified conclusion on the basis that we’ve got this huge body of data that’s all pointing in one direction, or it’s all pointing with this particular idea in mind, therefore we are much more comfortable concluding that this effect is real. Now, lawyers look at things like meta-analysis and randomised control trials outside of the medical litigation context where they’re more familiar, with a great deal of scepticism, and, well at least that has certainly been the historical trend, and there’s some pretty good reasons for that. |
| DT: 13:00 | You mentioned earlier a case where statistical evidence was, or the admissibility of statistical evidence was challenged on the basis that it constituted tendency evidence. Can you tell me what case that was? |
| NL:
14:00
15:00
16:00 | There’s a case called ALA 15 v The Minister of Immigration,[2] where a judge of the Federal Circuit Court was accused of, well an application had been brought against that judge on the basis that there was a reasonable apprehension of bias on migration applicants or migration appellants from the refugee review tribunal. The basis for the application was that he had decided, I believe it was something like 99.5 or something higher than that cases in favour of the minister whereas, and evidence was lead in the form of affidavits of these cases and sought to be compared to other judges in the Federal Circuit Court as to their percentage rates, which were a little bit lower. That was presented as statistical evidence. It’s fair to say it’s not strong statistical evidence for a number of reasons which the Full Court of the Federal Court picked up on, what’s curious is that the Full Court of the Federal Court said that evidence was irrelevant. Which to my mind seems to be a bit of a stretch, but they did so on the basis of certain authority in Gallagher and The Vietnam Veterans Tribunal where an allegation of reasonable apprehension of bias was brought against tribunal members,[3] on the basis of the way in which they had decided past cases, and in effect what these authorities are saying is that you can’t look at the past cases of a judge to say, even if they’re slated one way or another, to say well they’re definitely going to find, or they are very likely to find, that way again in the future because you’re not looking at what made the judge find that way in the first place. What underlies all of this was a finding actually by an earlier decision of the Full Court of the Federal Court, that the judge had dismissed a number of applications too early on without giving the applicants an opportunity to put forward their case, and so that was ignored by the Full Court in ALA 15 but you can see the link between that kind of conduct and the reversal of the decision in those first cases, and then when you get to ALA 15 there’s an attempt to say, well basically, if you think about it logically, the conclusion of saying this judge should have recused himself for reasonable apprehension of bias because of the aggregate of the past cases he decided, he would never have been able to decide subsequent cases on this issue, so you can see the policy rationale for why that sort of thing just can’t stand without really good evidence. But to say that the evidence is irrelevant, I think it is a bit of a stretch but then also it’s curious how you can see on the one hand similar evidence being used in the statistical sense, then being extrapolated into the coincidence sense. Slightly different causes of action but even so an almost identical reasoning process for that kind of evidence. |
| DT: | On its own or in the absence of other contextualising evidence, statistical evidence can’t itself fulfil either civil or criminal standard proof. Why is that? |
| NL:
17:00
18:00 | Statistical evidence has been used widely in civil litigation as part of the strand in the cable or link in the chain of proof to an ultimate question of causation for decades, if not longer. We really began to see it more clearly in the toxic tort litigation coming out of the United States and it’s used widely for asbestos cases and almost all forms of personal injury style litigation, where there’s difficult medical questions in issue. There’s a decision of the NSW Court of Appeal called Seltsam v McGuiness,[4] which in my view, correctly sets out the way in which courts should approach statistical evidence in civil cases, which is to say, that statistical evidence is evidence of a possibility and not evidence of a probability. Now what that means for lawyers is we have a standard of proof, which is to say a fact is found to be true if on balance of probabilities the fact finder believes that it is true. So it’s a subjective standard but it has an objective basis. It’s not just my pure caprice, whatever I think goes, I have to be able to argue and justify my conclusion. If you look at the example like arsenic. Arsenic is known to be a deadly toxin to humans. You could, for example, adduce a statistical study showing that arsenic is poisonous and results in a very high number of deaths when it is given to people. You could use that by itself to demonstrate that, this person who gave this other person arsenic caused their death, and I think it would be very difficult for the defendant to rebut that evidence and say, well that evidence isn’t good enough, you haven’t met the standard of proof because it’s such a high effect, the magnitude of the effect is so high. |
| DT: | And certainly, a lot safer than a demonstration, I suppose. |
| NL: | Yes, very much safer than a demonstration. A view in that kind of case, would be difficult. |
| DT: | Yes. |
| NL: 19:00 | Where this becomes a bit more difficult in the criminal law, is that the standard of criminal law is beyond a reasonable doubt. There are probably statistical studies that you could adduce to say beyond a reasonable doubt, this causal relationship between variable X and outcome Y is true. Arsenic might be one of those. But generally speaking, when you’ve got a case that’s laid down there in the sense that this effect is so known that it’s beyond a reasonable doubt that these two things are related to one another, you don’t need statistics at that point. |
| DT: | No, there’s probably judicial notice taken of the deadly effective arsenic at this stage. |
| NL:
20:00
21:00 | Yes, although that is actually why I changed my title to ‘Statistics in Adjudicated Fact Finding’ to get around the idea of how statistical evidence is used for the purposes of traditional notice, but that’s an aside. So, where you’ve got a correlation, a causal relationship between two variables extremely strong, you’re not going to need statistics. You don’t need statistics to prove that when you hit someone with a baseball bat their arm breaks, or where you shoot someone in the head it’s likely to cause them an injury from the bullet. You might want statistics to work out what is the probability of death or what is the probability of severe injury if someone is recklessly firing a gun in the air, but it would be surprising if a judge said, well you haven’t proved your case because you haven’t got epidemiological evidence of the number of people that die after being shot at point blank range. You just don’t need statistics for that kind of thing which is why you would rarely be using statistics by themselves to prove facts in criminal cases because the things that you’re trying to prove with statistics, are things outside of common knowledge. Transmission rates of HIV, for example. Nowadays, we know that HIV is a highly transmissible disease through blood but if you want to know the risks that are attached to that, well that’s outside of common knowledge and so that is where you would use statistics. |
| DT:
| I suppose in a civil context one way to understand it is that the balance of probabilities requires a real persuasion to be felt by the tribunal of fact and I suppose, that however compelling one statistical evidence about a quality or trait held by a general population, one might still not feel a real persuasion about the extent to which an individual in that population holds that trait, but one might very well feel a real persuasion about the characteristics of an illness or a poison or a drug, when observed in that same population. In your thesis you talk about the proof paradox, what is that paradox? |
| NL: 22:00
23:00
24:00 | The proof paradox is a species of naked statistical evidence. It is a reasoning device that tries to find the dividing line between where a fact is objectively likely, versus where it is objectively not likely. To give you the classic example of the proof paradox, it’s called Blue Bus. On an island that is entirely disconnected from everywhere else, there are 100 buses that are owned between 2 bus companies; the blue bus company and the red bus company. The blue bus company owns 51 buses. The red bus company owns 49 buses. A pedestrian is struck and gives evidence that she was hit by a bus but cannot give any evidence of colour. Is it permissible to adduce evidence of the number of buses owned by each company, in attempting to attribute liability to one or the other? So the strict formalist or someone who is fond of mathematics would say, well all other things being equal and in the absence of any other evidence, the probability that it was a blue bus is 51%. The probability that it was a red bus is 49%, ergo if the burden of proof is on balance of probabilities the probabilities favour it being a blue bus. Now this has caused a huge amount of consternation in legal scholarship and to some degree in judicial decisions, because even though superficially it does seem like there’s an attraction to saying it’s the blue bus, there is an immediate concern about whether the weight of that evidence or the probative value of that evidence is actually .51. Should it be something less than .51 because of all the things that we don’t know? We don’t know for example, whether or not the blue bus company had its Christmas party, we don’t know if that particular street was only driven by the red bus company, which would clearly fundamentally change how we view the probabilities. So that’s how the proof paradox has operated in the past and it’s been very controversial and quite useful reasoning device for articulating that dividing line between the burden of persuasion and the balance of probabilities expressed numerically, but if you actually think about it in context, it almost never arises in reality because you don’t have these constrained environments. If you did, then it’s fairly easily solved. |
| DT:
25:00 | One barrier to using statistical evidence is that many lawyers aren’t statisticians and may be unfamiliar with the terminology or language or syntax used to express statistical conclusions and therefore, may not understand its import or the significance of a conclusion that is expressed in that evidence. In your thesis introduction, you talk about the uncertainty in the probative value of statistical evidence being an understanding of N’s contribution to the probability of P in light of all the evidence. To a lay person who has no understanding of the shorthand of statistical studies, how would you describe that conundrum? |
| NL: | The conundrum being that, on the one hand statistical evidence might be very useful but on the other hand it’s very difficult to understand it in terms of how it’s being expressed. |
| DT:
| I might try that again, you say in your thesis, ‘uncertainty lies in how to understand N’s contribution to the probability of P in light of all the evidence’. I suppose I’m asking you to demystify some of the statistical language in that statement, the use of the denominators N and P I suppose. |
| NL: 26:00
27:00
28:00
29:00
30:00
31:00 | Understood. The way I have used terms in my thesis is that lower case ‘n’ is a sample that a statistical study draws from. Uppercase ‘N’ is the population and you use the same letters, one lower case and one upper case, to try to denote that the sample was meant to be drawn from the population and can therefore be inferred to be representative of that population. So you’re trying to see N as one entity because ultimately if your study is good, little n is showing exactly the same thing as what is being shown in big N. P is the individual. So what I’m talking about in that statement is, and really what the heart of the problem for courts, and lawyers in approaching statistical evidence, is that there isn’t a legal architecture for understanding how we take conclusions about N, or the sample and the population together, and extrapolating them to individuals, and I should say, when I say ‘individuals’, individuals is used in the loose sense because you could also be trying to prove things about groups. So then you’ve got the invidious task of saying, well I’ve got this evidence, this statistical evidence, I think it has a robust conclusion for the general proposition. I think it’s relevant in saying something about the individual, but I don’t think it controls it in the sense that the objective and subjective properly meet, so what do I do with it? And that’s where you get into really thorny trouble. The usual way is to take what’s called a frequentist approach, which is effectively to say, well it’s relevant to, it is helpful in how I’m approaching these issues but I’m going to put it together with all the other things and come up with the probability. In science that can be done by simply aggregating the different studies and not times-ing them together, but looking at them in the aggregate and saying, well this looks like it’s the kind of thing that’s happening as a trend. There’s another approach called well which draws heavily on Bayes’ theorem. It’s not necessarily always Bayes’ theorem but using formal methodologies to numerically provide a probability for the event occurring in P. TIP: Now Bayes Theorem comes up quite a bit in the next section, so let’s cover that quickly. Bayes Theorem is named from Reverend Thomas Bayes a pioneer in the field of statistical mathematics and is a way of thinking about probability mathematically and Bayes Theorem can be expressed mathematically and we will be putting the formula up in the show notes. Bayesian probability is an interpretation of probability as the rational expectation or belief in something occurring. Frequentist probability, an alternative which you’ve heard Nick talking about, describes probability differently as its relative frequency over many repeated experiments. When lay people talk about probability, we tend to talk about it in frequentist terms, ‘it’s a one in a million chance,’ for example. Now Bayes Theorem and the mathematical expression of probability make Bayesian inference possible. It’s a bit complicated but it basically means that because the probability of something is defined as your reasonable expectation of it occurring, you can update the probability of something occurring as you get more relevant information. For example, there might be a very low likelihood of a person getting a particular illness or disease based on the likelihood of the population as a whole getting that illness. But as you add that person’s age, genetic predispositions, vital signs and other health conditions into the mix, you can start to say whether it’s more or less likely for that person to contract that illness than the general population. Now Bayes’ theorem has a long and tortured history in litigation. The late Justice Hodgson of NSW Supreme Court and Court of Appeal, wrote extremely eloquently on the use of Bayes’ theorem and is probably some of the best articles around in understanding how it applies, or whether it can apply to the use or to understanding how the probabilities in N inform or can be used to find the probabilities of P. But Bayes’ theorem basically says you take the probabilities, and using a mathematical formula, you seek to determine what is the probability of P from N specifically, which is not what the usual legal approach is, which is to throw it into the mix. But that’s the overall structure of how I say, the objections to statistical evidence can be recorded and can be how courts should approach this idea of ‘well if I’m going to receive statistical evidence through an expert, this is the lens that I should use to test whether I’m going to rely on that evidence.’ |
| DT:
| I’m sure in your survey of the cases you’ll have seen a lot of different applications of statistical evidence, as we discussed earlier, many of those are in a medical context or in a toxic torts context for example, but what are some of the more unusual circumstances where you’ve seen statistical evidence used and perhaps, what are some of the scenarios where it can be used that some of our listeners might not be aware that it might be a useful tool? |
| NL: 32:00
33:00
34:00
35:00
36:00 | Well, I think it was in around about the late 1890s where Justice Holmes of the Supreme Court of the United States said that ‘the future are practitioners of economics and statistics.’ TIP: The Justice Holmes NL is referring to there is the ever quotable Oliver Wendell Holmes, Jr., and the full quote is, “For the rational study of the law, the black-letter man may be the man of the present, but the man of the future is the man of statistics and the master of economics.” Now that’s from a Harvard Law Review paper published by Oliver Wendell Holmes in 1897 so he was certainly ahead of his time there. So even back then, there was a recognition that the use of statistics would increase and it’s fair to say that they have been, certainly since the more intense use of forensic science in criminal cases. There’s a lot more attention that’s paid to statistical evidence and you can get some genuinely bizarre forms of forensic science that come through; e-print evidence, shoe print evidence is probably not quite as bizarre but it certainly can be problematic. Handwriting evidence for example. Now these are often just people making, what appeared to be making direct comparisons between one shoe print and another shoe print but they rely on databases to look at similarities to say, well this type of shoe appears to be similar to this type of shoe repeated over many instances, therefore we are more comfortable in the conclusion that they are similar. In the civil sphere, statistical evidence really became an issue and was used more frequently in the toxic tort litigation where you’re trying to prove causation of personal injuries. So that’s where it certainly had its breath of life in the civil process. Perhaps the critical case, when it comes to the use of bad statistics and unusual statistics, is the decision of People v Collins,[5] in the California Supreme Court. In that case, which by the way is to some extent a precursor to the proof paradoxes, a couple were accused of being involved in an armed robbery. There was actually quite compelling evidence outside of the statistics that could have been used to prosecute the case, to the extent where there were, they had admissions more or less. Nevertheless, the prosecution were concerned about identity and whether or not they could be identified as being at the scene of the crime. What they did is, they tried to calculate the probability that a black man with a beard, a blonde white woman, both of whom were seen in a yellow car would be in that one area of California at that particular time. Now that evidence was quite rightly torn to shreds by the California Supreme Court, not the least of which, it was just fundamentally flawed on the basis of the probabilities and the statistics from which was drawn, also because it was mathematically wrong, in the sense that they simply just timed the probabilities together which they shouldn’t have done because there was nothing to say that those probabilities were necessarily independent, but it was also part of the forerunner to the problem that is known as the prosecutor’s fallacy. So the prosecutor’s fallacy is where you take a particular piece of evidence, often its DNA is perhaps the best form, the most clear form, and you say, well there’s a 5 trillion to one chance that this person’s DNA was found at the scene, to put it simply. |
| DT: | Statistical conclusions are obviously drawn from data and we live in a world where we are collecting more data than ever before. The number of sensors that we carry around with us every day is one of the reasons why we’re collecting so much data. Where do you see statistical evidence being used in the future as we collect more and more data about our everyday activities? |
| NL:
37:00
38:00
39:00 | There’s probably two ways of answering that question. The first is, in relation to civil litigation between companies and between persons, consumers and companies, you can see how these sort of data collection exercises could be used to actually make more robust cases based on what really happens in the world, and could be used as a means of commencing, whether it’s class action lawsuits or individual lawsuits, but it’s certainly a means by which you can see how individuals can really begin to see wrongdoing or can be able to see trends in the population and in how people behave. I think more nefariously where these sorts of data collection exercises could lead to, in terms of legal proceedings, is the use of more and more strain risk assessment. So, assessments of risk are already quite difficult exercises. There are some very well established tools that are used by forensic psychiatrists and forensic psychologists to try and assess the individual propensities of people to reoffend, is the classic one, but you can see how this sort of evidence could be used in less well researched areas, like, for example one’s tendencies of accessing certain websites could be used to say well that person is more likely to be a terrorist or not, or more likely due to engage in certain kinds of distant behaviour and could then be used to bring proceedings on the basis of, well they pose a risk to society. So I suspect that if statistical evidence is going to be used in a problematic form, it’s in that sense of trying to find particular traits about individuals and then aggregating them so trying to find traits in a particular population of people that you are concerned by and then using them to apply to individuals, in a way that is not necessarily robust and not necessarily tested. The classic example being racial profiling. You can take numbers or you can skew samples and you can try and come up with results based on almost any characteristic a person possesses and then say well because they have that characteristic, even though they can’t change it, they’re more likely to commit a crime or they may be more likely to engage in conduct that leads to civil liability. |
| DT: | Nick Lennings, thanks so much for joining us on Hearsay. |
| NL: | You’re welcome and thank you. |
| DT:
40:00
| We’ve just heard from Nic Lennings on the use of statistical evidence, we’re now joined by John-Henry Eversgerd from FTI Consulting, a damages and loss specialist, about how he uses statistical evidence in his role as an expert witness. John-Henry, obviously your discipline is a quantitative one and I’ve been speaking to some of our other guests on Hearsay about the use of statistical evidence, in legal proceedings which is kind of an emerging discipline or an emerging practise. As I understand it there’s a real delicacy to drawing a conclusion from a statistical sample that’s actually relevant for a specific case, you know there’s maybe a conclusion that you can draw from a statistical study but the art of actually saying, well that conclusion applies in this particular case before the court, is something altogether different. You use statistical evidence in applying your expertise. How do you deal with that challenge of applying the general to the specific? |
| JE:
41:00 | Yeah I’ll say a couple of things, so, statistical evidence, when it is developed from studies, when it is developed from something that isn’t specific to the case, it’s often easy to pick holes in it because as you’re suggesting, the analysis isn’t necessarily directly comparable to the case at hand. But quite often, we do statistical analysis on the facts of the case at hand and there, you know as an example we will do in a shareholders class action we will use statistical analysis, regression analysis and what not to calculate the implied loss from misrepresentation of financials or something, some other incorrect information that’s been provided to the market in a shareholder litigation, There, even when there is statistical analysis that’s being performed on the facts that are very specific to the case. |
| DT: 42:00
43:00 | Again going back to a non-legal context, a scientist might say well this you know meta study is the best most reliable evidence because it’s drawn from such a large sample size and so it’s really reliable as a general proposition but in a legal context the general proposition is kind of irrelevant because the specific context might be an outlier right? The general proposition might be true 95% of the time but that doesn’t mean that Court’s going to feel a real persuasion that this case isn’t one of the 5% and so there’s a real challenge even though, increasingly, lawyers and experts and Courts are trying to use the power of data and statistics to draw conclusions. There’s a real challenge in I suppose challenging the way courts and lawyers think about the reliability of evidence to go from well I can draw some limited conclusion from the general proposition even though I can’t be entirely convinced beyond all reasonable doubt that the general proposition applies here and I suppose the question is when you’re looking at statistical evidence how do you draw as close a link as possible between the general proposition and the specific case? |
| JE:
44:00
45:00
46:00 | Yeah that’s a good question. I think you’re right to say that statistical analysis often comes up with an answer that is let’s say 95% statistically significant and the case the facts and circumstances of your particular case, that might be within the 5% that that’s missing from that so I think understanding that and I think preparing for that and if you are relying upon that statistical evidence to make sure you can explain or the expert is explaining you know, why is there a 5% you know why is it 95% and in 100% does that necessarily mean that it’s useless for this analysis? And I think if your expert puts their mind to that I think they can come up with a better explanation for why it either is or isn’t reliable in their particular situation. But with that being said, statistical analysis is often based on you know, theory accepted theory and theories and methodology, again accepted methodologies, that aren’t necessarily fact you know it is just that the experts in that area of specialisation have all agreed that this is the best way to do it but it may not necessarily be perfect, so when I use statistical analysis if I can I will try to cross check it against something else. TIP: John Henry is about to explain what a regression analysis is. But for those of our listeners who haven’t come across the term before it might be easier to hear an example first. In 2004, the Australian Government announced that families having children born after 1 July 2004 would receive a $3,000 “baby bonus”. There was significant discussion around the time of the introduction of this policy in subsequent years as to whether the payment provided an incentive for women to have children and whether the payment had any effect on the Australian birth rate. A number of studies were conducted to look into this, with regression analyses conducted. Regression analysis, to put it very simply, is a way of sorting out and determining which variables impact a particular outcome, and how different variables interact with each other. Let’s put this into context using the baby bonus. Many studies tried to study the impact of the baby bonus, being the dependent variable, the one we are trying to understand. One study, looking at the effect of the bonus in Western Australia, found that the introduction of the baby bonus in 2004 led to increased birth rates for mothers aged 20-24, mothers having their third or fourth child, and mothers living in remote areas. One startling implication of the bonus – which the analysis revealed, was that the bonus greatly impacted the timing of births. The announcement of the bonus saw the delay of over 1000 births from June until July, since only births in July were eligible for the bonus. There may have been many factors which influenced when and how many children were born. But a regression analysis allows the isolation of one variable to determine its individual impact on the result as a whole. |
| JE: 47:00 | As an example, statistical analysis, regression analysis, is used in shareholder class actions and it’s an elegant analysis that estimates what the share price drop might be based on the a certain piece of information being provided and it’s based on taking a look at history and taking a look at what we would see similar pieces of information being reported, you know what impact that had the past but it doesn’t necessarily prove that on that day, it would have necessarily had the same the same impact. A way to cross check that is to perform more of a bottom up analysis and say, okay that piece of information – what would a rational investor do with that piece of information? How would the decision-making process and how would a rational investor take that and put it into their valuation models, as an example, and how would they price that security differently and if you have the statistical analysis that shows, here’s the loss based on statistics and based on history, but it is also a valuation analysis that tries to attack it from a more logical perspective, using both of them particularly if they support each other is useful. |
| DT: 48:00 | Yeah a probabilistic analysis isn’t permissible to try and ascertain past events that to try and work out whether something really happened or not it’s not acceptable for a court to say well how probable is it based on a quantitative analysis but that it’s perfectly acceptable to establish whether or not something was likely to occur in the future or is likely to occur in the future or hypothetically in another world in a ‘but for’ scenario was likely to happen. In that hypothetical or future context it’s perfectly acceptable to use a quantitative analysis we’ve spoken a bit about the use of regression analysis in the context of a shareholder class action. Maybe you could tell our listeners in kind of lay terms, what a regression analysis is and how you use that to predict a future quantitative result? |
| JE: 49:00
50:00
51:00 | Sure, a regression analysis is a mathematical and economic analysis that basically looks at past events and tries to predict future events based on what we’ve seen in the past. So, one simple way to think about it is, okay if I want to predict what’s the share price is going to do on a particular day, I know from statistical analysis, I can tell from statistical analysis, that if the whole market goes up or goes down that the particular share of stock that I’m analysing is probably going to go up, is probably going to go down in a similar direction to the market as a whole but it may not go exactly in line. And a regression analysis will use the movement in the market as one indicator – okay I know that every time or on average this share of stock will increase in value about 10% for every time the market moves 20%. You know that’s one step in quantifying what I think that share stock might do in the future and what you do is the regression analysis also looks at other factors, maybe okay, not only am I going to look at the overall market to see if the share of stock kind of moves along with the market, I’m going to look at competitors. You know the top ten competitors in the market, I know that their shares, every time they go up, the stock that I am analysing that might go up a lot closer in line with the competitors than it does with the overall market. I’ll say okay every time those go up, my share of stock that I’m analysing will go up you know 95% of the movement of the other shares of stock. So how it’s used in shareholder class actions is the question is usually a piece of information, an impairment, or missing earnings, or something like that that’s reported and the claim is often that ‘oh, we should have known that a year ago, they should have told us that six months or a year ago.’ And the expert has to figure out okay what would have happened if the directors actually did report that piece of information when it is being alleged that they should have, would the share stock have gone down exactly the same amount as the share of stock actually did go down when the information was reported allegedly late? It’s not necessarily cut and dry and that’s why this statistical regression analysis is often used to calculate that. |
| DT: 52:00
53:00 | You’ve been listening to the Hearsay Podcast. I’d like to thank Nic Lennings, Senior Associate at Quinn Emmanuel Erkhart and Sullivan and PhD candidate at the University of Oxford and John Henry Eversgerd, expert valuer an economic and damage specialist at FTI Consulting for joining us on the show. If you liked this episode about statistical evidence, listen to my interview with John-Henry Eversgerd about briefing experts. Or for another litigation related episode, listen to my interview with Talitha Fishburn, barrister at Wardell Chambers, about advocacy in digital courtrooms. If you’re an Australian legal practitioner you can claim one continuing professional development point for listening to this episode. Now whether an activity entitles you to claim a CPD point is self-assessed but we suggest that this episode constitutes an activity in the substantive law or professional skills fields. If you’ve claimed five CPD points for audio content already this CPD year you may need to access our multimedia content to claim further points from listening to Hearsay. Visit htlp.com.au for more information on claiming and tracking your points on our platform. This episode was produced by Tim Edmeades, and researched by Kirti Kumar, Araceli Robledo is our resident marketing guru and I’m David Turner. Hearsay is the brainchild of Nicola Cosgrove and Chris Cruikshank, the cofounders of Assured Legal Solutions, making complex simple. You can find all of our episodes as well as summary papers, transcripts, quizzes and more at htlp.com.au. |
[1] Makita v Sprowles (2001) 52 NSWLR 705.
[2] ALA 15 v The Minister of Immigration [2015] FCCA 2047.
[3] Vietnam Veterans Association of Australia v Gallagher (1994) 52 FCR 34.
[4] Seltsam v McGuiness (2000) 49 NSWLR 262.
[5] People v Collins, 68 Cal 2d 319 (Cal Rptr 1968).
You must be a subscriber to access this content.