Grasping Causation: A Data Science Explanation of Causal Inference and the Role of Counterfactuals

The legal industry often relies on experts to assist in addressing causal questions, who may rely on simple before-and-after comparisons or correlations when refuting causal claims. Being able to effectively identify and assess the assumptions at hand when dealing with causal inferences, will place lawyers in a better position to both advocate for their clients and fulfil their duty to the court. As such, developing a data driven  understanding of causation is an immensely useful tool to add to your litigation toolbelt.

Uniform Civil Procedure Rules 2005 (NSW)

• The document discusses a case where the defendant was accused of multiple robberies. The main issue was the admissibility of tendency and coincidence evidence under ss 97(1)(a), 98(1)(a), and 101(2) of the Evidence Act 1995 (NSW). The court had to determine whether the similarities across the events, such as the description of the robber, the getaway vehicle, and the method of robbery, were distinct enough to provide significant probative value. The court found that when these features were considered together, they demonstrated substantial and relevant similarity across the events. The court concluded that the probative value of the evidence substantially outweighed any prejudicial effect it may have had on the defendant.

R v Milenkovic (2005) 158 A Crim R 4

• The accused was charged with the armed robbery of a Westpac Bank, and the Crown presented evidence linking him to a subsequent, similar robbery. Both robberies involved men dressed in dark clothing, armed with a shotgun and sledgehammers, and driving stolen vehicles. A key point was the use of the same changeover vehicle, owned by the father of one of the other men involved, in both robberies. DNA evidence connected the accused to the second robbery. Despite these similarities, the trial judge dismissed the connections as typical of armed robberies, and the Court of Criminal Appeal upheld the decision, finding that while the shared changeover vehicle gave the evidence some probative value, it was not significant enough to be deemed statistically meaningful in proving the accused’s involvement.

• Causal inference is the study of cause and effect relationships among variables and involves two main aspects: discovery, which focuses on identifying causal relationships between variables, and measurement of causal effects, which examines the size of causal effects.
• To determine a true causal effect, one would need to compare outcomes in different scenarios but whilst holding all other factors constant.
• The human brain is a predictive machine that relies on pattern recognition, leading to potential biases in causal inference.
• Counterfactual analysis is essential for establishing causal inference as data without a counterfactual can only provide observational insights. It is necessary to have a counterfactual to make better causal claims.
• Regardless of the assignment, an expert must understand the nature of the question being asked of them, especially if it is causal. It is crucial for the expert to acknowledge and address causal questions appropriately rather than ignoring them.
• Randomisation in creating groups for a study ensures balance not only in observable characteristics like gender and age but also in hidden factors like preferences and attitudes. This balancing act through randomisation leads to the elimination of potential influences, making it an efficient and effective method for achieving causal inference.
• In litigation, most data is backward-looking and observational, with some exceptions in areas like intellectual property or misleading advertising where experiments can be conducted. The key is often considered as finding situations with random events, called natural experiments, where the assignment of a treatment is external or random.
• Various causal inference techniques exist but the choice of technique depends on the specific circumstances of the case and the available data structure.

• It is essential that practitioners distinguish between correlation and causation. Where there are limits or invisible data at play, practitioners should address these as potential influences on any assertions made.
• When conducting research or analysis, it is essential to control for numerous factors to ensure the validity and reliability of the results. This requires a significant amount of variability in the data being used, as having diverse data points allows for a more comprehensive understanding of the phenomenon under study.
• It is important to ask questions early on in the research process, considering the available data sources and their potential value for exploration as a strong form of evidence.
• Practitioners should map out the causal structure by identifying variables and influences and then envisioning ideal settings for causal inference. By proactively thinking about data early on in the case and considering all possible variables and scenarios, one can prevent overlooking crucial information that could enhance the analysis and decision-making process.
• Lawyers should be conscious of biases and not projecting aspirations onto data when analysing it.
• Learning the basics of data analytics can help in asking the right questions, and ultimately making stronger claims.

Seven Up! (1964)

Council of Chief Justices Australia and New Zealand, ‘Harmonised Expert Witnesses Code of Conduct’ (2024)

Today on Hearsay, we’re joined by Hans Weemaes, Head of Economics and Data Analytics at Vincents. Now, Hans recently co-authored a paper entitled Causation, A Data Science Perspective for Lawyers – really incredible paper which delves into the intricacies of dealing with causal inferences and the role of counterfactuals.

With his extensive experience and expertise in economics and data analytics, Hans can provide a wealth of insight into these critical aspects of litigation. Hans, thank you so much for joining me today on Hearsay.

TIP: So I just mentioned a case involving Colgate. I was doing that off the top of my head. In that one, Procter & Gamble, the maker of Oral-B, filed a lawsuit against Colgate-Palmolive in the Federal Court, accusing the company of misleading and deceptive advertising for its Optic White Renewal Toothpaste.

The lawsuit challenged Colgate’s claim that their toothpaste removes 10 years of yellow stains, alleging that this statement lacks reliable scientific support. Procter & Gamble sought a court declaration that Colgate’s advertising breached Australian Consumer Law.

Oral-B argued that Colgate’s claims were damaging Oral-B’s sales, citing Woolworth’s data that showed that customers who bought the new Colgate product had previously purchased Oral-B or non-Colgate items. This is the causal inference exercise that I was discussing. Oral-B noted that Colgate had aggressively marketed the toothpaste across various platforms, including free to air television, with unusually high advertising spending between March and May.

In response, Colgate dismissed the Woolworths data as uninformative or without context, suggesting that factors like pricing or promotions at individual retailers could influence consumer choices.

TIP: So Hans just mentioned that humans have developed these pattern-recognising techniques that can occur very quickly in the human brain, but sometimes trick us into being convinced of things that aren’t true. We talk about these heuristics, these mental shortcuts that our brains sometimes take in the context of behavioural science and how we can harness this aspect of our psychology to make the law more effective in practice in episode 66 with Dr Alex Gyani called Don’t Shove – Nudge: Promoting Better Choices in the Legal System.

If I could just add one more thing, perhaps a different way of couching correlation versus causation and I think this is quite important to mention. The task of coming up with a measure of causal relationship is an order of magnitude greater than the rigour that’s required for coming up with a correlated relationship, and I like to think of it almost in a negative versus positive sense. So, in order to come up with a correlation, I can apply a statistical measure of how they move together, and I’m done. I can conclude whether they’re correlated, how strongly they’re correlated. But when it comes to causation, and I think this is the critical point, is that in order to come up with a causal effect, I have to eliminate every other influence that’s out there. It turns the problem on its head. And so that’s what I mean in terms of thinking about causation in a negative sense, that I’ve got to eliminate everything else and we’ll expand on that a bit later, but that is a key difference between the two concepts.

TIP: So Hans has just mentioned longitudinal data. For any listeners who might be unfamiliar with that term, longitudinal data, we just mean a collection of repeated observations over a long period of time. So in fields like medicine, social sciences, economics, longitudinal data is often used to study the long term effects of interventions or programs, like the progression of diseases or the impact of policy changes.

If anyone remembers that program called ‘Seven Up!’ that looked at people from the age of seven throughout their lives. That’s a great example of some longitudinal data over the life of a set of subjects.

TIP: Coincidence evidence is dealt with under Section 98 of the Evidence Act in NSW and equivalent provisions in the Evidence Act in the Commonwealth and other uniform law jurisdictions. Coincidence evidence means evidence that’s adduced to show that it’s unlikely that two or more events happened coincidentally, having regard to the respective features of each incident.

For example, if the victim of a hit and run recalls the vehicle in question was a blue and pink striped bus, and on a small island where the incident occurred, the alleged perpetrator also imported and reportedly drove around in a pink and blue striped bus, the chances that those events were not related is incredibly small, even though there’s no direct evidence that it was the perpetrator’s pink and blue striped bus that was involved in the incident.

Another example might be in a murder case where a person is charged with murdering their spouse, that that person has been married before several times to spouses all of whom have died in suspicious circumstances. It’s necessary that the person seeking to adduce coincidence evidence give notice to the other party of their intention to do so, you can’t do it by surprise, and that the evidence has significant probative value, something that a judge needs to make a determination about.

Now experts often draw conclusions from statistical data by extrapolating general patterns observed in groups to individual cases. This process involves establishing a general proposition based on a sample and inferring that that applies to the broader population and to the individual case. Now what lawyers then do, which causes scientists a bit of concern, is to then say that because a certain fact is true about the general population, then we can say that this and that particular individual in this particular individual case also followed that same path, or that that certain fact applies to that individual.

Even if individual similarities across multiple events could be dismissed as common, the key question is whether the combination of these similar features is distinct enough to provide significant probative value, a question that was highlighted in the case of R v Mason (2003) 140 A Crim R 274. One example of where a set of facts was not coincidental enough to be admitted is the R v Milenkovic (2005) 158 A Crim R 4.

In that case, the accused was charged with the armed robbery of a Westpac bank, and the Crown presented evidence linking him to a subsequent similar robbery. Both robberies involved men dressed in dark clothing, armed with a shotgun and sledgehammers and driving stolen vehicles.

A key point in the evidence was the use of the same changeover vehicle, owned by the father of one of the other men involved, in both robberies. DNA evidence connected the accused to the second robbery, but not the first. Despite these similarities, the trial judge dismissed the connections as typical of armed robberies.

Lots of armed robbers wear dark clothing, lots use shotguns and other weapons, lots drive stolen vehicles, and the Court of Criminal Appeal upheld that decision, finding that while the shared changeover vehicle gave the evidence some probative value, it wasn’t significant enough to be deemed statistically meaningful in proving the accused involvement.

Now we talk about this vexed question of using statistical data in individual cases in an earlier episode of Hearsay with Nicholas Lennings and John-Henry Eversgerd. That’s episode 14 called, fittingly enough, Statistics in Adjudicative Fact Finding. Go check out that one after you finish this one.

TIP: Now some of our listeners will already be familiar with the concept of a clean witness and a dirty witness when it comes to expert evidence, but colloquially, a clean witness is someone who’s offering unbiased expert opinions that are admissible to help the court or tribunal understand a specific issue. This person is called to give evidence in the party’s case. On the other hand, a dirty expert witness works directly for one of the parties in a dispute but they aren’t actually called to give evidence. The instruction that they’ve been given, the material with which they’ve been briefed… Well, it might bias them against giving evidence that’s admissible or probative or persuasive in a court hearing, but the dirty expert witness’ job isn’t to give evidence in court. No, it’s to give advice to a party and its legal advisors on how to strategically build and present their case and reduce expert testimony from the clean witness. They’re more in the nature of a litigation consultant, part of the litigation team, rather than an impartial expert witness who’s abiding by the Expert Witness Code of Conduct.

Tell me about the importance of the expert witness’ discipline, training, background in drawing causal inferences, measuring causal effects?

TIP: The specific professional and ethical obligations of expert witnesses vary between jurisdictions in Australia. For each jurisdiction, these obligations can be found both in the court rules and the practice notes of that jurisdiction. In 2015, the Council of Chief Justices of Australia and New Zealand approved a harmonised Expert Witness Code of Conduct.

And today, this harmonised code has been adopted in the Supreme courts of New South Wales, Victoria, Tasmania, the ACT, and the Northern Territory, as well as the Federal Court of Australia. So it’s Queensland, WA, and South Australia who are missing out. NSW listeners, the harmonised code appears in Schedule 7 of the NSW Uniform Civil Procedure Rules. The rules state that an expert’s report and oral testimony can’t be admitted as evidence unless the expert first confirms that they’ve read the Code of Conduct and agree to follow it. That Code of Conduct mandates a general duty to the court, a duty to comply with court orders, a duty to collaborate with other expert witnesses where possible, as well as specific requirements for the content of the expert’s report – things like enumerating the assumptions on which they’ve relied. For listeners outside of New South Wales, we’ll include a link to the show notes to a helpful summary provided by the Council of Chief Justices comparing the rules in each jurisdiction.

And by the way, if you’re interested in learning more about the ethical obligations of experts, you can check out episode 10 of Hearsay with John-Henry Eversgerd called How to Be an Expert at Briefing Experts.

If you’re an Australian legal practitioner, you can claim one continuing professional development point for listening to this episode. Whether an activity entitles you to claim a CPD unit is self assessed, as you know, but we suggest this episode entitles you to claim a professional skills point. For more information on claiming and tracking your points on Hearsay, go to our website.

Hearsay the Legal Podcast is brought to you by Lext Australia, a legal technology company that makes the law easier to access and easier to practise, and that includes your CPD.

I’d like to ask you a favour, listeners. If you like Hearsay the Legal Podcast, please leave us a Google review. It helps other listeners to find us, and that keeps us in business.

Thanks for listening, and I’ll see you on the next episode of Hearsay.