The Lawyer’s Guide to Generative AI: Where It Fits (and Doesn’t) in Legal Practice

In early 2024, LexisNexis conducted a survey with over 500 lawyers between Australia and New Zealand, and 1 in 2 respondents reported that they were already using generative AI tools in their day-to-day operational tasks. Further, a 2023 report by McKinsey Digital estimated that today’s AI technologies could automate tasks which presently occupy up to 60-70% of employees’ time around the globe. Goldman Sachs released a similar report but specific to the legal profession, though its findings, that up to half of all lawyers tasks could be automated by AI, were the subject of some debate.

However, the appropriate and effective use of generative AI in legal practice remains a complex and evolving issue. Misunderstanding AI’s capabilities and limitations can lead to ethical challenges, errors in legal work, and potential negligence. As such, there are important questions to be asked about when it is, and is not, appropriate to rely on AI in legal practice.

• Artificial intelligence can be defined in various ways, such as algorithms or robots that mimic human cognition and perform tasks like analysis and decision-making.
• Machine learning involves giving data to machines without explicit rules and allowing them to discover patterns and generate solutions based on the data.
• Proper terminology is crucial in generating accurate outputs in the context of next word prediction models, as they rely on the relationships between all words in the dataset they were trained on.
• The complexity of predicting the next word increases with a larger number of words, making it challenging to ensure accurate results, particularly for lengthy documents where the desired output may not be achieved due to the intricate calculations involved.
• The RAG (Retrieval-Augmented Generation) model has gained popularity among companies as it utilises embeddings instead of traditional language models, making it more reliable when analysing large documents. By identifying and presenting only relevant information to the language model the RAG model outperforms conventional language models in processing complex text data efficiently.
• Large language models can currently assist with a number of administrative legal tasks, but may be more or less effective with some tasks than others. The frequent release of updated models means that their performance can vary, making it important to evaluate each model individually for specific tasks.
• Many benchmarks used for testing AI models are unreliable and fail to accurately assess reasoning capabilities, often due to contaminated data and lack of proper question formulation.

• Law firms looking to implement generative AI must go beyond simply instructing people to use it, as there is a need to establish responsible practices and ensure compliance. This process involves ongoing training and reinforcement to make verification a habitual practice.
• To get the best out of AI technologies, it is important to try out new models and be persistent in experimenting with different ways of prompting.
• The design element of randomness in language models can result in varied outputs. It is important to carefully consider the terms being used in prompting a language model, to ensure the most accurate outputs.
• With the frequent releases of new language models, it is beneficial to check the release dates and differences between models to determine their suitability for a particular purpose. An older model may be more suitable for a specific use than a newer model.

ICLR Conference Paper (2024) ‘The Reversal Curse: LLMs Trained On “A Is B” Fail To Learn “B Is A”’