Canadian Legal Research Manual

While Generative AI can be a useful tool, it should never be your only tool for legal research and writing.

You should always evaluate and assess AI-generated content to ensure it is accurate before relying on it, and then take additional measures to supplement your research.

Use the following steps:

1. Assess system limitations
2. Verify all generated information
3. Compare with other sources
4. Update content for currency
5. Take steps to address bias

It can be difficult to assess AI-generated output without at least some understanding of how the system works. This will help you assess the strengths and limitations of the output. 

One way to start thinking about a system's limitations is to use the 3 layers method. Output is only the last layer of a system but it is the most obvious one—in reality, the input and analysis layers have major effects on the quality of the output.

►See How Does GenAI Work? for more about the 3 layers model.

For example, if the input layer is not focused on Canadian jurisdiction (e.g. includes American content, as is the case with most generic tools), the output may reflect a different jurisdiction or blend information from multiple jurisdictions. 

The following questions can help you reflect on the 3 layers of a genAI tool.

►Source: Christa Bracci & Erica Friesen, Legal Research Online (eCampus Ontario Open Library, 2024).

AI-generated text is not a substitute for reading the primary sources (case law and legislation) on your topic. 

This is true for any secondary source as well, but is especially dangerous for AI-generated content because they can include factually inaccurate information (often called hallucinations). 

Each AI system treats source citation differently—some may provide a link to a case, while others may only reference a style of cause. This can make it difficult to track down the source because of the lack of information needed to verify which case it is referencing. Never assume a source exists if you cannot find it.

For any sources referenced by a genAI system:

• Check that the source actually exists by finding a copy (e.g. finding the full-text of a case in Westlaw, Lexis, or CanLII). 
• Read each source carefully to ensure that the genAI tool has accurately summarized it (e.g. not misrepresented the case or summarized a dissent). 
• Take all steps you would normally take when researching case law or legislation, such as noting up. Don't assume the system will do any traditional legal research steps for you. For instance, it may summarize a case that was later overturned by a higher court or received negative treatment. 

►See Legislation Research Stage and Case Law Research Stage for more information about the steps involved in researching primary sources. 

You should never rely on only AI-generated content to answer a legal research question.

Even if you are able to identify and correct any hallucinations, the results may be incomplete. For example, a genAI tool might mislead you by:

• citing a number of relevant cases but not the leading case on an issue or binding cases from your jurisdiction;
• citing cases but neglecting to mention that there is applicable legislation; or
• answering your question in a narrow sense without alerting you to additional factors that are relevant.

You should always consult additional, human-authored sources to confirm that you have correct information that represents a fullsome approach to your legal research question.

►See the chapters on Secondary Sources, Legislation, and Case Law for detailed information on finding sources.

In Step 1, you assessed the genAI tool's limitations, including the crucial question of how current or up-to-date the input layer's dataset is.

Some genAI tools have a specific knowledge cut off date, such as ChatGPT's various models. This means that any legal answer provided by the system will not necessarily reflect current law.

Other systems, like Lexis+ AI, are hooked up to a broader platform's dataset, which means that they run on the most recent content available on that platform. However, even these systems have currency limitations. For example, Lexis' case databases are updated on a daily basis, but their legislation can often be a week or two behind the state of the law. The result for the researcher is that any legislative summary generated by Lexis+ AI will need to be updated to ensure no amendments have passed since Lexis last updated that statute in their system.

If you know the currency limitations of your genAI tool's input layer, you can simply note up case law and legislation as you would normally in the course of your research. If currency information is not available, do not assume that it is up-to-date information and focus on conducting additional research.

►See Noting Up Case Law and Noting Up Legislation for more information.

Generative AI is necessarily biased in that it reproduces patterns in language that already exist in an underlying dataset. Pattern projection is useful in law because the field relies on precedent, but it can also perpetuate bias in several ways.

Bias has long been a known issue with all types of AI-driven tools. One infamous example is risk assessment software COMPAS, which was found to predict greater recidivism for Black defendants than White defendants.  

►See Julia Angwin et al, "Machine Bias" (23 May 2016) ProPublica for more information on COMPAS. 

Think carefully about the following types of bias that may be present in your output:

Bias can be incredibly challenging to counteract in research of any kind, but awareness and reflectivity will allow you to take steps to counteract any biases with additional sources, such as those representing diverse perspectives.

►See the chapters on Secondary Sources, Legislation, and Case Law for more information on conducting additional research.