AI made-up case law is no longer a curiosity. One public database now lists 864 court decisions in which a judge found, or in some cases suspected, AI-invented material in a lawyer's filing. On 9 October a federal judge in New York ordered an attorney to pay nearly $39,000 to the lawyers who caught his. From 1 January 2027, California turns the lessons into statute. The surprising part is what none of these rules do: none of them stop a firm using AI. They make checking it a firm-wide job.
What does AI in law firms mean in 2026?
AI in law firms means using generative AI tools for research, drafting, document review and summarising under the same professional duties as any other work (competence, confidentiality, supervision, candour to the court and reasonable fees), and, in California from January 2027, under a statute that spells those duties out. Adoption is rising fast. Thomson Reuters reports that 41% of law firms said their legal teams were using generative AI in 2026, up from 28% in 2025.
This guide is for the people who run the firm rather than the people who argue the cases: managing partners, chief operating officers, firm administrators and IT leads at firms of 20 to 1,000 people. It covers what the ABA and California now require, what happens when it goes wrong, and how to roll AI out so it doesn't. It isn't legal advice, and your state's own rules and ethics opinions decide what applies to you.
Can lawyers use AI? What ABA Formal Opinion 512 says
Yes. The American Bar Association answered the question in Formal Opinion 512 on 29 July 2024. It doesn't create new duties. It applies the existing Model Rules to generative AI, and it's blunt about where the risks are.
| Duty | What Opinion 512 says it means for AI |
|---|---|
| Competence (Rule 1.1) | Relying on AI output "without an appropriate degree of independent verification or review" could breach the duty of competence. |
| Confidentiality (Rule 1.6) | Before putting information about a client's matter into a self-learning tool, you need the client's informed consent. "General, boiler-plate provisions" in an engagement letter aren't enough. Read each tool's terms of use and privacy policy. |
| Communication (Rule 1.4) | Even where consent isn't needed, consider telling the client. You must disclose your AI use if the client asks or their engagement terms or outside counsel guidelines require it, and consult them when it bears on your fee or when AI output will influence a significant decision. |
| Candour to the court (Rules 3.1, 3.3, 8.4(c)) | Review AI output, "including analysis and citations to authority", before anything goes to a court. "Even an unintentional misstatement to a court can involve a misrepresentation." |
| Supervision (Rules 5.1, 5.3) | Managerial lawyers "must establish clear policies regarding the law firm's permissible use" of AI. Supervising lawyers must make reasonable efforts to see that the lawyers and non-lawyers they oversee comply and are trained. |
| Fees (Rule 1.5) | Hourly billers must bill their actual time. You may not charge a client to learn a tool you'll use regularly. |
Two things stand out for whoever runs the firm. First, the supervision duty doesn't stop with the lawyer who used the tool. It reaches the managers who set policy and every lawyer who supervises others. Second, the confidentiality test turns on where the data goes. A tool that learns from your inputs and can surface them to other users needs informed consent. Opinion 512 warns that this includes other people inside your own firm, such as lawyers behind an ethical wall. Firm-only access isn't enough on its own. That makes choosing the tool a professional-responsibility decision, not only an IT one.
What is California SB 574?
SB 574 is the first state statute written specifically for lawyers' use of generative AI. Governor Newsom signed it on 30 September 2026 as Chapter 858 of the Statutes of 2026, after it passed the Assembly 75 to 0 and the Senate 39 to 0. It takes effect on 1 January 2027.
It does four things that matter to a law firm.
1. No delegation
New section 6068.1 of the Business and Professions Code opens with the line that sets the tone: "An attorney shall not delegate the practice of law to generative artificial intelligence." The statute doesn't define "delegate" or "the practice of law", so where help ends and delegation begins is still open.
2. Three duties whenever an attorney uses generative AI
An attorney who uses generative AI to assist in the practice of law must:
- Keep sensitive data out of open systems. Don't enter confidential, personal identifying or other nonpublic information into an AI system unless access to it is restricted to the attorney and people the attorney has authorised who are bound to keep it confidential. The statute's list of personal identifying information is specific: driver's licence numbers, dates of birth, Social Security numbers, criminal identification numbers, the addresses and phone numbers of parties, victims, witnesses and court personnel, medical or psychiatric information, financial information, account numbers, and anything sealed or deemed confidential.
- Verify and correct. Take reasonable steps to verify the accuracy of AI output, "including, but not limited to, the accuracy of all case and statutory citations", and to correct "any erroneous or hallucinated output in any material used by the attorney."
- Disclose. Disclose the use of generative AI to the court "for all documents submitted to the court", and consider disclosing it when AI helps create content for the public.
3. Every citation personally verified
SB 574 also amends section 128.7 of the Code of Civil Procedure, the rule on what an attorney certifies when presenting papers to a court. From 2027, a brief, pleading, motion or any other paper filed in any court "shall not contain any citations that an attorney responsible for submitting the pleading has not personally verified, including any citation provided by generative artificial intelligence."
Read that carefully. It covers every citation, whether or not AI produced it. The one carve-out is discovery: section 128.7 doesn't apply to discovery disclosures, requests, responses, objections or motions, though your other duties to check still do. And the same section says that, absent exceptional circumstances, "a law firm shall be held jointly responsible for violations committed by its partners, associates, and employees." Sanctions can include non-monetary directives, a penalty paid into court, or the other side's reasonable fees.
4. Arbitrators too
An arbitrator may not delegate any part of their decision-making to a generative AI tool, or rely on AI-generated information from outside the record without telling the parties first and, as far as practical, letting them comment on it. And the Judicial Council must publicly revisit Standard 10.80 of the California Standards of Judicial Administration to reflect developments in generative AI.
How SB 574 compares with Opinion 512
| ABA Formal Opinion 512 | California SB 574 | |
|---|---|---|
| What it is | Ethics guidance on the Model Rules | A statute |
| Who it binds | Persuasive; states adopt their own rules | The attorney duties bind California-licensed attorneys; the citation rule binds anyone filing in a California court |
| When | Since July 2024 | From 1 January 2027 |
| Delegating to AI | Says AI can't replace a lawyer's judgment | Prohibited outright |
| Client data | Informed consent before using self-learning tools | Keep it out of systems not restricted to your team |
| Checking output | "An appropriate degree of independent verification" | Reasonable steps to verify; every filed citation personally verified (discovery papers excepted) |
| Telling the court | Check the court's local rules | Disclose for all documents submitted |
| Firm responsibility | Managers set policy and train | Firm jointly liable for sanctions, absent exceptional circumstances |
What happens when AI-made citations reach a court?
The numbers keep climbing. Damien Charlotin's AI Hallucination Cases database lists 2,184 decisions worldwide in which a court addressed AI-generated material in a filing, as of 10 October 2026. Most are findings of hallucinated content, usually fake citations, though the database also includes some where AI use was alleged but not confirmed. Most involve people representing themselves, but 864 are tagged as involving lawyers.
The penalties have grown with the count. Flycatcher v. Affable Avenue, in the Southern District of New York, shows how far it can go. In August 2025, Judge Katherine Polk Failla was already weighing sanctions over a brief whose citation errors were "apparently the product of AI hallucinations", when defence counsel filed a reply citing a "Himmelstein" case that, as the court confirmed, "does not exist as such". In February 2026, according to a summary by Greenberg Traurig, the court struck the client's submissions and entered default judgment against it, and opened the way for the lawyers who exposed the errors to recover their fees. They acted for a co-defendant, not the other side. The client lost by default because its lawyer didn't check.
Then, on 9 October 2026, Law360 reported that a New York federal judge who had imposed case-terminating sanctions over AI-generated false citations ordered the attorney to pay nearly $39,000 to the lawyers who uncovered them, after he gave them "the runaround" when they tried to collect.
Three lessons carry over to any firm:
- The tool is never the defendant. Whatever invented the citation, the court sanctions the lawyer who filed it.
- The client pays too. A struck brief or a default judgment lands on the client, and then on the firm's relationship with them.
- Repetition is what turns a fine into a lost case. In Flycatcher, the court pointed to a lawyer who "was not dissuaded by court orders or the threat of sanctions". A firm that fixes its process after the first incident doesn't get there.
Verification is the work, not the overhead
Here's the part most AI rollouts at firms get wrong. If every citation has to be personally verified, the time AI saves in drafting doesn't vanish. Part of it moves to checking. A firm that counts the saved drafting hours and ignores the checking has measured the wrong thing, and the checking tends to land on the most senior people, who sign the filings.
Our framework treats this as a guardrail, not a style rule. Its output standard says anything AI helped make is labelled, owned, short and checked before it reaches someone else. For a law firm, "labelled" is now the disclosure duty, "owned" is the signing attorney, and "checked" is the verification the statute describes. The output standard, a 10-minute lesson in our free all-staff course, teaches all four rules. The data map, in module 2 of our leaders' course, sorts information into classes and says which tools each class may go into, which is the question SB 574's confidentiality duty asks.
How to roll out AI in a law firm
Six steps, in order. Each one produces what the next one needs. They work for any firm, and they meet California's rules if your lawyers are licensed there or file in its courts.
01 · Find out what's already in use
Before you write any rules, find out what people are already doing. Lawyers and staff are likely using chatbots on personal accounts, and AI features are switching on inside the research, practice management and document tools you already pay for. The free Shadow AI Audit covers both. The result is your starting list.
02 · Approve tools by where the data goes
For each tool on the list, answer the questions Opinion 512 and SB 574 both turn on. Who can see what you put in? Does the vendor train on it? Can access be limited to the people authorised on each matter, so it respects your ethical walls? Read the terms of use and privacy policy, as Opinion 512 tells every lawyer to, or have someone who has read them sign off. Tools that keep client data within the team authorised on the matter go on the approved list. SB 574 asks for exactly that: access restricted to the attorney and the people they authorise under a duty of confidentiality. Tools that can't do it are limited to public information. Approved tools and the vendor checklist has the full set of questions.
03 · Write the firm's AI policy
Opinion 512 says managerial lawyers must set clear policies. Keep yours short enough that people read it. It should cover:
- which tools are approved, and for which classes of data;
- the no-delegation rule: AI drafts and suggests, and a lawyer decides;
- how output is verified, and who signs that it was;
- when and how AI use is disclosed to courts and clients;
- how AI-assisted time is billed.
The AI Usage Policy generator gives you a first draft to adapt with your professional responsibility counsel.
04 · Make verification a named step with a sign-off
Write down what "checked" means for each kind of output. For a court filing, that's every citation pulled and read: that the case exists, says what you cite it for and is still good law. Put a line in the filing checklist that the responsible attorney initials. If a team can't spare the time to check, it can't use AI for that work yet. That's the honest trade.
05 · Train everyone who touches client work
Opinion 512 puts training on supervising lawyers, and it covers non-lawyers. Paralegals, legal assistants and business staff use these tools too, and they handle the same client information SB 574 protects. Our free course, Everyone's first hour with AI, gives every member of staff the approved tools, the data rules, the output standard and what to do when something goes wrong, with a completion record. For hands-on sessions built around your own matters, see our employee AI training.
06 · Fix the engagement letter and the billing
Under Opinion 512, a generic clause authorising AI doesn't amount to informed consent. Outside California, where a tool's terms mean client information could reach people beyond the matter team, explain the risk and get specific consent, or don't use that tool for that client. In California from 2027, consent isn't an alternative: SB 574 keeps confidential, personal and nonpublic information out of any system not restricted to the authorised team, whatever the client agrees to. Agree how AI-assisted work will be billed before you bill it, and bill hourly work at the time actually spent. Then give the whole programme an owner, a backup and a review date. If you want help with deploying AI across the firm, our guide to AI deployment covers the wider rollout.
What's still open
- What "delegate" means. SB 574 bans delegating the practice of law to AI without defining either term. It isn't yet clear how much drafting a tool can do, or how far an agent can act on its own, before an attorney has delegated.
- How far disclosure reaches. "All documents submitted to the court" could mean a line on every filing or only the ones AI touched. Until courts or the State Bar clarify it, document-by-document disclosure is the cautious reading.
- The courts' own standard. The Judicial Council must revisit Standard 10.80 of the California Standards of Judicial Administration in light of generative AI. What it changes, and when, isn't known yet.
- Other states. California is the first to legislate. Other states rely on ethics opinions and court orders for now, and local rules on AI disclosure differ from court to court.
We'll update this article when California's courts or State Bar issue guidance on SB 574, or when another state passes a similar law. The date at the top of the page shows when we last checked.
Sources
- SB 574, Chapter 858, Statutes of 2026, California Legislature, September 2026: the chaptered text, which we read in full, including Business and Professions Code section 6068.1 and the amendments to Code of Civil Procedure section 128.7.
- Formal Opinion 512: Generative Artificial Intelligence Tools, American Bar Association, July 2024: the duties of competence, confidentiality, candour, supervision and fees as they apply to AI.
- Flycatcher Corp. v. Affable Avenue LLC, order of 12 August 2025, US District Court for the Southern District of New York (via CourtListener): the nonexistent "Himmelstein" citation.
- AI Hallucination Cases database, Damien Charlotin, October 2026: the count of cases, and how many involve lawyers.
- California Enacts Rules Governing Lawyers' Use of Generative AI, Holland & Knight, October 2026: the effective date and the open questions on delegation and disclosure (secondary).
- Court Sanctions Highlight Potential Risks of Using Unchecked AI in Litigation, Greenberg Traurig, March 2026: the February 2026 sanctions in Flycatcher (secondary).
- Atty With False AI Citation Filings Ordered To Pay $39K, Law360, October 2026: the 9 October fee order (secondary).
- How AI is reshaping the legal profession, Thomson Reuters, July 2026: adoption figures from its 2026 AI in Professional Services Report (secondary).