AI in Judicial EducationExploratory Project
The process

From a question to a finished set of materials, in six steps

This is how the evidence-rulings example was made. Each step shows what the AI did, what the author decided, and a link to prompts you can adapt. The steps work for any topic where you have a lot of documents and a short time to teach.

1

Start with a question worth teaching

We asked: where do trial-court evidence rulings get appealed, and where do they actually get reversed? A good teaching question is specific, answerable from public sources, and interesting to your audience.

AI didHelped sharpen the question, listed ways a skeptic would attack it, and proposed what to measure: raised on appeal versus actually reversed.
The author didChose the audience and purpose, and decided what would count as an answer.
2

Gather the material, and count what you found

We collected the relevant appellate opinions for one year (July 1, 2025 to June 30, 2026), one month at a time.

AI didSearched month by month, listed every candidate before reading any, and reported “read X of Y found” each time.
The author didSet the time period and scope.
A mistake we caughtOne early pass covered only a fraction of a busy month but looked complete. The fix became a rule: make the AI report how many it found versus how many it actually processed, every time.
3

Break it down into a structure

Each opinion became one row in a table with the same columns: which rule was at issue, whether the court found error, whether the error mattered, and what happened.

AI didFilled in a fixed set of categories for each opinion from the text it retrieved (July to September 2025 only partly), run on full-text opinions from July 1, 2025 to June 30, 2026, then ran a second pass without seeing the first. That is a consistency check; no second-pass data is published, so it is not proof.
The author didApproved the categories before the AI started, and froze them so they couldn’t drift.
Agreement is not proofTwo AI passes that agree are consistent, not necessarily right. They used the same tools and tend to make the same mistakes.
4

Audit before you show anyone

Before building anything pretty, we asked a separate AI session to act as a skeptical auditor.

AI didRecomputed every number from the raw table and compared three of the eight reversals and one rule amendment with the courts’ own summaries; the other claims were not checked against sources. A second, primary-source audit on October 23, 2026 (twelve AI agents working from the court’s own opinions and the official Rules of Evidence) opened all eight reversals, 12 random criminal opinions and 14 other opinions, and compared all 63 rules word for word.
The author didDecided which findings to fix. No person has yet read every record; that read-through is still to be done.

The audits confirmed the arithmetic, and they found real problems in the coding, three truncated rule texts and several overstated claims. See what went wrong below.

5

Show it, so people can explore instead of just read

The result became an interactive heat map: every opinion is one mark, grouped by rule. Hover for a preview, click for details and the official court document.

AI didDesigned and built the page, built a page for each of the 63 rules from the rule text as supplied (checked against rules.incourts.gov; unofficial copy, confirm there), and linked every case to the rule it cites and to the court’s own opinion.
The author didDecided what story the picture should tell. Automated checks, not a person, tested the page with a keyboard and an accessibility tool; no screen-reader test has been done.

Design choices worth copying: one mark per item so nothing is hidden in an average; shapes and colors so it works without color vision; a plain table alternative; and links back to primary sources everywhere.

6

Package it for the room

The same data became a short discussion card, a poster, a field guide, a “how we know” sheet, and slide decks for 5, 10 or 15 minutes.

AI didDrafted each format from the same audited screening table, so the numbers match everywhere, and produced print-ready PDFs.
The author didDecided what to include and what to cut. The figures come from the table by code. The table has had AI-assisted screening and an AI primary-source audit of a sample; no person has yet read every record behind it.

See the handouts and slides →

What went wrong, which is the most useful part

Two audits caught statements in our own materials that were stronger than the data supported. Here are the headline ones. If you take one habit from this site, take this: ask for the audit.

The claimWhat the data showedThe fix
“All eight reversals are published opinions.”Seven are. One was an unpublished memorandum decision.Say “seven of eight.”
“Reversals never exceeded about 11% in any month.”March reached 16.7% (2 of 12). Months only hold 10 to 20 opinions each.Report the real range, 0% to 16.7%, and note the small counts.
“A second, independent pass confirmed the results.”The second pass used the same tools and model, so it shows consistency, not independence, and no second-pass data is published.Call it a consistency check, not proof, and plan a read-through by a person.
“4.5% of rulings are reversed.”It is 4.5% of the 176 opinions found, not the chance any trial ruling is reversed. Rulings never appealed aren’t counted. The interval (2.3% to 8.7%) is a Wilson score interval.Say “share of coded opinions,” with the interval.
“Reversed in fewer than 1 in 20.”That is a point estimate. 8 of 176 is about 1 in 22, and the upper end of the interval is 8.7%.Say “8 of 176 opinions, about 1 in 22, with the interval.”
(Unstated) Every field was fully filled in.49 records had placeholder values in some fields because that quarter was coded in a lighter first pass.Label those fields “not coded” and analyze only the 127 complete records.
“Every rule page is the official text.”Three rules (402, 604 and 801(c)) had lost text when the pages were built. The second audit found it by comparing every rule with the official site.Restore the text, say “as supplied,” and link the official page.
“Checked,” “verified,” “every figure on every page.”The audit verified arithmetic, not coding. In a random sample of 12 criminal opinions, two outcome codes were wrong.Say what was checked and what was not, and label the rest screening-grade.
Reporter citations printed as fact.None of the N.E.3d citations has been checked against a primary source, and some strings are shared by different cases.Label every citation “unverified” and lead with docket, date and the court’s link.

Where AI helps, and where you must

AI is strong at

  • Reading many long documents quickly, as a first pass
  • Sorting them into categories you define
  • Drafting handouts, slides, captions and speaker notes
  • Building a visual or interactive page from a table
  • Recomputing totals with code you can inspect, and flagging claims that don’t match the table

You must still

  • Decide the question, audience and categories
  • Verify anything that will be cited or relied on
  • Read the records behind the headline numbers
  • Keep confidential or personal information out of the tool
  • Say plainly what is and isn’t verified