I will vote to confirm:
I decided to do a more empirical test of the legal ability of AI. Specifically, I downloaded summaries of all Supreme Court cases decided so far this Term, put them into Claude 3 Opus (Claude’s best version), and asked a few follow-up questions. (Although I used Claude in this work, one can probably get similar results with GPT-4.)
The results were foreign. Claude is fully capable of serving as a High Court Judge at this time. When used as a law clerk, Claude is as easily intelligent and precise as a human clerk, while being superhuman.
Let’s start with the simple thing that I asked Claude to do: judge cases in the Supreme Court. Claude always decides cases correctly. When it gets a case “wrong”—that is, it decides it differently from the way the Supreme Court decided it—its decision is consistently reasonable…
Of the 37 benefit cases so far this time, Claude decided 27 in the same manner as the Supreme Court. In the other 10 (such as Campos-Chaves), I was more influenced by Claude’s analysis than by the Supreme Court. The few cases that Claude got “wrong” weren’t Claude’s fault, per se DeVillier v. Texaswhere the Court imposed a lesser sentence without deciding the question presented.
Although I’ve heard concerns that AI will be “awakened,” Claude moderated the study.
Here’s more from Adam Unikovsky. Many people are still in denial, or not far enough to count as “in denial.”
Source link