ChatGPT Bot Passes Law School Exams, New Research on AI Tool Reveals

The criticism-spurring ChatGPT bot passed several law school exams, newly released research shows.

“Alone, ChatGPT would be pretty mediocre law student,” lead study author Jonathan H. Choi of the University of Minnesota Law School said of the research, per a report from Reuters on Wednesday. Theoretically, Choi added, the research shows there is potential for a human lawyer to utilize the bot for rough drafts of documents to “make their practice that much more effective.”

Choi collaborated with Kristin E. Hickman (University of Minnesota Twin Cities School of Law), Amy Monahan (University of Minnesota Law School), and Daniel Schwarcz (University of Minnesota Law School) on a paper titled “ChatGPT Goes to Law School.” In it, researchers aimed to determine “how well” such models could do in a law school exam setting without help from humans.

Ultimately, the model managed to pass all four classes (Constitutional Law: Federalism and Separation of Powers, Employee Benefits, Taxation, and Torts) with an average of C+ across all tests. This average, it’s pointed out, would place this “student” on academic probation in a real-world scenario. However, if this level of performance was maintained throughout the duration of law school, this so-called “student” could make it to graduation.

In terms of particular strengths, researchers said that ChatGPT did better with essay questions than it did with multiple choice elements. However, even within the essay realm, the model performed in a “highly uneven” fashion.

Separately, additional research from a Wharton School professor showed that ChatGPT had passed a Master of Business Administration program. Also, CNET was recently revealed (via Futurism) to have used AI to pen a number of articles, although such efforts (thankfully) did not go smoothly.

ChatGPT, understandably, has inspired criticism from a number of experts regarding its potential for nefarious usages. Earlier this week, meanwhile, Microsoft was announced to have extended its collaborative relationship with ChatGPT company OpenAI.