But the new benchmark aims to better measure the models’ ability to perform legal tasks in the real world. business logic benchmarkPublished by ScaleAI in November, the leading LLMs were assessed on legal and financial tasks designed by professionals in the field. The study found that there are significant gaps in the reliability of models for professional adoption, with the best performing model scoring only 37% on the hardest legal problems, meaning it met just over a third of the possible points on the assessment criteria. The models often made incorrect legal decisions, and if they reached the correct conclusion, they did so through incomplete or opaque reasoning processes.
“The tools are really not meant to fundamentally replace your lawyer,” says Afra Feza Akyurek, lead author of the paper. “Even though many people think that an LLM has a good understanding of law, it still lags behind.”
This paper builds on other benchmarks measuring model performance on economically valuable tasks. AI Productivity IndexThe data, published by the firm Mercor in September and updated in December, found that the model has “substantial limitations” in performing legal tasks. The best performing model scored 77.9% on legal tasks, meaning it met almost four out of the five evaluation criteria. An early version of the study said a model with such a score could generate substantial economic value in some industries, but in areas where errors are costly, it might not be useful at all.
Professional benchmarks are a huge step forward in evaluating the real-world abilities of LLMs, but they still may not capture what lawyers actually do. “These questions, although more challenging than previous benchmarks, still do not fully reflect the type of subjective, extremely challenging questions lawyers deal with in real life,” says John Choi, a law professor at the University of Washington School of Law. Study On legal benchmark in 2023.
Unlike mathematics or coding, in which LLMs have made significant progress, learning legal reasoning for models can be challenging. Choi says the law deals with messy real-world problems full of ambiguity and subjectivity that often have no right answer. To make matters worse, he says, a lot of legal actions aren’t recorded in ways that can be used to train models. When this happens, documents can span hundreds of pages, scattered across statutes, regulations, and court cases that exist in a complex hierarchy.
But a more fundamental limitation may be that LLMs are not trained to think like lawyers. “Logic models still don’t reason fully about problems the way we humans do,” says Julian Nyarko, a law professor at Stanford Law School. models may be lacking mental models The ability to simulate the world—the ability to simulate a scenario and predict what will happen—and that ability can be at the heart of complex legal reasoning, he says. It is possible to provide training on the current model of LLM predict next word Only takes us so far.
