How we measure AI contract redlining at Crosby

MapQuest accelerates contract negotiations for our clients. A counterparty sends a draft (an MSA, an NDA, a DPA, etc.) and MapQuest reads it, applies redlines / inline comments against the client’s playbook, and hands it to a barred attorney at Crosby to review and send back. Most of our customers see MapQuest as a senior associate that does the first pass in minutes instead of hours.

AI improves quickly; so does MapQuest. New models ship every few weeks, agentic frameworks are rewritten every few months, and design principles we took for granted in the winter are obsolete by the spring. Several pods of engineers at Crosby work on different layers of the system in parallel: context engineering, OOXML editing, agent infrastructure, etc. Any one of them can cause the system to regress in a way that is invisible to the team shipping the change.

Without a measurement layer, “is this better?” becomes a vibes-based question. Two reasonable engineers could look at five redlined paragraphs, disagree, and ship whichever prompt they prefer.

The problem is that typical evaluation techniques do not transfer cleanly. Exact-match scoring is useless because two correct redlines can differ by a word. String similarity scores reward paraphrases of the wrong clause. Public benchmarks do not exist for this task, and even if they did, they would not reflect the actual distribution of our clients’ deals. We had to build the rubric, the data, and the harness ourselves.

Crosby is a hybrid law firm. Engineers sit next to lawyers, and the lawyers use MapQuest every day to close deals for the clients they advise. This matters more than it sounds. Most AI-for-legal companies have to hire outside counsel to label data or evaluate outputs. We walk across the office.

The first problem was writing down what “good” looks like. A redline has two surfaces we care about: the tracked change (the edit applied to the document) and the comment (the inline explanation, if any). Each can be correct, incorrect, missing, or superfluous. We iterated on a rubric with our attorneys that scores redlines and comments along the same axes they reason about — did MapQuest catch the right issue, did it edit the right span, is the legal substance correct, does the tone match what the lawyer would have sent, is the rationale in the comment defensible, etc.

Once the rubric was created, we labeled redlines and comments from past deals against it. These were not synthetic contracts. They were real MSAs, NDAs, DPAs, and bespoke agreements from real negotiations Crosby closed for our clients, redlined by the same attorneys who handled the live deal. The dataset spans the full range of variables that matter — deal size, contract type, template origin (client paper versus counterparty paper), turn number, business line, transaction side. That breadth is the reason we trust the numbers to generalize.

With a labeled set in hand, we built LLM-as-a-judge scorers to approximate the lawyers’ labels at scale. For each scorer, we measured agreement between the judge and the attorney labels on a held-out set; we iterated the judge prompt until parity was high enough that a change detected by the judge meaningfully predicted a change that the lawyer would flag, and only then trust the judge’s measure of the system’s performance. Our two main production scorers — comment_similarity and redline_similarity — are the result of roughly three rounds of this loop.

Two things make this process work at Crosby. First, the dogfooding loop is genuinely fast: an engineer can notice a case where the judge is wrong, walk over to the lawyer on the deal, get the correct label, and push a new judge prompt the same afternoon. Second, because our lawyers see the system they are evaluating as their own tool, the labeling doesn’t feel like a chore performed for the ML team, but how they improve the product they use every day.

Once the rubric and scorers were stable, we wrapped them in a platform designed to be used by the entire engineering, product, and design org — not just the MapQuest team. The platform has three surfaces, and everyone who ships anything that touches MapQuest uses all three.

Precisely filterable datasets of real-world contracts

The datasets page is where the corpus of labeled, real-world contracts lives. Every deal Crosby has closed contributes documents, which are indexed with rich metadata — client business, counterparty, contract type (MSA, NDA, DPA, and so on), template origin, transaction side, turn number, redlining lawyer, deal date, and more.