Every contract software vendor now has AI. What that means varies enormously, from a model that politely proposes a date to a chat window that states a wrong number with total confidence.
This article is about that difference: what language models are genuinely good at in contract management, where they predictably fail, and how to spot it during a demo.
What AI is good at
Understanding text that follows fixed patterns. The counterparty, the start date, the contract type: these appear in comparable phrasing across comparable documents. Models score high there.
Summarising what is already there. Reducing a long contract to its core commitments works well, as long as the model only summarises and adds nothing.
Turning language into a query. Translating "which contracts expire next quarter" into a filter on end date is exactly what language models are built for.
Translating and rephrasing. Making an English clause understandable to someone without legal training works fine.
Where AI systematically fails
Counting and calculating. This is the important one. A language model predicts words; it does not count. Ask "how many contracts expire within 90 days" and let the model count, and you get a plausible number that may well be wrong. And it sounds equally convincing either way.
Information that is not there. If a contract refers to general terms you did not upload, no model can know the notice period. A model that states one anyway is inventing it.
Legal judgement. "Is this clause favourable to us" is not an extraction question but a weighing of risk, negotiating position and context. That needs a human.
Context across turns. A follow-up like "and last year's?" requires the system to remember what you were discussing. In practice this often breaks, and the result is silently an empty or wrong answer.
The most dangerous answer is not "I don't know". It is a wrong number in a confident sentence. A system that takes its figures from the language model rather than the database will produce that kind of answer sooner or later.
The architectural choice that matters
There are broadly two ways to build AI into contract software.
The model searches for itself. You ask something, the model gets access to the data and decides what to retrieve and how to calculate. Flexible, and unreliable the moment numbers are involved.
The model only phrases the answer. The question is first translated into a fixed, verifiable database query. The result of that goes to the model, which turns it into readable prose, but calculates nothing itself.
The second approach is less impressive in a demo and considerably more reliable in use. That is how NORA works: every count comes from a deterministic query, and the answer is checked against that figure before you see it.
Questions to ask a vendor
- Where does the number come from when I ask how many contracts expire?
- What happens when the information is not in the document?
- Can I see which passage a suggestion is based on?
- What does the system do when unsure: guess, or offer options?
- Are my contracts used to train models?
- Where are they processed, and is that in the data processing agreement?
- Can I turn the AI features off if I want to?
The first question matters most and is the easiest to test. A vendor who explains that the figure comes from a fixed query has thought about it. A vendor who says "the AI looks at your data" has not.
Questions five and six are legal rather than technical. If an AI provider processes personal data from your contracts, that is a sub-processing arrangement that belongs in your data processing agreement.
A realistic expectation
In contract management, AI removes most of the work in entering data, not in deciding.
When building a register it saves hours: the model does the first pass, you verify. More on that in extracting contract data automatically with AI.
When using that register it saves search time: a question in plain language instead of setting three filters.
What it does not do: decide whether to renew this contract, whether the terms are acceptable, or who is responsible. Those are the decisions that matter, and they stay yours.
Evaluate AI features on the messiest contracts you have, not the tidy ones. A scanned PDF from 2019 with handwritten notes shows you the limits in one go.
Further reading
Frequently asked questions
- What is AI good at in contract management?
- Recognising data that follows fixed patterns, such as counterparty and start date; summarising; turning a plain-language question into a query; and rephrasing clauses for non-lawyers.
- Where does AI fail in contract management?
- At calculating and counting, at information not in the document, at legal judgement and at context across several questions. The most dangerous outcome is a wrong number stated with confidence.
- How do you recognise reliable AI in contract software?
- Ask where the number comes from when you ask how many contracts are expiring. If it comes from a fixed database query, it's reliable; if the language model counts, it isn't. Also ask whether you can see which passage a suggestion is based on.
