Key points
- After training, the 0.6B reached 87.5% on whole contracts and 93.7% when given the right excerpts, so finding the clause in the whole contract costs 6.2 points.
- We also tested having the model quote the deciding clause before answering, but this gave no gain, showing 85.6% against 85.9% on the dev set.
- The hand-off reached 89.6%, at 40% of the 8B's compute.
Can a small model match the best published results?
Can a small model review contracts as well as the best published model, whose best score in the ContractNLI paper is 89.2%?
Our trained 8B and our hand-off both came level with it.
Setting up a fair test for contract review
We used the official ContractNLI split of 423, 61 and 123 contracts. We started our work on a laptop with an 8 GB RTX 2070, but because whole contracts can reach 11,641 tokens, we moved to a GPU server with two 24 GB RTX 3090s.
We had to fix a dev score bug where the progress score during training was mixing examples together and reading 3.48 instead of 0.03.
The cost of finding the right clause
We started by looking at the untrained 0.6B model, which scored 46.3% on the 2,091 test questions, each scored once. After training, the 0.6B reached 87.5% on whole contracts and 93.7% when given the right excerpts, so finding the clause in the whole contract costs 6.2 points. When the model quotes the right clause it is right 95% of the time, and when it looks in the wrong place only 31%. Finding the clause is where the next gain is.
Trial and error with distillation
We tried several different ways to improve the 0.6B model using distillation, but many of our initial ideas did not help. We used Unsloth for training and found that QLoRA performed just as well as LoRA, with 88.0% compared to 87.8%. We also tested having the model quote the deciding clause before answering, but this gave no gain, showing 85.6% against 85.9% on the dev set. Attempting to use reasons from an untrained 8B teacher actually made the student model perform worse, dropping it to 85.1%.
The power of a hand-off system
Our hand-off system lets the 0.6B answer first and sends the 24% of questions it is unsure about to a trained 8B. The hand-off reached 89.6%, at 40% of the 8B's compute. This result includes a contradiction F1 of 0.437. By comparison, the 8B alone reached 89.7% with a contradiction F1 of 0.454.
What we learned about size and method
We learned that the method matters more than the size. While a bigger teacher gets you there, it costs more. We found that the 0.6B alone, 87.5% at 0.12 seconds, is the fastest option. For the best balance, we found the hand-off, 89.6% at 0.28 seconds. If cost is no concern, the 8B alone, 89.7% at 0.69 seconds, provides the top accuracy.
The paper's best model is about 0.9B, DeBERTa, trained on another contract dataset first, so it reached 89.2% at about a tenth of our teacher's size.