How to choose an AI model for working with your documents

Rankings are one thing. What about your proposal?

One model tops a ranking; a friend recommends another. You just need to organize a proposal without changing its price. Let’s start there: choose a document whose scope and numbers you know well, then give it to both models. It is easier to spot an invented delivery date when you know nobody agreed on one.

For the first trial, gather a few ordinary tasks, a few harder ones and two with missing information. Meeting notes or spreadsheet exports work too. For now, we want to get our bearings. A choice for an important deployment will later need a wider set of real users’ tasks.

What exactly earns a point?

For meeting summaries, we care about deadlines, task owners and places where no decision was made. For proposals, it is prices and scope. Write those conditions down before reading the answers. An elegant paragraph can distract from a promise that never appeared in the material.

Separately, define errors that disqualify a result. A wrong delivery date is hard to treat like an overlong introduction. Anthropic’s material on agent evaluations also emphasizes concrete tasks and clear success criteria. Below is a simple comparison card.

Task: extract agreements from a meeting.
Pass criteria:
- all deadlines match the notes;
- each person has only the tasks assigned to them;
- missing decisions are described as missing;
- no information is added from outside the material.
Rating: pass / needs a minor correction / incorrect.
Record the time spent correcting it manually and the specific error.

The same material, similar conditions

Both models receive the same material and instructions, with similar tool access. Record their exact names and the date. If possible, replace model names with letters before judging. It is easier to award a point to the answer that helps without cheering for a favorite brand.

Repeat the harder tasks. If a model preserves the agreements once but invents a deadline the next time, that difference is a result too. For each answer, record how long it took to make it usable. Include checking sources: that takes time as well.

How much work did the answer leave you?

Compare accepted results with correction time. Keep examples alongside the numbers: nine good summaries and one invented deadline differ from ten summaries that need shortening. The average may look similar, but the remaining work is very different.

Count repeated attempts too. A fast first answer is not much help if you spend ages rewriting it. You might end up choosing two models: one for working notes and another for harder documents.

Add this qualification to the decision: “for these tasks, materials and instructions.” After an update, return to the same set. You can see what changed instead of trying to remember why you chose that model in the first place.