Start with the workflow, not the model
Most AI projects fail before the first line of code, at the moment someone asks "what should we do with AI?" instead of "where do people wait?". The useful question is the second one. Walk the business and look for the places where a person reads something and types it somewhere else, waits for an answer that already exists in a document, or checks a file against a rule by eye. Invoices against purchase orders. Contracts against a checklist. Support tickets against a knowledge base. Those are AI projects with a measurable before and after.
Choose one. A single document type or a single kind of question beats a general assistant every time, because you can define what "correct" means and count it. "An assistant for the whole company" has no definition of correct, so it never finishes.
- Where do people wait, copy-paste or re-key data?
- One document type or one question, not a general assistant
- A good first scope has an owner, a volume and a cost of error you can name
Build the evaluation set first
Before anyone picks a model, collect thirty to fifty real examples with the right answer attached: the invoice and the values a person extracted from it, the question and the answer a senior colleague would give. This is the evaluation set, and it is the most important artefact of the project. It turns "the AI seems good" into "the AI gets 94 of 100 fields right, and here are the six it misses".
Agree the target with the people who own the workflow before the build starts. Ninety percent might be a triumph for a task humans get wrong a fifth of the time, and a failure for a payment amount. Accuracy claims without an evaluation set mean nothing, from a vendor or from your own team, and the set is also what tells you later whether a model update made things better or worse.
- Thirty to fifty real examples with the right answer
- Agree the target before the build
- Re-run the set on every model or prompt change
Pilot in production, not in a slide deck
A demo on hand-picked examples proves nothing about Tuesday's inbox. The pilot that matters runs on live inputs, in the real system, with a person reviewing anything below a confidence threshold. In our experience a scoped workflow goes from first meeting to a monitored pilot in about six weeks: the first week on the workflow and the evaluation set, two or three weeks building against it, one week measuring and tuning, then production with monitoring.
Human review is not a fallback to be embarrassed about. It is how the system earns trust and how the evaluation set grows: every correction is a new example. Measure after launch what you measured before it, plus latency, cost per document and how often a person had to step in.
- Six weeks from workflow to monitored pilot
- Human review below the confidence threshold
- Track accuracy, cost per item and review rate after launch
Who needs to be involved
Three people make a first AI project work: an owner of the workflow who can say what correct means and has the authority to change the process, an AI engineer who builds and evaluates, and a data engineer who gets the inputs out of wherever they live and the outputs back in. Everything should run in accounts you own: the cloud project, the model provider agreement, the prompts and the evaluation set. If the partner leaves, the system stays.
The part that is easy to skip is the business decision in front of the build: which workflow first, what to leave out, whether the economics work at your volume. That conversation is worth more than any model choice, and it is where an experienced team earns its place.
- A workflow owner, an AI engineer, a data engineer
- Cloud, provider agreement, prompts and evaluation set in your name
In short
Choose one measurable workflow, build an evaluation set from real data, agree a target, ship a monitored pilot in about six weeks, and keep the same team to run it.