How to hire an AI agency without getting a science project

AI agency pitches are full of impressive demos. Your job is to hire for a production outcome — a workflow that is faster, cheaper, safer, or more accurate for a real user — not a science project that looks clever in a slide and dies when it meets messy data, compliance, or day-two operations. The agencies worth shortlisting talk about evaluation, failure modes, and ownership as readily as they talk about models.

Start with the outcome and the constraint set, not the model menu. Write down who benefits, what decision or task improves, how you will measure it, and what is non-negotiable (PII handling, latency budget, human review, brand voice, on-prem vs vendor APIs). “We need generative AI” is not a brief. “Customer support deflection rises 15% on these five intents without increasing escalations or inventing policy” is a brief. Strong agencies will interrogate your data readiness and push you toward a thinner first slice; weak ones will expand scope into research because research is billable forever.

Separate PoC from production on purpose. A proof of concept should be time-boxed, paid, and judged against a pre-agreed bar — quality on a fixed eval set, cost per successful task, or human time saved on a real sample. It should answer “is this worth industrializing?” not “can we sparkle in a demo?” Production work then covers engineering for reliability: monitoring, evals that catch regressions, access control, logging, runbooks, and a path for your team to operate or take over the system. Mixing both into one fuzzy retainer is how science projects get booked as delivery.

Evaluate agencies on relevant proof and the people who will actually build. Ask for shipped systems that reached production users — not just lab work or wrapper apps — and dig into what broke after launch. Meet the engineers and ML practitioners assigned to you. Ask how they build eval harnesses, how they handle hallucinations and refusal behavior for your domain, and what they will refuse to automate. Score them on clarity about data pipelines and MLOps needs for your stage; you may not need a full platform on day one, but you do need honesty about what “production-ready” requires for your case.

Structure the commercial model to de-risk learning. Prefer a fixed-price discovery or PoC with a clear kill gate, then a scoped build for the winning path — with change orders when models, prompts, or data assumptions shift. Beware of open-ended “innovation retainers” with no acceptance criteria, and of guaranteed accuracy claims that ignore your data quality. Price should buy clarity and progress toward a production slice you control, not indefinite experimentation that never graduates.

Lock ownership and exit before kickoff. Your company should own application code, prompts, fine-tunes derived from your data, eval sets, and documentation; keep API keys, vector databases, and observability in accounts you administer. Spell out that your data is not used to train models for other clients. Confirm who holds third-party model subscriptions and what happens on offboarding — exports, runbooks, credentials rotation, and a brief support overlap. If the only runnable version lives on their laptop or private org, you bought a demo dependency, not a capability.

Run the hiring process like a product decision. Shortlist three to five firms with proof in your problem shape (LLM integration into an existing product, chatbots with guardrails, computer vision on your asset type, data pipelines into training). Give the same brief to each, score PoC proposals on specificity and risk honesty, and run a small paid spike with your hard data before a large SOW. Keep an internal owner who sets priorities and accepts or rejects go-live — agencies fill a leadership vacuum with billable experiments when no one on your payroll owns the outcome.

When you are ready to compare AI partners against a real brief, get matched with agencies that fit your outcome, data constraints, and production bar — or browse /agencies?category=artificial-intelligence — then insist on a path from PoC to something you can run, measure, and own.