AI agencies: when to hire and how to brief them

Give the agency the work that benefits from specialized patterns: model/provider integration, retrieval or feature design, evaluation harnesses, and the first production-ready slice behind clear acceptance criteria. Keep product priorities, user research, data access policy, security review, and long-term ownership on your side — especially repos, cloud accounts, API keys, and production deploy rights. Hybrid works well: agency ships a bounded workflow; your team integrates into the product surface and staffs ongoing iteration. Write the ownership split into the SOW so a polished demo never leaves you without a maintainable system.

Treat a happy-path chatbot UI as marketing, not evidence. Require evals on your domain data (or a realistic proxy), documented failure modes (hallucination, latency, cost spikes, PII leakage), and a written path from pilot to production — observability, rollback, and who owns prompts/models when the contract ends. Ask what breaks when inputs are messy or adversarial, and whether answers must be grounded with citations. Prefer a short paid spike with your constraints over a free demo that only works on their golden set.

Full MLOps is rarely day-one scope for a narrow LLM or automation pilot — but you still need monitoring, versioning (prompts, models, indexes), and a handoff plan before you scale traffic or touch sensitive data. Insist on basic logging of quality and cost, an environment story (dev/stage/prod), and clarity on who responds when quality drifts. Expand MLOps depth when you retrain custom models, run multi-model pipelines, or put AI on a critical path; until then, buy just enough operational discipline that the pilot can survive contact with real users.