Machine learning agencies: custom models & production ML

Custom ML is worth it when a decision repeats at scale, you have (or can collect) labeled examples, and offline error reduction maps to clear business impact — ranking, fraud, forecasting, recommendations, vision or NLP classifiers on proprietary assets. Skip custom training when a rules engine, analytics dashboard, or well-scoped LLM/RAG feature already hits the outcome cheaper. Ask agencies to defend “train” versus “buy/integrate” against your data volume and required accuracy bar — not to sell research by default.

A PoC should prove lift against a held-out baseline on your data (or a realistic proxy), under an agreed metric and operating constraints (latency, cost, human review). It should surface data risks — leakage, label noise, class imbalance — and end in an explicit go/no-go for productionization. Treat happy-path notebooks on clean samples as marketing. Prefer a four-to-eight-week fixed PoC with a kill gate over an open-ended “research retainer.”

Hire an agency when you need speed, proven modeling patterns, and a first production slice without staffing a full ML bench yet. Build in-house when the model is core IP, data access is highly restricted, and you can own labeling, evals, and on-call long-term. Hybrid is common: agency ships the first pipeline and eval harness; your team owns data connectors, product integration, and steady-state retraining. Either path requires your org to hold repos, cloud, and credentials.

You keep data access policy, product priorities, security review, and admin of cloud, warehouses, model registries, and endpoints. The agency typically owns modeling approach, feature/pipeline design for the scoped use case, eval harness design, and the first shippable slice per SOW. Write the split down so production knowledge does not live only in the vendor’s notebooks.

Healthy engagements split discovery/PoC (fixed, time-boxed) from production hardening (pipelines, monitoring, handoff) priced once the success bar is clear. Ballparks vary widely with data messiness and compliance; distrust a single low number that assumes perfect labels and no integration. Compare senior ML time, eval rigor, and ownership terms — not the sticker alone.