How do you get out of ML PoC Hell?

There’s no escaping them. Proof of Concepts are like machine learning’s training wheels.

Pilots, prototypes, and MVPs give ML pros a way to quickly validate the business value of an ML model and see what the challenges might be for getting it to production.

We’ve heard again and again, though, how easy it is to get caught up in an endless PoC loop: create a working prototype, get feedback, create a revised prototype, get more feedback, build a new one, and so on. The result is often frustration and ennui — and projects that never quite achieve lift-off as a result.

PoC Hell seems to come up whenever we talk about barriers to success in ML, so we thought it would be worth taking a look at some of the root causes.

Hurdles and pitfalls

I spoke with Oguzhan Gencoglu a few weeks back to get his thoughts. He called out these two big recurring issues.

01 Lack of executive buy-in at the outset

In the opening phases of an ML project, ‘it’s often led internally by enthusiastic people, not influential people,’ Oguzhan says. ‘Often, it’s a couple of switched-on people who know something is in the works and want to play around. They have the power to start things but in the end, aren’t influential enough to get the green light for budgets, or buy-in for things like product sizing, timelines, recruiting talent, and so on.’

Almost all of the projects that he’s put into production have had an executive-level sponsor, he says.

02 Disconnect between PoC metrics and business metrics

There can also be a mismatch in expectations around what success looks like.

‘Even if you’ve trained a model and got accurate outcomes, it doesn’t guarantee that the metrics are relevant for production.

‘Let’s say you want to detect certain objects. Maybe in the production line there are some defects you want to catch. Obviously, in that case, you want high accuracy, low missed rates, and things like that.

‘So you collect lots of data, annotate it, then train a model and get it working. Everyone’s happy and calls the proof of concept successful. But then later, you realize that you’ll have to use real-time data to make it into production.

‘That’s a completely different beast, and it pretty much guarantees that everything will have to change.’

To hell and back

How do you get out of PoC hell? Some people say ML should borrow from Lean Methodology — that is, aim for prototypes built on the lean model: use the leanest tools, gather the leanest data, and apply the leanest project management techniques — as a way to reduce complexity and slash the number of unseen variables that might skew an outcome.

For instance, if a clothing brand wanted to use ML to understand the demand drivers for winter wear, they might build a lightweight model using a limited number of external sources for things like meteorological data.

They could also ditch heavy-duty, enterprise-level platforms like Teradata and Oracle and replace them with leaner tools like Power BI or even Excel. In keeping with Lean thinking, the PoC could be run in a series of sprints, building in scheduled intervals for evaluation, learning and adaptation.

When a menswear brand tried this approach, they uncovered a direct sales trigger: customers were most likely to buy winter clothing anytime temperatures dropped 11-12 degrees within a 5-6 day period.

Sounds promising. But even if lean offers benefits as a general way of working, Oguzhan says the key to a successful PoC outcome is to plan for production during the proof of concept.

Data scientists need to be aware of the constraints they face and start with a big picture view from the beginning.

‘In ML, the whole is greater than the sum of its parts.’

‘You need to optimize the model first from the software side, quantize it, do model pruning compression and all the mathematics, then optimize for specific hardware, work with an inference engine, and so on.’

If you have at the back of your mind at the start ‘OK this thing will be running on a certain mode of hardware in very-close-to real-time, will have this false-negative rate, will be used by this user persona, with this budget …’ It’s a completely different mindset.

‘It changes your paradigm — starting with the desired outcome and then working backwards, rather than making something work and then iterating it later for production.’