Amnon Mishor is the CTO and founder of Leadspace, one industry recognized, an AI-powered buyer data platform used by B2B companies like Zoom and Salesforce.
In the past For a decade or so, the digital revolution has left us with an excess of data. This is exciting for a number of reasons, but especially in terms of how AI can further revolutionize the business.
However, in the world of B2B – the industry I am heavily in – we are still experiencing a lack of data, mainly because the number of transactions is significantly lower compared to B2C. So, um AI should keep its promise to revolutionize the company, it must also be able to solve these small data problems. Fortunately it can.
The problem is that many data scientists employ bad practices and create self-fulfilling prophecies that reduce the effectiveness of AI in small data scenarios – and ultimately hinder the influence of AI on the further development of the company.
The trick in correctly applying AI to small data problems is to follow proper data science practices and avoid bad ones.
The term “self-fulfilling prophecy” is used in psychology, investing, and elsewhere, but in the data science world it can be described simply as “predicting the obvious.” We see this when companies find a model that predicts what is already working for them, sometimes even “by design”, and apply it to different scenarios.
For example, a retail company finds that people who have their shopping carts filled online are more likely to buy than people who haven’t, and therefore markets heavily to that group. You predict the obvious!
Instead, they should employ models that help tweak things that don’t work well – converting first-time buyers who don’t have any items in their cart. By solving the latter – or predicting the non-obvious – this retail business is much more likely to impact sales and attract new customers rather than just keeping the same ones.
To avoid the trap of creating self-fulfilling prophecy, here’s how to apply AI to small data problems:
- Enrich your data: When you find that you don’t have a ton of existing data to manipulate, the first step is to enrich the data that you already have. This can be done by tapping external data to apply look-alike modeling. We are seeing this more than ever thanks to the advent of recommendation systems used by Amazon, Netflix, Spotify, and others. Even if you only make a purchase or two on Amazon, they have so much information about products in the world and the people who buy them that they can make reasonably accurate predictions about your next purchase. If you are a B2B company that uses a “single dimension” to categorize your business (e.g., “large corporations”), follow Pandora’s example and analyze each customer for the most accurate grades (e.g. Melody construction, beat, etc.). The more you know about your data, the richer it becomes. You can move from low-dimensional data with trivial predictions to high-dimensional knowledge with powerful prediction and recommendation models.
