Beyond the Chatbot: How to Turn AI Into Real Product Advantage

In 2006 Netflix offered a million dollars to anyone who could improve its movie recommendations by 10 percent. Three years later a team called BellKor's Pragmatic Chaos won. Then something odd happened: Netflix never fully used the winning solution. Its engineers wrote in 2012 that the extra accuracy wasn't worth the engineering effort it would take to run in production, and that the business had moved on to streaming, where the problem looked different.

People tell that story as a joke about wasted prize money. It's actually one of the clearest lessons in AI product strategy. Accuracy alone was never the advantage. Fit with the product, the data and the business model was.

Why recommendation became Netflix's moat anyway

The recommendation system Netflix did build was worth a great deal. In a 2015 paper, two of its product leaders, Carlos Gomez-Uribe and Neil Hunt, wrote that about 80 percent of hours streamed on the service came from recommendations, and estimated that personalization and recommendation together saved the company more than $1 billion a year by reducing cancellations.

That's the pattern we look for in AI-powered product innovation. The AI wasn't a feature bolted onto the side. It sat in the main path of the customer experience, it got better as more people used it, and it attacked the metric the business lived or died by.

The cost curve changed the math

For most of the last decade, putting a large model inside a product was expensive enough that only big companies tried. That has shifted fast.

Stanford's 2025 AI Index reported that the price of running a model that performs at roughly GPT-3.5's level fell from about $20 per million tokens in November 2022 to about 7 cents by October 2024. That is a drop of more than 280 times in under two years.

When a core input gets that cheap, two things happen. Ideas that used to fail on cost become viable. And features that once felt special become table stakes, because every competitor can afford them too. A chatbot on your site is no longer a differentiator. It's plumbing.

Most AI product ideas don't survive contact with reality

In July 2024 the research firm Gartner predicted that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, rising costs, weak risk controls and unclear business value. That matches what we see. The demo works. The product doesn't.

The failures tend to share a shape. The team started from "we should use AI" instead of from a painful customer problem. Or the AI produced something impressive but optional, so customers tried it once and went back to the old way.

A four-question test for any AI product idea

Before building, put every idea through these four questions. If an idea fails two of them, park it.

1. Is it in the main path? Does the customer pass through this moment every time they use your product, or is it a side feature? Netflix's recommendations were the home screen. Most failed AI features live in a menu.

2. Does it get better with use? If every customer interaction improves the next one, you're building a data flywheel that competitors can't copy by buying the same model. If it doesn't, your advantage lasts until a rival signs up for the same API.

3. Is a wrong answer cheap? Suggesting a movie someone doesn't like costs nothing. Suggesting the wrong medication dose is a lawsuit. The cheaper an error is, the more freedom you have to let the AI act on its own.

4. Can you measure the result the business cares about? "Users love it" isn't a metric. Retention, conversion, time saved or tickets avoided are.

Three levels of AI in a product

It helps to be specific about how deep the AI goes.

• Level 1, assist: AI helps the customer do something they were already doing. Drafting a message, filling a form, summarizing a long document.
• Level 2, decide: AI makes a choice inside the product. Which items to show, which leads to call first, which invoice to flag.
• Level 3, act: AI completes a task on the customer's behalf. Booking the appointment, sending the follow-up, reordering stock. This is where agentic AI lives.

Most small and mid-sized companies should start at level 1, prove value, then move one level deeper where the cost of a mistake is low. Jumping to level 3 because it demos well is how projects end up in Gartner's 30 percent.

A short case: Spotify's Discover Weekly

When Spotify launched Discover Weekly in 2015, it was a simple product: a playlist of 30 songs, refreshed every Monday, picked for you. There was no chat window and no visible "AI." The intelligence was invisible, and the format was a habit: one playlist, one day, every week.

Three things made it work. It sat on the main path (people already listened to playlists). It improved with every skip and save. And a bad pick cost the listener three seconds. It passed all four questions above, and that is why it became one of the company's best-known features.

How to run an AI opportunity session in your company

Set aside two hours with the people who talk to customers most, not just the people who build things.

1. List frictions. Have everyone write down the moments where customers get stuck, wait, repeat themselves or give up. Aim for 30.
2. Score them. Rate each on how often it happens and how much it costs you, on a simple 1 to 5 scale.
3. Match to a level. For the top five, decide whether AI would assist, decide or act.
4. Run the four questions. Kill anything that fails two.
5. Pick one. Build the smallest version that could prove the outcome, and set a date to look at the numbers.

Check your understanding

1. Why didn't Netflix fully use the algorithm that won its own million-dollar prize?
2. What does a "data flywheel" protect you against?
3. Why is it usually smarter to start an AI feature at the "assist" level?

The bottom line

Cheap models mean everyone can add AI. Very few will add it in the places that matter. The advantage goes to teams that put AI in the main path of the customer experience, let it learn from use and measure the result the business actually depends on.

---

Sources

• Netflix Technology Blog, "Netflix Recommendations: Beyond the 5 Stars," April 2012.
• Gomez-Uribe, C. A., Hunt, N. "The Netflix Recommender System: Algorithms, Business Value, and Innovation." ACM Transactions on Management Information Systems, 2015.
• Stanford Institute for Human-Centered AI, AI Index Report 2025.
• Gartner press release, July 29, 2024.
• Spotify company announcements, July 2015.