AI conversion rate optimization: what it can and can't do

What AI can and can't do, job by job

Work from the top. The first four cost little or nothing, and none of them needs the traffic a test does.

Four jobs worth handing over this week

1. Sort what customers already told you

Gather your reviews, support emails, chat logs and survey answers. Take out names, email addresses and order numbers, then paste the rest into a general AI assistant with something like this:

Here are 60 answers to "What almost stopped you from buying today?" Group them into themes. For each theme, give the count and three exact quotes. Don't add a theme that isn't in the answers. If there are too few answers to say anything useful, tell me.

The exact quotes matter. CXL, a marketing training company, has a name for one failure: research as decoration. The AI starts with a familiar best practice, then hunts through your data for support. Asking for quotes makes that easier to catch. Read 20 of the raw answers yourself as well, because you'll spot things a summary smooths over.

If you have nothing to sort, put one question on your busiest page and come back in a month. You can also ask your last ten customers what nearly put them off.

2. Get a first pass over your recordings

Microsoft Clarity is free for good and includes AI summaries. Its Heatmap insights feature tells you where people clicked first and last, where they clicked over and over or on things that did nothing, and how far they scrolled. You can also chat with Copilot about your dashboard. If you're still comparing conversion rate optimization tools, this is a sensible first install.

Microsoft's own FAQ tells you to double-check the output and not to treat the takeaways as expert recommendations. So use the summary as a list of places to look, then watch a few of those recordings yourself.

Shopify's assistant, Sidekick, can read your store's data, and it asks you to review a change before it goes through. Try asking which products got the most views and the fewest adds to cart in the last 30 days. Then open those product pages on your phone and see if you'd buy.

3. Ask for a page review, and ignore any score

CXL ran a workshop where an AI agent got through five full user journeys in about 30 minutes. Doing that with people usually takes hours of workshop time. The same write-up warns that a score like 42% looks like a measurement and is really someone's judgment. It also says AI can't really tell what drives your visitor.

There's a second trap. AI likes to please. Nielsen Norman Group warns that AI stand-ins for customers often give thin, flattering feedback and tend to treat every idea as a good one. Ask "is my page good?" and you'll probably hear yes. Ask for faults:

Here's a screenshot of my product page on a phone. Imagine you've never heard of this business. List the five places you'd hesitate or get confused, and quote the exact words or name the exact element. Don't give scores. Don't praise anything.

Treat the answer as a list of suspects. Run it against a conversion rate optimization checklist so nothing obvious gets missed. Then watch real people try the page. Jakob Nielsen has long advised that five people is plenty for a round of testing.

4. Draft new versions, then fact-check them

Give the AI your customers' own words and your real facts:

Write five headlines for this page. Use only the customer phrases and facts below. If a claim isn't in the facts, leave it out.

Check every number, guarantee and delivery promise before it goes live. Pick the version you'd be happy to defend to an annoyed customer.

There's some evidence behind writing for a specific person. A 2024 study in Scientific Reports had GPT-3 and ChatGPT write messages matched to people's personality and beliefs. The matched messages clearly did better in 20 of 33 comparisons. It measured what people said they thought and would pay, and never what they actually bought, so take it as encouraging rather than a forecast of your sales.

AI speeds up the work. The answer to how to increase website conversion rate still comes down to the same basics: a clear offer, a reason to trust you and an easy next step.

Five things it can't do for you

Make up traffic

This is the big one. You may see a software seller promise that AI gets reliable results from fewer visitors, cutting testing from weeks to hours. CXL's view is closer to ours: AI can help you launch more tests, but it can't conjure up the visitors or the evidence you need to trust the result.

AI can build the versions faster. The arithmetic of telling a real lift from luck stays where it was. Try it in Evan Miller's sample size calculator at the usual settings. Say a page makes 2 sales per 100 visitors and you want to spot a 20% lift, which is 2.4 sales per 100. You need 19,784 visitors for each version. Test three headlines and two layouts together and you have six versions. Here's how long a fair test would take:

"Multi-armed bandits" are the usual answer to this. A bandit moves traffic toward whichever version is ahead as the test runs, so fewer people see the weaker one. That's useful for something short-lived, like a sale banner, where you care more about earning than knowing. It's weaker proof that one version really is better. Optimizely's data also suggests you need more than two versions for it to help. With two, a bandit beat a plain even split just under half the time. With three or more, it won about 64% of the time. That's from its own customers, so it may not carry over to a small site.

Be careful with an AI tool that declares an early winner. Evan Miller showed what happens if you keep checking a test and stop as soon as it looks like a win. A test that should fool you 5 times in 100 can fool you about 26 times in 100. That's his worst case, with a check after every visitor.

If you're below the line, fix what's plainly broken, make bigger changes and compare the four weeks before with the four weeks after. It's weaker proof, and it's still far better than waiting ten years.

Personalize from data you don't have

Machine learning learns from examples, and a small business has few of them. Google's own purchase predictions in Analytics need at least 1,000 returning users who bought, and 1,000 who didn't, over a seven-day period. Google says it needs that many to train the model. Be skeptical when a smaller tool promises to learn each visitor from a few dozen orders.

Your data may also be dirtier than you think. Google Analytics says it filters out known bots, which leaves the ones it doesn't know about. If bots fill in your lead form, an AI told to get more leads will learn to please bots. Look through your last 30 leads before you let anything learn from them.

Below that scale, you can personalize by hand:

• Send each ad to a page that repeats the ad's promise.
• Show returning customers a quick way to reorder.
• Ask visitors what they're looking for, then show them that.

The last one gets you information no model could guess from a scroll pattern.

Know what people want without asking them

Some tools now offer AI "synthetic users" who answer your questions in place of customers. Nielsen Norman Group says they don't always act like real people. In one example, a synthetic user claimed to have finished all its online courses. The real person admitted they hadn't. Use these tools to prepare for research, and then go and talk to someone. CXL left fake interviews out of its workshop for the same reason.

Take responsibility for what it says

Air Canada's chatbot told a grieving passenger he could claim a bereavement fare after booking. That was wrong. The airline argued the chatbot was a "separate legal entity." A Canadian tribunal rejected that and ordered it to pay $812.02. The tribunal's point was simple: a company answers for everything on its website, whether a page says it or a bot does.

If AI talks to your customers:

• Give it only your real prices, policies and delivery times. Keep anything out of date away from it.
• Stop it making offers you haven't written. That includes discounts, refunds and anything about health or legal outcomes.
• Make a person easy to reach. Someone with an unusual problem shouldn't have to argue with it.
• Read 20 conversations a week at first. You'll catch mistakes, and you'll learn what people keep asking.
• Say it's an AI. Article 50 of the EU AI Act, listed as taking effect on 2 August 2026, says AI systems that talk to people must be built to let them know, unless it's obvious. It's a good habit wherever you sell.

Know which number matters

Tell an AI to raise conversions and it will try to raise conversions. CXL's example is a system that only counts completed purchases. It could overlook fraud, returns, complaints and profit. A bot handing out 20% off will look great on a conversion chart until you read your margins.

Choose the number that pays your bills, usually profit per visitor or booked appointments that actually show up. Check it alongside whatever the tool reports. A lower bounce rate on its own proves little, so questions like what is a good bounce rate can wait until sales have moved.

How to read the claims

The most useful numbers come from a vendor that says what they can't show. Optimizely's 2026 report covers 173,000 tests at more than 1,200 companies. Customers who used its AI a typical amount ran 70.2% more tests than before. They also scored 20.6% higher on what it calls "average test impact."

Now look closer. That score is how often a test wins, multiplied by how much the win adds. Across levels of AI use, it only climbs from 1.3% to 1.6%. Optimizely also says plainly that this doesn't prove AI caused the rise, because companies chose to adopt it. AI probably does help busy testing teams. It's a long way from "AI lifts your sales by 20%." The same report found that well set-up tests won 58% of the time and poorly set-up ones 35%, which is a useful reminder that the basics still count.

If you can't find where a number came from, don't plan around it.

Before you pay for anything, ask the vendor:

1. How many visitors or orders a month does this need to work properly? Can you show me a customer my size?
2. Was your headline result measured against a group that didn't get the tool?
3. What exactly does it optimize for, and can I change that to something closer to profit?
4. Can I see every change it made, and undo it?
5. Can I keep 10% of visitors on the unchanged page, so I can compare?

A good vendor will have answers. Be especially wary of guaranteed results. In 2024 the FTC went after several businesses that it alleged had used "AI-powered" online stores and false income promises to take millions from buyers.

What to use at your traffic level

If you make 5 sales per 100 visitors rather than 2, you reach the testing line sooner, at roughly 8,000 visitors a month. Personalization engines come after all of this, and only once you have plenty of repeat buyers for them to learn from.

When someone needs an answer while they're still on the page

Everything so far improves the page for the next visitor. The person on it right now still has a question. If your feedback keeps saying "I wasn't sure which one was right for me," those people needed a reply before they left.

That's what we built Possible to do, so take this with the obvious pinch of salt. Possible is our own product: an AI built on real sales psychology that asks each visitor the right question at the right moment, remembers the answers, and guides them to a sale, sign-up or booking. It's free to start, and you can read more about our AI sales assistant for online stores. The warnings above apply to any AI that talks to your customers, ours included, and if your page has a broken button, fix that first.

Your first week

1. Today: Install Clarity. Pick the one action that pays your bills.
2. Tomorrow: Paste your reviews and support emails into an AI assistant, using the prompt above.
3. Day 3: Run the page review prompt on your best page, and write down the three complaints you agree with.
4. Day 4: Watch five people try that page on their phones.
5. Day 5: Fix what they stumbled on, and have AI draft five new headlines from your customers' words.
6. A month later: Compare the four weeks after the change with the four weeks before.

Questions about AI conversion rate optimization