We built an AI shelf auditor. We’re publishing what it cost.
587 shelf photos. 13 yes/no checks on each one. That comes to roughly 7,600 separate calls in a single quarterly audit (587 × 13), and today a person makes every one of them.
The checks are specific
Is the product we negotiated with the retailer actually there? Does the number of facings match? Is the shelf fully stocked? A facing is one unit of a product visible from the front of the shelf. Contracts name the mandatory brands, and one contract puts two particular whiskies on the shelf in every store.
Making shelf recognition more versatile
Our Retail Image Recognition product already automates this work. The rep takes a photo, and the product turns it into finished KPIs. It’s live with manufacturers in dozens of countries, at up to 98% accuracy. The method has one limit: the model has to learn each product first, so a new brand or a new market means a round of training up front.
In the Asseco Platform Lab we’re testing how to make shelf recognition more versatile. We want it to handle new products and harder checks without training the model for each case. We also want it to read the whole shelf, our customer’s products and the competition’s. For that we used vision agents. An agent is a model that can reach for tools: it decides on its own to run a detector, crop a detail, look again, and only then answer. We call the project Vantage.
Why this is hard
A person with a checklist makes subjective calls, and the only way to cover more stores is to add more people. One AI model with one good prompt isn’t enough either. On easy checks, like whether a brand is on the shelf at all, a model does well. In our test on 100 photos it got 93.6% of the calls right. On hard checks it falls apart. Telling a 71 g pack from a 39 g pack, or custard powder from jelly powder, in a photo shot from a couple of meters away, drops accuracy to 71% to 77%. That’s a wrong call roughly one time in three or four.
The failure has a shape. The model answers “yes, it’s there” without evidence in the photo. On availability checks it did that 17 times out of 49 photos: products reported as present that the photo didn’t show. A report that’s wrong that often sends the rep back to the store to verify, and then the automation has saved nothing.
The model is a variable in the experiment
We built the setup so that any AI provider (Google, Anthropic, OpenAI) plugs in with one setting change, without rewriting any code. Every run uses the same prompt, the same photos, and the same answer key. That’s what lets us compare providers head to head, on accuracy and on cost.
We did it that way because the vision model market shifts about every quarter, and list price turned out to be misleading. The two models we compared were within twenty percent of each other on list price. One of them came out six times more expensive per photo in practice, and nine times slower. The reason is reasoning tokens, the text a model writes for itself before it answers. You pay for that text too.
Tools beat a better prompt
The biggest difference came from giving the model tools. Our agent decides step by step which tool to use. One detects products in a photo. The other is a magnifier: it crops a section of the photo and reads the label up close. On 49 photos and the 6 hardest checks (powdered desserts), the path looked like this.
Twenty-one percentage points. Wrong “yes” answers on availability fell from 17 to 2. One caveat: the 96% belongs to that particular hard set of 49 photos and 6 checks, not to the setup as a whole.
That accuracy costs money. Per photo, the full agent runs about six times what the bare prompt costs, and takes four times as long. That’s with the cheaper image format already in place.
Trust rests on outside evidence
A model can rate how confident it is in its own answer. The obvious move is to trust it when it says it’s sure and send the rest to a person. We measured that, and it doesn’t hold. Answers marked as confident were right 62% to 64% of the time for the model on its own, and 74% for the agent. Even at 74%, roughly one “sure thing” in four is an error. A filter built on the model’s own confidence would pass those errors through as settled.
So we don’t use confidence as the only signal. Comparing independent sources works better: whether two different models agree, what the tools returned, and how well this check has scored before. We haven’t built automatic handoff of uncertain cases to a person. It’s one of the next steps.
What it actually costs
How the image reaches the model changes the cost. A crop encoded as text inside the request costs far more tokens than the same crop sent as a native image, and it passes through the model’s context twice instead of once. Switching to the native format cut the cost per photo to about a quarter, with no drop in quality.
At the scale of our test, a few dozen photos, that cost is negligible. Image recognition in production works on a different volume: tens of thousands of stores, several photos per visit, repeated every month. At that volume even a small amount per photo becomes a real budget line. For this to work in production, we either optimize further or move to our own models.
Where this stands
Vantage is one of our internal research projects, a proof of concept, which means a check on whether the idea works at all. It isn’t a product, and no customer is running it. The whole path, from photo to finished call, passed an end-to-end test on data from three manufacturers, in beverages, spirits, and powdered desserts.
The technology underneath is also different from what we sell. Retail Image Recognition uses trained models, taught in advance on many photos of each product. It works offline, on the rep’s device. Vantage uses general-purpose models with no per-brand training, gives them tools, and lets them work step by step. What we don’t know yet is how well the result holds up on categories we haven’t tested.
Wojciech Koziej
System Architect, Asseco Platform
Works on image recognition for shelf execution. Vantage is his research project in the Asseco Platform Lab.
