1. The problem with search
Shoppers no longer arrive with a part number. They arrive with an intention they can barely put into words: outfit a backyard, plan a themed birthday party, find the pieces that go together for a project they can only describe by pointing at a photo. These are open-ended, multi-part needs. They rarely map to a single item, and they almost never map to the two or three keywords that catalog search was built to accept.
Keyword search was designed for short, structured queries against a structured index. It matches tokens, not intent. Ask it to reason about a whole scene — the mood of a room, the theme of a party, the visual style of a jacket someone photographed on the street — and it returns nothing useful, or a page of near-misses. The shopper leaves. For a retailer, every failed search is a lost conversion: no sale, no larger basket, no return visit, and no chance to earn loyalty.
We built a shopping assistant to close that gap. It understands natural language and images, works through discovery over several turns, and recommends from the retailer's own catalog. This paper describes the architecture, not the specific models — every component is defined by its role, and any model can be swapped for another hosted or self-hosted equivalent.
2. An agentic pipeline
At the centre is a reasoning model acting as an agent. It does not answer from its own memory; it interprets the request, decides what needs to be retrieved, and calls the right tools to retrieve it. An orchestration framework binds the reasoning model to those tools and holds the state of the conversation.
Real requests are compound. "Something to serve drinks in the garden, and glasses to match" is two searches, not one. The agent decomposes a request into several focused sub-queries, runs them against the catalog, and assembles the results into a single coherent set of recommendations. This multi-query approach is what lets the assistant handle a whole scene instead of one item at a time.
2.1 Image-aware routing
Requests carry images as well as text — a photo of a sofa, a screenshot of a colour scheme, a picture of a dish to recreate. The agent routes on the modalities present. A text-only request goes to semantic text retrieval. A request that includes an image triggers image-to-image similarity search. A request that mixes both — "a rug like this, but in blue" — is split so each part reaches the retrieval path that can serve it, and the results are combined. Routing is a decision the agent makes per request, not a fixed path.
3. Retrieval
3.1 Semantic product discovery
We embed the catalog — titles, descriptions and attributes — with a text embedding model and store the vectors in a vector database. At query time the shopper's phrasing is embedded the same way, and nearest-neighbour search returns products by meaning rather than by matching tokens. "Cosy reading nook" finds the right lamps and chairs without those words appearing in any product title.
3.2 Image-to-image similarity
A vision embedding model of the CLIP style maps images and text into one shared space, so a photograph and a description can be compared directly. Every catalog image is embedded once, offline. When a shopper supplies an image, we embed it and run nearest-neighbour search over the catalog vectors to surface the closest-looking products. This skips the lossy step of describing an image in words before searching, which is where visual detail is usually lost.
3.3 Language-agnostic matching
The vocabulary the assistant matches against is derived from the catalog itself, not from a fixed dictionary. Because retrieval works in an embedding space grounded in the retailer's own product data, the assistant carries over to a non-English catalog with no extra configuration. The catalog defines the language of the store; the assistant follows it.
4. Multi-turn cart management
Discovery is a conversation, and the cart is its running state. The assistant manages the cart in natural language — add, remove, update, swap — and tracks prices as it goes.
The difficulty is reference. Across turns, shoppers speak in shorthand: "add the second one", "make it two", "drop the cheaper one", "actually, the blue instead". Each of those points back to something said earlier. The agent performs coreference resolution against the conversation and the current cart, mapping every pronoun and ellipsis to a specific product before it acts. Get this right and the exchange feels like talking to a person who was listening; get it wrong and the cart fills with the wrong items and the shopper gives up.
5. Safety guardrails
A public-facing assistant needs boundaries. We place two guard models around the reasoning model. A content-safety model inspects both the shopper's input and the assistant's output and blocks unsafe content in either direction. A topic-control model keeps the conversation inside the retailer's domain, so the assistant stays a shopping assistant and does not get steered off-task. Both are separate from the reasoning model, so their policies can be tuned — or the models replaced — without touching the core of the system.
6. Bringing your own catalog
The assistant ships with a sample catalog and imagery to run against, but it is built to ingest a retailer's own data. Product text and images are embedded through the same two pipelines — text into the semantic index, images into the visual index — and from that point the assistant reasons and recommends over the retailer's real inventory. No manual tagging or taxonomy work is required to start.
7. Key capabilities
- Multimodal, multi-query agentic retrieval. Text and images in one request, decomposed into focused sub-queries and routed per modality.
- Image-to-image similarity search. Shop from a photo against catalog imagery, with no intermediate captioning step.
- Semantic product discovery. Embedding-plus-vector-database retrieval that matches on meaning, not keywords.
- Conversational cart management. Natural-language add, remove and update with persistent price tracking and coreference resolution across turns.
- Language-agnostic matching. Vocabulary derived from the catalog; portable to non-English catalogs with no extra configuration.
- Dual safety guardrails. Separate content-safety and topic-control models on input and output.
- Bring-your-own catalog. Ingest a retailer's own product text and images.
- Production-ready. Unit and integration test coverage and CI/CD from the start.
8. Deployment requirements
The architecture is defined by roles, so the models behind each role can be hosted or self-hosted to suit the retailer's constraints.
| Component | Role | Notes |
|---|---|---|
| Reasoning model | Natural, multi-turn interaction and agentic control | Hosted or self-hosted; swappable |
| Text embedding model + vector database | Semantic product retrieval | Catalog-derived, language-agnostic |
| Vision embedding model | Image-to-image similarity search | CLIP-style shared text/image space |
| Orchestration framework | Tool calling and conversation state | Holds cart and turn context |
| Guard models (×2) | Content safety and topic control | Applied on input and output |
| Compute | Serving self-hosted models | Multiple GPUs for locally hosted models |
| Packaging | Reproducible deployment | Container-based |
9. Closing
The shift is from matching tokens to understanding intent. A shopper describes an outcome, or shows a picture of one, and the assistant reasons across text and images, retrieves from the retailer's real catalog, and builds the cart in conversation. The requests that keyword search turned away — the open-ended, visual, multi-part ones — become the ones that convert.