Ferremundo AI: replacing OpenSearch with fine-tuned embedding search, and building the AI function behind it

What I own

Three things, in the order the business feels them.

The AI roadmap: what gets built, in what sequence, and what it is worth. The AI engineering function: the people, the workflow, and the review culture that turn that roadmap into shipped features. And the retrieval and matching stack end to end: catalog modelling on Postgres, the embedding strategy for text and image, the FastAPI matching surface, the LangGraph ordering agent and its checkpointing model, the incremental warehouse sync that keeps the catalog and embedding matrices fresh, and the production posture across Datadog APM, structured JSON logs, AWS CDK, and ECS rolling deploys. On top of matching, the agentic ordering flows that take a reseller from natural-language intent to a structured order without leaving the conversation.

Why not OpenSearch

The incumbent was an OpenSearch-backed app search, and it failed in the specific way lexical search always fails in this market. Hardware store owners search from a job site on a phone, in one or two words. They type “tubo” when the product is “Tubo PVC presión 1/2” pulg PN10”, or “plastig” for a brand, or a dimension in whichever of four formats they happen to use. Before this shipped, 15.8% of searches returned nothing at all, about 22,000 dead ends a month.

The deeper problem is that 60% of queries are a single word, so the bottleneck is disambiguation, not retrieval. When someone types “tubo” the catalog has dozens of tubes and the entire job is ranking the right one first. That is a ranking problem, and it is won with signals OpenSearch has no access to: ABC velocity, view popularity, and this customer’s own purchase history.

So the decision was to build custom rather than adopt it, and the decision was measured afterwards rather than asserted: Top-1 59.1% against OpenSearch’s 46.5% on the same query set, at roughly a tenth of the monthly cost. What we gave up is real, just not binding at this size: automatic horizontal scaling, sharding for million-item catalogs, and built-in dashboards. At 11,500 SKUs the catalog fits in RAM.

Current production sits at Top-1 64.8%, Top-5 82.3%, p50 571 ms on a standing 385-query labelled benchmark. The sections below are how that pipeline works.

Building the AI function

There was no AI team when this started. I built it: hiring, onboarding, and mentoring the engineers, then establishing the engineering culture, development workflow, and code review practice around them. That scaffolding is what makes the accuracy numbers above repeatable rather than a one-off. A retrieval system only its author can safely change is a liability, so the review practice and the standing benchmark matter as much as the model does.

The roadmap: from answering to initiating

I own the AI roadmap and translate the founder and CTO’s requirements into phases. The arc is a single sentence: turn an assistant that answers into an agent that initiates.

The phases are sequenced so each one ships value alone and makes the next cheaper. Search and disambiguation came first, because nothing downstream works without it. Conversational search and image search followed, then full cart operations against the live B2B API, with stock and quantity rules matching the app’s own. Purchase and credit are the current frontier: today the agent takes a retailer all the way to a filled cart and stops there, and closing that gap means order creation, credit term selection, and invoice visibility, with a hard confirmation step before anything is created. WhatsApp comes after that, deliberately, because a retailer arriving on WhatsApp should be able to buy rather than only build a cart. Since both channels mount the same agent definition, that is a channel mount rather than a rebuild.

The destination is a replenishment agent: detect when a store is due to reorder, assemble the complete basket with the right SKUs, quantities, and that customer’s live prices, and propose it over WhatsApp for one-tap approval. Four constraints shape it, and all four are deliberate. A human always confirms, and confirmation fills the cart rather than placing an order, so every purchase rule and credit decision stays exactly where it is today. The agent works only inside the retailer’s own catalog and prices, never across tenants. Prediction comes first and the language model second: a statistical engine decides what and when, and the LLM only composes and explains. And every action is a permissioned tool rather than logic buried in a prompt.

The retrieval pipeline

A hybrid pipeline with a fine-tuned multilingual E5-base as the first-class encoder. BM25 anchors rare drug-name-style tokens, the dense encoder handles paraphrase and Spanish slang, a BGE cross-encoder reranks the merged pool, and a synonym and alias table corrects long-tail miscalls. Embedding-first routing skips the heavier fuzzy and reranker work for queries that confidently resolve at the encoder layer, which is what produced the +3.5 pp top-1 and -190 ms latency move. A separate Doc2query enrichment pass, which concatenates synthetic queries into catalog text, gave another lift to 81.5% top-5.

Embedding finetune (text)

Backbone intfloat/multilingual-e5-base, fine-tuned with CachedMultipleNegativesRankingLoss on a four-source training set: GPT-4o-mini synthetic Spanish queries (~46k), labeled benchmark positives, synonym-derived positive pairs, and hard negatives mined by nearest neighbors over catalog embeddings in Postgres. Best checkpoint chosen by cosine NDCG@10 on a held-out slice. The frozen baseline lands at Acc@1 45.3%, Acc@5 64.0%, NDCG@10 55.0%; the fine-tuned model is what unlocks the production accuracy and latency posture.

Image search (vision)

A separate pipeline trains a contrastive image-text encoder on top of google/siglip2-base-patch16-256 using SigLIP’s sigmoid loss, with image-to-text Recall@1/5/10 on a deterministic held-out 10% split. Training data is 6,251 active-SKU thumbnails downloaded from the staging API and aligned with synthetic-query text. The fine-tuned weights are exported to ONNX and serve at about 99 ms per query on CPU. INT8 quantization was tried, recorded, and rejected: Recall@5 went from 64% to 13% for only a 1.3x speedup. This is the canonical example I now reach for when someone proposes “just quantize the encoder.”

The conversational agent, and why it moved to Agno

The agent handles product search, image search, follow-ups, and full cart operations (add, update, remove, view) against the live B2B API, with stock and quantity validation matching the app’s own rules. It routes deterministically first: small talk, SKU-shaped strings, and confident follow-ups are answered with zero LLM calls, and the model runs only when reasoning is genuinely needed. On an ordering surface the agentic path costs a couple of seconds for a fraction of a point of accuracy, which is the right trade there and the wrong one for search-only, so the same service exposes both.

It started as a hand-built LangGraph ReAct loop with its own checkpointing across in-memory, Postgres, and Redis backends. That worked, and I migrated off it anyway. LangGraph ships excellent primitives, but primitives were the problem: WhatsApp, scheduling, human-in-the-loop approvals, tracing, and evals were each going to be a separate infrastructure project, and every one of them is on the roadmap above. Agno makes all five first-class, mounts the WhatsApp channel from the same agent definition as the app channel, and self-hosts on our own AWS with sessions, runs, and traces in our own Postgres.

The migration replaced plumbing only. Routing rules, cart rules, the search pipeline, and the prompts all lived in framework-free modules and moved across unchanged, which is the entire reason swapping the framework was tractable at all. The trade-off is accepted rather than hidden: Agno is young and has shipped breaking API changes, so versions are pinned and business logic stays out of framework classes. Alternatives were evaluated and rejected on record, including Google ADK (optimized for Gemini on Google Cloud when we run OpenAI and Claude-class models through our own gateway), LangChain deep agents (a planning harness for open-ended work, where ours is tightly guardrailed and transactional), and the multi-agent frameworks (they optimize agent-to-agent collaboration, and there is one agent here with a small tool set and hard business rules).

The operator console

Built rather than bought. Any conversation can be replayed end to end: chat as a real customer, see product cards and live cart state, read per-reply execution traces, inspect individual tool calls, browse session history, and switch across local, staging, and production. When someone asks why the agent proposed something, the answer is in the same place the team already works, and no conversation data leaves our infrastructure to a third-party tracing vendor.

Where the work runs

The split matters more than it sounds. Retrieval and ranking run in the API process against in-memory structures, because at 11,500 SKUs the catalog fits in RAM and the retrieval logic itself is a few milliseconds. Every model inference (query embeddings, image embeddings, cross-encoder reranking) is a call to a remote GPU serving stack, because that is the latency that actually matters and it does not belong in an API container.

At startup the match service hydrates catalog rows, ranking signals, the alias map, the BM25 index (pickled from Postgres or rebuilt), the text embedding matrix with its SKU ordering, and the vocabulary and synonym tables. A nightly sync job upserts warehouse data, embeddings, image embeddings, and BM25 artifacts, and operators roll the API tasks afterward so workers reload from the new state. Treating “the cache is reloading fat matrices after batch sync” as a first-class concern rather than a side effect is what keeps the system honest in production.

Production posture

Multi-stage Docker images, ddtrace-run wrapping the entrypoint when Datadog is enabled, AWS RDS Postgres, Alembic migrations on startup with a SKIP_MIGRATIONS escape hatch, S3 model sync from a ferremundo-models bucket, CircleCI for black/pytest plus ECR build and ECS rolling deploy to a dev cluster, and structured JSON logs with Datadog trace correlation. The health endpoint exposes embedding coverage and image index size, which is what I look at first when something has shifted.

Lessons

B2B hardware search fails in ways leaderboard models never measure. Customers type one word or a slang synonym, and the right SKU is often buried behind a popular default, so you spend more time on disambiguation signals and alias learning than on raw retrieval tricks. Fine-tuned E5 only pays off once you stop treating embeddings as a side channel: embedding-first routing was the largest accuracy move and it lowered latency by skipping fuzzy work most of the time. Hard negatives from your own catalog matter as much as synthetic positives; otherwise the cross-encoder and the alias table fight you when expansion changes surface forms across query and title. Agents look like a small wrapper, but they need checkpointed state and deterministic bypasses for short inputs, or you burn seconds on LLM routing for every SKU lookup. Image search is not “quantize and forget”; INT8 wrecked recall with almost no speed win, which is the kind of result you only get from measuring on real product photos. The “cache” is mostly reloading fat matrices after batch sync, which is simple, but it means deployments and cron are part of the correctness story, not an afterthought.

This engagement is ongoing; this page will be updated as the system evolves.