This weblog is collectively written by Md Rahman, Arkaprabho Ghosh, Navin Bilwar, and Desh Shukla.
Government abstract
Cisco IT just lately evaluated fine-tuning embedding fashions utilizing NVIDIA Nemotron RAG fine-tuning recipe as a part of an effort to enhance retrieval accuracy for domain-specific enterprise knowledge. The target was to not redesign current retrieval-augmented technology (RAG) techniques, however to know whether or not focused embedding fine-tuning may materially enhance semantic search high quality with affordable effort and quick turnaround. Via this experiment, Cisco was capable of validate firsthand that embedding fine-tuning, mixed with artificial knowledge technology, can ship measurable accuracy good points inside a short while body. The experiment additionally demonstrated sturdy time-to-value, enabling speedy iteration and clear efficiency alerts with out lengthy coaching cycles or intensive guide labeling. The diminished turnaround of only some days to know the quick advantages was a key end result of this collaboration.
The embedding mannequin coaching and analysis workflow was executed on Cisco AI PODs operating Cisco UCS 885A infrastructure powered by NVIDIA HGX platform.
Drawback assertion
Previous to conducting this experiment, Cisco had performed related embedding fine-tuning experiments utilizing earlier technology fashions and smaller scale infrastructure. These prior efforts required important guide tuning of hyperparameters comparable to batch dimension and variety of epochs, and outcomes have been usually troublesome to stabilize. Iteration cycles have been lengthy, making it difficult to discover totally different configurations or scale experiments. Regardless of some localized enhancements, key phrase search remained mandatory for a lot of domain-specific retrieval situations. There was additionally no standardized, end-to-end workflow that engineering groups may execute shortly and consider persistently throughout runs. Usually, these efforts would take weeks to months of guide effort for unsure good points.
How the nice‑tuning went and time to worth
On this experiment, Cisco used the NVIDIA NeMo Retriever embedding finetuning recipe, leveraging artificial knowledge technology to supply coaching alerts from current corpora. The recipe runs via 5 distinct levels: artificial knowledge technology (SDG), knowledge preparation with hard-negative mining, contrastive fine-tuning, BEIR analysis, and ONNX mannequin export. The workflow was capable of run end-to-end efficiently. All experiments ran on a single NVIDIA H200 143 GB GPU hosted inside Cisco AI Pods constructed on Cisco UCS 885A techniques. Finetuning runs accomplished inside hours of coaching time, enabling speedy experimentation throughout a number of dataset sizes and configurations. Using artificial knowledge technology eradicated the necessity for guide labeling, considerably decreasing overhead. This strategy allowed Cisco to iterate shortly, observe efficiency traits early, and validate whether or not embedding fine-tuning was value additional funding. The general time-to-value was considerably shorter than earlier efforts, with significant insights gained after solely a small variety of runs.
The five-stage pipeline structure:
Timings primarily based on ~925 paperwork / ~9,200 QA pairs / ~7,800 coaching examples on a single NVIDIA H200 GPU operating on Cisco AI Pods with Cisco UCS 885A infrastructure. Precise period scales with knowledge quantity.
Accuracy good points noticed
Throughout a number of experiments, the outcomes confirmed constant, measurable enhancements. Superb-tuning the NVIDIA 1-billion-parameter NV-EmbedQA mannequin on artificial domain-specific knowledge yielded good points throughout all retrieval metrics, with NDCG@1 good points of +7.1 to +7.3 absolute factors (+9.9% to +11.1% relative). Recall@10 improved by as much as +6.8 factors (+8.5%), and MAP@10 by as much as +6.5 factors (+9.7%). Utilizing an on-premise 120B-parameter LLM for artificial knowledge technology, the complete pipeline ran with zero exterior API prices and with the info staying fully on prem ensured knowledge privateness. These good points held at the same time as dataset dimension elevated and retrieval duties turned more difficult. Importantly, enhancements have been noticed on domain-specific queries that beforehand carried out poorly with base embedding fashions. Whereas these outcomes signify an preliminary baseline somewhat than a completely optimized end result, they offered sturdy affirmation that embedding fine-tuning can materially enhance retrieval high quality for enterprise-specific knowledge.
Abstract of experiments
Desk 1. Retrieval efficiency comparability between the bottom embedding mannequin and the contrastively fine-tuned mannequin throughout two dataset sizes (334 and 925 paperwork). Superb-tuning persistently improves rating high quality throughout all BEIR analysis metrics.
Key Observations:
- Superb-tuning persistently improved retrieval high quality throughout all metrics.
- NDCG@1 confirmed the biggest enchancment in top-level relevance.
- Positive factors have been secure throughout the 2 dataset sizes (334 and 925 paperwork).
- Improved Recall@10 and Map@10 good points indicative of higher protection and rating than the bottom embedding mannequin.
What stunned us
Essentially the most surprising discovering was how shortly the recipe delivered actionable outcomes. Inside a couple of days of beginning the experiment, we had measurable accuracy enhancements — a stark distinction to earlier efforts that took weeks to months. The artificial knowledge technology strategy produced coaching alerts of adequate high quality to drive significant good points with out a single manually labeled instance. We have been additionally stunned by how effectively the enhancements generalized throughout question sorts, together with the rare-token identifier queries that had traditionally been the weakest level for semantic search.
Subsequent steps with engagement
Constructing on these outcomes, Cisco will proceed working with NVIDIA to systematically push accuracy additional. The following section of labor will focus on:
- Utilizing a hard and fast analysis set throughout runs in order that metrics shall be immediately comparable
- Tuning the educational price (making an attempt default, half, and double) and rising epochs from 3 to five
- Scaling coaching knowledge to ~100K QA pairs to seek out the saturation level for the area
- Utilizing a bigger or higher-quality LLM for artificial knowledge technology to enhance QA pair constancy
- Making use of 10% warmup with cosine decay for extra secure convergence
- Rising hard-negative mining from 5 to 10 negatives per question for a stronger contrastive sign
- Refining artificial knowledge technology prompts to raised emphasize uncommon and domain-specific phrases — bug IDs, product identifiers, firmware variations — the place base fashions battle most
- Exploring chunk-aware coaching: utilizing actual doc chunks from a manufacturing vector database because the retrieval corpus, producing questions in opposition to these chunks through the LLM, and mapping every query to its optimistic chunk and hard-negative chunks — coaching the mannequin on the identical knowledge distribution it will encounter in manufacturing, the place solutions could also be buried in longer textual content and chunking methods will differ
Long run, the engagement will increase to incorporate re-ranker fine-tuning and broader retrieval optimization as a part of a full end-to-end RAG enchancment effort.
Worth of the fine-tuning embedding mannequin
This experiment helps that leveraging a fine-tuning embedding mannequin can speed up time to manufacturing by offering a validated, end-to-end fine-tuning workflow that delivers measurable enhancements in days somewhat than months. The concepts and findings from this work are actively shaping the recipe’s evolution, whereas Cisco good points early entry to a maturing pipeline that shortens the trail from experimentation to manufacturing. The work additionally demonstrates how Cisco AI Pods primarily based on Cisco UCS 885A techniques and NVIDIA H200 GPUs can present an efficient enterprise infrastructure basis for speedy embedding mannequin adaptation.
Key fine-tuning embedding mannequin advantages for companies
- Defend proprietary knowledge (on-premises execution)
- Cut back assist prices (quicker decision, fewer escalations)
- No cloud API dependency (zero exterior prices)
- Quick time-to-value (full end-to-end pipeline — all 5 levels together with SDG, mining, coaching, analysis, and export — completes in 2-5 hours on a single GPU)
Key fine-tuning embedding mannequin advantages for builders
- No guide annotation required (artificial knowledge technology)
- Modular, hackable structure (5 distinct levels: SDG → Knowledge Prep → Superb-Tune → Consider → Export)
- Manufacturing-ready outputs (ONNX export)
- Constructed-in analysis (BEIR — Benchmarking Info Retrieval — framework)
- Onerous unfavourable mining included (computerized high quality increase)
Get began
The fine-tuning recipe for Llama Nemotron Embed 1B mannequin is obtainable now as an entire, production-ready pipeline. Whether or not you’re constructing enterprise search, RAG functions, or domain-specific retrieval techniques, this recipe offers a transparent path from uncooked paperwork to deployed, domain-adapted embeddings.
Able to fine-tune your individual embedding mannequin?
👉 Discover the Nemotron Embed Superb-Tuning Recipe on GitHub
