Dance Between Dense and Sparse Embeddings: Enabling Hybrid Search in LangChain-Milvus | Omri Levy and Ohad Eytan

Should you swap the queries between the 2 examples above, and use every one with the opposite’s embedding, each will produce the improper outcome. This demonstrates the truth that every technique has its strengths but additionally its weaknesses. Hybrid search combines the 2, aiming to leverage the perfect from each worlds. By indexing information with each dense and sparse embeddings, we are able to carry out searches that think about each semantic relevance and key phrase matching, balancing outcomes based mostly on customized weights. Once more, the inner implementation is extra sophisticated, however langchain-milvus makes it fairly easy to make use of. Let’s take a look at how this works:

vector_store = Milvus(
embedding_function=[
sparse_embedding,
dense_embedding,
],
connection_args={"uri": "./milvus_hybrid.db"}, 
auto_id=True,
)
vector_store.add_texts(paperwork)

On this setup, each sparse and dense embeddings are utilized. Let’s check the hybrid search with equal weighting:

question = "Does Sizzling cowl climate modifications throughout weekends?"
hybrid_output = vector_store.similarity_search(
question=question,
okay=1,
ranker_type="weighted",
ranker_params={"weights": [0.49, 0.51]},  # Mix each outcomes!
)
print(f"Hybrid search outcomes:n{hybrid_output[0].page_content}")
# output: Hybrid search outcomes:
#         In Israel, Sizzling is a TV supplier that broadcast 7 days every week

This searches for related outcomes utilizing every embedding operate, provides every rating a weight, and returns the outcome with the perfect weighted rating. We will see that with barely extra weight to the dense embeddings, we get the outcome we desired. That is true for the second question as effectively.

If we give extra weight to the dense embeddings, we’ll as soon as once more get non-relevant outcomes, as with the dense embeddings alone:

question = "When and the place is Sizzling energetic?"
hybrid_output = vector_store.similarity_search(
question=question,
okay=1,
ranker_type="weighted",
ranker_params={"weights": [0.2, 0.8]},  # Observe -> the weights modified
)
print(f"Hybrid search outcomes:n{hybrid_output[0].page_content}")
# output: Hybrid search outcomes:
#         Immediately was very heat in the course of the day however chilly at evening

Discovering the suitable steadiness between dense and sparse isn’t a trivial activity, and could be seen as a part of a wider hyper-parameter optimization downside. There may be an ongoing analysis and instruments that attempting to resolve such points on this space, for instance IBM’s AutoAI for RAG.

There are lots of extra methods you’ll be able to adapt and use the hybrid search strategy. As an illustration, if every doc has an related title, you could possibly use two dense embedding features (probably with completely different fashions) — one for the title and one other for the doc content material — and carry out a hybrid search on each indices. Milvus at the moment helps as much as 10 completely different vector fields, offering flexibility for complicated functions. There are additionally further configurations for indexing and reranking strategies. You’ll be able to see Milvus documentation concerning the obtainable params and choices.