Milvus: Open-Source Vector Database for Similarity Search

Overview

Milvus is an open-source vector database system specifically engineered for managing and searching embedding vectors derived from unstructured data. Founded in 2019, it addresses the challenge of similarity search across vast datasets, which is foundational for many artificial intelligence and machine learning applications Milvus Overview documentation. Unstructured data, such as images, audio, video, and natural language text, can be transformed into numerical vectors (embeddings) using machine learning models. Milvus then stores these vectors and provides mechanisms to quickly find the most similar vectors to a given query vector.

The system is designed for high performance and scalability, capable of handling billions of vector embeddings. Its architecture separates storage and compute resources, allowing for independent scaling to meet varying workload demands. This makes Milvus suitable for use cases requiring real-time similarity search, such as recommendation engines, where users are matched with similar items; image and video search, where visual content is retrieved based on similarity; and generative AI applications, which often rely on retrieving contextually relevant information from large knowledge bases Milvus use cases. For instance, in a retrieval-augmented generation (RAG) system, Milvus can store and retrieve relevant documents to provide context to a large language model LlamaIndex Milvus integration example.

Milvus offers both a self-hosted open-source version and a managed cloud service called Zilliz Cloud. The open-source variant provides developers with control over their infrastructure, while Zilliz Cloud simplifies deployment, scaling, and operational management, offering a free tier for initial exploration. The database supports various indexing algorithms (e.g., HNSW, IVF_FLAT) to optimize search performance and accuracy based on specific application needs. Its comprehensive SDKs for popular languages like Python, Java, and Go, coupled with extensive documentation, aim to facilitate developer integration and enhance the developer experience Milvus Quickstart guide.

Key features

• Vector Indexing Algorithms: Supports multiple indexing algorithms (e.g., HNSW, IVF_FLAT, ANNOY) to optimize for query speed, recall, and storage efficiency based on dataset characteristics and application requirements Milvus Index documentation.
• Scalable Architecture: Features a distributed architecture that separates compute and storage, enabling independent scaling of components to handle large-scale vector datasets and high query throughput Milvus architecture overview.
• High Availability and Reliability: Designed for fault tolerance with data replication and automatic failover mechanisms to ensure continuous operation and data durability.
• Multi-Language SDKs: Provides client SDKs for Python, Java, Go, Node.js, and C++ to facilitate integration into diverse application environments Milvus SDKs overview.
• Data Management Operations: Supports CRUD (Create, Retrieve, Update, Delete) operations on vector data, allowing for dynamic updates to datasets Milvus data insertion guide.
• Filtering and Hybrid Search: Offers attribute filtering alongside vector similarity search, enabling more precise queries by combining vector comparisons with structured metadata conditions Milvus search operations.
• Cloud-Native Deployment: Can be deployed on Kubernetes, enabling containerized operation and integration with cloud-native ecosystems Deploy Milvus on Kubernetes.
• Managed Service (Zilliz Cloud): A fully managed service that abstracts away operational complexities, offering simplified deployment, automated scaling, and enterprise-grade features and compliance Zilliz Cloud homepage.

Pricing

Milvus is an open-source project and is free to use. Zilliz Cloud, the managed service for Milvus, offers usage-based pricing with a free tier. Pricing is determined by compute units (CU-hours) and storage (GB-months).

For detailed and current pricing information, refer to the Zilliz Cloud pricing page.

Common integrations

• LangChain: Integration with LangChain allows Milvus to serve as a vector store for large language model (LLM) applications, facilitating retrieval-augmented generation (RAG) and semantic search LangChain Milvus integration.
• LlamaIndex: Milvus can be used as a vector database within LlamaIndex applications for indexing and querying external data sources to augment LLMs LlamaIndex Milvus blog post.
• PyTorch/TensorFlow: Embeddings generated by machine learning models built with PyTorch or TensorFlow can be directly ingested into Milvus for similarity search Milvus PyTorch integration example.
• Apache Spark: Used for processing and transforming large datasets before ingesting them into Milvus, particularly in big data pipelines Milvus Spark integration documentation.
• MinIO/AWS S3: Object storage solutions like MinIO or AWS S3 can be used for storing raw unstructured data that is then vectorized and indexed in Milvus.
• Grafana/Prometheus: Monitoring tools like Grafana and Prometheus can be integrated to track Milvus performance metrics and operational health Milvus Prometheus monitoring documentation.

Alternatives

• Pinecone: A fully managed vector database service focused on performance and ease of use for AI applications.
• Weaviate: An open-source vector database that combines vector search with graph-like data capabilities and supports various data types.
• Qdrant: An open-source vector similarity search engine that provides a production-ready service with a Go-based core.
• Elasticsearch with k-NN: While primarily a search engine, Elasticsearch offers k-Nearest Neighbor (k-NN) search capabilities for vector embeddings, enabling similarity search on existing data Elasticsearch k-NN search.
• PostgreSQL with pgvector: An extension for PostgreSQL that adds vector similarity search capabilities, suitable for integrating vector search into existing relational database workflows pgvector GitHub repository.

Getting started

To get started with Milvus using its Python SDK, you can install the client and then connect to a Milvus instance (either local or Zilliz Cloud). The following example demonstrates creating a collection, inserting data, and performing a basic vector similarity search.

from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection

# 1. Connect to Milvus
# For a local Milvus instance, use target=\"localhost:19530\"
# For Zilliz Cloud, replace with your endpoint and API key:
# connections.connect(
#   alias=\"default\", 
#   uri=\"YOUR_CLUSTER_ENDPOINT\", 
#   token=\"YOUR_API_KEY\"
# )
connections.connect(alias="default", host="localhost", port="19530") # Connect to a local Milvus instance

# 2. Define schema for a collection
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=False),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128)
]
schema = CollectionSchema(fields, "This is a test collection for Milvus SDK tutorial")

# 3. Create a collection
collection_name = "hello_milvus"
hello_milvus = Collection(collection_name, schema, consistency_level="Strong")

# 4. Prepare data for insertion
# In a real application, 'data' would be actual vector embeddings from your unstructured data.
# Here, we generate some dummy data.
import random

num_entities = 100
data = [
    [i for i in range(num_entities)],  # IDs
    [[random.random() for _ in range(128)] for _ in range(num_entities)]  # 128-dim vectors
]

# 5. Insert data
hello_milvus.insert(data)
hello_milvus.flush()
print(f"Number of entities in collection: {hello_milvus.num_entities}")

# 6. Create an index for the vector field
# Using IVF_FLAT for demonstration, HNSW is often preferred for high performance.
index_params = {
    "index_type": "IVF_FLAT",
    "metric_type": "L2",
    "params": {"nlist": 128}
}
hello_milvus.create_index("embedding", index_params)

# 7. Load the collection into memory for searching
hello_milvus.load()

# 8. Prepare a query vector
query_vector = [[random.random() for _ in range(128)]]

# 9. Perform a vector similarity search
search_params = {"metric_type": "L2", "params": {"nprobe": 10}}
results = hello_milvus.search(
    data=query_vector,
    anns_field="embedding",
    param=search_params,
    limit=3,  # Return top 3 most similar results
    output_fields=["id"]
)

print("\nSearch Results:")
for hit in results[0]:
    print(f"ID: {hit.id}, Distance: {hit.distance}")

# 10. Release the collection from memory (optional)
hello_milvus.release()

# 11. Drop the collection (optional)
# hello_milvus.drop()

This Python example demonstrates the basic workflow: connecting to Milvus, defining a schema, creating a collection, inserting vector data, building an index, loading the collection, and performing a similarity search. For more advanced scenarios and detailed API usage, refer to the Milvus API reference and official documentation.