11. October 2023
Vector Representation: Unlocking the Power of AI and ML in Data Processing
AI and ML are technologies that help computers learn from data and make decisions or predictions. They can work with various types of data, including:
- Images: AI/ML can analyze and understand pictures, which is useful in applications like image recognition, image search, or medical imaging.
- Unstructured Texts: This involves working with text that doesn't follow a specific format, like social media posts, business contracts, blogs, or academic articles. AI/ML can extract meaning and insights from this kind of data.
- Audio/Video: These technologies can process audio and video data, making it possible to, for example, transcribe spoken words or recognize patterns and actions in videos.
- Users/Items Profiles: AI/ML can analyze user profiles (like your social media or shopping behavior) and item profiles (such as product descriptions) to make recommendations or personalize content.
- Histories: AI/ML can learn from historical data, like past sales records or user interactions, to make predictions or optimize processes.
In the world of AI and ML, data is often represented as vectors, which are arrays of numbers. Conversion of various types of data (like text, images, or audio) into vectors enables machines to understand and work with such data efficiently. Technically, the vector representation in AI/ML, which is frequently called vector embedding, as it embeds (or puts) unstructured data into vector space and aids the neural network or another ML model to process the data.
Many AI/ML models use vectors as their primary way of representing and processing data because they are versatile and can capture complex patterns and relationships. For example, words in a text document can be turned into numerical vectors, making it possible for computers to analyze, visualize, and compare them mathematically. This numerical representation enables AI/ML models to perform tasks like text classification, image recognition, or recommendation systems effectively.
Why vector databases?
This effectiveness, however, is not the only reason why vector databases have become an extremely popular tool of data management, and there are several reasons for that:
- ChatGPT Hype: With the skyrocketing popularity of ChatGPT and similar AI chatbots, vector databases play a vital role. They enable these chatbots to understand and respond to user queries more intelligently by efficiently storing and retrieving vast amounts of text data.
- Semantic Search: When you're searching for something on the Internet, you definitely want to achieve precise results. Vector databases power semantic search engines, making your online searches more accurate. They can find not just what you ask for but also what you mean, improving your search experience.
- Recommendations Whether it's Netflix suggesting your next binge-watch or Amazon recommending products you might love, recommendations are powered by vector databases. They analyze your preferences and past behavior to offer you tailored suggestions, making your online interactions more personalized and enjoyable.
In essence, vector databases are the backbone of modern technology, enhancing the capabilities of AI, improving search accuracy, and making our digital experiences smarter and more convenient.
The process that happens and makes vector databases so efficient is visually presented on this scheme:
The idea of the vector index
One of the main features that make vector databases so efficient within the semantic search context is the vector index. Let’s find out what it is and why it is so important.
First, we should notice that vector databases, like usual SQL databases, use indexing for faster access to information. For proper understanding imagine the index at the end of a book, which enables you to find important concepts, chapters, and pages faster. But, as vectors cannot be indexed alphabetically (as they are just sequences of numbers, and no individual number having special meaning, the semantics is contained in a vector as a whole), it is based on special indexes and proper use of vector database frequently requires setting and fine-tuning of the index.
In the simplest case, one can just directly compare the query vector with all vectors in a database. Though sounding naïve and inconvenient, it is the most accurate search and, with modern computers and for a reasonably small number of records (up to tens of thousand), even such a simple approach works perfectly. Moreover, this is the simplest method when we talk about the possibility to add new vectors to a database. Technically, this approach is known as the ‘flat’ index.
When the number of vectors grows (modern applications require millions and billions of records), the flat index becomes truly inefficient. Through that, several efficient approaches were developed, such as grouping (clustering) of vectors.
Modern vector indexes enable both fast search and compression of data, so an instance with less RAM can work with larger database. For a better understanding, you can look at the scheme.
However, the performance comes with two undesirable additions:
- First, the advanced vector index takes significant time. Usually, a combination of flat index and more advanced index is used. In other words, the vector database adds new vectors into flat index and only optimizes (reindexes) the data during off-peak times.
- Second is that advanced index types have accuracy/performance trade-off. And this is the side effect of a fast search through millions of data. The faster the required result is produced, the higher is the chance not all relevant vectors are returned.
- First, the advanced vector index takes significant time. Usually, a combination of flat index and more advanced index is used. In other words, the vector database adds new vectors into flat index and only optimizes (reindexes) the data during off-peak times.
- Second is that advanced index types have accuracy/performance trade-off. And this is the side effect of a fast search through millions of data. The faster the required result is produced, the higher is the chance not all relevant vectors are returned.
In many cases (such as recommendation systems, online advertising, and many types of search), the decreased accuracy can be not an issue. For more critical applications, it is recommended to regularly test and monitor both performance (search time) and accuracy of search and optimize the hardware (for example, number and type of cloud instances) and the vector database settings according to the project needs.
Examples of vector indexes
Currently, there are few vector databases on the market, with new solutions appearing every few months. Roughly, and for the proper understanding, the vector databases can be divided into the following groups:
- Open-source solutions, usually with a cloud hosting option. Examples: Milvus/Ziliz, Qdrant, Weaviate;
- NoSQL databases or search engines with vector search functionality added to recent versions. Examples: Redis, Elastic search;
- Vector search extensions for SQL databases, such as pgvector or sqlite-vss;
- SaaS solutions, such as Azure Semantic Search or Pinecone;
- Vector search (indexing) libraries, mostly focused on developers; Examples: FAISS or HNSWlib.
Usually, all vector databases store a vector for similarity search, ID of record for linking to other data storages, and metadata (such as text fragments, publication date, author, and similar). Many vector databases support filtering of records by metadata, thus enabling hybrid search – when semantic similarity is combined with SQL-like query for even more flexible information retrieval.
Was this information useful?
If all this information seems sophisticated and you don’t see the profit for your business, you can always apply for the help of professionals. Silk Data offers you comprehensive expertise and many years of practical commercial experience. You can ask for mere consulting or full-time project cooperation – Silk Data can cover all these tasks. Contact us, and we are ready to help.
Frequently Asked Questions
Almost all modern vector databases, both open source and proprietary, are based on similar well-proven algorithms. The main difference between them is functionality related to metadata filtering, hybrid search, scaling, and integration with SQL database or search index. Therefore, the selection of a vector database for your case will be mostly governed by the business requirements of your project. Examples of the most important business requirements are the scale (number of images, texts, or other items), requirements for filtering or special handling of domain-specific keywords, and integration with an existing infrastructure.
Scaling of vector database is no different from other software solutions: one can choose the vertical (that is, switch to a server with more CPU and RAM), horizontal (using several servers and data splitting or sharding), or both. Most modern vector databases support both vertical and horizontal scaling in the cloud. The specific choice of the scaling strategy depends on the project’s needs.
All vector databases support well-documented REST API, and adding the vector database to the project is no different from adding another external service. However, the important point of vector databases is to create the vectors. Usually, vectors are calculated by an AI/ML model, such as a neural network. Therefore, each image, document, or another item of information should be processed by this model. For millions and billions of data items, populating vector database with data may take a lot of time. Various scaling solutions exist to overcome this bottleneck, from using several models in parallel to running a neural network on server with GPU/TPU (graphical/tensor processing unit, usually a special hardware from NVIDIA or Google). The selection of the optimal approach depends on the type and size of data, the type of AI model, and other details of the project.
Surely, this part of the AI landscape is mature enough. There are special kind of software frameworks, named ETL (usually meaning extract, transform, load or embed, transform, load) that are tailored to get the data from existing storage (be it a database or a data lake), process it by ML model and put to a database. The use of ETL tools is related mainly to MLOps, as the deployment, monitoring, and support of AI solutions. As before, the selection of tools depends on the data and business requirements for your project.
