Large-scale image search system
Non-fungible tokens (NFTs) are cryptographic assets on a blockchain with unique identification codes and metadata that distinguish them from each other.
Time: 8 months
Key idea
There is a vast amount of NFTs traded every day. The majority of NFTs today contain a vivid image, such as an avatar or football card. And it would be cool if it was possible to search for NFTs or for collections of NFTs in different ways:
- find NFT by a similar image;
- free-text search by images (using AI, without any text descriptions);
- using their names;
- using traits (building blocks, such as a hat or glasses);
- by contract;
- by other different filters.
Steps
1. Data collection.
The data is collected from existing APIs by separate scripts, according to the following algorithm:
- the primary data was loaded at the start of the project;
- in the process of work, scripts regularly fetch new data;
- after collection, the data is indexed for search.
2. Scripts request NFT data from a public API and NFT images from an URL in IPFS distributed data storage.
Two types of search are currently supported:
- full text, using ElasticSearch/OpenSearch;
- AI-based, using the neural network.
How it works
The full-text search allows us to include single or multiple words or phrases in a search query that returns NFTs that match the search condition.
The second type of search uses a neural network, namely the CLIP model developed by OpenAI research and development company.
CLIP is one of a class of deep neural networks that can process both images and text and, for example, find a suitable caption (or tag) for an image or find similar images. Such neural networks can be used to find the text snippet that best represents a given image and vice versa. In other words, modern AI provides a very powerful bridge between computer vision and natural language understanding.
The use of this technology gives a lot of freedom to search against NFTs. It allows to search for the necessary or similar NFTs by image, search for NFT images by description, and so on. In other words, it allows us to link NFT names and descriptions with their images.
Features:
- searching;
- real-time rarity calculation;
- dynamic metadata updates;
- all NFTs in one spot.
Challenges
The client approached our company with a request to develop a system for NFT images that would solve three main functions:
- search for similar images;
- search by description;
- search by collections.
Solution
It is worth noting that the system can be used both for searching for NFTs and for regular pictures. The system's scalability was tested on at least 6 million indexed NFTs. In addition to image search based on ML, you can check how our team has developed an AI-assisted text search.
Project info
