Youtube: Open NLP meetup #7
Slides: A Practical Introduction to Image Retrieval
Colab: MultiModalRetriever - Live coding
All the material can also be found here.
A Practical Introduction to Image Retrieval Link to heading
by Sara Zanzottera from deepset
Search should not be limited to text only. Recently, Transformers-based NLP models started crossing the boundaries of text data and exploring the possibilities of other modalities, like tabular data, images, audio files, and more. Text-to-text generation models like GPT now have their counterparts in text-to-image models, like Stable Diffusion. But what about search? In this talk we’re going to experiment with CLIP, a text-to-image search model, to look for animals matching specific characteristics in a dataset of pictures. Does CLIP know which one is “The fastest animal in the world”?
For the 7th OpenNLP meetup I presented the topic of Image Retrieval, a feature that I’ve recently added to Haystack in the form of a MultiModal Retriever (see the Tutorial).
The talk consists of 5 parts:
- An introduction of the topic of Image Retrieval
- A mention of the current SOTA model (CLIP)
- An overview of Haystack
- A step-by-step description of how image retrieval applications can be implemented with Haystack
- A live coding session where I start from a blank Colab notebook and build a fully working image retrieval system from the ground up, to the point where I can run queries live.
Towards the end I mention briefly an even more advanced version of this image retrieval system, which I had no time to implement live. However, I later built a notebook implementing such system and you can find it here: Cheetah.ipynb
The slides were generated from the linked Jupyter notebook with jupyter nbconvert Dec_1st_OpenNLP_Meetup.ipynb --to slides --post serve.
This was my most popular talk to date, with almost a hundred attendees watching live.