Creating vector embeddings

Overview

To use Aerospike Vector Search (AVS), you must build an application that generates vector embeddings. This page outlines some general approaches for generating vector embeddings using Python and a machine learning model.

Generate embeddings using a hosted service

Using a hosted model like OpenAI provides ease of use, quick deployment, and access to cutting-edge technology without the need for significant infrastructure investment. It ensures scalability, automatic updates, and maintenance, allowing organizations to focus on application development rather than managing the underlying model infrastructure.

1. Install the OpenAI Python client library if you haven't already:

   pip install openai

2. Use the following Python code to generate a vector embedding:

   import openai
   
   # Set your OpenAI API key
   openai.api_key = 'your-api-key-here'
   
   # Define the text chunk for which you want to generate an embedding
   text_chunk = "OpenAI's GPT-4 is a powerful language model capable of performing a wide range of natural language processing tasks."
   
   # Generate the embedding
   response = openai.Embedding.create(
       input=text_chunk,
       model="text-embedding-ada-002"
   )
   
   # Extract the embedding vector
   embedding_vector = response['data'][0]['embedding']
   
   # Print the embedding vector
   print(embedding_vector)

Self-host an open-source model

Self-hosting a machine learning model offers enhanced data privacy, security, and control over the environment, making it easier to comply with regulatory requirements and optimize performance. It can also be more cost-effective for high usage scenarios, eliminating dependency on third-party providers and reducing latency.

The following example shows how you can generate a vector embedding from a chunk of text using the LLaMA model. You can use the generated vector for various downstream tasks such as similarity searches and other vector computations.

1. Install the required libraries:

   pip install transformers
   pip install torch

2. Use the following Python code to generate a vector embedding:

   from transformers import AutoTokenizer, AutoModel
   import torch
   
   # Load the LLaMA model and tokenizer
   model_name = "facebook/llama-7b"  # Replace with the correct model name
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   model = AutoModel.from_pretrained(model_name)
   
   # Define the text chunk for which you want to generate an embedding
   text_chunk = "LLaMA is a powerful language model capable of performing a wide range of natural language processing tasks."
   
   # Tokenize the text chunk
   inputs = tokenizer(text_chunk, return_tensors="pt")
   
   # Generate the embeddings
   with torch.no_grad():
       outputs = model(**inputs)
       # The embeddings are typically in the 'last_hidden_state' tensor
       embeddings = outputs.last_hidden_state
   
   # Average the token embeddings to get a single vector representation
   embedding_vector = torch.mean(embeddings, dim=1).squeeze().numpy()
   
   # Print the embedding vector
   print(embedding_vector)

Additional resources

1. Hugging Face Model Hub: A comprehensive repository of pre-trained models for various natural language processing (NLP) tasks, computer vision, and more.

2. TensorFlow Hub: A library of reusable MLLs for TensorFlow, offering models for text, image, and audio processing.

3. PyTorch Hub: A repository of pre-trained PyTorch models, facilitating easy integration and deployment for various machine learning tasks.

4. Kaggle: An online community for data scientists and machine learning practitioners, providing datasets, notebooks, and pre-trained models.

5. GitHub: A vast platform hosting a multitude of open-source projects, including repositories for MLLs, tools, and frameworks.

6. OpenVINO Model Zoo: A collection of pre-trained models optimized for Intel hardware, supporting various AI tasks.

7. Model Zoo for Caffe, TensorFlow, PyTorch, MXNet: Collections of pre-trained models specific to different deep learning frameworks, offering models for diverse applications.

8. NVIDIA NGC: A platform offering GPU-optimized deep learning frameworks and pre-trained models for various AI tasks.