Using the Weaviate Vector Store connector (Preview)

Overview

The Weaviate Vector Store connector can be used to access and manage data in Weaviate. The connector has the following characteristics.

Limitations

Notable Weaviate connector functionality limitations.

Getting started

Add the Weaviate Vector Store connector NuGet package to your project.

dotnet add package Microsoft.SemanticKernel.Connectors.Weaviate --prerelease

You can add the vector store to the dependency injection container available on the KernelBuilder or to the IServiceCollection dependency injection container using extension methods provided by Semantic Kernel. The Weaviate vector store uses an HttpClient to communicate with the Weaviate service. There are two options for providing the URL/endpoint for the Weaviate service. It can be provided via options or by setting the base address of the HttpClient.

This first example shows how to set the service URL via options. Also note that these methods will retrieve an HttpClient instance for making calls to the Weaviate service from the dependency injection service provider.

using Microsoft.SemanticKernel;

// Using Kernel Builder.
var kernelBuilder = Kernel
    .CreateBuilder()
    .AddWeaviateVectorStore(options: new() { Endpoint = new Uri("http://localhost:8080/v1/") });

using Microsoft.SemanticKernel;

// Using IServiceCollection with ASP.NET Core.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddWeaviateVectorStore(options: new() { Endpoint = new Uri("http://localhost:8080/v1/") });

Overloads where you can specify your own HttpClient are also provided. In this case it's possible to set the service url via the HttpClient BaseAddress option.

using System.Net.Http;
using Microsoft.SemanticKernel;

// Using Kernel Builder.
var kernelBuilder = Kernel.CreateBuilder();
using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") };
kernelBuilder.AddWeaviateVectorStore(client);

using System.Net.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.SemanticKernel;

// Using IServiceCollection with ASP.NET Core.
var builder = WebApplication.CreateBuilder(args);
using HttpClient client = new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") };
builder.Services.AddWeaviateVectorStore(client);

You can construct a Weaviate Vector Store instance directly as well.

using System.Net.Http;
using Microsoft.SemanticKernel.Connectors.Weaviate;

var vectorStore = new WeaviateVectorStore(
    new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") });

It is possible to construct a direct reference to a named collection.

using System.Net.Http;
using Microsoft.SemanticKernel.Connectors.Weaviate;

var collection = new WeaviateVectorStoreRecordCollection<Hotel>(
    new HttpClient { BaseAddress = new Uri("http://localhost:8080/v1/") },
    "skhotels");

If needed, it is possible to pass an Api Key, as an option, when using any of the above mentioned mechanisms, e.g.

using Microsoft.SemanticKernel;

var kernelBuilder = Kernel
    .CreateBuilder()
    .AddWeaviateVectorStore(options: new() { Endpoint = new Uri("http://localhost:8080/v1/"), ApiKey = secretVar });

Data mapping

The Weaviate Vector Store connector provides a default mapper when mapping from the data model to storage. Weaviate requires properties to be mapped into id, payload and vectors groupings. The default mapper uses the model annotations or record definition to determine the type of each property and to do this mapping.

• The data model property annotated as a key will be mapped to the Weaviate id property.
• The data model properties annotated as data will be mapped to the Weaviate properties object.
• The data model properties annotated as vectors will be mapped to the Weaviate vectors object.

The default mapper uses System.Text.Json.JsonSerializer to convert to the storage schema. This means that usage of the JsonPropertyNameAttribute is supported if a different storage name to the data model property name is required.

Here is an example of a data model with JsonPropertyNameAttribute set and how that will be represented in Weaviate.

using System.Text.Json.Serialization;
using Microsoft.Extensions.VectorData;

public class Hotel
{
    [VectorStoreRecordKey]
    public ulong HotelId { get; set; }

    [VectorStoreRecordData(IsFilterable = true)]
    public string HotelName { get; set; }

    [VectorStoreRecordData(IsFullTextSearchable = true)]
    public string Description { get; set; }

    [JsonPropertyName("HOTEL_DESCRIPTION_EMBEDDING")]
    [VectorStoreRecordVector(4, DistanceFunction.EuclideanDistance, IndexKind.QuantizedFlat)]
    public ReadOnlyMemory<float>? DescriptionEmbedding { get; set; }
}

{
    "id": 1,
    "properties": { "HotelName": "Hotel Happy", "Description": "A place where everyone can be happy." },
    "vectors": {
        "HOTEL_DESCRIPTION_EMBEDDING": [0.9, 0.1, 0.1, 0.1],
    }
}

Coming soon

More info coming soon.

Coming soon

More info coming soon.