Defining your storage schema using a record definition (Preview)

Here is an example of how to create a record definition.

using Microsoft.Extensions.VectorData;

var hotelDefinition = new VectorStoreRecordDefinition
{
    Properties = new List<VectorStoreRecordProperty>
    {
        new VectorStoreRecordKeyProperty("HotelId", typeof(ulong)),
        new VectorStoreRecordDataProperty("HotelName", typeof(string)) { IsFilterable = true },
        new VectorStoreRecordDataProperty("Description", typeof(string)) { IsFullTextSearchable = true },
        new VectorStoreRecordVectorProperty("DescriptionEmbedding", typeof(float)) { Dimensions = 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw },
    }
};

When creating a definition you always have to provide a name and type for each property in your schema, since this is required for index creation and data mapping.

To use the definition, pass it to the GetCollection method.

var collection = vectorStore.GetCollection<ulong, Hotel>("skhotels", hotelDefinition);

Record Property configuration classes

VectorStoreRecordKeyProperty

Use this class to indicate that your property is the key of the record.

new VectorStoreRecordKeyProperty("HotelId", typeof(ulong)),

VectorStoreRecordKeyProperty configuration settings

VectorStoreRecordDataProperty

Use this class to indicate that your property contains general data that is not a key or a vector.

new VectorStoreRecordDataProperty("HotelName", typeof(string)) { IsFilterable = true },

VectorStoreRecordDataProperty configuration settings

VectorStoreRecordVectorProperty

Use this class to indicate that your property contains a vector.

new VectorStoreRecordVectorProperty("DescriptionEmbedding", typeof(float)) { Dimensions = 4, DistanceFunction = DistanceFunction.CosineDistance, IndexKind = IndexKind.Hnsw },

VectorStoreRecordVectorProperty configuration settings

Here is an example of how to create a record definition, for use with a pandas DataFrame.

There are a couple of important things to note, other then the fields definitions themselves. The first is the container_mode parameter. When set to True, this indicates that the data model is a container type, like a DataFrame, and that the data model is therefore a container of records, instead of a single one, a container record can be used in the exact same way, the main difference is that get and get_batch will return the same data type, with a single record for a get and one or more for a get_batch. When you want to do a upsert, upsert and upsert_batch can be used interchangeably, in other words, passing a container to upsert will result in multiple upserts, instead of a single one.

The second is the addition of the to_dict and from_dict methods, which are used to convert between the data model and the storage schema. In this case, the to_dict method is used to convert the DataFrame to a list of records, and the from_dict method is used to convert a list of records to a DataFrame. There can also be a serialize and deserialize method (not shown in the example below), for details on the difference between those see the serialization documentation.

from semantic_kernel.data import (
    VectorStoreRecordDataField,
    VectorStoreRecordDefinition,
    VectorStoreRecordKeyField,
    VectorStoreRecordVectorField,
)

hotel_definition = VectorStoreRecordDefinition(
    fields={
        "hotel_id": VectorStoreRecordKeyField(property_type="str"),
        "hotel_name": VectorStoreRecordDataField(property_type="str", is_filterable=True),
        "description": VectorStoreRecordDataField(
            property_type="str", has_embedding=True, embedding_property_name="description_embedding"
        ),
        "description_embedding": VectorStoreRecordVectorField(property_type="list[float]"),
    },
    container_mode=True,
    to_dict=lambda record, **_: record.to_dict(orient="records"),
    from_dict=lambda records, **_: DataFrame(records),
)

When creating a definition you always have to provide a name (as the key in the fields dict) and type for each property in your schema, since this is required for index creation and data mapping.

To use the definition, pass it to the GetCollection method or a collection constructor, together with the data model type.

collection = vector_store.get_collection(
    collection_name="skhotels", 
    data_model_type=pd.DataFrame, 
    data_model_definition=hotel_definition,
)

Here is an example of how to create a record definition.

var hotelDefinition = VectorStoreRecordDefinition.fromFields(
    Arrays.asList(
        VectorStoreRecordKeyField.builder().withName("hotelId").withFieldType(String.class).build(),
        VectorStoreRecordDataField.builder()
            .withName("name")
            .withFieldType(String.class)
            .isFilterable(true).build(),
        VectorStoreRecordDataField.builder()
            .withName("description")
            .withFieldType(String.class)
            .isFullTextSearchable(true).build(),
        VectorStoreRecordVectorField.builder().withName("descriptionEmbedding")
            .withDimensions(4)
            .withIndexKind(IndexKind.HNSW)
            .withDistanceFunction(DistanceFunction.COSINE_DISTANCE)
            .withFieldType(List.class).build()
    )
);

When creating a definition you always have to provide a name and type for each field in your schema, since this is required for index creation and data mapping.

To use the definition, pass it to the GetCollection method.

var collection = vectorStore.getCollection("skhotels",
        JDBCVectorStoreRecordCollectionOptions.builder()
            .withRecordDefinition(hotelDefinition)
            .build()
    );

Record Field configuration classes

VectorStoreRecordKeyField

Use this class to indicate that your field is the key of the record.

VectorStoreRecordKeyField.builder().withName("hotelId").withFieldType(String.class).build(),

VectorStoreRecordKeyField configuration settings

VectorStoreRecordDataField

Use this class to indicate that your property contains general data that is not a key or a vector.

VectorStoreRecordDataField.builder()
    .withName("name")
    .withFieldType(String.class)
    .isFilterable(true).build(),

VectorStoreRecordDataField configuration settings

VectorStoreRecordVectorField

Use this class to indicate that your field contains a vector.

VectorStoreRecordVectorField.builder().withName("descriptionEmbedding")
    .withDimensions(4)
    .withIndexKind(IndexKind.HNSW)
    .withDistanceFunction(DistanceFunction.COSINE_DISTANCE)
    .withFieldType(List.class).build(),

VectorStoreRecordVectorField configuration settings