vector-db-sink
This agent writes vector data to vector databases. LangStream currently supports AstraDB and Pinecone.
Astra DB and Pinecone are both of the type "vector-db-sink" in a LangStream pipeline, but the databases require different configuration values to map the vector data from the sink into the database.
Astra DB example
The Astra DB vector database connection is defined in configuration.yaml:
The "Write to Astra DB" pipeline step takes embeddings as input from "input-topic" and writes them to the configured datasource "AstraDatasource":
AstraDB Topics
Input
Output
None, it’s a sink.
AstraDB Configuration
Label | Type | Description |
---|---|---|
datasource | String | The datasource is defined in the Resources section of configuration.yaml. |
table | String | The `keyspace.table-name` the vector data will be written to |
mapping | String | How the data from the input records will be mapped to the corresponding columns in the database table. "id=value.id" maps the "id" value in the input record to the "id" value of the database. |
Pinecone Example
The "Write to Pinecone" pipeline step takes embeddings as input from "vectors-topic" and writes them to a Pinecone datasource.
The Pinecone vector database connection is defined in configuration.yaml:
The "Write to Pinecone" pipeline step takes embeddings as input from "input-topic" and writes them to the configured datasource "PineconeDatasource":
Pinecone Topics
Input
Output
None, it’s a sink.
Pinecone Configuration
Label | Type | Description |
---|---|---|
datasource | String | The datasource is defined in the Resources section of configuration.yaml. |
vector.id | String | Maps id to vector.id |
vector.vector | String | Maps the input value "vector" to "vector.vector" in the database. |
vector.namespace | String | Maps the input value "namespace" to "vector.namespace" in the database. |
vector.metadata.{metadataField} | String | Maps the input value "metadata.{metadataField}" to "vector.metadata.{metadataField}" |
Last updated