Comment on page
Solr
LangStream allows you to use Apache Solr as a vector database. This is useful if you want to use a vector database that is hosted on-premises or in your own cloud environment.
To use Apache Solr as a vector database, create a "vector-database" resource in your configuration.yaml file.
resources:
- type: "vector-database"
name: "SolrDatasource"
configuration:
service: "solr"
username: "${ secrets.solr.username }"
password: "${ secrets.solr.password }"
host: "${ secrets.solr.host }"
port: "${ secrets.solr.port }"
collection-name: "${ secrets.solr.collection-name }"
Explanation for the parameters:
- username: the username
- password: the password
- host: the host
- port: the port, usually it is 8983
- collection-name: the name of the collection to connect to
Currently LangStream supports connecting to one Collection at a time, so you need to create a separate resource for each Collection.
LangStream uses the official Apache Solr Java client to connect to Solr.
You can use both Solr and Solr Cloud. If you use Solr Cloud then you can also manage the collections using the Solr Cloud API.
To do that, you need to create special assets in your pipeline: "solr-collection".
Here is an example of creating a collection named "documents" with a dense vector, as required to perform Vector Similarity Searches:
assets:
- name: "documents-table"
asset-type: "solr-collection"
creation-mode: create-if-not-exists
deletion-mode: delete
config:
collection-name: "documents"
datasource: "SolrDataSource"
create-statements:
- api: "/api/collections"
method: "POST"
body: |
{
"name": "documents",
"numShards": 1,
"replicationFactor": 1
}
- "api": "/schema"
"body": |
{
"add-field-type" : {
"name": "knn_vector",
"class": "solr.DenseVectorField",
"vectorDimension": "1536",
"similarityFunction": "cosine"
}
}
- "api": "/schema"
"body": |
{
"add-field":{
"name":"embeddings",
"type":"knn_vector",
"stored":true,
"indexed":true
}
}
- "api": "/schema"
"body": |
{
"add-field":{
"name":"text",
"type":"string",
"stored":true,
"indexed":false,
"multiValued": false
}
}
As you can see in the "create-statements" section above, you can configure a number of "commands" that translate to Solr API calls. You can invoke any of the APIs, for each command you have to declare:
- the "api" you want to call: "/schema" or "/api/collections"
- the body of the HTTP Request, as a JSON string
- you can also set the HTTP method, if you want to use something different than "POST"
If you set "api" to "/schema" then you are using the Solr Schema API. If you set "api" to "/api/collections" then you are using the Solr Collections API.
Use the "query-vector-db" agent to query Solr with the following parameters:
- name: "lookup-related-documents"
type: "query-vector-db"
configuration:
datasource: "SolrDataSource"
query: |
{
"q": "{!knn f=embeddings topK=10}?"
}
fields:
- "fn:toListOfFloat(value.question_embeddings)"
output-field: "value.related_documents"
As usual you can use the '?' symbol as a placeholder for the fields that you specify in the "q" section.
For Apache Solr, the "query" field requires a JSON that describes the parameters to pass in the Solr query. Usually you provide a value for "q" for the main query string, and then add other parameters.
Use the "vector-db-sink" agent to write to Solr with the following parameters:
- name: "Write to Solr"
type: "vector-db-sink"
input: chunks-topic
configuration:
datasource: "SolrDataSource"
collection-name: "documents"
fields:
- name: "id"
expression: "fn:concat(value.filename, value.chunk_id)"
- name: "embeddings"
expression: "fn:toListOfFloat(value.embeddings_vector)"
- name: "text"
expression: "value.text"
Set the collection-name to the name of the collection you want to write to, and define the fields in the "fields" list. This works similarly to the 'compute' agent, where you define the name of the field and the expression to compute the value of the field.
Last modified 1mo ago