Links

Configuration

A manifest of additional resources and dependencies to run the pipeline.
Example of a data source that needs drivers as well as credentials:
configuration:
resources:
- type: "datasource"
name: "PGDataSource"
configuration:
service: "jdbc"
driverClass: "org.postgresql.Driver"
url: "jdbc:postgresql://postgresql.default.svc.cluster.local:5432/"
user: "postgres"
password: "xxxxxx"
dependencies:
- name: "PostGRES JDBC Driver"
url: "https://jdbc.postgresql.org/download/postgresql-42.6.0.jar"
sha512sum: "ec3b57d8377715ef6286d457b610a2e056aa99db….1"
type: "java-library"
LangStream has built-in support for a few Databases and Vector databases (no need for dependency), for example:
Pinecone

Cassandra (with Vector support)

configuration:
resources:
- type: "datasource"
name: "Cassandra"
configuration:
service: "cassandra"
username: ""
password: ""
contact-points: ""
loadBalancing-localDc: ""

DataStax Astra DB

configuration:
resources:
- type: "vector-database"
name: "AstraDBDatasource"
configuration:
service: "astra"
secret: ""
clientId: ""
token: ""
database: ""

Pinecone

configuration:
resources:
- type: "vector-database"
name: "PineconeDatasource"
configuration:
api-key: ""
environment: ""
index-name: ""
project-name: ""

OSS Milvus and Zillis

configuration:
resources:
- type: "datasource"
name: "MilvusDatasource"
configuration:
service: "milvus"
## OSS Milvus
username: "${secrets.milvus.username}"
password: "${secrets.milvus.password}"
host: "${secrets.milvus.host}"
port: "${secrets.milvus.port}"
## Set to "upsert" for OSS Milvus, on Zills use "delete-insert"
write-mode: "${secrets.milvus.write-mode}"
## Zillis
url: "${secrets.milvus.url}"
token: "${secrets.milvus.token}"

Apache Solr

configuration:
resources:
- type: "vector-database"
name: "SolrDataSource"
configuration:
service: "solr"
user: "${secrets.solr.username}"
password: "${secrets.solr.password}"
host: "${secrets.solr.host}"
port: "${secrets.solr.port}"
collection-name: "documents"

OpenSearch

configuration:
resources:
- type: "vector-database"
name: "OpenSearch"
configuration:
service: "opensearch"
username: "${secrets.opensearch.username}"
password: "${secrets.opensearch.password}"
host: "${secrets.opensearch.host}"
port: "${secrets.opensearch.port}"
index-name: "my-index-000"

Manifest

Root
Node
Type
Description
configuration
Top level node
dependencies
object
A collection of artifacts that a pipeline step of resource may need to run. Refer to the spec below.
Example collection:
  • type: “xxx” name: “xxx” configuration: …
  • type: “xxx” name: “xxx” configuration: …
resources
object
A collection of resources. Refer to the spec below.
Example collection:
  • type: “xxx” name: “xxx” sha: “xxx” …
  • type: “xxx” name: “xxx” sha: “xxx” …

dependencies

The given artifact will be downloaded, validated, and made available to the pipeline. \
Label
Type
Description
type
string (required)
The type of dependency. Supported values are:
  • java-library
Example: “java-library”
name
string (required)
The name of the dependency. It is used for display and as a reference pointer.
Example: "Postgres JDBC Driver"
url
string (required)
A fully qualified URL to the dependency artifact.
Example: "https://jdbc.postgresql.org/download/postgresql-42.6.0.jar"
sha512sum
string (required)
The downloaded artifact is validated against this value.
Example: "ec3b57d8377715ef6286d457…”
These dependencies are downloaded by the LangStream CLI when you run "apps deploy", "docker run" and similar commands. This mechanism is especially useful for JDBC Drivers and for Kafka Connect connectors.
To handle Python dependencies, check out the documentation about developing custom Python agents.

resources

Label
Type
Description
type
string
The type of resource. Refer to the configuration resources reference for naming.
Example: “datasource”
name
string
The name of the resource. It is used for display and as a reference pointer.
Example: "PGDataSource"
configuration
object
Custom configuration for the given resource. Refer to the configuration resources reference for options.
Last modified 4mo ago