Configuration

A manifest of additional resources and dependencies to run the pipeline.

Example of a data source that needs drivers as well as credentials:

configuration:
  resources:
    - type: "datasource"
      name: "PGDataSource"
      configuration:
        service: "jdbc"
        driverClass: "org.postgresql.Driver"
        url: "jdbc:postgresql://postgresql.default.svc.cluster.local:5432/"
        user: "postgres"
        password: "xxxxxx"
  dependencies:
    - name: "PostGRES JDBC Driver"
      url: "https://jdbc.postgresql.org/download/postgresql-42.6.0.jar"
      sha512sum: "ec3b57d8377715ef6286d457b610a2e056aa99db….1"
      type: "java-library"

LangStream has built-in support for a few Databases and Vector databases (no need for dependency), for example:

Cassandra (with Vector support)

DataStax AstraDB

Pinecone

OSS Milvus and Zillis

Apache Solr

OpenSearch

Cassandra (with Vector support)

configuration:
  resources:
    - type: "datasource"
      name: "Cassandra"
      configuration:
        service: "cassandra"
        username: ""
        password: ""
        contact-points: ""
        loadBalancing-localDc: ""

DataStax Astra DB

configuration:
  resources:
    - type: "vector-database"
      name: "AstraDBDatasource"
      configuration:
        service: "astra"
        secret: ""
        clientId: ""
        token: ""
        database: ""

Pinecone

configuration:
  resources:
    - type: "vector-database"
      name: "PineconeDatasource"
      configuration:
        api-key: ""
        environment: ""
        index-name: ""
        project-name: ""

OSS Milvus and Zillis

 configuration:
  resources: 
  - type: "datasource"
    name: "MilvusDatasource"
    configuration:
      service: "milvus"
      ## OSS Milvus
      username: "${secrets.milvus.username}"
      password: "${secrets.milvus.password}"
      host: "${secrets.milvus.host}"
      port: "${secrets.milvus.port}"
      ## Set to "upsert" for OSS Milvus, on Zills use "delete-insert"
      write-mode: "${secrets.milvus.write-mode}"
      ## Zillis
      url: "${secrets.milvus.url}"
      token: "${secrets.milvus.token}"

Apache Solr

configuration:
  resources:
  - type: "vector-database"
    name: "SolrDataSource"
    configuration:
      service: "solr"
      user: "${secrets.solr.username}"
      password: "${secrets.solr.password}"
      host: "${secrets.solr.host}"
      port: "${secrets.solr.port}"
      collection-name: "documents"
      

OpenSearch

configuration:
resources:
    - type: "vector-database"
      name: "OpenSearch"
      configuration:
        service: "opensearch"        
        username: "${secrets.opensearch.username}"
        password: "${secrets.opensearch.password}"
        host: "${secrets.opensearch.host}"
        port: "${secrets.opensearch.port}"
        index-name: "my-index-000"
      

Manifest

Root	Node	Type	Description
configuration			Top level node
	dependencies	object	A collection of artifacts that a pipeline step of resource may need to run. Refer to the spec below. Example collection: type: “xxx” name: “xxx” configuration: … type: “xxx” name: “xxx” configuration: …
	resources	object	A collection of resources. Refer to the spec below. Example collection: type: “xxx” name: “xxx” sha: “xxx” … type: “xxx” name: “xxx” sha: “xxx” …

dependencies

The given artifact will be downloaded, validated, and made available to the pipeline. \

Label	Type	Description
type	string (required)	The type of dependency. Supported values are: java-library Example: “java-library”
name	string (required)	The name of the dependency. It is used for display and as a reference pointer. Example: "Postgres JDBC Driver"
url	string (required)	A fully qualified URL to the dependency artifact. Example: "https://jdbc.postgresql.org/download/postgresql-42.6.0.jar"
sha512sum	string (required)	The downloaded artifact is validated against this value. Example: "ec3b57d8377715ef6286d457…”

These dependencies are downloaded by the LangStream CLI when you run "apps deploy", "docker run" and similar commands. This mechanism is especially useful for JDBC Drivers and for Kafka Connect connectors.

To handle Python dependencies, check out the documentation about developing custom Python agents.

resources

Label	Type	Description
type	string	The type of resource. Refer to the configuration resources reference for naming. Example: “datasource”
name	string	The name of the resource. It is used for display and as a reference pointer. Example: "PGDataSource"
configuration	object	Custom configuration for the given resource. Refer to the configuration resources reference for options.