Instances

This is a declaration of what infrastructure will support the “streaming” and “compute” of the pipeline. Streaming refers to what messaging platform the topics will be created in. Compute refers to where the agents will run the pipeline steps.

instance:
  globals:
    tableName: "vsearch.products"
  streamingCluster:
    type: "kafka"
    configuration:
      admin:
        bootstrap.servers: kafka-gcp-useast4.dev.streaming.datastax.com:9093
        security.protocol: SASL_SSL
        sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='${ secrets.astra-token.tenant }' password='token:${ secrets.astra-token.token }';"
        sasl.mechanism: PLAIN
        session.timeout.ms: "45000"
  computeCluster:
    type: "kubernetes"

Globals

Within instance.yaml, use "globals" to define values for parameters across your application.

For example, this instance defines topicName as a global parameter with the value "input-topic":

instance:
  globals:
    topicName: "input-topic"
    otherTopicName: "${OTHER_TOPIC_NAME:-default-topic-name}"

The second global otherTopicName uses an alternate declaration method where the value is loaded from a dotenv file containing a OTHER_TOPIC_NAME="value" line. The :- characters allow you to designate a default value - in this case, default-topic-name.

The topicName parameter can now be referenced wherever you need it, perhaps in your application's pipeline.yaml file:

topics:
  - name: "${globals.topicName}"
    creation-mode: create-if-not-exists
    deletion-mode: delete
  - name: "${globals.otherTopicName}"
    creation-mode: create-if-not-exists
    deletion-mode: delete

You can also use these parameters when creating assets, as in CREATE TABLE IF NOT EXISTS ${globals.vectorKeyspace}.${globals.vectorTable}.

Manifest

Root	Node	Type	Description
instance			Top-level node
	globals	object	A set of name:value pairs that should be applied to all clusters. Example: tableName: "vsearch.products"
	streamingCluster	object	The settings of the messaging platform use to stream data. See the ref below for more.
	computeCluster	object	The settings of the cluster where agents process data. See the ref below for more.

streamingCluster

Label	Type	Description
type	string	The type name of messaging platform to be used. Refer to the instance clusters area for supported types.
configuration	object	Configurations of the streaming platform. Refer to the instance clusters area for supported configuration.

DataStax Astra users

To use your Astra streaming tenant as the streaming cluster with LangStream, enable the Starlight for Kafka feature. Doing so will provide you with the needed bootstrap and security information to use the kafka type.

Read more about enabling Starlight for Kafka in Astra Streaming in the documentation and also in the learning site. Learn more about the Starlight for Kafka project here.