LangStream Documentation
Langstream.aiLangStream GitHub RepoChangelog
  • LangStream Documentation
  • ❤️Langstream.ai
  • ⭐LangStream GitHub Repo
  • 📜Changelog
  • about
    • What is LangStream?
    • License
  • Get Started
  • installation
    • LangStream CLI
    • Docker
    • Minikube (mini-langstream)
    • Kubernetes
    • Build and install from source
  • Building Applications
    • Vector Databases
    • Application structure
      • Pipelines
      • Instances
      • Configuration
      • Topics
      • Assets
      • Secrets
      • YAML templating
      • Error Handling
      • Stateful agents
      • .langstreamignore
    • Sample App
    • Develop, test and deploy
    • Application Lifecycle
    • Expression Language
    • API Gateways
      • Websocket
      • HTTP
      • Message filtering
      • Gateway authentication
    • API Reference
      • Agents
      • Resources
      • Assets
  • LangStream CLI
    • CLI Commands
    • CLI Configuration
    • Web interface
  • Integrations
    • Large Language Models (LLMs)
      • OpenAI
      • Hugging Face
      • Google Vertex AI
      • Amazon Bedrock
      • Ollama
    • Data storage
      • Astra Vector DB
      • Astra
      • Cassandra
      • Pinecone
      • Milvus
      • Solr
      • JDBC
      • OpenSearch
    • Integrations
      • Apache Kafka Connect
      • Apache Camel
    • LangServe
  • Pipeline Agents
    • Agent Messaging
    • Builtin agents
      • Input & Output
        • webcrawler-source
        • s3-source
        • azure-blob-storage-source
        • sink
        • vector-db-sink
        • camel-source
      • AI Agents
        • ai-chat-completions
        • ai-text-completions
        • compute-ai-embeddings
        • flare-controller
      • Text Processors
        • document-to-json
        • language-detector
        • query
        • query-vector-db
        • re-rank
        • text-normaliser
        • text-extractor
        • text-splitter
        • http-request
      • Data Transform
        • cast
        • compute
        • drop
        • drop-fields
        • merge-key-value
        • unwrap-key-value
      • Flow control
        • dispatch
        • timer-source
        • trigger-event
    • Custom Agents
      • Python sink
      • Python source
      • Python processor
      • Python service
    • Agent Developer Guide
      • Agent Types
      • Agent Creation
      • Configuration and Testing
      • Environment variables
  • Messaging
    • Messaging
      • Apache Pulsar
      • Apache Kafka
      • Pravega.io
  • Patterns
    • RAG pattern
    • FLARE pattern
  • Examples
    • LangServe chatbot
    • LlamaIndex Cassandra sink
Powered by GitBook
On this page
  • Configuration
  • Authentication
  • Examples
  • Using Apache Kafka in "docker run"
Edit on GitHub
  1. Messaging
  2. Messaging

Apache Kafka

PreviousApache PulsarNextPravega.io

Last updated 1 year ago

Configuration for a LangStream streaming instance.

Configuration

To connect to a Kafka Broker, provide the address of the broker with the bootstrap.servers property.

Please refer to the Apache Kafka Java client (external link) for more details about the configuration properties.

Authentication

Usually in Apache Kafka you would provide Authentication using the JAAS configuration file, but in LangStream you can provide the configuration directly in the instance.yaml file.

  • security.protocol: the security protocol to use, for example SASL_SSL

  • sasl.jaas.config: the JAAS configuration to use for authentication

  • sasl.mechanism: PLAIN

  • session.timeout.ms: "45000"

Examples

This is a configuration for a Kafka instance without authentication, on PLAINTEXT connections:

instance:
  streamingCluster:
    type: "kafka"
    configuration:
      admin:
        bootstrap.servers: my-cluster-kafka-bootstrap.kafka:9092

This is how it looks like when you use SASL_SSL for authentication:

instance:
  streamingCluster:
    type: "kafka"
    configuration:
      admin:
        bootstrap.servers: "${ secrets.kafka.bootstrap-servers }"
        security.protocol: SASL_SSL
        sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='${ secrets.kafka.username }' password='${ secrets.kafka.password }';"
        sasl.mechanism: PLAIN
        session.timeout.ms: "45000"

Please always use secrets to store sensitive information like passwords and usernames.

Using Apache Kafka in "docker run"

When you are running you application in mode, you can use the default Kafka instance provided by the LangStream docker image, and you don't need to provide any configuration.

documentation
Docker