Application structure
Last updated
Last updated
A LangStream application is composed of multiple yaml manifests. Each manifest complements the others to create a data processing environment. You will typically use the to interact with the Kubernetes control plane.
Your development environment should follow a specific folder structure:
At the root there should be instance.yaml and optionally a secrets.yaml. These files cannot be in the "application" directory, which is passed as a zip at runtime.
The root should also contain a directory for the application files. We typically name this “application”.
The “application” directory should contain a pipeline.yaml, and optionally a configuration.yaml and gateways.yaml.
The hierarchy should look like this:
This application would be deployed out of the project-folder directory with:
This is referred to as the “application directory” because the main pipeline and its configuration are declared within.
Typically the values for an instance are specific to the environment running the application. As you promote the application off of your desktop to higher environments, other instance files may be needed. In this case, it is recommended to add a suffix to the file name, representing the environment it is configured for. A few examples:
Environment: local
Filename: instance-local.yaml
Environment: staging
Filename: instance-staging.yaml
Environment: production
Filename: instance-production.yaml
This is a holder for secrets used within pipeline steps, configuration dependencies, and instance declarations. It is an optional value.
pipeline.yaml: this is the declaration of topics and pipeline steps. It is required.
gateways: gateways interact with an agent via a message topic.
configuration.yaml: this is the declaration of additional services a step in the pipeline depends on. It is optional.
The instance is a declaration of the application’s processing infrastructure. This includes where “streaming” is processed and where “compute” takes place. This is a required value.
Secrets are held separately to create a separation of information. It’s not a good practice to hard code tokens in pipeline manifests. Instead, include a pointer to a secret that is managed in different ways.