Comment on page
ai-chat-completions

Given the AI model specified in an application's configuration resources, this agent will use its completion API to submit message prompts and return the result. This agent will discover its type (i.e. OpenAI, Hugging Face, VertexAI) and use the corresponding library to generate the embedding. It is up to the developer to match the correct mode name with the configured AI model.
Using OpenAI chat models
The ai-chat-completions for OpenAI uses the /v1/chat/completions endpoint. Refer to the OpenAI documentation to know which models are compatible.
Setup the OpenAI LLM configuration. Add the ai-chat-completions agent:
pipeline:
  - name: "ai-chat-completions"
    type: "ai-chat-completions"
    output: "history-topic"
    configuration:
      model: "gpt-3.5-turbo"
      # on the log-topic we add a field with the answer
      completion-field: "value.answer"
      # we are also logging the prompt we sent to the LLM
      log-field: "value.prompt"
      # here we configure the streaming behavior
      # as soon as the LLM answers with a chunk we send it to the answers-topic
      stream-to-topic: "output-topic"
      # on the streaming answer we send the answer as whole message
      # the 'value' syntax is used to refer to the whole value of the message
      stream-response-completion-field: "value"
      # we want to stream the answer as soon as we have 10 chunks
      # in order to reduce latency for the first message the agent sends the first message
      # with 1 chunk, then with 2 chunks....up to the min-chunks-per-message value
      # eventually we want to send bigger messages to reduce the overhead of each message on the topic
      min-chunks-per-message: 10
      messages:
        - role: user
          content: "You are a helpful assistant. Below you can find a question from the user. Please try to help them the best way you can.\n\n{{ value.question }}"
Using VertexAI chat models
Refer to the VertexAI documentation to know which models are compatible.
Setup the Vertex LLM configuration. Add the ai-chat-completions agent:
pipeline:
  - name: "ai-chat-completions"
    type: "ai-chat-completions"
    configuration:
      model: "chat-bison"
      max-tokens: 100
      completion-field: "value.chatresult"
      log-field: "value.request"
      messages:
        - role: user
          content: "You are a helpful assistant. Below you can find a question from the user. Please try to help them the best way you can.\n\n{{ value.question }}"
Using Ollama models
Refer to the ollama documentation to find a list of models.
Setup the Ollama configuration. Add the ai-chat-completions agent:
pipeline:
  - name: "ai-chat-completions"
    type: "ai-chat-completions"
    configuration:
      model: "llama2"
      max-tokens: 100
      completion-field: "value.chatresult"
      log-field: "value.request"
      messages:
        - role: user
          content: "You are a helpful assistant. Below you can find my question. Please try to help them the best way you can.\n\n This is my question: {{ value.question }}"
Using Amazon Bedrock AI21 Jurassic-2 models
Refer to the Amazon documentation for other parameters and options.
Set up the Amazon Bedrock LLM configuration. Add the ai-chat-completions agent:
pipeline:
  - name: "ai-chat-completions"
    type: "ai-chat-completions"
    configuration:
      model: "ai21.j2-mid-v1"
      completion-field: "value.answer"
      options:
        request-parameters:
          # here you can add all the supported parameters
          temperature: 0.5
          maxTokens: 300
        # expression to retrieve the completion from the response JSON. It varies depending on the model 
        response-completions-expression: "completions[0].data.text"
      messages:
        - content: "{{ value.question }}"
Using Amazon Bedrock Anthropic Claude models
Refer to the Amazon documentation to learn other parameters and options.
Setup the Amazon Bedrock LLM configuration. Add the ai-chat-completions agent:
pipeline:
  - name: "ai-chat-completions"
    type: "ai-chat-completions"
    configuration:
      model: "anthropic.claude-v2"
      completion-field: "value.answer"
      options:
        request-parameters:
          # here you can add all the supported parameters
          temperature: 0.5
          max_tokens_to_sample: 300
          top_p: 0.9
          top_k: 250
        # expression to retrieve the completion from the response JSON. It varies depending on the model 
        response-completions-expression: "completion"
      messages:
        - content: "{{ value.question }}"
Prompt limitations
Most public LLMs have character limits on message content. It is up to the application developer to ensure the combination of preset prompt text and an input message stays under that limit.
Costs incurred
Some public LLMs offer a free tier and then automatically begin charging per prompt (or by prompt chunk). It is up to the application developer to be aware of these possible charges and manage them appropriately.
Topics
Input
Structured and unstructured text ?​
Implicit topic ?​
Templating ?​
Output
Structured text ?​
Implicit topic ?​
Configuration
Check out the full configuration properties in the API Reference page.
AI Agents
ai-text-completions
Last modified 14d ago