Chat Models

Chat models are a variation on language models. While chat models use language models under the hood, the interface they expose is a bit different. Rather than expose a "text in, text out" API, they expose an interface where "chat messages" are the inputs and outputs.

Chat model APIs are fairly new, so we are still figuring out the correct abstractions.

Getting Started

This section covers how to get started with chat models. The interface is based around messages rather than raw text.

import { ChatOpenAI } from "langchain/chat_models";
import { HumanChatMessage, SystemChatMessage } from "langchain/schema";

const chat = new ChatOpenAI({ temperature: 0 });

Here we create a chat model using the API key stored in the environment variable OPENAI_API_KEY. We'll be calling this chat model throughout this section.

Message in, Message out

You can get chat completions by passing one or more messages to the chat model. The response will also be a message. The types of messages currently supported in LangChain are AIChatMessage, HumanChatMessage, SystemChatMessage, and a generic ChatMessage -- ChatMessage takes in an arbitrary role parameter, which we won't be using here. Most of the time, you'll just be dealing with HumanChatMessage, AIChatMessage, and SystemChatMessage.

const response = await chat.call([
  new HumanChatMessage(
    "Translate this sentence from English to French. I love programming."
  ),
]);

console.log(response);

AIChatMessage { text: "J'aime programmer." }

Multiple Messages

OpenAI's chat model supports multiple messages as input. See here for more information. Here is an example of sending a system and user message to the chat model:

response = await chat.call([
  new SystemChatMessage(
    "You are a helpful assistant that translates English to French."
  ),
  new HumanChatMessage("Translate: I love programming."),
]);

console.log(response);

AIChatMessage { text: "J'aime programmer." }

Multiple Completions

You can go one step further and generate completions for multiple sets of messages using generate. This returns an LLMResult with an additional message parameter.

const responseC = await chat.generate([
  [
    new SystemChatMessage(
      "You are a helpful assistant that translates English to French."
    ),
    new HumanChatMessage(
      "Translate this sentence from English to French. I love programming."
    ),
  ],
  [
    new SystemChatMessage(
      "You are a helpful assistant that translates English to French."
    ),
    new HumanChatMessage(
      "Translate this sentence from English to French. I love artificial intelligence."
    ),
  ],
]);

console.log(responseC);

{
  generations: [
    [
      {
        text: "J'aime programmer.",
        message: AIChatMessage { text: "J'aime programmer." },
      }
    ],
    [
      {
        text: "J'aime l'intelligence artificielle.",
        message: AIChatMessage { text: "J'aime l'intelligence artificielle." }
      }
    ]
  ]
}

PromptTemplates to encapsulate reusable tokens

You can make use of templating by using a MessagePromptTemplate. You can build a ChatPromptTemplate from one or more MessagePromptTemplates. You can use ChatPromptTemplate's format_prompt -- this returns a PromptValue, which you can convert to a string or Message object, depending on whether you want to use the formatted value as input to an llm or chat model.

Continuing with the previous example:

import {
  SystemMessagePromptTemplate,
  HumanMessagePromptTemplate,
  ChatPromptTemplate,
} from "langchain/prompts";

First we create a reusable template:

const translationPrompt = ChatPromptTemplate.fromPromptMessages([
  SystemMessagePromptTemplate.fromTemplate(
    "You are a helpful assistant that translates {input_language} to {output_language}."
  ),
  HumanMessagePromptTemplate.fromTemplate("{text}"),
]);

Then we can use the template to generate a response:

const responseA = await chat.generatePrompt([
  await translationPrompt.formatPromptValue({
    input_language: "English",
    output_language: "French",
    text: "I love programming.",
  }),
]);

console.log(responseA);

{
  generations: [
    [
      {
        text: "J'aime programmer.",
        message: AIChatMessage { text: "J'aime programmer." }
      }
    ]
  ]
}

Model + Prompt = LLMChain

This pattern of asking for the completion of a formatted prompt is quite common, so we introduce the next piece of the puzzle: LLMChain

const chain = new LLMChain({
  prompt: chatPrompt,
  llm: chat,
});

Then you can call the chain:

const responseB = await chain.call({
  input_language: "English",
  output_language: "French",
  text: "I love programming.",
});

console.log(responseB);

{ text: "J'aime programmer." }

Stateful chains

You can also use the chain to store state. This is useful for eg. chatbots, where you want to keep track of the conversation history. MessagesPlaceholder is a special prompt template that will be replaced with the messages passed in each call.

const chatPrompt = ChatPromptTemplate.fromPromptMessages([
  SystemMessagePromptTemplate.fromTemplate(
    "The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know."
  ),
  new MessagesPlaceholder("history"),
  HumanMessagePromptTemplate.fromTemplate("{input}"),
]);

const chain = new ConversationChain({
  memory: new BufferMemory({ returnMessages: true, memoryKey: "history" }),
  prompt: chatPrompt,
  llm: chat,
});

The chain will internally accumulate the messages sent to the model, and the ones received as output. Then it will inject the messages into the prompt on the next call. So you can call the chain a few times, and it remembers previous messages:

const responseD = await chain.call({
  input: "hi from London, how are you doing today",
});

{
  response: "Hello! As an AI language model, I don't have feelings, but I'm functioning properly and ready to assist you with any questions or tasks you may have. How can I help you today?"
}

const responseE = await chain.call({
  input: "Do you know where I am?",
});

console.log(responseE);

{
  response: "Yes, you mentioned that you are from London. However, as an AI language model, I don't have access to your current location unless you provide me with that information."
}

Agents and Tools

Finally, we introduce Tools and Agents, which extend the model with other abilities, such as search, or a calculator.

A tool is a function that takes a string (such as a search query) and returns a string (such as a search result). They also have a name and description, which are used by the chat model to identify which tool it should call.

class Tool {
  name: string;
  description: string;
  call(arg: string): Promise<string>;
}

An agent is a stateless wrapper around an agent prompt chain (such as MRKL) which takes care of formatting tools into the prompt, as well as parsing the responses obtained from the chat model.

interface AgentStep {
  action: AgentAction;
  observation: string;
}

interface AgentAction {
  tool: string; // Tool.name
  toolInput: string; // Tool.call argument
}

interface AgentFinish {
  returnValues: object;
}

class Agent {
  plan(steps: AgentStep[], inputs: object): Promise<AgentAction | AgentFinish>;
}

To make agents more powerful we need to make them iterative, ie. call the model multiple times until they arrive at the final answer. That's the job of the AgentExecutor.

class AgentExecutor {
  // a simplified implementation
  run(inputs: object) {
    const steps = [];
    while (true) {
      const step = await this.agent.plan(steps, inputs);
      if (step instanceof AgentFinish) {
        return step.returnValues;
      }
      steps.push(step);
    }
  }
}

And finally, we can use the AgentExecutor to run an agent:

// Define the list of tools the agent can use
const tools = [new SerpAPI()];
// Create the agent from the chat model and the tools
const agent = ChatAgent.fromLLMAndTools(new ChatOpenAI(), tools);
// Create an executor, which calls to the agent until an answer is found
const executor = AgentExecutor.fromAgentAndTools({ agent, tools });

const responseG = await executor.run(
  "How many people live in canada as of 2023?"
);

console.log(responseG);

38,626,704.

Streaming

You can also use the streaming API to get words streamed back to you as they are generated. This is useful for eg. chatbots, where you want to show the user what is being generated as it is being generated. Note: OpenAI as of this writing does not support tokenUsage reporting while streaming is enabled.

const chatStreaming = new ChatOpenAI({
  streaming: true,
  callbackManager: CallbackManager.fromHandlers({
    handleLLMNewToken(token: string) {
      console.log(token);
    },
  },
});

const responseD = await chatStreaming.call([
  new HumanChatMessage("Write me a song about sparkling water."),
]);

Verse 1:
Bubbles in the bottle,
Light and refreshing,
It's the drink that I love,
My thirst quenching blessing.

Chorus:
Sparkling water, my fountain of youth,
I can't get enough, it's the perfect truth,
It's fizzy and fun, and oh so clear,
Sparkling water, it's crystal clear.

Verse 2:
No calories or sugars,
Just a burst of delight,
It's the perfect cooler,
On a hot summer night.

Chorus:
Sparkling water, my fountain of youth,
I can't get enough, it's the perfect truth,
It's fizzy and fun, and oh so clear,
Sparkling water, it's crystal clear.

Bridge:
It's my happy place,
In every situation,
My daily dose of hydration,
Always bringing satisfaction.

Chorus:
Sparkling water, my fountain of youth,
I can't get enough, it's the perfect truth,
It's fizzy and fun, and oh so clear,
Sparkling water, it's crystal clear.

Outro:
Sparkling water, it's crystal clear,
My love for you will never disappear.