Special Tokens in LLM Chatbots

It turns out that special tokens help us solve a certain mystery about LLM chatbots...

The Chatbot Conundrum

LLM stands for Large Language Model. As the name implies, these are in fact language models, and they fit into the exact same mathematical framework we've been discussing: they model the probability \( P(w_1,w_2,\dots,w_k) \). The thing that distinguishes them from the simple Markov models we've been working with so far is that they use much fancier techniques than smoothing, backoff, or interpolation to handle contexts that weren't seen during training. We'll cover these techniques later in this class. For now, you just need to know that this allows LLMs to support much larger context windows (think hundreds of thousands of tokens, instead of 2 or 3) while reducing (but not eliminating) the possibility of memorization.

And yet, there's something strange about the (currently) most popular mainstream use of LLMs, chatbots like ChatGPT. If you were to take any language model (even a cutting edge LLM) and ask it to generate text starting from a context like "Help me find interesting new recipes.", there's no reason to think the model would treat this as a command. After all, it's only job is to generate plausible next tokens; it could just as easily decide that this is a line of dialogue from a novel and generate the continuation ",' said John to the master chef".

But if you feed that line to ChatGPT, Claude, Gemini, etc., you'll find that they universally do treat it as a command, and will respond with a list of recipes or possibly even conduct a web search on your behalf. How is that possible?

The History of Chatbots

To figure out what's going on, it might help to build a little intuition through the following quick activity:

While there's many ways to interpret the context above, as humans we might think it looks like a snippet from a script for a play, and we would imagine that the next line must start with the name of another character, e.g., JULIET. And since modern LLMs have been trained on a bunch of data that includes scripts, they tend to share this intuition.

As an unintended side effect, back in OpenAI's early days, researchers realized that if you gave GPT (one of OpenAI's earlier language models) a context like the following:

Below is a transcript of a conversation between a human user and an AI assistant.

USER: Help me find interesting new recipes.
ASSISTANT:

Then the language model would complete the context with a plausible response from the AI assistant. This is because it's seen a lot of scripts in its training data so it recognizes the format, and it's seen a lot of science fiction so it knows what AI assistants "should" act like.

However, colons are still a bit ambiguous since they're used in contexts other than scripts. So to reduce the ambiguity, researchers came up with the idea of using...special tokens! They started using contexts like:

Below is a transcript of a conversation between a human user and an AI assistant. 
The human user's lines are denoted by <user>, </user> tags. 
The assistant's lines are denoted by <assistant>, </assistant> tags.

<user>Help me find interesting new recipes.</user>
<assistant>

The special tokens <user> and </user> denote the start and end of the user's text, just like <SOS> and <EOS> denote the start and end of sentences in our simpler models. <assistant> and </assistant> do the same for the AI assistant's text, and just like how <EOS> acts a signal to stop generating a sentence, once the model generates an </assistant> token, we can have it stop generating.

Upon realizing how well this worked, OpenAI decided to scale up their experimentation by opening it up to the broader public. They wrote a simple web wrapper that, effectively:

1. Reads input from the user;
2. wraps it in <user> and </user> tags;
3. adds the preamble at the start that talks about how "Below is a transcript...", and
4. lets the language model generate until it hits </assistant>, at which point it waits for more user input.

They called this system ChatGPT, released it in late 2022, and the rest is history.

Special Tokens Let LLMs Reason and Act!

Today, the specialized chatbot-inducing contexts like the one shown above are called system prompts. Modern-day system prompts are much more complex than our toy example, and for closed-source systems like ChatGPT, they are a closely guarded trade secret.

One way that system prompts have gotten more complicated than the toy example is that there are now many more special tokens than <user> and <assistant>. Each special token controls one of the extra capabilities that modern LLMs have gained since ChatGPT's first release.

One example is <think> tokens. For reasoning-enabled models like GPT Astra, text inside <think> is supposed to represent a sort of scratchpad where the LLM can "reason" and work out problems. This empirically seems to improve their ability to answer complex questions.

More recently, models have gained the ability to use tools and skills to take concrete actions (e.g., running a terminal command, operating a web browser, etc.). Today, tool use has been standardized under the Model Context Protocol (MCP), which defines an explicit standard for how LLMs can communicate with servers that actually run the tools. While MCP is a huge specification with a whole bunch of detail that you don't need to know about for the purposes of this class, the key thing to understand is that at its heart it is also powered by special tokens that act as control sequences.

For example, here is actual raw output from asking Google's open-source Gemma model (inside the LM Studio chatbot interface) to execute a JavaScript snippet using the run_javascript tool:

<channel|><|tool_call>call:run_javascript{javascript:<|"|>var x = [1,2,3];
var y = x;
y += [4];
console.log("Value of x:", x);<|"|>}<tool_call|><|tool_response>response:run_javascript{value:<|"|>{"stdout":"Value of x: [ 1, 2, 3 ]","stderr":""}<|"|>}<tool_response|><|channel>

Notice all the special tokens that appear in this output! There's <|tool_call>...<tool_call|> indicating that the text between those tokens should be interpreted as a command to be run, rather than as actual text to output to the user. Likewise, there's <|tool_response>...<tool_response|> indicating that the text between those tokens should be interpreted as the raw output from the command that was run.

What actually happens when the <|tool_call>...<tool_call|> tokens are generated is that the "harness" software (in this case, LM Studio) detects that these tokens were generated, and then hands control over to an MCP server to run the requested command. You don't need to know all the thorny software engineering details of how MCP works (it's literally pages and pages of specification!). For the purposes of CS 159, the only thing we need you to understand is that from the LLM's perspective, all it did was generate some tokens—the LLM is, after all, still a language model, and generating tokens is the only thing it actually knows how to do!

(When logged in, completion status appears here.)