My Understanding of How Large Language Models Work

Lately, I've been trying to learn more about AI, particularly large language models (LLMs). My usual approach to learning a new subject is to read the classic books on the topic. However, since this version of generative and agentic AI is so new and evolving so rapidly, it's been difficult to find good, established books that covers it and explains it so a web developer like me could understand it.

So I decided to try something different by learning AI with AI. That is, I asked Claude to explain how LLMs work over a series of back-and-forth conversations over a few days to build my understanding of the topic. This post is my attempt to explain what I've learned about how LLMs work.

I'm a web developer, not an AI or machine learning researcher, and I'm writing this from the perspective of a student learning a new subject, not an expert teaching it. In fact, one of my motivations for writing this post is to clarify my own understanding of LLMs. If you notice any mistakes or inaccuracies, please let me know. I hope you find it useful!

---

A large language model (LLM) is a program that takes a piece of text and predicts what comes next. You give it some text, it gives you back a list of possible next pieces of text, each with a probability, and one of them gets picked. Then the picked piece is appended to the text and the whole thing runs again. Chatbots, coding agents, ticket classifiers, and everything else you've seen built on Claude or GPT are this one loop, run many times, with a lot of engineering around it.

You don't need to understand the math to build good AI features. But you do need an accurate mental model of what's going on, because almost every confusing thing a model does (making up facts, forgetting what you said two messages ago, answering the same question differently each time, miscounting letters in a word) follows directly from how it works. Once you see the mechanism, those behaviors stop being mysterious and start being things you can work around.

The One Thing An LLM Does

Imagine the text "The customer could not log" and ask yourself what word comes next. You'd probably say "in". You might also say "on" or "into". You almost certainly wouldn't say "Rails". Without thinking about it, you assigned probabilities to possible next words based on everything you've read in your life.

An LLM does the same thing, explicitly. Given "The customer could not log", it produces a probability for every piece of text it knows about. Something like:

" in"     0.91
" on"     0.05
" into"   0.03
...
" Rails" 0.0000001

One of these gets chosen, say " in", and appended to the input. Now the input is "The customer could not log in", and the model runs again to predict the piece after that. This repeats until the model produces a special "I'm done" marker or hits a length limit you set.

This is what people mean when they call these models autoregressive: each output is fed back in as input for the next step. It's also why responses stream in word by word in a chat interface. The model really is producing them one at a time, and the API is just sending each one to you as soon as it exists. As a result, the model never plans out an answer and then types it.

Tokens, Not Words

I've been saying "piece of text" instead of "word", as models don't read words or characters. They read tokens.

A token is a chunk of text from a fixed vocabulary that the model was built with. Common words are usually a single token. Rarer words are split into several. OpenAI's docs describe a token as a commonly occurring sequence of characters, and give a rough rule of thumb for English: about 4 characters, or three quarters of a word, per token.

The exact split depends on the model. A token on older Claude models is about 3.5 English characters, and Claude Opus 4.7 and later use a newer tokenizer that produces about 30% more tokens for the same text. So the same paragraph costs a different number of tokens on different models, and you can't compare token counts across providers or even across model generations. When you need an exact number, both providers have token counting endpoints that will tell you, for the specific model you plan to use.

Before the model sees your text, a tokenizer converts it into a list of integers, one per token. The model only ever works with those integers. It never sees the letters.

Tokens are the unit of everything. You're billed per token, in and out. Context limits are measured in tokens. Speed is measured in tokens per second. When someone says a model is expensive, they mean its price per million tokens is high.

A Next-Word Predictor In Ruby

The best way I've found to understand next-token prediction is to build a very simple version of it. Here's a model that predicts the next word by counting which words followed which in some training text:

# A toy, for understanding only. Real models work very differently inside.
class Bigram
  def initialize(text)
    @counts = Hash.new { |hash, word| hash[word] = Hash.new(0) }

    text.downcase.scan(/[\w']+|[.,?!]/).each_cons(2) do |word, next_word|
      @counts[word][next_word] += 1
    end
  end

  def probabilities(word)
    followers = @counts[word]
    total = followers.values.sum.to_f

    followers.transform_values { it / total }.sort_by { -it.last }.to_h
  end
end

This Ruby class implements a simple Bigram language model. In plain English, it reads some text, learns which words tend to follow other words, and calculates the probability of each possible next word.

First, we initialize a nested hash to keep track of how often each word follows another:

@counts = Hash.new { |hash, word| hash[word] = Hash.new(0) }

The outer hash stores the current word, while the inner hash stores the words that follow it, along with their frequency. Using Hash.new(0) lets us increment the counts without having to initialize them first.

Next, we break the input text into individual words and punctuation marks:

text.downcase.scan(/[\w']+|[.,?!]/)

We convert everything to lowercase so that "Ruby" and "ruby" are treated as the same word.

For example, "I love Ruby." becomes:

["i", "love", "ruby", "."]

The each_cons(2) method iterates over consecutive pairs of words.

.each_cons(2) do |word, next_word|
  @counts[word][next_word] += 1
end

For our example, it produces:

["i", "love"]
["love", "ruby"]
["ruby", "."]

For each pair, we increment the count of how often the second word follows the first. That's essentially what a bigram is: a sequence of two consecutive words (or tokens).

Finally, the probabilities method calculates how likely each word is to follow a given word:

def probabilities(word)
  followers = @counts[word]
  total = followers.values.sum.to_f

  followers.transform_values { it / total }.sort_by { -it.last }.to_h
end

It retrieves all the words that followed the given word, adds up their occurrences, and divides each word's count by the total.

For example, suppose our input text is:

"I love Ruby. I love Rails. I love programming."

The word "love" is followed by "ruby", "rails", and "programming" exactly once each.

So calling:

bigram.probabilities("love")

Returns approximately:

{
  "ruby" => 0.333,
  "rails" => 0.333,
  "programming" => 0.333
}

Each word has a 33.3% probability of following "love", based on the input text. The results are sorted from highest to lowest probability. That's the model's prediction: given a word, here's how likely each possible next word is.

This is a simple language model that learns word patterns from text and uses them to predict what might come next.

Of course, this model only considers the immediate word before the target, without understanding the meaning or broader context of a sentence. But it shows the fundamental idea behind statistical language modeling: using patterns in existing text to predict the next word.

Let's train it on a few sentences from a support inbox:

corpus = <<~TEXT
  The customer could not log in. The customer reset the password.
  The agent replied to the customer. The agent closed the ticket.
  The ticket was about billing. The customer was happy with the reply.
TEXT

model = Bigram.new(corpus)
model.probabilities("the")
# => {"customer" => 0.4, "agent" => 0.2, "ticket" => 0.2, "password" => 0.1, "reply" => 0.1}

"The" was followed by "customer" 4 times out of 10, so the model says there's a 40% chance the next word is "customer". That's the model, and it really is a language model, just a tiny and dumb one.

Now we need a way to generate text with it. The obvious approach is to always pick the most likely word:

class Bigram
  def generate(word, length: 12)
    words = [ word ]

    length.times do
      candidates = probabilities(words.last)
      break if candidates.empty?

      words << candidates.keys.first
    end

    words.join(" ")
  end
end

model.generate("the")
# => "the customer could not log in . the customer could not log in"

This is the autoregressive loop. It also shows our first problem. Always taking the top choice gets the model stuck in a loop, repeating the most probable path forever. The fix is to pick randomly, weighted by probability, so that "customer" is picked 40% of the time and "password" 10% of the time:

class Bigram
  def generate(word, length: 12)
    words = [ word ]

    length.times do
      candidates = probabilities(words.last)
      break if candidates.empty?

      words << sample(candidates)
    end

    words.join(" ")
  end

  private
    def sample(candidates)
      roll = rand

      candidates.each do |word, probability|
        roll -= probability
        return word if roll <= 0
      end

      candidates.keys.last
    end
end

sample rolls a random number between 0 and 1 and walks through the candidates, subtracting each probability until the roll runs out. A word with probability 0.4 covers 40% of the range, so it's chosen 40% of the time. (The last line is there in case floating-point rounding leaves a tiny remainder.)

Running it three times gives three different outputs:

the customer reset the password . the ticket . the agent replied to
the customer was happy with the agent replied to the reply . the
the ticket was about billing . the agent closed the ticket . the

We used the same model and the same input, however got a different text. Because it's exactly why a real LLM gives you a different answer each time you ask.

You can also see what's fundamentally wrong with this model. It only looks at one word back. "The agent replied to the reply" is a perfectly probable sequence of word pairs and complete nonsense as a sentence. To write sensibly, a model needs to consider the whole text, not just the last word. If the text is about a customer who can't log in, "password" should become far more likely than "billing", even if "password" is many words back.

What A Real LLM Does Differently

A real LLM keeps the same contract as our toy (text in, probabilities for the next token out), but differs in three big ways.

First, it looks at everything. When it predicts the next token, it takes the entire input into account, every token from the beginning of the conversation. The architecture that makes this possible is called the transformer, and its key idea, attention, lets the model decide for each position which earlier tokens are relevant. When predicting what comes after "Can you reset my", attention lets the model pick up on "log in" from three sentences ago, which pushes "password" up. You don't need to know how attention works to build AI features. It's enough to know that the model weighs the whole context, and that the amount of context it can handle at once is limited.

Second, it doesn't store counts. Our toy model memorizes a table of word pairs. An LLM instead consists of billions of numbers called parameters (or weights). There's no table to look up, and no place inside where a particular fact is written down. Your input tokens are converted into numbers and pushed through layer after layer of arithmetic using those parameters, and out the other end comes a probability for every token in the vocabulary. Whatever the model "knows" is spread across all of those numbers.

Third, it generalizes. Because it isn't looking up exact sequences, a large model can handle text it has never seen before. Our bigram model has nothing to say about "the invoice", because it never saw "invoice". An LLM has seen enough text to have learned that invoices are related to billing, that people who can't log in often want password resets, and an enormous amount of other structure about language and the world.

That leaves the obvious question. If nobody writes those billions of parameters by hand, where do they come from?

Where The Probabilities Come From: Training

Language models like Claude aren't programmed directly by humans, they're trained on large amounts of data. Training happens in stages.

The first stage is pretraining. The model starts with random parameters, so its predictions are garbage. It's then shown a huge amount of text (a large part of the public internet, books, code, and more) and, for each position in that text, asked to predict the next token. Since the real next token is right there in the text, it's easy to check how wrong the prediction was. The parameters are then nudged slightly in the direction that would have made the correct token more likely. Repeat that trillions of times and the predictions get very good.

There's no labeling and no human telling the model what's true. The text itself provides the answers. And it turns out that to predict the next token in a physics textbook, a Ruby file, or a legal contract really well, a model has to pick up a lot of physics, Ruby, and law along the way.

What you get at the end of pretraining is called a base model, and it's surprisingly unhelpful. Anthropic's says that pretrained models "are not inherently good at answering questions or following instructions." A base model completes documents. Give it "What is the capital of France?" and it might continue with "What is the capital of Germany? What is the capital of Spain?", because in its training data, a question like that is often part of a quiz. It's predicting what document this is likely to be, not trying to help you.

Turning A Base Model Into An Assistant

The second stage turns the document completer into an assistant. There are a few techniques, and providers combine them.

Fine-tuning means continuing to train the model, but on a smaller, carefully chosen dataset. If you train it on thousands of example conversations where a user asks something and an assistant answers helpfully, the model learns that pattern. Now when it sees a question, the most likely continuation is a helpful answer.

Reinforcement learning from human feedback (RLHF) goes further. The model generates several responses to the same prompt, people rank them from best to worst, and training pushes the model toward responses like the higher-ranked ones. This is how models learn the fuzzier qualities that are hard to show with examples alone, like being concise, admitting uncertainty, or declining harmful requests.

Newer models also go through reinforcement learning on tasks where the result can be checked automatically, like math problems with known answers or code that has to pass tests. This is a big part of how reasoning models learned to work through problems step by step, which we'll come back to.

Choosing The Next Token: Sampling

Back to our toy model's sample method. Real models face the same choice we did. Always taking the most likely token tends to produce repetitive, flat text. Sampling randomly by probability produces more natural text but less predictable results.

Historically, APIs exposed a parameter called temperature to control this. A low temperature sharpens the probabilities so the top choices win almost every time. A high temperature flattens them so less likely tokens get picked more often. Anthropic's glossary describes lower temperatures as producing "more conservative and deterministic outputs" and higher ones as encouraging "rare, uncommon, or surprising word choices."

There are two things worth knowing about temperature in practice. Even at temperature 0, results aren't fully deterministic. Identical inputs can produce different outputs across API calls. Small numerical differences in how the calculation runs on the provider's hardware can tip a close call between two tokens, and once one token differs, everything after it can differ too.

The second is that, as of November 2026, the current Claude models don't accept temperature at all. Sending temperature, top_p, or top_k to Claude Opus 4.7 or later returns an error, and the docs recommend steering behavior through the prompt instead. If you're reading older tutorials that set temperature: 0 to get consistent output, that advice doesn't carry over.

So you should design every AI feature assuming the output will vary. That's why you can't test an LLM feature the way you test a method that returns a fixed value, and why evals (tests that score many outputs instead of asserting one) exist.

The Model Remembers Nothing

HTTP is a stateless protocol. The server doesn't remember you between requests, so your browser sends a session cookie every time, and Rails uses it to rebuild who you are.

LLM APIs work the same way. The model has no memory between calls. The Messages API from the model providers is is stateless, which means that you always send the full conversational history to the API. Each text generation request is independent and stateless.

When you chat with Claude and it seems to remember your name from five messages ago, that's because the app sent all five messages again, along with your new one. Here's what that looks like at the HTTP level:

require "net/http"
require "json"

def ask(messages)
  uri = URI("https://api.anthropic.com/v1/messages")

  request = Net::HTTP::Post.new(uri)
  request["x-api-key"] = ENV.fetch("ANTHROPIC_API_KEY")
  request["anthropic-version"] = "2026-10-21"
  request["content-type"] = "application/json"
  request.body = { model: "claude-sonnet-5-5", max_tokens: 1024, messages: messages }.to_json

  response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { it.request(request) }
  JSON.parse(response.body)
end

messages = [ { role: "user", content: "Hi, I'm John. I can't log in to my account." } ]
reply = ask(messages)

messages << { role: "assistant", content: reply["content"].find { it["type"] == "text" }["text"] }
messages << { role: "user", content: "What was my name again?" }

ask(messages)

The ask method sends a single POST request with the model, a maximum number of output tokens, and a list of messages. Each message has a role: user for what the person said and assistant for what the model said.

To continue the conversation, we append the model's reply to messages as an assistant message, then append the new user message, and send the whole array again. The second request contains John's name only because we put it there. If you sent [ { role: "user", content: "What was my name again?" } ] alone, the model would have no idea.

(The find is there because the response's content is an array of blocks, and on models that think before answering, the first block can be the model's reasoning rather than its reply.)

The model's reply also comes with a usage field reporting how many tokens went in and how many came out. Watch the input count grow as the conversation gets longer, because it changes how you think about cost.

The Context Window

Everything you send in a request, plus everything the model generates in response, has to fit in the model's context window. Think of it as the model's "working memory", distinct from the huge dataset it was trained on. The system prompt, every message, any documents or images, tool definitions, and the output all count toward it.

As of November 2026, the current Claude models have a 1M-token context window, which Anthropic estimates at roughly 555,000 English words on the current tokenizer. That sounds like plenty, and for a single request it usually is. But there are two things to keep in mind.

The first is cost. Because you resend the whole conversation each turn, the input grows with every message. Say each turn adds 500 tokens. By turn 40, a single request sends 20,000 input tokens, and the total input across all 40 turns is about 410,000 tokens. On Claude Sonnet 5.5 at $2 per million input tokens (as of November 2026), that's about 82 cents for one conversation. That's fine for one user, and adds up for thousands. Long conversations get expensive in a way that short ones don't hint at.

The second is quality. Anthropic's docs say that "more context isn't automatically better. As token count grows, accuracy and recall degrade, a phenomenon known as context rot." A model given 500 pages and asked about one sentence is more likely to miss it than a model given the right two pages. Choosing what goes into the context is one of the main skills of building AI features, and a lot of the techniques you'll hear about (summarizing old messages, retrieval, compaction) exist to keep the context small and relevant.

What The Model Knows, And What It Makes Up

Everything a model knows from training was frozen when training ended. Anthropic lists a "reliable knowledge cutoff" for each model, the date through which its knowledge is most reliable. For the current Claude models, that's June 2026 (as of November 2026). The model doesn't know about anything after that unless you tell it.

More importantly for your app, the model has never seen your data. It doesn't know your customers, your tickets, your pricing, or your refund policy. The only way it can use that information is if you put it in the request. When people talk about retrieval-augmented generation (RAG), they mean: find the relevant records in your database and include them in the prompt, so the model reads them in its context window instead of relying on what it memorized.

What happens when the model doesn't know something? Ideally, it says so. Sometimes it confidently makes something up instead. This is called hallucination, and if you've followed along, you can probably guess part of the reason. The model is trained to produce a probable continuation. A plausible-sounding wrong answer can be a very probable continuation.

For your application's AI features, the practical answer is the same one RAG gives: don't ask the model to recall specific facts. Give it the facts and ask it to work with them. "Here's the customer's order history; is this order eligible for a refund under this policy?" is a much safer question than "What's our refund policy?"

Thinking Models

There's one more piece that connects back to the autoregressive loop. Since each token is computed from the tokens before it, a model that writes out intermediate steps before the final answer gives itself more to work with. Writing "36 + 59: 6 + 9 is 15, carry the 1..." puts those steps into the context, where they can inform the following tokens.

That's what thinking (or reasoning) models do. Before answering, they generate a stream of reasoning tokens, trained through the reinforcement learning I mentioned earlier to be useful for getting the final answer right. In the Claude API, this shows up as thinking blocks in the response. As of November 2026, the current Claude models use adaptive thinking, where the model decides how much to think based on the problem and an effort setting you control.

Thinking isn't free. Thinking tokens are billed as output tokens and they take time to generate, so a model that thinks for a while before answering is both slower and more expensive. For a hard question, it's worth it. For classifying a ticket as billing or technical, it usually isn't.

What This Means For a Rails App

Let me pull this together into what it changes about how you build things.

An LLM call is an HTTP request to a slow, metered, nondeterministic function. A response can take anywhere from under a second to a minute or more, depending on how many tokens are generated. That alone means model calls belong in background jobs, not in the request cycle, with results streamed or broadcast back to the page.

The model knows nothing about your app, so the quality of a feature depends mostly on what you put in the context. Most of the real work in AI features is the Rails work of gathering the right records, formatting them well, and keeping out what doesn't belong (including data the current user shouldn't see).

Output varies and can be wrong, so you treat it like user input. Validate it, parse it into structured objects when you need structure, and keep a human in the loop when mistakes are costly. Test the parts you control, like the prompt you build, with ordinary unit tests, and use evals for the parts you don't.