Beam search keeps several unfinished sequences alive while a model generates text. Instead of committing to the most likely next token immediately, it expands a small set of candidates and keeps the strongest partial sequences. That can find a more probable complete sequence than greedy decoding, although it still discards possibilities and does not establish whether an answer is correct.
The easiest way to understand the difference is to calculate a very small example. This guide uses invented token labels and probabilities so every choice can be checked by hand. It is an explanation of an algorithm, not a measurement of a commercial chatbot, and the numbers do not describe real language-model behavior.
Start with greedy decoding
Imagine a model must generate exactly two tokens. At the first step, it offers A with probability 0.60 and B with probability 0.40. A greedy decoder chooses A because 0.60 is larger. It does not keep B as a backup. Once A has been chosen, the next decision is limited to continuations after A.
For the second step, suppose the model offers x with conditional probability 0.51 and y with conditional probability 0.49 after A. Greedy decoding selects x. The finished sequence is A x. Its probability is the probability of A multiplied by the probability of x given A: 0.60 × 0.51 = 0.306. The multiplication matters because the second probability is conditional on the first choice.
This is a locally sensible result. The decoder selected the strongest option at both decisions it encountered. However, the unvisited B branch may contain a continuation strong enough to produce a better overall score. A locally strongest first token does not necessarily begin the strongest complete sequence.
Keep two paths with beam width two
Set the beam width to two. At the first step, both A and B survive because there are only two candidates. At the next step, expand both branches. The A branch still offers x at 0.51 and y at 0.49. The B branch offers z at 0.95 and w at 0.05. Those second-step values each sum to one within their own branch.
| Sequence | Calculation | Joint probability |
|---|---|---|
| A x | 0.60 × 0.51 | 0.306 |
| A y | 0.60 × 0.49 | 0.294 |
| B z | 0.40 × 0.95 | 0.380 |
| B w | 0.40 × 0.05 | 0.020 |
The decoder ranks the expanded sequences together. With width two, B z and A x survive. Generation ends here because the exercise specifies exactly two tokens. B z wins with probability 0.380. Greedy decoding returned A x at 0.306. The example shows a search difference, not a percentage improvement in quality, and neither sequence represents a meaningful sentence.
What gets pruned, and why that matters
Beam search is not exhaustive search. At each step it removes candidates that fall outside the beam. In a longer sequence, a discarded path could eventually have become the best finished result. A wider beam retains more options, but it also requires more candidate scoring and memory. There is no universal width that makes every application better.
Width one behaves like greedy search under the same scoring and stopping assumptions. Width two describes how many candidates survive, not how many tokens the response contains. In a real vocabulary, each surviving sequence can have many possible next tokens. The decoder must compare those expansions before keeping the next beam.
To make the pruning visible, change the first-step probabilities to A = 0.70, B = 0.20, and C = 0.10. With width two, C disappears immediately. Even if C would have led to an excellent continuation later, that continuation will not be evaluated. This is why a beam is a bounded search strategy rather than a guarantee of the globally best sequence.
Read scores without confusing them with accuracy
Sequence probability describes a model's preference under a particular scoring setup. It does not measure whether a factual claim has a source, a translation preserves meaning, or a product caption names the right material. A fluent, probable sentence can still contain a false detail. Keep a separate review standard for the task you actually care about.
Real implementations often add log probabilities rather than multiplying small values repeatedly. That changes the arithmetic representation, not the basic idea of combining conditional scores. Comparing responses of different lengths introduces additional choices, including length normalization or penalties. Our two-token example deliberately avoids that complication so the first comparison remains hand-checkable.
Stopping also matters. A production decoder may encounter an end-of-sequence token, a maximum generation length, or other stopping criteria. A partially generated sequence and a finished sequence need a consistent comparison rule. When reproducing a published result, record the implementation, score settings, stopping policy, and model version rather than recording only the beam width.
Try the original calculation yourself
Download the small Python exercise. It uses Python's exact fractions for the four paths above, prints the greedy result, and ranks the expanded candidates. It does not download a model, call a paid service, or need an API key.
- Run the unmodified exercise and confirm A x = 0.306 and B z = 0.380.
- Change the probabilities after B to z = 0.50 and w = 0.50. Predict the winner before running it.
- Restore the original values and set
beam_width = 1to keep only one first-step candidate. Notice that B z is no longer considered. - Write down which assumptions made the comparison simple: fixed length, two branches, no sampling, and no additional score penalties.
The value of the exercise is being able to explain the decision. If the printed winner surprises you, inspect whether you multiplied conditional probabilities correctly or compared a second-step probability with a complete-sequence probability. Comparing 0.95 directly with 0.306 would mix two different quantities.
How this connects to creative prompting
You normally do not choose beam width in a consumer creative tool. Check the actual controls available in your chosen interface instead of adding “use beam search” to a natural-language prompt and assuming that changes the underlying decoder. Prompt wording, model selection, generation settings, and decoding implementation are different layers.
In Prompt Lab, the useful next step is to save the creative instructions and acceptance criteria you can control. For example, require a caption to preserve the product name and avoid unsupported claims, then compare the finished text against the approved brief. Use the prompt-testing workflow for that review. Read the tokens and context-window guide if the unfamiliar part is how the request is represented.
For implementation-level details, Hugging Face's generation-strategy documentation distinguishes greedy decoding, sampling, and beam search. Apply its controls only in a compatible implementation. The toy exercise here is intentionally much smaller than a complete text-generation system.
Frequently asked questions
What does beam width mean?
It is the number of partial sequences retained after a decoding step. It is not the response length or a factual-confidence score.
Is beam search always better than greedy decoding?
No. It can find a stronger sequence under a chosen scoring rule, but that does not guarantee better factual accuracy, usefulness, or creativity.
Does width one equal greedy search?
Under matching deterministic scoring and stopping assumptions, keeping only one candidate gives greedy behavior.
Can I turn it on by writing a prompt?
Do not assume so. A service must expose a supported decoding control. Ordinary prompt instructions are not proof that the generation algorithm changed.
Why use invented tokens instead of real model output?
The tiny example lets you verify every conditional probability and pruning decision. It teaches the mechanism without presenting an untested model benchmark.
Apply the distinction to a creative brief
Draft a precise creative instruction in Prompt Lab. Beam settings are implementation controls; this handoff does not imply that Prompt Lab exposes a beam-width switch.
