# Generation

### temperature

Temperature is applied by dividing by `temperature`

right before softmax. Higher temperature leads to more
diversed output. However, do note that `temperature`

does not change the magnitude orders of the probabilities,
hence zero impact on decoding when `do_sample=False`

.

### top_p

Top_p keeps the first few most probable candidates that the sum of their probabilities exceeds `top_p`

.
The rest are masked with a filter value such as `NaN`

so that they don’t get picked.

### top_k

Top_k simply keeps the `top_k`

most probable candidates. The rest are masked with a filter value such as `NaN`

so that they don’t get picked.

### repetition_penalty

Tokens that already occur, their probabilities got scaled down by dividing by `self.penalty`

.

## Different Decoding Strategies

### greedy search

The model selects the token with the highest conditional probability in each step during generation.

### beam search

In each time step, $k$ most probable candidates are kept to the next time step where the model generates the next token for $k$ different sequences, after which the most probable sequence overall is chosen.

As illustrated from the graph above, in time step 1, the sequence (“The”, “dog”) and (“The”, “nice”) are reserved. In time step 2, the model generates next token for both sequences and the new sequence (“The”, “dog”, “has”) is selected because its overall probability is higher: $0.4*0.9 > 0.5 * 0.4$.

### constrained beam search

In open-ended generation, a couple of reasons have been brought forward why beam search might not be the best possible option:

- Beam search can work very well in tasks where the length of the desired generation is more or less predictable as in machine translation or summarization - see Murray et al. (2018) and Yang et al. (2018). But this is not the case for open-ended generation where the desired output length can vary greatly, e.g. dialog and story generation.
- We have seen that beam search heavily suffers from repetitive generation. This is especially hard to control with
*n-gram*- or other penalties in story generation since finding a good trade-off between inhibiting repetition and repeating cycles of identical*n-grams*requires a lot of finetuning. - As argued in Ari Holtzman et al. (2019), high quality human language does not follow a distribution of high probability next words. In other words, as humans, we want generated text to surprise us and not to be boring/predictable. The authors show this nicely by plotting the probability, a model would give to human text vs. what beam search does.

## Quantization

An example to load a model in 4bit using NF4 quantization below with double quantization with the compute dtype bfloat16 for faster training:

## sampling

### stopping criteria

## Speed Up Generation

### speculative decoding

In speculative decoding, a small but competent draft model is used to start generating tokens. The base model(big one) is used to examine the output tokens. Tokens are accepted according to the log probabilities. The base model is then used again to actually generate tokens starting from the rejected token. It can be mathematically proved that the probability distribution is the same as if it was just the base model the whole time. Hence no performance loss while a significant speed up in generation is obtained.

Recent approaches include LLMLingua which speeds up inference by compressing prompt and KV cache with minimal performance loss, and Medusa which achieves the same performance as speculative sample by attaching and training multiple medusa heads, hence eliminating the need for small yet competent draft models. Note that Medusa indeed requires extra training.