Applying Merges In Rank Order
A BPE tokenizer was trained and produced this merge table. Merges are applied to a word in rank order, rank 1 first, and each merge is applied everywhere it matches before the next one is considered.
| rank | merge | new symbol |
|---|---|---|
| 1 | u + g |
ug |
| 2 | h + ug |
hug |
| 3 | p + u |
pu |
| 4 | n + </w> |
n</w> |
The word pug never appeared in training. At inference it is first split into characters, p u g </w>. How is it tokenized?
Sign in to answer questions and track your progress
Sign In