Applying Merges In Rank Order

A BPE tokenizer was trained and produced this merge table. Merges are applied to a word in rank order, rank 1 first, and each merge is applied everywhere it matches before the next one is considered.

rank merge new symbol
1 u + g ug
2 h + ug hug
3 p + u pu
4 n + </w> n</w>

The word pug never appeared in training. At inference it is first split into characters, p u g </w>. How is it tokenized?

A

pug </w>

B

p ug </w>

C

pu g </w>

D

p u g </w>