3 / 4 Natural Language Processing

Two Washingtons, One Vector

A named entity recogniser feeds each token's pretrained word2vec vector into a classifier that predicts a tag for that token. On these two sentences

  • "Washington signed the treaty in Paris." (gold tag: PERSON)
  • "Washington raised its fuel tax this year." (gold tag: LOCATION)

the model emits the same tag for both occurrences of Washington, no matter how long it trains. What is the actual cause?

A

Washington is polysemous, so the fix is to store one embedding row per sense and select the row at inference using the surrounding part-of-speech tags.

B

word2vec stores one vector per word type, so the classifier receives numerically identical input in both sentences and cannot produce different outputs; the fix is a contextual encoder that computes a token's representation from the sentence it appears in.

C

The classifier is underfitting: the two sentences are separable in the embedding space, and a deeper network over the same word2vec features would learn the distinction.

D

The skip-gram context window used during pretraining was too narrow to see words such as treaty, so retraining word2vec with a wider window fixes it.