Lesson 5: Under the Hood
Finding word connections
In this step, each token looks at the other tokens around it in the sentence and works out which ones matter most for understanding its meaning here. This is called attention.
A word's sticker gives it a general meaning, but that meaning can change depending on the sentence. So the model checks nearby words for clues and updates the sticker on the spot.
In the sentence:
"The bat flew out of the cave"
The embeddings for "bat", "flew" and "cave" would be very close suggesting they are connected, leading to the model predicting "bat" means the animal.