Input Embedding and Position Embedding

In the context of Transformer models, which are widely used in natural language processing, two key concepts are “input embeddings” and “position embeddings.” These embeddings are crucial for understanding how Transformers process sequential data like text.

Input Embeddings

1. Purpose: Input embeddings are used to convert tokens (usually words or subwords) into vectors of fixed size. In natural language processing, words are represented as discrete, symbolic entities. However, neural networks work better with continuous, dense vectors. Input embeddings transform these discrete tokens into a format that the neural network can process.
2. Training: In models like BERT or GPT, these embeddings are learned during the training process. The model adjusts the embeddings to capture semantic and syntactic information about the words.
3. Dimensionality: The dimension of these embeddings is a hyperparameter of the model. Larger dimensions can capture more information but also increase computational complexity.

Position Embeddings

1. Purpose: The Transformer architecture, unlike models like RNNs or LSTMs, does not inherently process tokens in a sequence one after the other. Instead, it processes all tokens simultaneously. Position embeddings are added to input embeddings to provide the model with information about the position of each token in the sequence.
2. Representation: Position embeddings are vectors that encode the position of each token in the sequence. Each position in the sequence has a corresponding position embedding. These embeddings are added to the input embeddings to create a combined representation that carries both the semantic meaning of the token and its position in the sequence.
3. Variants: There are different ways to create position embeddings. The original Transformer model used a specific mathematical formula to generate fixed position embeddings. However, some variants of Transformers, like BERT, use learned position embeddings, where the position embeddings are parameters that are learned during the training process.

The positional encoding formula is as follows:

For position pos and dimension i in the embedding, the positional encoding PE(pos, i) is given by:

• PE(pos, 2i) = $\sin(\frac{pos}{10000^\frac{2i}{d}}​)$
• PE(pos, 2i+1) = $\cos(\frac{pos}{10000^\frac{2i}{d}}​)$

Where:

• pos is the position of the token in the sequence.
• i is the dimension, ranging from 0 to d − 1, where d is the dimensionality of the embeddings.
• The embeddings are alternately encoded with sine and cosine functions at each even and odd index, respectively.

Combined Role in Transformers

In a Transformer model, each token of the input sequence is first converted into an input embedding. Then, a position embedding corresponding to the position of the token in the sequence is added to this input embedding. The result is a combined embedding that carries both the meaning of the token and its position in the sequence. This combined embedding is then fed into the subsequent layers of the Transformer model for further processing.

This approach allows the Transformer to understand both the content of the input (through the input embeddings) and how each piece of content relates to others in the sequence (through the position embeddings), which is essential for tasks like language understanding, translation, and text generation.