transformer embed模型详解

### Transformer Embedding Model Detailed Explanation

In the context of transformers, embeddings serve as crucial components that convert discrete tokens into continuous vector spaces. Each token from an input sequence is transformed into a dense vector representation through this process[^1]. The embedding layer captures semantic meanings and relationships among words or patches (in case of images), which are then fed into subsequent layers.

For text-based tasks using Hugging Face’s `transformers` library, word embeddings typically include positional encodings to preserve order information since self-attention mechanisms do not inherently account for positionality[^2].

#### Positional Encoding
Positional encoding adds absolute or relative position information to each token's embedding so that the model can distinguish different positions even when identical tokens appear multiple times within one sentence. This approach ensures that the attention mechanism understands where each part belongs spatially without relying solely on content similarity[^3].

import numpy as np

def get_positional_encoding(max_len, d_model):
    pe = np.zeros((max_len, d_model))
    position = np.arange(0, max_len)[:, None]
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    pe[:, 0::2] = np.sin(position * div_term)
    pe[:, 1::2] = np.cos(position * div_term)

    return pe

The above code snippet demonstrates how sinusoidal functions generate fixed positional encodings based on token indices and dimension sizes. These values remain constant during training but help maintain sequential structure awareness throughout processing stages.

#### Tokenization & Vocabulary Mapping
Before applying any transformation operations like those mentioned earlier, raw inputs must first undergo preprocessing steps such as tokenizing sentences into individual units suitable for numerical computation while mapping them against predefined vocabularies maintained inside specific models provided by libraries similar to huggingface/transformers.

--related questions--

1. How does positional encoding contribute to maintaining temporal dependencies in sequences?
2. What alternatives exist besides sine/cosine waves for generating positional codes?
3. Can you explain why adding rather than concatenating position-specific signals works better with transformer architectures?
4. In what scenarios might custom vocabulary mappings be necessary over default ones offered by popular NLP frameworks?