Word Embeddings in Detail: How Machines Turn Language Into Geometry

Computers cannot process words natively — they process numbers. To pass text into a machine learning model, you must map string tokens into a numerical format.

However, how you perform this mapping dictates whether your model treats words as isolated, arbitrary symbols or as points in a rich semantic space where geometric distance reflects meaning.

Here is how word embeddings evolved from basic counting tricks to modern contextual vectors, explained in detail with clear Python implementations.

1. The starting point: one-hot encoding and its flaws

The simplest way to numericalize text is one-hot encoding. You define a vocabulary of size $V$, assign an index to each unique word, and represent every word as a sparse vector of size $V$ containing a single 1 at its assigned index and 0s everywhere else.

import numpy as np
 
# Vocabulary: ["cat", "dog", "mat"] (V = 3)
vocab = {"cat": 0, "dog": 1, "mat": 2}
 
def one_hot_encode(word, vocab):
    vec = np.zeros(len(vocab))
    if word in vocab:
        vec[vocab[word]] = 1.0
    return vec
 
print("cat ->", one_hot_encode("cat", vocab))  # [1., 0., 0.]
print("dog ->", one_hot_encode("dog", vocab))  # [0., 1., 0.]

Why one-hot encoding fails in practice

• High dimensionality & sparsity — if your vocabulary has 1,000,000 unique words, every single word is a 1,000,000-dimensional vector consisting of 99.9999% zeros.
• Zero semantic understanding — in vector spaces, semantic similarity is measured via dot products or cosine similarity. For any two distinct one-hot vectors $\mathbf{u}$ and $\mathbf{v}$:

$\mathbf{u} \cdot \mathbf{v} = 0$

Under one-hot encoding, "cat" and "feline" are completely orthogonal — the model has no way to infer that they share meaning.

2. Counting semantics: co-occurrence matrices & TF-IDF

To capture meaning, models need to observe context. John Rupert Firth famously summarized this in 1957: "You shall know a word by the company it keeps."

Term-frequency vectorization

A co-occurrence matrix tracks how often words appear together within a fixed window. Words with similar meanings (e.g., "coffee" and "tea") end up with similar context profiles.

However, raw term frequency over-indexes on high-frequency stop words like "the", "is", and "and".

TF-IDF reweighting

TF-IDF (Term Frequency–Inverse Document Frequency) penalizes terms that appear everywhere across your corpus, highlighting words that carry distinctive information.

from sklearn.feature_extraction.text import TfidfVectorizer
 
corpus = [
    "the cute cat sat on the mat",
    "the energetic dog ran in the yard",
    "a feline rested on the rug",
]
 
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)
 
feature_names = vectorizer.get_feature_names_out()
print("Vocabulary size:", len(feature_names))
print("TF-IDF matrix shape:", tfidf_matrix.shape)

Limitation: while TF-IDF captures document-level topic similarity, the resulting matrix remains large, sparse, and unable to capture fine-grained word relationships (like analogies).

3. Dense representations: Word2Vec (CBOW and Skip-gram)

Introduced by Mikolov et al. at Google in 2013, Word2Vec revolutionized NLP by learning dense, low-dimensional vectors (typically 100–300 dimensions) using a simple shallow neural network.

Instead of counting, Word2Vec frames word representation as a predictive task across a moving context window:

CBOW       Context: ("the", "sat", "on")  ──predict──►  Target: "cat"
Skip-gram  Target:  "cat"                 ──predict──►  Context: ("the", "sat", "on")

The Skip-gram architecture

Given a sequence of words $w_1, w_2, \dots, w_T$, Skip-gram maximizes the average log probability of predicting context words within a window size $c$:

$\mathcal{L} = \sum_{t=1}^{T} \sum_{-c \le j \le c,\, j \neq 0} \log P(w_{t+j} \mid w_t)$

The probability is computed using a softmax over inner products of input matrix $\mathbf{W}$ and output matrix $\mathbf{W}'$:

$P(w_o \mid w_i) = \frac{\exp(\mathbf{v}'_{w_o}\!^\top \mathbf{v}_{w_i})}{\sum_{w=1}^{V} \exp(\mathbf{v}'_w\!^\top \mathbf{v}_{w_i})}$

Negative sampling

Computing the full softmax denominator over a vocabulary of 1,000,000 words for every token update is computationally intractable. Word2Vec solves this via Negative Sampling (SGNS), turning the multi-class classification problem into binary logistic regressions: distinguish the true context word $(w_o, w_i)$ from $k$ randomly sampled noise words $(w_n, w_i)$.

Training Word2Vec with Gensim

from gensim.models import Word2Vec
import re
 
raw_text = [
    "The quick brown fox jumps over the lazy dog",
    "The brown feline leaps across the sleepy hound",
    "Cats and dogs are common domestic animals",
]
 
sentences = [
    [word.lower() for word in re.findall(r"\w+", doc)] for doc in raw_text
]
 
# Train a 50-dimensional Skip-gram model
model = Word2Vec(
    sentences=sentences,
    vector_size=50,
    window=3,
    min_count=1,
    sg=1,           # 1 for Skip-gram, 0 for CBOW
    negative=5,     # Number of negative samples
    seed=42,
)
 
cat_vector = model.wv["cat"]
print("Vector dimension:", cat_vector.shape)  # (50,)
 
similarity = model.wv.similarity("cat", "dog")
print(f"Similarity between 'cat' and 'dog': {similarity:.4f}")

4. Global matrix factorization: GloVe

While Word2Vec focuses on local context windows, it ignores structural global co-occurrence statistics across the entire corpus.

GloVe (Global Vectors for Word Representation), developed by Pennington et al. at Stanford, combines the advantages of global matrix factorization (like SVD) and local context-window algorithms.

How GloVe works

GloVe trains directly on a global word-word co-occurrence matrix $X$, where $X_{ij}$ represents how many times word $i$ appears in the context of word $j$. The objective function minimizes the weighted least-squares difference:

$J = \sum_{i,j=1}^{V} f(X_{ij}) \left( \mathbf{w}_i^\top \mathbf{\tilde{w}}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2$

Where $f(X_{ij})$ is a weighting function that prevents rare co-occurrences from acting as noise, and over-represented co-occurrences from dominating.

def glove_weighting_function(x, x_max=100, alpha=0.75):
    """Downweights extremely frequent co-occurrences."""
    return (x / x_max) ** alpha if x < x_max else 1.0
 
print("Weight for count 5:", glove_weighting_function(5))
print("Weight for count 1000:", glove_weighting_function(1000))

5. Linear vector arithmetic: the geometry of meaning

One of the most remarkable properties of static embedding spaces (Word2Vec, GloVe, FastText) is their linear structure. Relationships between words map to vector offsets:

$\mathbf{v}_{\text{King}} - \mathbf{v}_{\text{Man}} + \mathbf{v}_{\text{Woman}} \approx \mathbf{v}_{\text{Queen}}$

[King] ───────────────────► [Queen]
  │                            │
  │ - (Man)                    │ - (Man)
  │ + (Woman)                  │ + (Woman)
  ▼                            ▼
[Man] ─────────────────────► [Woman]

Subword embeddings: FastText

A core flaw of standard Word2Vec/GloVe models is the out-of-vocabulary (OOV) problem: if a word wasn't seen during training, it cannot receive a vector.

Meta's FastText fixes this by representing words as bags of character $n$-grams. The vector for "apple" with $n=3$ includes representations for <ap, app, ppl, ple, le>.

• Out-of-vocabulary handling — unseen words are represented by summing their constituent character $n$-gram vectors.
• Morphological awareness — handles prefixes and suffixes automatically ("running" shares subwords with "runner").

6. The modern era: static vs. contextual embeddings

Static embeddings assign one fixed vector to a word, regardless of how it is used in a sentence:

• "I deposited money at the bank."
• "We walked along the river bank."

In static models (Word2Vec/GloVe), both instances of "bank" share the exact same vector representation.

Transformer contextual embeddings

Modern models like BERT, RoBERTa, and GPT compute contextual embeddings. Using self-attention mechanisms, every word's vector representation is dynamically generated based on the entire surrounding sentence.

from transformers import AutoTokenizer, AutoModel
import torch
 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
 
sentence_1 = "I deposited money at the bank."
sentence_2 = "We walked along the river bank."
 
inputs_1 = tokenizer(sentence_1, return_tensors="pt")
inputs_2 = tokenizer(sentence_2, return_tensors="pt")
 
with torch.no_grad():
    outputs_1 = model(**inputs_1)
    outputs_2 = model(**inputs_2)
 
# Target token: "bank" (index 5 in sentence 1, index 6 in sentence 2)
vec_bank_finance = outputs_1.last_hidden_state[0, 5, :]
vec_bank_river = outputs_2.last_hidden_state[0, 6, :]
 
cos_sim = torch.nn.functional.cosine_similarity(
    vec_bank_finance.unsqueeze(0),
    vec_bank_river.unsqueeze(0),
)
 
print(f"Cosine similarity between financial 'bank' and river 'bank': {cos_sim.item():.4f}")

Comparative matrix

Summary

Choosing the right word embedding strategy depends on your resource constraints and system objectives:

• TF-IDF remains a strong, fast baseline for basic text classification and keyword search.
• Static embeddings (Word2Vec, FastText) offer high inference speeds for lightweight deployments and similarity lookups.
• Contextual embeddings (Transformers) are required for complex downstream tasks requiring deep semantic resolution across polysemous words.

Reference links

• Efficient Estimation of Word Representations in Vector Space (Word2Vec paper)
• GloVe: Global Vectors for Word Representation (Stanford)
• Enriching Word Vectors with Subword Information (FastText paper)
• Hugging Face Transformers Documentation