Embeddings & Vector Arithmetic¶

Agentic AI · L-Embed

Text as a vector: the geometry that turns "similar in meaning" into "close in $\mathbb{R}^d$", how that geometry is trained rather than found, and the nearest-neighbour search an agent runs over it.

Prerequisite: the recap Linear Algebra.

Learning Objectives¶

After this week you can:

1. Compute cosine similarity and Euclidean distance, and state when the two rank neighbours identically.
2. Explain what an embedding is and why semantic similarity ≈ geometric closeness.
3. Distinguish a vector per word type, per word token and per text, and say which problem each granularity solves.
4. Describe how a word vector is trained from co-occurrence — CBOW, skip-gram, the co-occurrence-ratio argument of GloVe, and subword $n$-grams — and read the cost count that decides between the architectures.
5. Explain why a text vector must be trained rather than pooled from an untuned contextual model, and read the InfoNCE objective that trains it.
6. Perform semantic search by nearest neighbour over embeddings, and explain why a bi-encoder makes that search possible where a cross-encoder does not.
7. Distinguish symmetric from asymmetric embedding tasks and apply the matching task prefixes.
8. Say where agents use embeddings — retrieval (RAG), memory, tool selection.

Overview¶

An agent compares text by meaning rather than by exact words: to retrieve a relevant document, to recall a stored fact, or to select one tool out of many. The construction that makes such a comparison computable is the embedding — a map from text to a vector in which closeness is similarity.

Vectors, the dot product, norms and the angle between two vectors are recapped in Linear Algebra; Part I adds the two measures by which embeddings are compared, Part II states how one vector per word type is learned from co-occurrence, Part III how a vector is computed for a word token and for a whole text, and Part IV how an agent searches a collection of text vectors:

Part I — Vector Math¶

Every construction in this lecture reduces to a comparison of two vectors in $\mathbb{R}^d$. Vectors, the dot product, norms and the angle between two vectors are a prerequisite of the lecture, recapped in Linear Algebra; this part adds the two measures by which embeddings are compared.

Cosine Similarity¶

Cosine similarity uses the cosine of the angle between two vectors (Linear Algebra) as a measure of similarity: it compares the directions of two embeddings and ignores their lengths.

$$ S_C(\mathbf{u},\mathbf{v}) = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|_2\,\|\mathbf{v}\|_2} = \cos\theta \;\in\; [-1, 1] $$

• $S_C = 1$: the same direction, the most similar pair; $S_C = 0$: orthogonal vectors, an unrelated pair; $S_C = -1$: opposite directions.
• Scale invariance: multiplying either vector by a positive factor leaves $S_C$ unchanged, so the length of an embedding does not enter the comparison.

Euclidean Distance¶

The Euclidean distance of two vectors is the length of their difference vector.

$$ \|\mathbf{u}-\mathbf{v}\|_2 = \sqrt{\sum_i (u_i - v_i)^2} $$

Expanding the square gives $$ \|\mathbf{u}-\mathbf{v}\|_2^2 = \|\mathbf{u}\|_2^2 + \|\mathbf{v}\|_2^2 - 2\,\mathbf{u}\cdot\mathbf{v}. $$ For unit vectors this is $2 - 2\cos\theta$ — so on normalized embeddings, Euclidean distance and cosine similarity rank neighbours identically.

Part II — Sparse Vectors and Word Embeddings¶

A word embedding assigns one vector to each word type of a fixed vocabulary; the vector is learned from the co-occurrence statistics of a corpus, so that words which occur in similar contexts receive nearby vectors. The embedding is a dense vector of a few hundred learned real numbers; the alternative is a sparse vector with one dimension per word of the vocabulary, whose entries are word counts weighted by how rare each word is across documents (tf-idf) (Jurafsky & Martin, 2026, §11.1).

Sparse and Dense Text Vectors¶

A text becomes a vector in one of two ways: as a sparse vector with one dimension per word of the vocabulary, whose entries are weighted word counts, or as a dense vector of a few hundred learned real numbers — an embedding. Retrieval systems divide along the same line: sparse retrieval compares the weighted word counts of a query and a document, dense retrieval their embeddings (Jurafsky & Martin, 2026, §11.1).

Sparse vectors¶

The bag-of-words vector. The vector space model represents a document $d$ by the counts of the words it contains; the order of the words is discarded and only their frequency is kept, which is why the model is also called the bag-of-words model (Jurafsky & Martin, 2026, §11.1.1). The vector has one entry per word of the vocabulary $V$, so its length is $|V|$, typically 10,000 to 50,000, and almost all of its entries are zero (Jurafsky & Martin, 2026, §5.3). With $\mathbf{e}_w$ the one-hot vector of the word $w$ — a 1 in the dimension of $w$ and 0 in every other (see What Is an Embedding?) — the count vector of a text $w_1 \dots w_n$ is the sum of the one-hot vectors of its words:

$$\mathbf{x}_d = \sum_{i=1}^{n} \mathbf{e}_{w_i}, \qquad x_{d,t} = \operatorname{count}(t, d).$$

The count vectors of a collection, placed side by side as columns, form the term-document matrix: one row per word, one column per document.

Example — length and direction. A 5-page document about cats and a one-sentence tweet about cats point in nearly the same direction ($\theta \approx 0^\circ \Rightarrow S_C \approx 1$). The length of a count vector grows with the text, so the long document's vector has a much larger norm $\|\mathbf{x}_d\|_2$: the Euclidean distance of the two is large, while their cosine similarity is close to 1. After both vectors are normalised, the two measures give the same ranking, by the identity for unit vectors (see Euclidean Distance).

Weighting: tf-idf. Retrieval does not use the raw counts. It multiplies two weights, each of which corrects one defect of the count (Jurafsky & Martin, 2026, §11.1.2):

• Term frequency measures how important a word is within one document. A word that occurs more often in a document is likely to be informative about the content of the document, but a word occurring 100 times is not 100 times as informative as a word occurring once. The weight therefore grows with the count, but more slowly than the count.
• Inverse document frequency measures how well a word distinguishes one document from the others in the collection. A word that occurs in only a few documents separates those documents from the rest; a word that occurs in every document separates none. The weight therefore falls as the number of documents containing the word rises.

The term frequency replaces the count by its logarithm, so the weight rises by 1 for every tenfold increase of the count (counts 1, 10, 100 give weights 1, 2, 3):

$$\mathrm{tf}_{t,d} = \begin{cases} 1 + \log_{10} \operatorname{count}(t,d) & \text{if } \operatorname{count}(t,d) > 0, \\ 0 & \text{otherwise.} \end{cases}$$

In a collection of $N$ documents, of which $\mathrm{df}_t$ (the document frequency of $t$) contain the word $t$,

$$\mathrm{idf}_t = \log_{10} \frac{N}{\mathrm{df}_t},$$

so a word that occurs in every document has $\mathrm{idf}_t = \log_{10} 1 = 0$.

The entry of the document vector is $\text{tf-idf}(t,d) = \mathrm{tf}_{t,d} \cdot \mathrm{idf}_t$. In the 37 plays of Shakespeare, Romeo occurs in one play ($\mathrm{idf} = \log_{10} 37 = 1.57$), battle in 21 ($0.246$) and good in all 37 ($0$): good contributes nothing to any play's vector, however often it occurs.

BM25. tf-idf leaves two properties of the count uncorrected. BM25, the weighting of current sparse retrievers and a variant in the tf-idf family (Jurafsky & Martin, 2026, §11.1.2), corrects both (Robertson & Zaragoza, 2009, §3.4):

• Saturation. The logarithmic term frequency grows more slowly than the count, but it has no upper bound: $1 + \log_{10} \operatorname{count}(t,d)$ tends to infinity (Robertson & Zaragoza, 2009, §3.5). The occurrences of a word can show at most that the document is about the word; once that is established, further occurrences carry no information about relevance (Robertson & Zaragoza, 2009, §3.4). BM25 therefore uses a term-frequency weight that rises towards a fixed maximum.
• Document length. A long document contains more occurrences of every word. If the author uses more words to say the same thing (verbosity), the count should be divided by the length; if the document covers more topics (scope), it should not. Real collections contain both kinds of long document, so BM25 normalises by the length only in part (Robertson & Zaragoza, 2009, §3.4).

With $|d|$ the length of document $d$ in words and $|d_{\text{avg}}|$ the average length of a document in the collection, the BM25 score of $d$ for a query $q$ is (Jurafsky & Martin, 2026, Eq. 11.12)

$$\operatorname{score}(q,d) = \sum_{t \in q} \underbrace{\log \frac{N}{\mathrm{df}_t}}_{\text{idf}} \cdot \underbrace{\frac{\operatorname{count}(t,d)}{k \left(1 - b + b\,\frac{|d|}{|d_{\text{avg}}|}\right) + \operatorname{count}(t,d)}}_{\text{saturated, length-normalised tf}}.$$

• $k \ge 0$ sets the saturation. At $k = 0$ the fraction is 1 for every query word the document contains, so only the presence of the word counts; a large $k$ makes the weight nearly proportional to the count. With $k = 1.2$ and $b = 0$, counts of 1, 10 and 100 give 0.45, 0.89 and 0.99, against a maximum of 1; the logarithmic term frequency gives 1, 2 and 3.
• $b \in [0, 1]$ sets the length normalisation. $b = 1$ divides the count by the relative length $|d| / |d_{\text{avg}}|$ (verbosity), $b = 0$ ignores the length (scope).
• Usual values are $k \in [1.2, 2]$ and $b = 0.75$ (Jurafsky & Martin, 2026, §11.1.2).

What Is an Embedding?¶

An embedding is a learned function that maps text to a vector in $\mathbb{R}^d$ so that semantically similar text lands nearby. In one-hot coding, by contrast, each word receives an axis of its own: every pair of words is then orthogonal, and cat is as dissimilar from kitten as it is from car. An embedding instead compresses meaning into a dense vector, in which cat and kitten are nearly aligned.

A central difficulty of natural-language processing (NLP) is that a word form need not carry one meaning:

• Polysemy: one word with several related meanings — play as a theatrical performance against play in a sports game.
• Homonymy: one word form with entirely unrelated meanings — the bank of a river against a financial bank.

Three answers have been given to the question what receives a vector, and they arrived in this order:

The word-level static models are where the geometry is easiest to derive; the text-level models are what a modern retrieval system calls.

Meaning ≈ Geometry¶

Because similar meanings map to nearby vectors, geometry encodes semantics:

• Similarity = high cosine (small angle).
• Analogies as vector arithmetic — the classic $\text{king} - \text{man} + \text{woman} \approx \text{queen}$ (the vector-offset method, Mikolov, Yih & Zweig 2013).

This property makes it possible to search by meaning instead of by keywords.

Word Embeddings and the Distributional Hypothesis¶

A word vector is learned from co-occurrence alone. No model in this section reads a dictionary, a parse tree or a morphological analysis; each one is a loss function built on a single assumption, the distributional hypothesis (Harris 1954; Jurafsky & Martin, 2026, ch. 5): a word is characterised by the words it occurs with.

Three training signals implement that assumption:

1. Predict, locally — word2vec: predict a word from its neighbours, or its neighbours from it.
2. Fit the counts, globally — GloVe: fit the logarithm of a word–word co-occurrence count.
3. Decompose the word — fastText: build the word's vector from its character $n$-grams.

All three produce a space that is searched with the cosine similarity.

CBOW and Skip-gram¶

word2vec (Mikolov et al. 2013) proposes two architectures, both log-linear: an input layer, a projection, and an output layer, with no non-linear hidden layer.

• CBOW (continuous bag of words) predicts the middle word from its context. The context vectors are averaged, so word order inside the window is discarded — hence bag of words — while the representation stays continuous. The reported best configuration uses four words of history and four of future.
• Skip-gram inverts the direction: it predicts the context from the middle word. For each training word a radius $R$ is drawn uniformly from $\{1,\dots,C\}$ and the $R$ words on each side become the labels, which weights near words more heavily than distant ones. The experiments use $C = 10$.

The word vectors are the rows of the projection matrix: the prediction task serves only to train the matrix, and the matrix is the result.

Why the hidden layer was removed. Mikolov et al. (2013) measure a model by the number of parameters accessed per training example, written $Q$, with total cost $O = E \times T \times Q$ for $E$ epochs over $T$ training words. With vocabulary size $V$, vector dimension $D$, context size $N$ and hidden layer size $H$:

The letters are the paper's own: its $D$ is the dimension written $d$ elsewhere in this lecture, its $V$ is the vocabulary size (written $|V|$ in the course notation), and its $T$ counts training words rather than denoting a temperature.

Deleting the non-linear hidden layer deletes the $N D H$ term; the $V$-sized output is replaced by a hierarchical softmax over a Huffman tree of the vocabulary, which evaluates about $\log_2 V$ units instead of $V$. Mikolov et al. state the trade-off: models "that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently". The removal made it possible in 2013 to train on a 6-billion-word corpus with a vocabulary of one million words.

Semantics vs. Syntax for Word Embeddings¶

The analogy test set of Mikolov et al. (2013) divides its questions into semantic and syntactic relations:

Which of the two, and what the numbers say. Trained on the same 320 M-word corpus with 640-dimensional vectors, the two architectures separate by question type (Mikolov et al. 2013, Table 3; accuracy in per cent on the paper's own Semantic-Syntactic Word Relationship set):

Skip-gram is much stronger on semantics, CBOW slightly stronger on syntax and about three times cheaper (one day against three for 300-dimensional vectors on 783 M words).

Further results of the same study.

• Dimension and data must grow together. Holding one fixed exhausts the other: at $d = 50$ the accuracy stops improving beyond a few hundred million words, and at 24 M words it stops improving beyond $d = 100$. Under the cost formula above, doubling the corpus costs about as much as doubling $d$, which makes the allocation an explicit decision rather than an accident.
• One epoch over twice the data beats three epochs over the same data, and takes less time (53.8 % against 53.3 %, two days against three).

One point of vocabulary. This paper trains its output layer with the hierarchical softmax described above. The negative sampling objective usually associated with word2vec — the normalised softmax replaced by a set of binary decisions against sampled non-neighbours — is introduced in the same authors' follow-up paper (Mikolov et al. 2013b), together with the subsampling of frequent words that makes it fast, and it is the objective fastText builds on.

Global Co-occurrence Counts: GloVe¶

GloVe (Global Vectors) is a word-embedding model that is fitted to the global word–word co-occurrence counts of a corpus, where word2vec is trained on local context windows (Pennington et al. 2014).

word2vec (Mikolov et al. 2013) produces vectors whose differences encode analogies, but does not explain why. GloVe (Pennington et al. 2014) supplies the argument, and it begins with a co-occurrence table rather than a predictive model.

Let $X_{ij}$ count how often word $j$ occurs in the context of word $i$ — throughout this section $X$ is a matrix of counts, not a random variable — and let:

$$P_{ij} = P(j \mid i) = \frac{X_{ij}}{X_i}$$

where $X_i = \sum_k X_{ik}$ is the total number of times any word appears in the context of word $i$.

The ice / steam Co-occurrence Ratios¶

The relationship between target words $i = \textit{ice}$ and $j = \textit{steam}$ can be examined by studying the ratio of their co-occurrence probabilities with various probe words $k$ over a 6-billion token corpus:

The ratio $\frac{P(\textit{solid} \mid \textit{ice})}{P(\textit{solid} \mid \textit{steam})}$ is much larger than 1, revealing that $\textit{ice}$ has a much stronger association with $\textit{solid}$ than $\textit{steam}$ does. Conversely, for $k = \textit{gas}$, the ratio is much smaller than 1. Ratios for non-discriminative context words ($\textit{water}$ or $\textit{fashion}$) remain close to $1$.

Compared to raw probabilities, the ratio isolates relevant words ($\textit{solid}$ and $\textit{gas}$) from noise words ($\textit{water}$ and $\textit{fashion}$). The key quantity word vectors should encode is therefore the ratio of co-occurrence probabilities, not the raw probabilities themselves.

From Intuition to the Log-Bilinear Model¶

Three constraints leave the model only one form: the relation between two words is carried by their difference $(\mathbf{w}_i - \mathbf{w}_j)$, because a vector space is a linear structure; that difference must reduce to a scalar, because the ratio it has to reproduce is one; and exchanging a word with a context word must change nothing, because the count matrix is symmetric. What survives is a dot-product relationship

$$\mathbf{w}_i^\top \tilde{\mathbf{w}}_k + b_i + \tilde{b}_k = \log X_{ik},$$

with $\tilde{\mathbf{w}}_k$ the separate vector of the context word, and $b_i$ and $\tilde b_k$ absorbing the terms that depend on one side alone.

Because $75\%\text{--}95\%$ of the co-occurrence matrix is zero, GloVe optimizes a weighted least-squares loss strictly over non-zero entries ($X_{ij} > 0$):

$$J = \sum_{i,j=1}^{|V|} f(X_{ij}) \left( \mathbf{w}_i^\top \tilde{\mathbf{w}}_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2$$

where $f(X_{ij})$ downweights rare, noisy co-occurrences and caps frequent stop words. Since $f(0) = 0$, zero entries are ignored and the fit costs $\mathcal{O}(|C|^{0.8})$ in the corpus length $|C|$ rather than $\mathcal{O}(|C|)$.

GloVe vs. word2vec: Counting vs. Predicting¶

Counting (GloVe) and predicting (word2vec) are usually presented as opposing paradigms, but they use the same corpus statistics:

• word2vec (Skip-gram): Scans local context windows sequentially, an $\mathcal{O}(|C|)$ pass over the $|C|$ tokens of the corpus. Summed over the corpus, its objective equals $\sum_i X_i\, H(P_i, Q_i)$: the cross-entropy $H$ between the co-occurrence distribution $P_i$ of word $i$ and the model's softmax distribution $Q_i$, weighted by the row count $X_i$ (Pennington et al. 2014, §3.1).
• GloVe: Tabulates global counts upfront into matrix $X$, then fits a log-bilinear model via weighted least-squares over $X_{ij} > 0$.

The two differ in their distance measure and their weighting, and recover the same linear structure from the same global co-occurrence distribution.

Subword Vectors: fastText¶

fastText represents a word as the sum of the vectors of its character $n$-grams, so that a word absent from the training corpus still receives a vector and related word forms share parameters.

Both word2vec and GloVe assign a single vector to each word in a fixed vocabulary. A word absent from the training corpus (Out-of-Vocabulary / OOV) receives no vector at all, and morphologically related words like walking and walked are treated as entirely unrelated tokens. For morphologically rich or compounding languages (e.g., German Tischtennis, Finnish noun cases), this vocabulary bottleneck is severe.

fastText (Bojanowski et al. 2017) modifies the scoring function while retaining the underlying Skip-gram architecture. A target word is represented as a bag of character $n$-grams alongside the special boundary-wrapped word itself.

For example, with $n = 3$, the word where yields: $$\mathcal{G}_{\text{where}} = \{\texttt{<wh}, \texttt{whe}, \texttt{her}, \texttt{ere}, \texttt{re>}, \texttt{<where>}\}$$

where < and > mark word boundaries to distinguish prefixes/suffixes from internal character sequences.

Scoring Function & Context Representation¶

Context words are not decomposed into $n$-grams—a context word $c$ is represented by a single, standard word vector $\mathbf{v}_c$.

Given $\mathbf{z}_g$ as the vector representation for subword $g \in \mathcal{G}_w$, the scoring function between target word $w$ and context word $c$ becomes:

$$s(w,c) = \sum_{g \in \mathcal{G}_w} \mathbf{z}_g^\top \mathbf{v}_c$$

Core Advantages¶

• Parameter Sharing: Words sharing prefixes, suffixes, or roots share underlying subword parameters, enabling robust representations for rare inflections.
• Out-of-vocabulary words: An unseen word at test time still receives a vector by summing the vectors of its constituent $n$-grams.

The experiments of Bojanowski et al. (2017) extract $n$-grams for $3 \le n \le 6$, which makes training about $1.5\times$ slower than skip-gram: 105,000 against 145,000 words per second and thread.

Form and Meaning in Subword Vectors¶

Character $n$-grams encode orthographic form — the surface shape of a word — and form correlates much more strongly with grammar than with meaning. The evaluation splits accordingly (accuracy in per cent, Bojanowski et al. 2017, Table 2; sisg, subword information skip-gram, is the paper's own name for the model):

Syntax improves, semantics does not. Czech syntactic accuracy rises by 25.0 percentage points, from 52.8 % to 77.8 %; German semantic accuracy falls by 4.2 percentage points, from 66.5 % to 62.3 %. A shared character sequence is strong evidence of a shared grammatical form and weak evidence of a shared meaning.

Subwords make the model more data-efficient. Every word containing a gram trains that gram's vector, so the parameters are shared: a model trained on 5 % of German Wikipedia reaches a higher Spearman rank correlation with human similarity judgements on the German word-similarity set GUR350 than a CBOW baseline trained on all of it (Bojanowski et al. 2017, Fig. 1).

Part III — Contextual and Text Embeddings¶

A contextual embedding assigns a vector to each word token, computed from the sentence in which the token occurs; a text embedding assigns one vector to a whole text, and contrastive training arranges these vectors so that cosine similarity measures the similarity of the texts.

From Word Types to Word Tokens¶

A word-type vector must average every use of the word. Consider play. Its nearest neighbours under GloVe are playing, game, games, played, players, plays, player, football, multiplayer — several parts of speech at once, and all of them sporting. The theatre sense is present in the training corpus and absent from the vector, because one vector cannot hold two senses.

A contextual embedding assigns a vector to a word token — a word in a particular sentence — so that the vector may differ from occurrence to occurrence. ELMo, Embeddings from Language Models (Peters et al. 2018), obtains one from a bidirectional language model: a recurrent language model reading left to right and a second one reading right to left, trained jointly on the same corpus. The trained pair is then frozen and used as a feature extractor — the embedding of a token is a weighted sum of the model's own internal states at that position, one vector per layer. The nearest neighbours of a word in context are the most similar token vectors in a corpus: for play in "made a spectacular play on Alusik's grounder" the nearest token vector comes from a baseball sentence, for play in "signed to do a Broadway play" from a theatre sentence (Peters et al. 2018).

The layers of such a model are not interchangeable. Used directly as features, the lower layer tags parts of speech at 97.3 % accuracy against the top layer's 96.8, while the top layer disambiguates word senses at 69.0 $F_1$ — the harmonic mean of precision and recall — against the lower layer's 67.4. Syntax is represented low, meaning high, which is why ELMo hands the downstream task a learned weighting over all layers rather than the topmost one.

The Bidirectional Transformer Stack: BERT¶

BERT is a bidirectional Transformer stack: every position attends to every other position, so the vector of a token is computed from its left and its right context in one model (Devlin et al. 2019).

ELMo concatenates two models that each read in one direction; BERT (Bidirectional Encoder Representations from Transformers) replaces them by a single stack. Most of the text embedders in this lecture are built from it, so three of its properties are fixed here. The attention mechanism itself is not needed for that: the only property an embedder uses is that the attention is bidirectional.

• A masked training objective. Predicting the next word is unusable when every position may already read every other one, so BERT is trained as a masked language model — a fraction of the input tokens is hidden and predicted from the text that remains. A second objective, next-sentence prediction, asks of two given sentences whether they were adjacent in the corpus.
• The [CLS] token. Every input sequence begins with a special reserved token, [CLS] (short for classification). Rather than representing a specific word, its final-layer output vector aggregates contextual information from the entire input sequence. Downstream classification tasks (such as sentiment analysis or topic labeling) use this pooled vector as the input to a final linear prediction layer.
• A frozen BERT is a feature extractor. Its hidden states are usable without further training, and, as in ELMo, the layers specialise — the context-independent input layer is far weaker than the upper ones.

From Word Vectors to Text Vectors¶

Retrieval compares a query with a passage, not a word with a word, so what is needed is one vector per text. The obvious construction is to run a contextual encoder over the text and reduce its token vectors to one — by taking the mean, or by taking the [CLS] vector. Measured against human similarity judgements on seven standard similarity sets, that construction fails (Reimers & Gurevych 2019; rank correlation $\times 100$):

An untuned contextual encoder is beaten by averaged word vectors, and the [CLS] vector is far behind both. BERT's own authors state the reason in a footnote:

"The vector $C$ is not a meaningful sentence representation without fine-tuning, since it was trained with NSP." — Devlin et al. (2019), footnote 6

Here $C$ is the [CLS] output and NSP the next-sentence prediction objective: the vector is an aggregate for that task, and for no other.

Information Content and Metric Geometry¶

A text vector can contain the information a task needs and still be arranged so that its cosine does not measure similarity. A supervised classifier fitted on mean-pooled contextual vectors performs well because its linear layer learns a weight for each dimension and so selects the dimensions that carry the task.

Cosine similarity weights every dimension equally and has no parameters with which to make such a selection. Unless the model was trained so that similar texts lie at small angles, the cosine of two of its vectors is determined largely by dimensions that carry no information about the similarity of the texts.

An embedding model is a model whose geometry has been trained for retrieval by cosine similarity, and contrastive training is the method by which that geometry is built.

From Geometry to Systems: Bi-Encoders vs. Cross-Encoders¶

A bi-encoder computes one vector per text and scores a pair of texts by a fixed similarity function of the two vectors; a cross-encoder reads the two texts together as one sequence and outputs the score of the pair directly, without a vector for either text (Reimers & Gurevych 2019). Only the bi-encoder is an embedding model.

Cross-Encoders: Joint Attention, Non-Embedding Architecture¶

A cross-encoder feeds both texts through the network together as a single concatenated sequence ([CLS] Text A [SEP] Text B).

Self-attention over the concatenated sequence lets every token of one text attend to every token of the other, in every layer; this joint reading is the source of the cross-encoder's accuracy. A cross-encoder produces no sentence embedding. The score is indecomposable—no representation is ever computed for one text in isolation. Every pair requires a full forward pass through the model:

$$\text{Cost} = \frac{n(n-1)}{2} \text{ forward passes}$$

To find the most similar pair among $n = 10{,}000$ sentences, a cross-encoder requires 49,995,000 forward passes, about 65 hours (Reimers & Gurevych, 2019).

Bi-Encoders: Decomposable Scoring & Vector Embeddings¶

A bi-encoder routes each text through the Transformer separately to generate independent vector embeddings $f(a)$ and $f(b)$, then compares them using a fixed, parameter-free metric like cosine similarity ($S_C$):

$$\text{score}(a,b) = S_C\big(f(a), f(b)\big)$$

By factorizing the calculation into distinct vector representations, each text is encoded once and cached:

$$\text{Cost} = n \text{ forward passes} + \text{vector comparisons}$$

For $n = 10{,}000$ sentences, this requires 10,000 encodings (about 5 seconds) plus the cosine similarities of all $n(n-1)/2$ pairs (about 0.01 seconds; Reimers & Gurevych 2019). The time falls from 65 hours to about 5 seconds because the expensive step, the forward pass, runs once per text instead of once per pair.

Sentence-BERT builds such an encoder out of BERT (Reimers & Gurevych 2019), by fine-tuning it in a siamese structure: two copies of the network with tied weights — one set of parameters, applied twice — a pooling layer reducing each copy's token outputs to one vector (the mean, by default), and a training objective comparing the two vectors. At inference one copy and the cosine remain, and the second copy was never a second model.

Contrastive Training¶

Contrastive training shapes the geometry of a bi-encoder directly. The similarity function $S_C$ of a bi-encoder is fixed, so training can change only the vectors: a contrastive objective raises $S_C$ for matched pairs and lowers it for unmatched ones. A language-modelling loss, by contrast, trains the model to predict tokens and places no constraint on the cosine between two text vectors.

The contrastive objective in general use is InfoNCE, used for example by SimCSE (Gao, Yao & Chen 2021) and by cpt-text (Neelakantan et al., 2022).

The geometry is not discovered — it is optimised for. The training data are pairs $(q_i, p_i)$: a query and a passage that belongs to it. Write $\mathbf{q}_i,\mathbf{p}_i \in \mathbb{R}^d$ for their embeddings. Over a batch of $n$ such pairs the model minimises the InfoNCE loss — the name is from noise-contrastive estimation, learning by distinguishing a true pair from sampled noise:

$$\mathcal{L}_C = -\frac{1}{n}\sum_{i=1}^{n} \log \frac{\exp\left(S_C(\mathbf{q}_i, \mathbf{p}_i) / T\right)}{\sum_{j=1}^{n} \exp\left(S_C(\mathbf{q}_i, \mathbf{p}_j) / T\right)}$$

with $S_C$ the cosine similarity and $T > 0$ a temperature — the same quantity that sharpens a sampling distribution, and written $\tau$ in this literature.

Reading the loss. The fraction is a softmax over cosine similarities, so the expression as a whole is the cross-entropy of an $n$-way classification: which of these $n$ passages belongs to this query? What the loss does is decided by its denominator, its temperature and the sign of each term.

• The denominator is the batch. The competitors $\mathbf{p}_j$ are the passages of the other pairs in the same batch — in-batch negatives. Nothing has to be collected in advance: each pair supplies a positive to itself and a negative to every other pair, so a batch of $n$ pairs yields $n$ classification problems with $n-1$ competitors each.
• The temperature decides how much of the ranking is punished. Dividing by a small $T$ spreads the cosines apart before the softmax, so the loss is dominated by the single closest competitor; a large $T$ flattens them and distributes the gradient over all of them.
• Each term has one sign. Minimising $\mathcal L_C$ raises $S_C(\mathbf{q}_i,\mathbf{p}_i)$ and lowers every $S_C(\mathbf{q}_i,\mathbf{p}_j)$: a positive pair is pulled together, everything else is pushed apart.

Consequence. Cosine similarity is the search metric at inference time because it is the score the loss optimises during training. "Similarity" in vector space reflects whichever relationships were defined by the training pairs (Nussbaum et al. 2025).

The Text-Embedder Family¶

The models named here are trained by the same contrastive principle and differ in two places: the pre-trained stack they start from, and what counts as a positive pair. The constant is not BERT — it is bidirectional attention together with contrastive training.

The last row locates the property. A causally masked stack — the architecture of a generative language model, in which a position attends only to the positions up to itself — is a poor embedder as it stands: the earlier tokens of a passage never see the later ones. Deleting that mask makes the model bidirectional, and the ablation reports that "the bidirectional mask consistently outperforms the causal mask" for every pooling choice (Lee et al. 2025). What an embedder needs is bidirectional attention, not a particular architecture family.

One entry in the table deviates in the score rather than in the pre-trained stack: DPR compares its two vectors by the plain inner product and not by the cosine, so a long vector can win on magnitude alone. Every other model in the table normalises and compares by cosine.

Tokenization in an Embedder¶

An embedder reads neither characters nor words: its input is a sequence of subword tokens, produced by the same kind of learned tokenizer a generative model uses — byte-pair encoding (Sennrich et al. 2016), or the WordPiece vocabulary of 30,000 token types introduced with BERT (Devlin et al. 2019). How such a vocabulary is constructed is covered in → L-Att; three of its consequences belong here.

• The tokenizer belongs to the checkpoint. It is fixed during pre-training, together with the embedding matrix whose rows its token identifiers index, so a model must be called with its own tokenizer and with no other. nomic-embed-text is trained on text tokenized with BERT's bert-base-uncased vocabulary (Nussbaum et al. 2025); an embedder initialised from a causally masked generative model inherits that model's vocabulary together with its weights, as NV-Embed does from Mistral-7B (Lee et al. 2025).
• The token is the unit of every length limit. A model accepts a maximum number of tokens, not of words: nomic-embed-text is trained at 2048 tokens, and its own evaluation truncates every text to 512 (Nussbaum et al. 2025). Truncation raises no error — the part beyond the limit is simply not represented — so the chunk length of a retrieval corpus is measured in the embedder's tokens. A task prefix is ordinary text and is counted in the same budget.

• The embedder and the generator do not share a vocabulary. They are two models with two tokenizers, so one passage has two token counts, and a chunk that fits the generating model's context window may still exceed the embedder's input.

Subword tokenization is the second answer to the unseen word. fastText composes an out-of-vocabulary word from the character $n$-grams it contains; a subword tokenizer splits that word into pieces of a fixed vocabulary before the encoder sees it, so every input is representable and no word needs an entry of its own.

Part IV — Semantic Search¶

Semantic search returns the stored texts whose vectors are nearest to the vector of a query; an agent uses it for retrieval, for long-term memory and for tool selection.

Semantic Search as Nearest-Neighbour Search¶

Semantic search is nearest-neighbour search in the embedding space: the query and every document are embedded, and the documents whose vectors are closest to the query vector are returned.

1. Embed the query and every document → vectors.
2. Score each document by cosine similarity to the query.
3. Return the top-$k$ (the nearest neighbours).

Brute force costs $O(n\,d)$ per query over $n$ documents. That is affordable for thousands of documents; for millions it is replaced by approximate nearest-neighbour (ANN) search over a vector index (→ L-RAG).

Limits of nearest-neighbour search over embeddings.

• Asymmetric embeddings — query and document are embedded differently by models trained with task prefixes such as search_query: and search_document:, so the two sides are not mapped by the identical function. The section Symmetric and Asymmetric Tasks treats this case.
• Keyword blindness — dense vectors miss exact identifiers and error codes (e.g. ERR_2043); the remedy is hybrid search, a sparse keyword score (BM25, Robertson & Zaragoza 2009) + the dense one.
• Negation — "cats like milk" and "cats do not like milk" can have a high cosine (same topic, same words) despite opposite meaning.
• Uncalibrated scores — a cosine is meaningful within one query's ranking, not as an absolute number: a fixed cut-off such as if cos > 0.8 discards good matches for one query and admits noise for the next. Rank first, then cut by $k$ (Nussbaum et al. 2025).

Symmetric and Asymmetric Tasks¶

Not every use of a text embedding compares two things of the same kind, and the difference decides how the model must be called.

In a symmetric task — detecting duplicate questions, measuring semantic similarity, clustering — the two sides share structure, length and intent: two questions, two captions, two summaries. In retrieval, and therefore in RAG, they do not. A query is short, interrogative and states an information need; a passage is long, declarative and states facts. Nothing about the two texts is alike except the answer relation between them.

One encoder serving both is required to satisfy two different meanings of belongs together at once. The remedy is to tell the model which side it is encoding, and the marker has grown more expressive over time:

For a symmetric task the two sides receive the same marker; for an asymmetric one they do not (Su et al. 2023).

A marker is not the only mechanism, and it is not an architectural one. The designs differ in where the asymmetry is placed.

• Two encoders. Dense Passage Retrieval (DPR, Karpukhin et al. 2020) trains a separate, untied network for each side — $E_Q$ for questions, $E_P$ for passages — and scores a pair by the inner product $E_Q(q)^\top E_P(p)$. The asymmetry then lives in the weights rather than in the input, at the price of two networks to train and to serve.
• One network, two roles. E5, INSTRUCTOR and nomic-embed-text share a single set of weights and break the symmetry with a string prepended to the text. The encoder has no notion that the string is a marker: it is ordinary input, and the behaviour comes entirely from training pairs that carried the same strings.

The second is what is in general use, and its two failure modes are therefore silent. The corpus and the queries must follow the same convention — a wrongly marked query raises no error, it only ranks worse — and changing the convention means re-embedding the corpus, not re-running the query.

Embeddings in Agents¶

• Retrieval-augmented generation (RAG) — fetch documents relevant to the query and add them to the context.
• Long-term memory — store and recall facts by similarity (→ L-Memory).
• Tool selection — with many tools, embed the tool descriptions and retrieve only the relevant ones per request. Every tool's schema is re-sent on each call (→ L-ReAct) and the context window is finite, so this keeps the prompt small.

All three are nearest-neighbour search over embeddings.

In Practice / This Week's Lab¶

lecture-Embed-exercises.ipynb — calculations: cosine similarity & Euclidean distance by hand.

lecture-Embed-lab.ipynb: implement the distance functions and a brute-force nearest-neighbour search from scratch, then do semantic search over tool descriptions with real embeddings (Ollama nomic-embed-text), and check what a framework's embedder sends to the same model: task prefixes and token counts (LlamaIndex).

Summary¶

• Cosine similarity = angle only (scale-invariant); Euclidean distance = straight line; on unit vectors they rank the same.
• An embedding maps text → a vector so that meaning ≈ geometry. Three granularities, in historical order: one vector per word type (word2vec, GloVe, fastText), per word token (ELMo, BERT), per text (Sentence-BERT and the contrastively trained embedders in use today).
• A word vector is learned from co-occurrence alone: predict the context (word2vec), fit the logarithm of the count (GloVe), or sum the vectors of the character $n$-grams (fastText, which also gives an unseen word a vector).
• A contextual encoder does not yield a text vector: an untuned [CLS] vector is a worse similarity representation than averaged word vectors. The geometry is trained, by the InfoNCE objective over cosine similarity with in-batch negatives.
• The embedders differ in the pre-trained stack they start from — BERT, the bidirectional stack of T5, even a generative model whose causal mask has been removed — and agree on bidirectional attention and that objective.
• Semantic search = nearest neighbour by cosine, which is affordable only because a bi-encoder score factorises into one stored vector per text.
• Symmetric tasks compare two texts of the same kind; asymmetric ones compare a short query with a long passage, and the two sides must be marked accordingly.
• Agents use all of this for RAG, memory, tool selection.

References¶

Peer-reviewed papers

• Bojanowski, Piotr, et al. "Enriching Word Vectors with Subword Information." Transactions of the Association for Computational Linguistics, vol. 5, 2017, pp. 135–146. arXiv:1607.04606. — fastText: a word as a bag of character $n$-grams whose vectors are summed, so that related word forms share parameters and an out-of-vocabulary word still has a vector.
• Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019, pp. 4171–4186. arXiv:1810.04805. — masked language modelling, why a bidirectional model cannot be trained on the next word, and the footnote recording that the [CLS] vector is not a sentence representation without fine-tuning.
• Gao, Tianyu, Xingcheng Yao, and Danqi Chen. "SimCSE: Simple Contrastive Learning of Sentence Embeddings." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021, pp. 6894–6910. arXiv:2104.08821. — the contrastive objective for sentence embeddings, applying the two diagnostics of Wang & Isola (2020) to it.
• Harris, Zellig S. "Distributional Structure." Word, vol. 10, no. 2–3, 1954, pp. 146–162. doi:10.1080/00437956.1954.11659520. — the distributional hypothesis: a word is characterised by the contexts it occurs in, which is the assumption every word-embedding objective implements.
• Karpukhin, Vladimir, et al. "Dense Passage Retrieval for Open-Domain Question Answering." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 2020. arXiv:2004.04906. — DPR: a retrieval bi-encoder built from two untied BERT towers, and the argument that a retrieval score must be decomposable before the passage vectors can be precomputed.
• Lee, Chankyu, et al. "NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models." International Conference on Learning Representations. Vol. 2025. 2025. arXiv:2405.17428. — a causally masked generative model turned into an embedder by removing its causal attention mask: evidence that bidirectional attention, rather than the architecture family, is what an embedder needs.
• Liu, Yizhou, Ziming Liu, and Jeff Gore. "Superposition Yields Robust Neural Scaling." Advances in Neural Information Processing Systems. Vol. 38. 2025. arXiv:2505.10465. — superposition: more features than dimensions, placed at small angles, and the $1/m$ scaling of the loss that the interference between them produces.
• Mikolov, Tomas, et al. "Distributed Representations of Words and Phrases and their Compositionality." Advances in Neural Information Processing Systems. Vol. 26. 2013. arXiv:1310.4546. — negative sampling and the subsampling of frequent words; the objective every later embedder builds on.
• Mikolov, Tomas, et al. "Linguistic Regularities in Continuous Space Word Representations." Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013, pp. 746–751. ACL Anthology N13-1090. — the vector-offset method and the $\text{king}-\text{man}+\text{woman}$ analogy itself (the word2vec paper cites this one for it).
• Nussbaum, Zach, et al. "Nomic Embed: Training a Reproducible Long Context Text Embedder." Transactions on Machine Learning Research, 2025. arXiv:2402.01613. — the model this week's demo and lab both run: the InfoNCE objective over cosine similarity, hard negatives, and the four task prefixes (search_query / search_document / classification / clustering) with the training-signal conflict that motivates them.
• Pennington, Jeffrey, Richard Socher, and Christopher D. Manning. "GloVe: Global Vectors for Word Representation." Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014, pp. 1532–1543. ACL Anthology D14-1162. — the co-occurrence-ratio argument, the weighted least-squares fit of $\log X_{ij}$, and the reconciliation of the count-based and predict-based families.
• Peters, Matthew E., et al. "Deep contextualized word representations." Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018, pp. 2227–2237. arXiv:1802.05365. — ELMo: a vector per word token, the play nearest-neighbour comparison, and the measured split between syntax in the lower layer and word sense in the upper one.
• Reimers, Nils, and Iryna Gurevych. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019, pp. 3982–3992. arXiv:1908.10084. — text → one vector: the siamese bi-encoder compared by cosine, which is what makes semantic search a nearest-neighbour problem (65 h → 5 s for the most similar pair among 10,000 sentences).
• Robertson, Stephen, and Hugo Zaragoza. "The Probabilistic Relevance Framework: BM25 and Beyond." Foundations and Trends in Information Retrieval, vol. 3, no. 4, 2009, pp. 333–389. doi:10.1561/1500000019. — BM25, the sparse half of hybrid search.
• Sennrich, Rico, et al. "Neural Machine Translation of Rare Words with Subword Units." Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016. arXiv:1508.07909. — byte-pair encoding: the subword vocabulary an encoder is called with, and the reason an unseen word still has a representation.
• Su, Hongjin, et al. "One Embedder, Any Task: Instruction-Finetuned Text Embeddings." Findings of the Association for Computational Linguistics: ACL 2023. 2023, pp. 1102–1121. arXiv:2212.09741. — the symmetric versus asymmetric task distinction, and a written instruction per side as the marker that encodes it.
• Wang, Tongzhou, and Phillip Isola. "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere." International Conference on Machine Learning. Vol. 119. 2020. arXiv:2005.10242. — the alignment and uniformity terms the contrastive objective decomposes into, and therefore why distributionally matched pairs are pulled together whatever their truth value.

Preprints & other non-peer-reviewed papers

• Mikolov, Tomas, et al. "Efficient Estimation of Word Representations in Vector Space." arXiv preprint arXiv:1301.3781 (2013). — word2vec: the CBOW and skip-gram architectures behind word-level embeddings.
• Neelakantan, Arvind, et al. "Text and Code Embeddings by Contrastive Pre-Training." arXiv preprint arXiv:2201.10005 (2022). — contrastive training at scale, initialised from a language model, with in-batch negatives.
• Wang, Liang, et al. "Text Embeddings by Weakly-Supervised Contrastive Pre-training." arXiv preprint arXiv:2212.03533 (2024). — E5: text pairs harvested from the structure of web documents, and the query: / passage: prefixes that break a shared encoder's symmetry.

Textbooks & monographs

• Jurafsky, Daniel, and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd ed., online manuscript released August 19, 2026. https://web.stanford.edu/~jurafsky/slp3/ — the distributional hypothesis, count-based word vectors, the cosine as similarity measure and word2vec (ch. 5); the bag-of-words vector, the term-document matrix, tf-idf weighting and the distinction between sparse and dense retrieval (ch. 11).

Blogs, documentation & other web sources

• LlamaIndex. "Ollama Embeddings." LlamaIndex Developer Documentation, 2026. https://developers.llamaindex.ai/python/examples/embeddings/ollama_embedding/ — the class OllamaEmbedding, which prepends a task prefix only when query_instruction or text_instruction is set.
• Pydantic. "Embeddings." Pydantic AI Documentation, 2026. https://pydantic.dev/docs/ai/guides/embeddings/ — the methods embed_query and embed_documents, and the providers onto whose task marker they are mapped.