Explainer 1 of 5

Word and title embeddings

Part of the interactive companion to Mapping the Market for Ideas in Europe, 1450–1650, by Noel Johnson and Alexander Taylor

An embedding turns a word into a list of numbers so that words used in similar ways get similar lists. We use embeddings to measure what a book was about from its title alone. This page explains how the vectors are built, in four steps, and why we built them this way.

Why a vector and not a subject class

The Universal Short Title Catalogue assigns each edition one of about 20 subject classes. A class supports counts and shares. It cannot say how far apart two books are. Erasmus on the freedom of the will and Luther's reply are both filed under Religious, and so are a Catholic devotional manual and a Calvinist catechism. A vector gives every title a position, and positions have distances. Every measure in the paper is a distance between titles or between averages of titles.

Step 1. Count which words appear together

The idea behind every embedding is that a word is known by the company it keeps (Mikolov et al. 2013, Pennington, Socher, and Manning 2014). In Latin titles medicus appears near chirurgia and rex appears near imperator. For each language we count how often each pair of words appears within five words of one another in a title. Words that appear fewer than 20 times are left out. The result is a large table with one row and one column for every word.

Step 2. Correct for how common each word is

Raw counts mostly track frequency. Two common words appear together often whether or not they are related. Pointwise mutual information compares how often a pair appears together with how often it would if the two words were unrelated (Church and Hanks 1990).

$$\mathrm{PMI}(w, c) \;=\; \log \frac{p(w, c)}{p(w)\,p(c)}$$

A pair that appears together exactly as often as chance predicts has a PMI of zero. Negative values are set to zero because a negative value for a rare pair rests on very few observations. The result is called positive PMI.

times together if the two words were unrelated
pointwise mutual information
positive PMI, the value we keep
Figure 1. A toy calculation with 100,000 word positions in total. Raise both word counts and hold the joint count fixed. The same 60 joint appearances go from strong evidence of association to none.

Step 3. Compress the table to 200 dimensions

The table has thousands of columns and most cells are zero. We reduce it to 200 columns with a truncated singular value decomposition. The decomposition keeps the 200 directions that account for the most variation in the table and drops the rest, which is mostly noise. Each word now has a vector of 200 numbers. Levy and Goldberg (2014) show that the neural method word2vec implicitly factors a table of this kind, so the two approaches are close relatives. Ours has no random initialization and the same input always gives the same vectors.

Two words are compared by the cosine of the angle between their vectors.

$$\cos(u, v) \;=\; \frac{u \cdot v}{\lVert u \rVert\,\lVert v \rVert}, \qquad d_{\cos}(u, v) \;=\; 1 - \cos(u, v)$$

The cosine is 1 when two vectors point the same way, 0 when they are at right angles, and −1 when they point in opposite directions. The cosine distance runs from 0 to 2. Length does not enter, so a long title and a short title on the same subject are close.

cosine similarity
cosine distance
straight-line distance between the tips
Figure 2. Two vectors in two dimensions. Changing the length of a vector changes the straight-line distance and leaves the cosine where it was. The paper's vectors have 200 dimensions and the arithmetic is the same.

Step 4. Average the words of a title

A title's vector is a weighted average of its word vectors. We use the smooth inverse frequency weights of Arora, Liang, and Ma (2017). A word that makes up the share \(p(w)\) of all words in the language gets the weight

$$\text{weight}(w) \;=\; \frac{a}{a + p(w)}, \qquad a = 10^{-3}.$$

Common words such as de and et get little weight and rare words get nearly full weight. After averaging we remove the first principal component of all title vectors in the language. That component carries what all titles share, such as common syntax, and removing it leaves the differences in content.

Figure 3. The weight a word gets in a title's average, by how common the word is. A word that is one in a thousand gets half weight. A word that is one in ten gets about one percent.

What comes out

Each title has a vector of 200 numbers that can be compared with any other title in the same language. Each language is trained on its own titles so a Latin vector and a German vector lie in different spaces and are never compared. Every measure in the paper is a distance inside one language.

SettingValue
Context window5 words
Minimum word frequency20
Dimensions kept200
Smoothing constant in the title average0.001
Languages trained separatelyLatin, German, French, Italian, English, Dutch, Spanish

Why not a modern language model

Pretrained neural models are trained on modern text. None exists for sixteenth-century Latin, German, or Dutch titles in their original spelling. Ganguli et al. (2026) show for patents that the choice of text representation can reverse a measured trend so a text measure has to be validated before its trend is read. We train on the catalog's own titles and check the result against things we know. The nearest titles to William Harvey's De motu cordis are reprints of Harvey and other works on the circulation of the blood, and the catalog files them under three different classes. You can search any word's neighbors in the companion.

Word embeddings are now a standard tool in economics. They have been used to measure the similarity of firms and of patents (Hoberg and Phillips 2010, Kelly et al. 2021) and to trace change in the meaning of words over time (Hamilton, Leskovec, and Jurafsky 2016). Gentzkow, Kelly, and Taddy (2019) survey text as data for economists.

What to keep in mind

References